PaperlayerView exact revision on arXiv

Source-linked AI summary

More Memory, Worse Agents: Error Reproduction and Anti-Persistence in LLM Agents

Ruhan Wang, Kishan Panaganti, Dongruo Zhou

arXiv:2610.27406v1cs.AIcs.CL

TL;DR

Memory-based self-improvement raises the question of whether agents should retain only trajectories and outcomes or also inferred abstractions, whose errors can shape later experience. Anti-Persistence stores experience while generating task-specific guidance on demand and discarding it after execution. Across the evaluated benchmarks and backbones, it achieves higher overall held-out success with fewer average LLM calls, though indirect feedback through trajectories remains possible.

  • Problem

    Retained abstractions can ease reuse of experience, but their applicability depends on task conditions and errors may persist through reuse and later synthesis.

    Source: §1 · pp. 1–2

  • Method

    Anti-Persistence stores trajectories and outcomes, uses an adaptive selector to construct guidance on demand, and discards that guidance after execution.

    Source: §1 · p. 2

  • Results

    Across three benchmarks and three backbones, Anti-Persistence achieved higher overall held-out success with fewer LLM calls on average.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The findings support separating experience accumulation from abstraction persistence while recognizing that retained trajectories can still mediate indirect feedback.

    Source: §5 · p. 10

    The main iterative experiments cover three text-based benchmarks with fixed backbones and a fixed improvement procedure, leaving broader generalization and longer-term gains untested.

    Source: H Limitations · p. 31

Abstract

from arXiv · show

LLM agents can improve across tasks without updating model parameters by storing experience from their own executions in external memory and reusing it to guide later decisions. A central design question is whether to retain only observed trajectories and outcomes or also the rules, summaries, and skills inferred from them. Retaining these abstractions allows agents to reuse strategies without repeatedly deriving them from raw trajectories. However, errors in stored abstractions can propagate to subsequent decisions through retrieval and reuse. They can also shape execution trajectories, which then become the basis for new abstractions that reproduce the original errors. We term this feedback mechanism error reproduction: errors are regenerated through a cycle of abstraction, execution, and re-abstraction, extending their persistence beyond direct reuse of the original faulty knowledge. We propose Anti-Persistence, a memory design that accumulates trajectories and outcomes while reconstructing task-specific guidance on demand. An adaptive selector uses prior outcomes to decide whether and how to synthesize guidance, which is discarded after execution. This prevents its direct reuse as a stored object, although indirect feedback through retained trajectories remains possible. Across different benchmarks and backbones, Anti-Persistence achieves higher overall held-out success and the lowest average LLM-call cost among the evaluated memory designs. These results support separating experience accumulation from abstraction persistence in memorybased self-improvement.

1 Introduction

Memory-based self-improvement reuses stored experience, but inferred abstractions can encode patterns whose applicability depends on task conditions. The paper proposes retaining trajectories and outcomes while generating task-specific guidance temporarily, and reports higher held-out success at lower average call cost.

1 Introduction

  • Abstractions can make long histories easier to retrieve, but a pattern that succeeds on one task may depend on conditions absent from another.Their value therefore depends on reassessing applicability when reused.Source: §1 · p. 1
  • A mistaken abstraction may shape later actions, whose records can inform synthesis of a new abstraction that reproduces the same error.This indirect persistence through execution history is distinguished from propagation by direct retrieval and reuse.Source: §1 · p. 2
  • Deleting an abstraction alone does not revise trajectories generated under its influence or remove abstractions later derived from those trajectories.Whether reproduction continues depends on retrieval, synthesis, and which outputs enter memory.Source: §1 · p. 2
  • Anti-Persistence retains trajectories and outcomes but discards task-specific abstractions after execution, with an adaptive selector deciding whether and how to synthesize guidance.The synthesis procedure remains fixed while guidance is reconstructed for each task.Source: §1 · p. 2
  • Across multiple rounds of self-improvement on three benchmarks and three backbones, Anti-Persistence achieved a higher overall success rate on held-out tasks at a lower average LLM-call cost.Source: §1 · p. 2

2 Related Work

Prior work studies both trajectory-based memory and retained abstractions, as well as feedback from repeated use of generated content. This paper distinguishes direct error propagation from error reproduction through trajectories and subsequent synthesis, while treating abstraction persistence as complementary to memory maintenance.

2 Related Work

  • Memory-based agents variously retain raw or lightly processed episodes, reflections, cross-task rules, summaries, or executable skills [2] [3] [5] [6] [7] [9] [10] [11].Related abstractions also appear in production agent systems [13] [14].Source: §2 · p. 2
  • Error propagation is an existing error affecting behavior, whereas error reproduction is its encoding in a newly generated abstraction after trajectories inform later synthesis.Prior work also studies repeated-use bias [6] [15] [16] and model collapse from training on self-generated data [18] [19], a different setting from external-memory updates.Source: §2 · p. 2
  • Pruning, scoring, and refinement address memory maintenance [8] [23] [24] [25] [26] [27], while Anti-Persistence addresses whether inferred abstractions persist and still retains trajectories for later synthesis.It removes direct abstraction reuse but permits indirect feedback through experience.Source: §2 · p. 3

3 Anti-Persistence: A Memory Design for Self-Improving Agents

Anti-Persistence accumulates task trajectories and outcomes, then retrieves records to guide later tasks. Its selector adapts how experience is used, while synthesized guidance is temporary and stored trajectories can still carry indirect feedback.

3 Anti-Persistence: A Memory Design for Self-Improving Agents

  • Anti-Persistence separates accumulating execution experience from temporarily using abstractions, which are discarded after the current task.The synthesis procedure remains fixed while guidance is constructed on demand.Source: §3 · p. 3, §3.2 · p. 4

3.1 Preliminaries

  • The agent has fixed backbone parameters and processes tasks into trajectories and outcomes; evaluation follows training updates with memory and related state frozen.Trajectory memory stores prior tasks, executions, and outcomes, while the abstraction store contains currently retained summaries, rules, strategies, or skills.Source: §3.1 · pp. 3–4
  • For a new task, retrieval draws context from trajectory and abstraction stores, and the policy generates an execution conditioned on that task and context.After execution, the task, trajectory, and outcome are added to trajectory memory; abstraction-retaining methods separately update their abstraction store.Source: §3.1 · p. 4
  • An abstraction may influence a trajectory whose record later informs synthesis, allowing an earlier error to reappear in newly generated guidance.Anti-Persistence keeps the trajectory records but does not retain synthesized guidance for direct reuse.Source: §3.1 · p. 4

3.2 Algorithm Design

  • The design retains trajectories and outcomes without certifying that their actions or explanations are correct; temporary guidance can still influence records used in later synthesis.Discarding prevents direct reuse of guidance as a stored object, not indirect feedback through trajectories.Source: §3.2 · p. 4
  • The selector chooses between direct record use and synthesis; its four modes are none, imitation, strategic, and corrective.The none mode reuses retrieved records directly without an additional synthesis call.Source: §3.2 · p. 4
  • Mode-selection feedback combines task outcome with synthesis cost, and retrieved attempts are weighted by task similarity and recency.Older feedback decays for mode selection without changing trajectory retrieval; the controller estimates mode utility from weighted rewards and adds an exploration bonus.Source: §3.2 · pp. 5–6
  • During training, execution records enter trajectory memory and eligible feedback enters the outcome log, while guidance is discarded and evaluation leaves both stores unchanged.The backbone, mode definitions, prompts, retrieval procedures, and selection rule remain fixed.Source: §3.2 · pp. 4, 6

4 Experiments

The experiments evaluate memory designs across benchmarks, backbones, and ten update iterations, then test individual design choices. Anti-Persistence achieves the strongest overall held-out results and lowest average call cost among the evaluated memory configurations.

4.1 Experimental Setup

  • The evaluation covers ALFWorld, WebShop, and ScienceWorld with three backbones, ten training-memory updates, and held-out evaluation on frozen state.Success and mean LLM calls per held-out task are measured over three seeds.Source: §4.1 · p. 7

4.2 Held-Out Performance and LLM-Call Cost

  • 59.9% mean held-out success across nine settings, versus 53.5% for trajectory memory, 49.0% for abstraction memory, and 49.3% for joint memory; Anti-Persistence also has the lowest mean LLM-call cost.It has the highest success rate in seven of nine settings and exceeds Reflexion’s 52.9% average success.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

4.3 Dynamics Across Self-Improvement Rounds

  • Anti-Persistence has the highest final mean success in all six main-text settings, while AutoRefine exceeds abstraction memory in one, matches it in one, and trails it in four.Anti-Persistence also exceeds AutoRefine in all six settings; because AutoRefine changes retrieval and synthesis, this comparison does not isolate maintenance effects.Source: §4.3 · p. 8

4.4 Ablations and Design Analysis

  • At γ = 0.998, mean success is 96.0% on ALFWorld and 54.0% on WebShop; faster decay and no decay underperform this tested default.Success does not vary monotonically with γ.Source: §4.4 · p. 8
  • The contextual bandit achieves the highest success on Qwen3-Coder-Next for both ALFWorld (96.0%) and WebShop (54.0%), outperforming fixed and random mode selection.The strongest fixed policy differs by benchmark: strategic on ALFWorld and imitation on WebShop.Source: §4.4 · p. 9
  • With λ = 0.05, mean success is highest among tested cost weights on ALFWorld (96.0%) and WebShop (54.0%); removing the penalty increases synthesis-mode usage but reduces success.Stronger penalties (λ ∈{0.20, 0.50}) reduce both success and synthesis-mode usage.Source: §4.4 · p. 9
  • Mode use varies by setting: stronger backbones favor imitation on WebShop, Llama-4-Scout favors strategic there, and the none mode has nonzero mean frequency in every setting.The reported averages describe overall allocation, not whether modes were selected in every round or whether the policy converged.Source: §4.4 · p. 9
  • Discarding synthesized guidance after the current task improves success over persisting it in all four tested settings.The reported gains are larger for Llama-4-Scout and positive for Qwen3-Coder-Next.Source: §4.4 · p. 9, Table 2 · p. 10

5 Conclusion

Anti-Persistence retains trajectories and outcomes while using task-specific abstractions temporarily. The appendix formalizes how historical exposure, flagged generation, and persistent admission relate within a single synthesis invocation.

5 Conclusion

  • Anti-Persistence accumulates trajectories and outcomes, constructs temporary guidance with a fixed backbone and synthesis procedure, and achieves higher held-out success with fewer average LLM calls across three benchmarks and three backbones.It removes direct storage and reuse of synthesized abstractions, while indirect feedback through trajectories remains possible.Source: §5 · p. 10

A Reproduction Dynamics in Abstraction Memory

  • The appendix distinguishes historical exposure, operational bias labels, and persistent admission, while limiting its claims to a single synthesis invocation rather than long-run improvement or degradation.Its results do not imply that all maintenance policies fail.Source: A Reproduction Dynamics in Abstraction… · p. 14

A.1 Operational Bias Labels and Historical Provenance

  • Operational bias is protocol-relative: unflagged abstractions may be unclassified, and recorded exposure to retrieved parents does not itself establish behavioral influence.The evaluation labels do not identify an intrinsic error rate or the causal effect of an abstraction.Source: §A.1 · p. 14

A.2 Exposure, Generation, and Persistent Admission

  • For one synthesis invocation, α is the flagged-output rate conditional on flagged-parent exposure; a positive α alone does not establish error accumulation.These quantities may vary with the state, retrieval policy, synthesis operator, and maintenance procedure.Source: §A.2 · pp. 14–15

A.3 What Deletion and Maintenance Do Establish

Deletion-only operations remove selected abstractions and descendants without altering execution records. The appendix gives a one-step probability result and bounds what it establishes for maintenance and anti-persistence.

A.3 What Deletion and Maintenance Do Establish

  • Deleting a parent alone leaves its retained descendants and historical records unchanged, while deleting descendants can remove them.The identities do not guarantee that descendants survive other maintenance steps or that retained records are retrieved again.Source: §A.3 · p. 15
  • When q, α, and η are positive, flagged-parent exposure has positive probability of producing and admitting a flagged candidate.The decomposition concerns one synthesis invocation and does not establish inevitable amplification or a long-horizon performance ordering.Source: §A.3 · p. 15

A.4 Implications for Anti-Persistence

  • Anti-persistence makes persistent admission of synthesized candidates impossible, but temporary flagged guidance may still influence trajectories and later synthesis.Its structural guarantee is absence of stored abstractions and their direct reuse, not absence of all feedback.Source: §A.4 · pp. 15–16

B Method Details

This appendix details Anti-Persistence’s guidance reconstruction, experience retrieval, mode selection, and outcome logging, with a worked example of on-demand corrective guidance. It also specifies how training experience updates memory and controller state while held-out outcomes remain separate.

B.1 Abstraction Modes

  • Anti-Persistence offers direct factual context and three on-demand abstractions—imitation, strategic, and corrective—that are discarded after execution.Imitation extracts successful behavioral patterns, strategic distills multi-trace plans, and corrective generates failure-avoidance rules; missing required records can skip synthesis, and corrective does not fall back to strategic guidance.Source: §B.1 · pp. 16–17

B.2 Mode-Specific Prompts

  • Strategic prompts distill transferable structure from successful trajectories, corrective prompts derive avoidance rules from failures, and imitation extracts adaptable action patterns rather than copying traces.The direct-context wrapper supplies factual records without synthesis, while synthesized guidance is presented under mode-specific headers.Source: §B.2 · pp. 17–19

B.3 Retrieval Similarity

  • Trajectory retrieval selects factual records or mode-specific successes and failures, while bandit retrieval uses similar outcome-log entries to estimate local rewards.Both pathways use a task embedding combining the instruction and a truncated initial observation; the bandit falls back to a global-best policy when neighborhood similarity is below its threshold.Source: §B.3 · pp. 19–20

B.4 Hyperparameters

  • The bandit uses round-robin selection for initial mode coverage, then ε-greedy exploration at 0.1; when average cosine similarity falls below ρ = 0.7, it switches to global-best fallback.Rewards use r = clip(y − λκ, −1, 1), with λ = 0.05 and κ equal to the realized synthesis-call count, which is zero when synthesis is skipped.Source: §B.4 · p. 20

B.5 Outcome-Log Schema and Update Protocol

  • The outcome log records seven-field entries including task embeddings, selected modes, cost-adjusted rewards, task families, and truncated instructions.The log records observed mode choices and outcomes, not synthesized abstractions.Source: §B.5 · p. 21
  • Logging skips initial empty-memory attempts, embedding failures, and infrastructure failures, but remaining observations may still have task-selection effects or other confounding.These exclusions are intended to avoid attributing baseline noise or outcomes that cannot be assigned to the selected mode.Source: §B.5 · p. 21

C Worked Example: A Walkthrough of On-Demand Synthesis

  • The constructed ALFWorld walkthrough illustrates a local-selection case with adequate neighborhood similarity, positive weights for every mode, and no exploration override.Its candidate records include three related failures involving heating an unheld object, placing an object before heating, and inefficient microwave search; the example is illustrative, not a measured execution trace.Source: C Worked Example: A Walkthrough of… · p. 22
  • The controller selects corrective synthesis using cost-adjusted UCB scores, then analyzes three failure records and a contrasting success in one LLM call.The example uses an exploration bonus with c = 0.5; corrective mode has the highest score.Source: C Worked Example: A Walkthrough of… · pp. 22–23
  • Corrective reflection derives rules to hold an object before heating, heat before placing it, and go directly to the microwave listed in the observation.The rules address failures caused by heating an unheld object, premature placement, and searching for a microwave already listed in the initial observation.Source: C Worked Example: A Walkthrough of… · p. 23
  • Applying the three rules yields an illustrative six-step trajectory that heats an egg and places it on the dining table successfully.The trace takes the egg, goes directly to the microwave, heats it, then transports and places it on the table.Source: C Worked Example: A Walkthrough of… · p. 23
  • After execution, the trajectory and mode outcome are retained, but synthesized guidance is discarded; retained trajectories may still inform later synthesis and outcomes guide mode selection.The abstraction is not stored in either trajectory memory or the outcome log, preventing its direct reuse as a stored object.Source: C Worked Example: A Walkthrough of… · p. 23

D Experimental Setup Details

D.1 Per-Benchmark Task Counts and Partitioning

The study uses controlled training and held-out partitions across three benchmarks, with fixed backbone parameters and shared task samples. It also evaluates three backbones, specifies an AutoRefine baseline, and extends the results across backbones.

D.1 Per-Benchmark Task Counts and Partitioning

  • ALFWorld uses 180 training tasks and 134 held-out tasks; WebShop uses 200 training instructions and a fixed 100-instruction held-out set.The WebShop evaluation set differs from the benchmark’s original 500-instruction test split.Source: §D.1 · p. 24
  • ScienceWorld uses 150 training and 60 held-out instances, with both partitions covering all 30 task types through nonoverlapping variations.All methods use the same fixed-seed partition, supporting controlled comparisons within this study despite differences from prior protocols.Source: §D.1 · p. 24
  • Anti-Persistence is evaluated over ten iterations, with 50 training tasks sampled per iteration and the full held-out set evaluated after each update while memory remains frozen.Held-out episodes cannot update memory or related state, and episodes exceeding 50 environment steps count as failures.Source: §D.1 · p. 24

D.2 Backbone Configurations

  • Three open-weight backbones—Llama-4-Scout, Qwen3-Coder-Next, and Gemma-4-31B-it—share the agent prompt and per-step output budget.Qwen3-Embedding-8B supplies task embeddings across backbones.Source: §D.2 · p. 25

D.3 AutoRefine Baseline Implementation

  • AutoRefine adapts Qiu et al.’s [8] maintenance procedure but also changes retrieval, synthesis, and the update loop; the appendix extends results to Gemma-4-31B-it, cost breakdowns, and flagged-parent associations.Its implementation includes contrastive skill evolution and periodic pruning; the extended results cover all nine backbone–benchmark settings for cost.Source: §D.3 · pp. 25–26, E Extended Cross-Backbone Results · p. 26

E.1 Per-Iteration Curves on Gemma-4-31B-it

On Gemma-4-31B-it, Anti-Persistence leads on final ALFWorld and WebShop success, while Joint Memory leads on ScienceWorld. The appendix also reports trajectory lengths, cost coverage, observational provenance associations, and additional controlled analyses.

E.1 Per-Iteration Curves on Gemma-4-31B-it

  • 97.3% on ALFWorld and 53.3% on WebShop were Anti-Persistence’s final mean success rates; Joint Memory led on ScienceWorld with 56.3% versus 54.1%.The values match Table 1, while the relative ordering varied across earlier rounds.Source: §E.1 · p. 26, Figure 9 · p. 27
  • Joint Memory had the shortest ALFWorld trajectories, while Trajectory Memory had the shortest on WebShop and ScienceWorld.Anti-Persistence followed closely on WebShop; on ScienceWorld its trajectory lengths were comparable to Joint Memory’s.Source: §E.1 · p. 27, Figure 9 · p. 27

E.2 Per-Task LLM-Call Cost

  • Table 6 reports average per-task LLM-call counts for all four memory configurations across nine backbone–benchmark settings, including execution and synthesis calls.The supplied table caption specifies final-iteration means and standard deviations over three seeds; lower cost is better.Source: §E.2 · p. 27, Table 6 · p. 27

E.3 Provenance Associations and Operational Bias Flags

  • Flagged-parent descendants had higher flag rates in most of six settings, with the largest reported gap, +0.55, on Llama-4-Scout with ALFWorld; this association establishes no causal effect.The pooled rates are observational, condition on prior storage and labeling eligibility, and lack group counts and an uncertainty-estimation procedure.Source: §E.3 · pp. 27–28

F Additional Controlled and Statistical Analyses

  • Three additional analyses examine supplied skill content on AppWorld, selective skill retention on WebShop, and consistency across the nine main evaluation settings.The controlled analyses report aggregate success rates without task-level or seed-level observations for uncertainty estimation.Source: F Additional Controlled and… · p. 28

F.1 Effects of Skill Content on AppWorld

On AppWorld, useful supplied skills improve success while toxic skills do not; on WebShop, filtering improves the abstraction-memory baseline but a frozen skill layer reduces Anti-Persistence’s success.

F.1 Effects of Skill Content on AppWorld

  • 40.0% success with useful skills exceeded 23.0% with no skills and 20.0% with toxic skills on 30 AppWorld tasks.Qwen3-Coder-Next evaluated the same tasks with the skill libraries frozen; useful and toxic versions differed in procedural-rule correctness.Source: §F.1 · pp. 28–29, Figure 10 · p. 29

F.2 Selective Skill Retention on WebShop

  • On WebShop, filtering raised abstraction memory’s final success from 38.7% to 46.0%, while adding the filtered skills as a frozen layer lowered Anti-Persistence from 56.0% to 44.0%.Because the added layer remained frozen, this result does not establish descendant generation or error reproduction across iterations.Source: §F.2 · p. 29, Figure 11 · p. 29

F.3 Setting-Level Consistency Across the Main Evaluation Grid

Across nine fixed backbone–benchmark settings, Anti-Persistence generally outperformed the baselines. The accompanying statistical summaries are exploratory, and the next sensitivity analyses vary trajectory and outcome-log retrieval budgets.

F.3 Setting-Level Consistency Across the Main Evaluation Grid

  • The setting-level comparison summarizes paired Anti-Persistence-minus-baseline differences across nine fixed settings using equally weighted mean percentage-point differences and counts of positive differences.The analysis also reports nominal 95% t-intervals, paired t-tests, and Wilcoxon signed-rank tests with Holm correction within each test family.Source: §F.3 · p. 29
  • Anti-persistence had higher success in all nine settings against five baselines and in eight settings against Reflexion and joint memory, with mean differences of +6.4 to +19.2 percentage points.Table 7 reports exploratory comparisons across the nine settings.Source: §F.3 · p. 29, Table 7 · p. 30
  • The nine settings share backbones and benchmarks, so their nominal intervals and p-values are exploratory rather than evidence of population-level significance.All seven Holm-adjusted p-values were below 0.05 in each test family, but the analysis neither estimates seed-level uncertainty within settings nor supports extrapolation to untested models and benchmarks.Source: §F.3 · p. 30, Table 7 · p. 30

G Sensitivity and Robustness Studies

  • The sensitivity studies vary the trajectory retrieval budget B for records used in direct reuse or synthesis and the outcome-log budget ˆB for records used in mode selection.These budgets govern distinct parts of the process.Source: G Sensitivity and Robustness Studies · p. 30

G.1 Trajectory Retrieval Budget B Sensitivity

On both benchmarks, the default retrieval budgets performed best among the tested values for trajectories and outcome-log neighborhoods. Smaller and larger tested budgets yielded lower success in the reported sensitivity sweeps.

G.1 Trajectory Retrieval Budget B Sensitivity

  • At B = 1, success rates were 86.0% on ALFWorld and 46.0% on WebShop, below the default.Direct reuse received at most one record; synthesis received at most one success for imitation or strategic mode, or one failure and an optional success for corrective mode.Source: §G.1 · p. 30
  • At B = 10, success rates were 91.3% on ALFWorld and 48.7% on WebShop, again below the default.The sweep does not isolate the proposed explanation that adding weakly related trajectories may dilute useful context.Source: §G.1 · p. 30
  • Across both benchmarks, the default budgets yielded the highest observed success rates within the tested ranges and settings; these results do not establish a general optimum.Source: §G.2 · p. 31

G.2 Outcome-Log Retrieval Budget ˆB Sensitivity

  • At ˆB = 3, held-out success was 90.0% on ALFWorld and 45.3% on WebShop; at ˆB = 5, it was 92.0% and 46.7%, respectively, both below default.The lower-budget results may reflect sensitivity to individual outcomes in small neighborhoods.Source: §G.2 · p. 31
  • At ˆB = 20, success rates were 92.0% on ALFWorld and 50.0% on WebShop, below the default.The sweep does not directly test whether including more weakly related tasks reduces the relevance of local reward estimates; it establishes no general optimum.Source: §G.2 · p. 31

H Limitations

Anti-Persistence prevents direct reuse of stored abstractions, but its scope and evidence have important boundaries.

H Limitations

  • The main iterative experiments cover three text-based benchmarks, fixed backbones, and a fixed improvement procedure; broader generalization and longer-term gains remain untested.The controlled auxiliary analyses lack task-level and seed-level uncertainty estimates, and setting-level statistics remain exploratory.Source: H Limitations · p. 31
  • Temporary guidance can still influence later synthesis through retained trajectories, while discarding abstractions also forgoes reusing reliable knowledge across tasks.Performance also depends on retrieval quality and experience coverage.Source: H Limitations · p. 31

Related papers

Loading 2610.27406v1…