PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Continual Experience Internalization for Self-Evolving LLM Agents

Jingwen Chen, Wenkai Yang, Shengda Fan, Wenbo Nie, Chenxing Sun, Shaodong Zheng, Yangen Hu, Lu Pan, Ke Zeng, Yankai Lin

arXiv:2606.04703v1cs.CLcs.LG

TL;DR

Existing experience-internalization methods show strong single-iteration gains but can fail during iterative self-evolution. This paper analyzes three transfer dimensions and finds a stable recipe combining principle-level experience, step-wise injection, and off-policy context-distillation.

  • Problem

    Existing methods largely emphasize single-iteration transfer, leaving iterative experience internalization insufficiently examined despite its importance for continual learning.

  • Method

    The paper systematically analyzes experience representation, teacher-supervision shaping, and trajectory distribution, then combines the findings into a recipe for multi-iteration transfer.

  • Results

    Across self-evolution cycles, principle-level experience is more durable, step-wise injection outperforms global injection, and off-policy context-distillation provides more coherent supervision than on-policy distillation.

  • Takeaways & Limitations

    The findings provide a stable recipe and practical guidance for designing LLM agents that sustain experience-based self-evolution across iterations.

  • Takeaways & Limitations

    The experiments focus on web-reasoning agent tasks, so generalization to other domains, languages, and agent settings remains unassessed.

Abstract

from arXiv · show

Experience internalization converts contextual experience from past interactions into reusable parametric capability, offering a promising path toward continual learning in large language models (LLMs). While prior work has predominantly focused on single-iteration transfer, we discover that under multi-iteration experience learning, existing methods suffer from a progressive capability collapse rather than compounding improvement. We systematically examine this failure through three vital dimensions of experience internalization: (1) Experience Granularity: We find that principle-level experience is more durable than instance-level experience, as it effectively abstracts transferable strategies away from trajectory-specific details. (2) Experience Injection Pattern: Our analysis reveals that step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states, a property that is critical for long-horizon tool use. (3) Internalization Regime: We demonstrate that off-policy context-distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-distillation, which is inherently limited by local corrections on student-induced flawed states. Together, these insights yield a simple yet robust recipe for stable and sustainable experience internalization, providing concrete guidance for engineering self-evolving and continually learning LLMs.

1 Introduction

Experience internalization is motivated by the limits of in-context learning and the need to convert context-dependent experience use into parametric capability. This study examines why existing approaches fail under multi-iteration learning and identifies experience representation, teacher supervision, and trajectory distribution as key transfer stages.

  • Motivation: Continual learning enables autonomous, adaptive LLM agents to acquire generalizable knowledge from past interactions and improve through future interactions.In-context learning directly exploits experience but is bounded by in-context capacity.
  • Motivation: Experience internalization converts context-dependent experience use into parametric capability, but prior approaches largely target strong performance in a single iteration.Existing approaches predominantly adopt on-policy context-distillation while overlooking iterative experience learning.
  • Failure analysis: Multi-iteration failures arise across experience representation, teacher supervision, and trajectory distribution during behavior transfer into the student.These three stages organize the study’s analysis of why current experience internalization paradigms fail.
  • Experience Granularity: Principle-level experience is more suitable than instance-level experience because it abstracts transferable strategies and failure patterns from trajectory-specific details.This abstraction provides a more generalizable signal and reduces reinforcement of instance-specific behaviors across iterations.
  • Contributions: The study systematically analyzes experience granularity, experience injection pattern, and internalization transfer to propose a simple recipe for sustainable internalization.The findings are intended to guide LLM agents toward experience-based self-evolution across iterations.

2 Related Work

Related work studies experience reuse through contextual guidance, context distillation, and self-evolving agent systems. These approaches span trajectory storage, reflection, abstraction, parameter internalization, and iterative evolution of models and external components.

  • Context-Based Experience Learning: Context-based experience learning organizes prior work into storage, reflection, and abstraction methods for reusing interaction trajectories as contextual guidance.Storage preserves trajectories for retrieval, reflection refines experience through self-feedback, and abstraction generalizes it into skills, strategies, or summaries.
  • Experience Internalization: Context distillation internalizes experience by aligning an experience-free student with an experience-aware teacher, with early approaches typically using teacher-generated trajectories off-policy.Off-policy formulations may suffer from training–inference mismatch, motivating recent movement toward on-policy context distillation.
  • Self-evolving LLM Agents: Self-evolving LLM agents iteratively improve behavior using interaction data, feedback signals, and self-generated experience at policy and component levels.Policy-level methods update agent models, while component-level methods evolve external structures such as memory, tools, skills, or experience libraries.
  • Self-evolving LLM Agents: Recent work couples model training with experience evolution in a closed loop, refreshing experience pools with trajectories from updated models across rounds.Effective self-evolution requires experience evolution and model improvement to reinforce each other iteratively.

3 Formulation

The paper formalizes continual experience internalization as an iterative loop in which experience-aware teachers train experience-free students. It defines a design space spanning experience granularity, injection pattern, and internalization regime.

  • Agent Trajectories and Experience Pool: An agent policy πθ interleaves thoughts and actions while interacting with an environment, producing trajectories evaluated by task-level reward and summarized into an experience pool E.Tool calls produce observations, and trajectories contain the query, thought-action pairs, and observations.
  • Experience Distillation: Experience distillation trains an experience-free student πθ to match an experience-aware teacher πT that accesses injected experience during supervision.The formulation distinguishes off-policy teacher-generated trajectories from on-policy student-generated trajectories supervised on student-induced states.
  • Continual Experience Internalization: Continual internalization iterates over policies, trajectories, and experience pools, using the current policy conditioned on newly summarized experience to train the next student policy.The process is indexed by k = 0, 1, . . . , K and evaluates whether gains persist across iterations rather than only after one update.
  • Dimensions of Experience Internalization: The design space varies experience granularity, injection pattern, and internalization regime to study sustained experience internalization.Granularity contrasts trajectory-specific instance experience with reusable principle-level strategies; injection contrasts global and history-selected step-wise contexts.

4 Experimental Setup

The experiments use Qwen3 student agents in a five-tool ReAct environment, train on a 15K-example web-reasoning corpus, and evaluate on in-domain and out-of-domain benchmarks. Training and inference follow fixed optimization, rollout, interaction-step, and context-window settings.

  • Models and Environment: Students are Qwen3-4B-Instruct-2507 and Qwen3-8B, with thinking mode disabled for Qwen3-8B, operating in ReAct format with five tools.The tools are Search, Visit, Python, Scholar, and File Parser.
  • Training Data and Experience: The 15K-example training corpus combines five public web-reasoning QA datasets and supports trajectory generation, experience extraction, and experience-conditioned supervision.The datasets are WebWalkerQAsilver, DeepDive, WebShaper, WebDancer, and SailorFog-QA.
  • Benchmarks and Metrics: Evaluation covers WebWalkerQA as in-domain and GAIA-Text-103 and BrowseComp-ZH as out-of-domain benchmarks, using Pass@1 or three-rollout average accuracy as specified.Pass@1 uses one rollout per query on WebWalkerQA and BrowseComp-ZH; GAIA-Text-103 accuracy averages three rollouts and is abbreviated as GAIA in tables.
  • Training and Inference: All methods use verl, with learning rate 1 × 10^-5, batch size 128, 5 epochs on 8× NVIDIA A800 GPUs, temperature 0.7, Tmax = 100, and 32,768-token context.Inference allows at most 100 interaction steps.

5 Toward Stable Continual Experience Internalization

Stable continual experience internalization depends on principle-level experience, step-wise injection, and off-policy context-distillation. Together, these choices preserve reusable experience use, reduce failure and interaction costs, and sustain self-evolution across iterations.

  • Experience Granularity: Instance-level experience produces transient gains that diminish below the base model, whereas principle-level experience offers a more stable basis for multi-iteration self-evolution.Instance-level items contain specific URLs or domains (74.4%), concrete numbers (57.3%), and query- or entity-specific strings (93.9%); principle-level items contain reusable strategy-like statements (84.0% versus 3.7%).
  • Experience Injection Pattern: At Iteration 1, step-wise injection consistently outperforms global injection by shaping the teacher distribution used for distillation.Under Qwen self-generated experience, WebWalkerQA increases from 23.2% to 31.2% with step-wise rather than global injection.
  • Experience Injection Pattern: Across iterations, step-wise injection maintains stronger performance and preserves experience-use ability, whereas global injection degrades with and without experience context.This difference is especially evident on WebWalkerQA and GAIA, and global injection can destabilize the model–experience loop.
  • Internalization Regime: Off-policy context-distillation provides proactive, end-to-end experience-guided supervision, while on-policy distillation reactively corrects student-induced states that may already be inefficient or off target.Rejection sampling filters low-quality variants and trains the student on compact successful trajectories generated by the experience-aware teacher.
  • Internalization Regime: After one internal weight update, on-policy students average 21.9 assistant turns versus 2.5 for the base model and 4.5 for the experience-aware teacher, increasing practical interaction cost.Off-policy distillation avoids this trajectory inflation by sampling shorter teacher trajectories and applying rejection sampling.
  • Stable Self-Evolution: The combined design—principle-level experience, step-wise injection, and off-policy context-distillation—sustains robust gains while preserving standalone parametric execution and in-context responsiveness across iterations.The updated model retains its ability to exploit the experience pool and serve as the experience-aware teacher in the subsequent iteration.

6 Conclusion

The paper shows that experience internalization can fail to sustain improvement across self-evolution iterations. It identifies principle-level experience, step-wise injection, and off-policy context-distillation as more durable or coherent alternatives.

  • Failure Across Iterations: Existing experience-internalization methods may fail to sustain improvement across self-evolution iterations.The study extends analysis beyond single-iteration transfer.
  • Experience Granularity: Principle-level experience provides a more durable signal than instance-level experience.Principles abstract transferable strategies beyond trajectory-specific details.
  • Experience Injection Pattern: Step-wise injection better aligns experience with intermediate decision states than global injection.This alignment is especially relevant to intermediate decisions in long-horizon tool use.
  • Internalization Regime: Off-policy context-distillation offers more coherent supervision than on-policy context-distillation.Off-policy training uses high-quality teacher trajectories, whereas on-policy learning is constrained by student-induced flawed states.

Limitations

The study is limited to web-reasoning agent tasks and three dimensions of experience internalization, leaving generalization and other stability factors for future evaluation.

  • The experiments focus on web-reasoning agent tasks, so generalization to other domains, languages, and agent settings remains unevaluated.Further evaluation is needed beyond the studied task setting.
  • Experience-pool size, selector quality, and filtering criteria may also affect internalization stability but were not explored.The study leaves comprehensive investigation of these factors to future work.

Broader Impact · A Statement on the Use of LLMs

The work aims to support stable experience internalization for self-evolving LLM agents while acknowledging that repeated internalization can reinforce incorrect, biased, or unsafe behaviors. It also specifies how LLMs assisted manuscript preparation and experiments, while authors verified the technical work and reported results.

  • Broader Impact: Stable experience internalization may help self-evolving agents transform accumulated experience into reusable capability for long-horizon tool-use applications.Examples include web reasoning, information seeking, and research assistance requiring search, evidence verification, and behavioral updates.
  • Broader Impact: More stable internalization may reinforce undesirable behaviors when accumulated experience contains incorrect, biased, or unsafe patterns.The risk is especially relevant when self-evolving systems repeatedly generate, internalize, and reuse their own experience.
  • Broader Impact: Practical deployment should use trajectory filtering, experience-pool auditing, human oversight, and restrictions in highrisk settings.
  • A Statement on the Use of LLMs: LLMs served as writing assistants for manuscript polishing, grammar improvement, and presentation refinement.The authors made and verified all technical claims, experimental designs, analyses, and final writing decisions.
  • A Statement on the Use of LLMs: DeepSeek-V4 summarized trajectories, selected relevant experience for step-wise injection, and generated experience-conditioned teacher trajectories for distillation.These uses placed LLMs within the experimental pipeline rather than evaluation generation.
  • A Statement on the Use of LLMs: In the Qwen self-generated setting, the student-side Qwen model extracted and selected experience, producing experience pools and teacher supervision.
  • A Statement on the Use of LLMs: No LLM generated evaluation questions, reference answers, or reported results; evaluated agent models produced all reported metrics under the stated experimental settings.The authors take responsibility for the paper’s content, experiments, and conclusions.

B Implementation Details

The implementation uses a ReAct-style agent with five tools, up to 100 interaction steps, and a 32,768-token context window. Training compares on-policy student trajectories with teacher supervision against rejection-filtered teacher-generated trajectories, using standardized experience extraction and distillation settings.

  • Agent Environment and Tools: The ReAct-style agent uses Search, Visit, Python, Scholar, and File Parser, with a maximum of 100 interaction steps and a 32,768-token context window.
  • Trajectory Collection: On-policy trajectories come from the current student and receive experience-aware teacher supervision, whereas off-policy trajectories come from the teacher and undergo rejection sampling before training.
  • Experience Extraction and Selection: DeepSeek-V4 summarizes trajectories and selects experience for step-wise injection by default, while Qwen performs both operations in the Qwen self-generated setting.Instance-level experience retains trajectory-specific observations and tool-use traces, whereas principle-level experience abstracts reusable strategies, search principles, and failure patterns.
  • Distillation Training: Students are trained with verl and AdamW at a learning rate of 1 × 10−5, batch size 128, and 5 epochs on 8× NVIDIA A800 GPUs.On-policy distillation uses student-induced trajectories with teacher supervision at each step; off-policy distillation uses rejection-filtered teacher-generated trajectories.
  • Self-Evolution Procedure: Self-evolution runs for three internalization iterations, refreshing the experience pool from current-model trajectories before distilling experience-conditioned behavior into the next model.
  • Inference and Evaluation: Evaluation omits inference-time experience unless explicitly marked, uses temperature 0.7, and reports Pass@1 for WebWalkerQA and BrowseComp-ZH versus average accuracy over three rollouts for GAIA-Text-103.

C Experience-Use Ability under Different Injection Patterns

Across both externally generated and self-generated principle-level experience, global injection becomes unstable or degrades during iterative self-evolution. Step-wise injection instead preserves stronger internalized performance and experience-use ability.

  • Experience-Use Ability under Different Injection Patterns: Step-wise injection maintains stronger internalized performance and better preserves experience-use ability across iterations than global injection.This result holds with DeepSeek-generated principle-level experience and off-policy context-distillation.
  • Experience-Use Ability under Different Injection Patterns: With Qwen-generated principle-level experience, global injection degrades both experience-free inference and in-context experience use.The degradation indicates that global injection does not reliably preserve the ability to use the updated experience pool during iterative self-evolution.

D Complete Self-Evolution Results

Table 4 presents complete self-evolution results spanning experience sources, injection patterns, distillation regimes, and model backbones. The results are consistent with the main findings, including greater stability for step-wise than global injection across iterations.

  • Complete self-evolution results: Table 4 reports complete self-evolution results across experience sources, injection patterns, distillation regimes, and model backbones.It includes both internalized and in-context inference results.
  • Complete self-evolution results: The table provides the full set of internalized and in-context inference results used to complement the main comparisons.The main text focuses on comparisons involving experience granularity, injection pattern, and internalization regime.
  • Complete self-evolution results: Step-wise injection is more stable than global injection across iterations.This complete-results trend is consistent with the paper’s main findings.
Loading 2606.04703v1…