Source-linked AI summary
EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, Botian Shi
TL;DR
LLM agents often fail to systematically learn from their own interactions, treating tasks episodically and limiting strategy refinement. EvolveR introduces a closed-loop lifecycle that distills trajectories into reusable principles, applies them online, and updates the policy through reinforcement learning. Across complex QA benchmarks, it consistently outperforms strong baselines, while its self-distillation quality remains bounded by the base model’s capabilities.
Problem
LLM agents often treat sequential tasks independently, limiting their ability to learn from prior successes, avoid mistakes, and refine reusable problem-solving strategies.
Method
EvolveR alternates online interaction and offline self-distillation, curates deduplicated strategic principles, retrieves them for new tasks, and evolves the policy through reinforcement learning.
Results
EvolveR consistently and significantly outperforms a wide range of strong baseline methods across a comprehensive suite of QA benchmarks.
Takeaways & Limitations
EvolveR provides a blueprint for agents that learn from the consequences of their own experiences by building and evolving strategic expertise.
Takeaways & Limitations
Self-distillation quality is bounded by the agent’s model capabilities, and broader task coverage and computational efficiency for lifelong learning remain open challenges.
Abstract
from arXiv · showhide
Current Large Language Model (LLM) agents show strong performance in tool use, but lack the crucial capability to systematically learn from their own experiences. While existing frameworks mainly focus on mitigating external knowledge gaps, they fail to address a more fundamental limitation: the inability to iteratively refine problem-solving strategies. In this work, we introduce EvolveR, a framework designed to enable agent to self-improve through a complete, closed-loop experience lifecycle. This lifecycle comprises two key stages: (1) Offline Self-Distillation, where the agent's interaction trajectories are synthesized into a structured repository of abstract, reusable strategic principles; (2) Online Interaction, where the agent interacts with tasks and actively retrieves distilled principles to guide its decision-making, accumulating a diverse set of behavioral trajectories. This loop employs a policy reinforcement mechanism to iteratively update the agent based on its performance. We demonstrate the effectiveness of EvolveR on complex multi-hop question-answering benchmarks, where it achieves superior performance over strong agentic baselines. Our work presents a comprehensive blueprint for agents that learn not only from external data but also from the consequences of their own actions, paving the way for more autonomous and continuously improving systems. Code is available at https://github.com/Edaizi/EvolveR.
1 INTRODUCTION
LLM agents often treat interactions as isolated episodes, limiting their ability to learn reusable strategies from experience. EvolveR addresses this gap with a closed-loop lifecycle that distills trajectories into principles, applies them online, and evolves the policy.
- Sequential LLM-agent interactions are often handled independently, causing operational amnesia and preventing learning from prior successes or mistakes.
- Existing retrieval methods address knowledge gaps but do not systematically extract strategies from the consequences of the agent’s interactions.
- Raw-trajectory retrieval can guide decisions but struggles to generalize because agents mimic past solutions instead of abstracting reusable principles.
- EvolveR alternates online interaction with offline self-distillation, transforming collected trajectories into a curated repository of strategic principles for new tasks.
- EvolveR closes the experience lifecycle by using reinforcement learning to evolve the agent’s policy rather than merely replaying prior interactions.
- EvolveR significantly outperforms strong agentic baselines on complex question-answering benchmarks.
- Ablations identify curation and self-distillation as critical, with self-distillation surpassing a stronger external teacher model at the 3B scale.
2 RELATED WORK
Related work includes continual-learning, prompting, reflection, and external-memory approaches for improving LLM agents. These methods generally emphasize knowledge preservation, reflection, or trajectory reuse rather than systematic long-term accumulation of strategic knowledge.
- Continual-learning methods address sequential learning and catastrophic forgetting but commonly assume predefined task boundaries and prioritize knowledge preservation.
- ReAct and Reflexion support reasoning, actions, or self-reflection, but their primarily prompt-based and stateless designs limit long-term strategic accumulation.
- External-memory frameworks such as ExpeL reuse past trajectories but do not enable systematic self-improvement across tasks.
3 METHOD
EvolveR forms a closed-loop lifecycle in which agents distill interaction trajectories into strategic principles, retrieve them during online reasoning, and improve policies through reinforcement learning.
- Experience Lifecycle: EvolveR alternates frozen-parameter offline self-distillation with online interaction, where reinforcement learning updates the agent’s policy.The lifecycle transforms raw interaction data into reusable strategic principles and gathers new experience-guided trajectories.
- Agent Interaction: The agent’s action space includes searching internal experience, searching external knowledge, and producing a final answer.Internal search retrieves distilled principles, while external search retrieves factual information.
- Offline Experience Self-Distillation: Self-distillation uses the agent’s policy model to convert successful and failed trajectories into guiding or cautionary principles.Each principle pairs a natural-language description with structured knowledge triples.
- Experience-Base Curation: New principles are deduplicated and matched against existing entries, then either added as novel insights or merged with their best semantic match.The update uses embedding similarity and a semantic-equivalence judgment, with cosine similarity threshold θsim governing the update rule.
- Experience-Base Curation: Principles receive empirical utility scores from usage and success counts, and low-scoring entries are periodically pruned to maintain a compact repository.The score is intended to reflect each principle’s historical effectiveness.
- Online Interaction: Retrieved principles guide subsequent reasoning and produce structured trajectories that feed the next offline refinement cycle.The online phase therefore serves both immediate problem solving and future experience distillation.
- Policy Evolution: The composite trajectory reward combines task outcome and procedural format, with outcome correctness and reasoning-process quality contributing separately.The format reward includes think and search components, while structurally incomplete trajectories receive no format reward.
4 EXPERIMENTS
EvolveR is evaluated across seven in-domain and out-of-domain question-answering benchmarks using Qwen2.5 models and Exact Match as the primary metric. It achieves the strongest reported average performance across the main evaluation and shows consistent results across diverse tasks.
- Evaluation Setup: EvolveR is evaluated on seven QA benchmarks spanning in-domain datasets NQ and HotpotQA and out-of-domain datasets including TriviaQA, PopQA, 2Wiki-MultiHopQA, and Musique.The benchmark suite covers both standard and complex multi-hop question answering.
- Evaluation Setup: The comparison includes prompting-based, supervised fine-tuning, and reinforcement-learning agent baselines built on Qwen2.5 models.The baselines represent methods with different levels of parameter adaptation and experience use.
- Evaluation Setup: Exact Match is the primary evaluation metric, while F1 is additionally reported for model-scale generalizability analysis.Exact Match requires the normalized prediction to exactly match the ground truth; F1 accommodates multiple valid answers or aliases.
- Implementation: The experiments use Qwen2.5 models with a cold-start fine-tuning stage based on approximately 700 NQ and HotpotQA trajectories before GRPO evolution.LoRA fine-tuning is used for the cold-start stage.
- Main Results: 0.382 for 3B and 0.417 for 7B are EvolveR’s average scores, outperforming all baselines including Searcher-R1.The reported performance remains strong across diverse benchmarks rather than being concentrated in one task.
- Main Results: EvolveR secures the best scores on NQ, PopQA, and Bamboogle while remaining highly competitive on the other evaluated benchmarks.These results span in-domain, out-of-domain, and adversarial settings.
5 FURTHER ANALYSIS
Further analyses show that EvolveR’s effectiveness depends on model scale, self-distilled experience, retrieval, reinforcement learning, and experience curation. The framework remains robust as its experience base grows, while principle quality improves through scoring and pruning.
- Model-scale generalizability: 0.150, 0.270, and 0.382 average performance are achieved by EvolveR with Qwen2.5 models of 0.5B, 1.5B, and 3B parameters, respectively.Performance improves monotonically across all evaluated benchmarks as the base model grows.
- Self-distillation: 0.382 avg. for self-distillation exceeds 0.370 avg. for teacher-distillation at the 3B scale.The external teacher provides a benefit for smaller models, but the relationship reverses at 3B.
- Experience retrieval: 0.382 versus 0.340 average performance at 3B shows substantial degradation when experience retrieval is disabled during evaluation.The compared models received identical experience-driven reinforcement-learning training; only evaluation-time access to the experience base differed.
- Component attribution: 0.325 from RL alone is insufficient for optimal performance, while retrieval and RL together reach 0.382 on Qwen2.5-3B.The ablation reports 0.340 for the RL-only baseline and 0.357 for non-RL retrieval, indicating complementary contributions.
- Principle quality: 24% to 8% incorrect or misleading principles and 26% to 82% highly actionable principles follow utility-based scoring and pruning.The lower-score tier’s primary failure mode is vague advice, at 50%.
- Scalability: Nearly 50,000 principles incur only 0.20s retrieval latency, while performance remains stable and peaks at the 45k scale.The evaluation used a 10% validation subset sampled across six benchmarks.
6 CONCLUSION
EvolveR introduces a closed-loop experience lifecycle for self-evolving LLM agents. Experiments show strong benchmark performance and validate the value of self-distilled experiences, while principle quality remains tied to the base model’s capabilities.
- Contribution: EvolveR is a closed-loop experience lifecycle for self-evolving LLM agents.The conclusion presents the framework as a paradigm centered on continuous use of experience.
- Results: EvolveR consistently and significantly outperforms a wide range of strong baseline methods on comprehensive QA benchmarks.The conclusion also reports that ablations validate the value of self-distilled experiences and the self-distillation mechanism.
A.1 EXPERIMENTAL IMPLEMENTATION DETAILS
The implementation uses Qwen2.5 models with staged supervised training, retrieval, self-distillation, and GRPO policy optimization. The experience base is designed for efficient retrieval, while the full lifecycle requires substantial multi-GPU training.
- Qwen2.5 models use 8192-token inputs, 1024-token responses, BGE-M3 embeddings, and GPT-4o-mini as the teacher in the corresponding ablation.
- Cold-start SFT teaches the required interaction format, including structured <think> and <search> actions, before later policy evolution.It runs for 3 epochs with Adam, a 1 × 10^-4 learning rate, 0.1 warm-up ratio, and batch size 16.
- Online interaction retrieves the top-3 documents for <search knowledge> and top-3 principles for <search experience>.
- Offline self-distillation uses the agent’s own policy model, semantic pre-filtering at θsim = 0.85, LLM equivalence checks, and pruning below θprune = 0.3.
- The composite reward weights outcome reward at wo = 1.0 and format reward at wf = 0.1, while GRPO uses ϵ = 0.2 and β = 0.001.
- 0.06 seconds is the measured latency for retrieving the top-3 principles from an experience base containing approximately 14,000 principles.The full Qwen2.5-3B training lifecycle requires approximately 39.4 hours on 8 NVIDIA A100 GPUs.
A.2.1 NECESSITY OF THE RL (GRPO) STAGE
The ablation compares GRPO with standard SFT using the same successful trajectories. The RL-based variant performs better because it learns from both successful and failed trajectories rather than reproducing surface-level action sequences.
- The ablation reuses the same successful online trajectories while replacing GRPO policy training with standard SFT.
- 7% relative improvement is achieved by the RL-based variant over the SFT-only version on the Qwen2.5-3B model.
- SFT encourages reproduction of successful trajectory patterns, whereas GRPO uses positive and negative rollouts to learn which actions to perform or avoid.
A.2.2 HYPERPARAMETER SENSITIVITY ANALYSIS
The pruning-threshold sensitivity analysis evaluates EvolveR-1.5B across four retention thresholds. Performance remains robust, and the default threshold filters low-quality principles to limit unbounded experience-base growth.
- Four pruning thresholds, θprune ∈{0.1, 0.3, 0.7, 0.9}, are evaluated on EvolveR-1.5B.
- Performance remains robust across the tested pruning thresholds.
- θprune = 0.3 is the default setting and removes low-quality principles to prevent indefinite database growth.
A.2.3 LONGITUDINAL ANALYSIS OF LEARNING DYNAMICS
The longitudinal analysis examines how RL changes action use and principle quality over four training intervals. It also compares experience handling choices and illustrates how retained principles support concrete reasoning behavior.
- Evolution of Action Frequencies: RL training increases <think> and <search knowledge> frequencies during optimization, followed by convergence rather than indefinite action-count growth.
- Intrinsic Improvement in Principle Distillation: 0.500 is the average metric score for principles generated in the final interval, compared with 0.462 for the earliest interval.The analysis reports usage counts above 12 for statistical stability.
- Experience Internalization: Allowing gradients through retrieved <experience> tokens in EvolveR w/ exp-absorb causes slight performance degradation compared with treating principles as external context.
- Experience Self-Distillation: EvolveR uses distinct self-distillation prompts for successful and failed trajectories to extract guiding and cautionary principles.
- Principle Integration: The agent’s semantic judge decides whether a newly distilled principle is equivalent to an existing one, determining whether to merge it or create a new principle.
- Qualitative Principle Quality: High-scoring principles provide specific, actionable heuristics, whereas low-scoring principles give abstract advice lacking sufficient action guidance.
- Limitations and Broader Impact: EvolveR’s self-distillation quality is bounded by the capabilities of its own model, and broader task coverage remains necessary to delineate applicability.
- Rollout Example: A rollout example applies retrieved principles and external documents to identify Kate Warne as the first female detective employed by the Pinkerton Detective Agency.