Source-linked AI summary
OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation
Guibin Zhang, Xun Xu, Yanwei Yue, Zikun Su, Wangchunshu Zhou, Xiaobin Hu, Shuicheng Yan
TL;DR
Existing memory agents may improve in specific settings without becoming holistic agent evolvers. OPD-Evolver co-evolves memory-use abilities through on-policy self-distillation, outperforming memory-augmented and training-based baselines across diverse benchmarks.
Problem
Existing agents may improve within particular settings yet lack the holistic competencies required to become agent evolvers.
Method
OPD-Evolver uses fast online memory-conditioned interaction and slow outcome-calibrated on-policy self-distillation for experience selection, execution, writing, and management.
Results
OPD-Evolver-4B/9B consistently improves compact open models over memory-augmented and training-based baselines while remaining competitive with giant counterparts.
Takeaways & Limitations
The results support cultivating agents that transform experience into continuing evolutionary capacity rather than merely storing it.
Takeaways & Limitations
Evaluation starts with an empty experience repository, with memory accumulated only from the evaluation stream.
Abstract
from arXiv · showhide
Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it. Existing memory agents can store trajectories, retrieve reflections, or accumulate skills, but often lack the holistic competence to select useful experience, act on it, write reusable knowledge, and maintain a growing repository. We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation. In the fast loop, OPD-Evolver interacts with a four-level memory hierarchy to read, use, write, and maintain experience for rapid test-time evolution. In the slow loop, outcome-calibrated memory attribution and privileged hindsight distill these four abilities into the deployable policy. Across multi-domain benchmarks, OPD-Evolver surpasses memory systems such as ReasoningBank by up to 11.5%, and training-based methods such as Skill0 by ~5.8%. Further analysis shows that OPD-Evolver internalizes high-value experience and memory management, enabling OPD-Evolver-9B to challenge giant counterparts such as Qwen3.5-397B-A17B and Step-3.5-Flash, pointing beyond memory-augmented agents toward genuinely qualified agent evolvers.
1 Introduction
The introduction defines a qualified agent evolver as one that jointly selects, uses, writes, and manages experience rather than merely storing memory. OPD-Evolver addresses this goal with fast online evolution and slow on-policy self-distillation, and its 4B/9B models surpass contemporary memory systems while remaining competitive with giant counterparts.
- Motivation: Memory provides the substrate for self-evolution, but preserving trajectories, retrieving lessons, and reusing skills does not itself define a self-evolving agent.Memory has become an indispensable component of modern agentic foundation models and agent systems, especially in interactive environments where feedback reveals constraints and reusable strategies.
- Motivation: Existing memory-, skill-, reflection-, and self-improvement-based agents may improve within particular settings yet still fall short of becoming holistic agent evolvers.The introduction frames this shortcoming as a lack of integrated competence across the four experience-centered abilities.
- Objective Formulation: A qualified agent evolver requires four coupled capabilities: experience selection, experience-grounded execution, experience writing, and experience management.These capabilities respectively identify useful memories, convert them into actions, extract reusable knowledge, and score, consolidate, update, or retire memories.
- Practical Solution: OPD-Evolver combines a fast loop for online interaction with environments and memory and a slow loop that distills interaction-derived behaviors through on-policy self-distillation.The framework operates over trajectories, tips, skills, and tools, then uses attribution-calibrated hindsight supervision to shape selection, execution, writing, and management.
- Experimental Validation: OPD-Evolver-4B/9B surpass contemporary memory systems and remain competitive with giant counterparts such as QWEN3.5-397B and STEP-3.5-196B.These results come from evaluations on four self-evolving benchmarks.
2 Related Work
Prior self-evolving agents organize around experience selection and experience-grounded execution, while on-policy distillation reduces train–inference mismatch by supervising students on their own visited states. OPD-Evolver instead applies OPD to jointly cultivate the four capabilities of a holistic agent evolver.
- Self-Evolving Agents: Prior self-evolving agents select useful memories through embedding retrieval, utility scoring, learned routers, or policy-based ranking.These approaches are framed within the experience lifecycle defined by this work.
- On-Policy Distillation: On-policy distillation trains students on their own visited states using dense teacher supervision, reducing train–inference mismatch relative to off-policy distillation.Recent studies apply OPD to mathematical reasoning, knowledge QA, and tool-use tasks.
- On-Policy Distillation: OPD-Evolver uses on-policy distillation not merely to strengthen execution, but to jointly cultivate the four capabilities required by a holistic agent evolver.Its fast loop interacts with a four-level memory hierarchy, while its slow loop converts outcome-calibrated hindsight into self-distillation signals.
3 Methodology
OPD-Evolver trains a closed-loop agent evolver that retrieves, uses, writes, and maintains reusable experience while optimizing both current task returns and future repository utility. Its fast loop performs online evolution, and its slow loop converts outcome-calibrated hindsight into unified on-policy supervision for these lifecycle abilities.
- 3.1 Problem formulation: The agent evolver maps each task-memory state to task behavior and an updated repository through retrieval, selection, execution, experience writing, and maintenance.The repository must provide relevant evidence, actionable context, reusable updates, and growing future utility.
- 3.1 Problem formulation: The streaming objective maximizes task return while preserving the downstream usefulness of the updated repository, with λ controlling the value assigned to future evolvability.The objective is instantiated by a fast online loop and a slow hindsight-distillation loop.
- 3.2 Fast loop: The fast loop performs test-time evolution without changing θ, transforming the repository into actionable context before execution and episodes into reusable experience afterward.Its four-tier hierarchy stores trajectories, tips, skills, and tools at different reuse granularities.
- 3.2 Fast loop: High-recall retrieval surfaces top-K candidates from each memory tier, after which the selector compresses potentially stale, redundant, or mismatched items into the operational context.The same agent conditions execution on this context, chooses which tiers to update, writes variable numbers of memories, and periodically merges or deletes repository items.
- 3.3 Slow loop: Outcome-calibrated attribution estimates each memory’s value only on tasks where it was retrieved, using selection reliability and delayed environment feedback to produce hindsight labels.High-value memories guide selection, execution, writing, and preservation, while low-value memories indicate noise to ignore.
- 3.3 Slow loop: Unified on-policy self-distillation trains selection, execution, writing, and maintenance by giving the teacher privileged hindsight while deploying only student-facing behavior in the fast loop.The teacher evaluates lifecycle decisions using memory value, successful trajectories, usage statistics, redundancy diagnostics, and merge/delete trajectories.
4 Experiments
OPD-Evolver outperforms same-backbone memory systems and training-based methods across diverse self-evolving agent benchmarks, while also challenging much larger models. Ablations and analyses show that jointly distilling memory attribution, selection, writing, maintenance, and useful experience is central to reliable self-evolution.
- Comparison with Memory Systems: OPD-Evolver achieves the best result on all 10 benchmark subsets among same-backbone memory baselines for both 4B and 9B.For 9B, it beats the strongest memory baseline on OS (65.00% vs. 61.00%), AMA-SA (52.92% vs. 48.00%), and InterCode-CTF (57.00% vs. 53.00%).
- Comparison with Memory Systems: OPD-Evolver-9B exceeds STEP-3.5-FLASH (196B) on 9/10 subsets and surpasses QWEN3.5-397B-A17B on 6/10 subsets.Examples include AMA-SA (52.92% vs. 51.41%), InterCode-CTF (57.00% vs. 56.00%), and SQL (64.01% vs. 62.74%).
- Comparison with Training-based Methods: OPD-Evolver achieves the best average performance among training-based methods, winning on 5/6 subsets.Compared with GRPO, it improves Maze (27.45% vs. 23.53%), KeyRoom (9.80% vs. 3.92%), Bash (64.01% vs. 63.69%), CTF (59.00% vs. 58.00%), and SQL (49.55% vs. 47.77%).
- Ablation Study: Similarity-only selection reduces SQL from 45.86% to 42.04%, while removing writing distillation lowers CTF from 34.00% to 29.00%.Together with the ablation findings, these results support jointly training calibrated attribution, learned selection, memory writing, and repository maintenance.
- Memory Selection and Writing: Selection distillation raises median selected-memory scores from 0.66/0.69/0.66 to 0.79/0.76/0.76 on SQL/CTF/Bash.Writing distillation likewise increases median written-memory scores from 0.80/0.82/0.82 to 0.91/0.90/0.89 on SQL/CTF/Bash, with the lower quartile reaching 0.83 or higher.
- Impact of Experience Internalization: OPD-Evolver raises success by about 3–4 points for 4B and 3–7 points for 9B on Bash/CTF/SQL while reducing trajectories by up to 2.5 steps.This experience internalization is measured without external memory during task execution.
5 Conclusion · A Related Work Discussion · B Potential Risks
OPD-Evolver is presented as a slow-fast co-evolution framework that improves compact open models across diverse benchmarks and transfers learned evolver abilities beyond training environments. The discussion distinguishes its unified experience lifecycle from related work and notes that standard agentic-LLM risks remain relevant.
- 5 Conclusion: OPD-Evolver combines online memory-conditioned interaction with outcome-calibrated attribution and privileged on-policy self-distillation.The framework is described as a slow-fast co-evolution approach.
- 5 Conclusion: OPD-Evolver consistently improves compact open models over memory-augmented and training-based baselines across diverse evaluation benchmarks.The paper states that these improvements transfer beyond the training environments.
- A Related Work Discussion: SKILL0 and OpenClaw-RL internalize agent execution behaviors through RL or OPD, whereas OPD-Evolver additionally addresses experience selection, writing, and maintenance.This comparison places OPD-Evolver’s focus beyond execution-behavior internalization.
- A Related Work Discussion: SkillRL, Trace2Skill, and Skill-R1 mainly improve skill or lesson generation, corresponding most closely to OPD-Evolver’s experience-writing capability.These works are characterized as focusing primarily on experience writing within the broader lifecycle.
- A Related Work Discussion: Skill-SD uses skills as privileged context during distillation, while SLIM and Skill1 take broader views of skill lifecycle management.The cited works emphasize experience-guided execution or broader skill lifecycle management.
- A Related Work Discussion: OPD-Evolver studies a unified evolver policy over trajectories, tips, skills, and tools rather than a single experience form.The related-work discussion presents this as a broader unified treatment of experience forms.
- B Potential Risks: No direct evidence indicates that OPD-Evolver introduces qualitatively new social risks beyond those associated with LLM-based agents.The risk discussion nevertheless emphasizes continued attention to deployment safety.
- B Potential Risks: Improved experience reuse and memory maintenance leave concerns about unsafe tool use, privacy leakage, biased behavior, and unintended task generalization requiring careful attention.Deployments should follow safeguards expected for agentic LLM systems.
C Benchmark Details
The evaluation spans lifelong interaction, multi-session memory reasoning, completed-trajectory memory, interactive coding, and procedurally generated navigation benchmarks. These benchmarks vary in domains, test-set sizes, session structures, interfaces, and action spaces.
- LifelongAgentBench: LifelongAgentBench evaluates MySQL database and Ubuntu-container operating-system tasks, with 100 test tasks per subset.Database agents receive natural-language instructions and database context, acting through SQL execution followed by final submission.
- MemoryArena: MemoryArena evaluates multi-session reasoning on Math and Physics subsets containing 354 and 86 test tasks, respectively.Each task provides shared background and sequential question-answer sessions requiring preservation and reuse of prior information.
- AMA-Bench: AMA-Bench uses 208 episodes with 2,496 question-answer pairs, focusing on Causal Inference, State Updating, and State Abstraction.The full taxonomy also includes Recall, with 839 pairs, alongside 596 Causal Inference, 647 State Updating, and 414 State Abstraction pairs.
- InterCode: InterCode evaluates SQL, Bash, and CTF subsets containing 314, 224, and 100 test tasks, respectively, through isolated execution environments.Agents issue SQL queries, execute bash commands, or attempt capture-the-flag challenges depending on the subset.
- MiniHack: MiniHack evaluates Room, Maze, and KeyRoom, with 51 test tasks for each procedurally generated grid-world environment.The tasks require reaching goals, systematic maze exploration, or locating a key and opening a locked door before reaching the goal.
C.1 Evaluation Protocol
The evaluation protocol tests memory-based agents in a streaming setting that begins with empty repositories and permits growth only from experience observed during evaluation. Task-level success or failure feedback supports memory updates and attribution, without providing ground-truth answers or expert guidance.
- C.1 Evaluation Protocol: Evaluation starts each memory-based method with an empty experience repository, M0 = ∅.
- C.1 Evaluation Protocol: Agents solve tasks sequentially in streaming order, growing their repositories only from trajectories, memories, and feedback observed during the current evaluation stream.
- C.1 Evaluation Protocol: After each task, task-level success-or-failure feedback is used for memory update and attribution, while ground-truth answers and expert guidance are withheld.
D Training Details · D.1 Training Data
OPD-Evolver’s training data comprises 7,000 interactive agent tasks drawn from three heterogeneous sources. These trajectories expose memory selection, writing, memory-conditioned execution, and repository maintenance across executable, terminal, and scalable tool-interactive scenarios.
- D.1 Training Data: 7,000 interactive agent tasks form the training dataset, spanning diverse executable and tool-interactive scenarios.The data are designed to expose OPD-Evolver to heterogeneous trajectories for memory selection, writing, memory-conditioned execution, and repository maintenance.
- D.1 Training Data: 3,000 tasks come from Agent World Model (AWM), 2,000 from nvidia/Nemotron-Terminal-Corpus, and 2,000 from EnvScaler.This allocation totals the 7,000-task training set.
- D.1 Training Data: AWM supplies synthetic multi-turn tool-use environments with executable code-driven transitions and database-backed application scenarios.Its trajectories provide reliable task-level outcomes and rich tool interactions.
- D.1 Training Data: nvidia/Nemotron-Terminal-Corpus contributes terminal-style interactions involving command execution, shell operations, and multi-step problem solving.It complements AWM with command-line workflows and environment-feedback patterns.
- D.1 Training Data: EnvScaler broadens the training distribution with scalable tool-interactive environments and varied interaction structures.Its trajectories reduce reliance on a single benchmark style.
- D.1 Training Data: For Table 2’s training-based comparison, all methods are trained on MiniHack using identical training data and evaluated on identical test tasks.This protocol is intended to ensure a fair comparison.
D.2 Training Parameters
The experiments use standardized backbone, precision, context-length, sampling, and episode-length settings, with method-specific hyperparameters for SFT, GRPO, and OPD-Evolver. OPD-Evolver uses a four-type memory pool with staged retrieval, filtering, selection, and periodic maintenance.
- Shared Settings: Shared settings use Qwen3.5-9B and Qwen3-4B backbones with bf16, 8192-token prompts, and 40-step maximum episodes.GRPO uses temperature 1.0 and top-p 1.0; OPD-Evolver distillation data generation uses temperature 1.0 and top-p 0.95.
- SFT: SFT fine-tunes the base model with LoRA using learning rate 1 × 10^-5, per-device batch size 2, gradient accumulation 4, and 3 epochs.The LoRA rank is 32, scaling factor 64, and dropout 0.05.
- GRPO: GRPO rewards use environment cumulative reward, with penalties of −0.1 for invalid actions and −0.2 for premature submission.The KL term is not included in the reward.
- OPD-Evolver: OPD-Evolver stores skills, tips, tools, and trajectories, retrieves 50 teacher-side candidates, injects 20, and filters memories below score 0.01.Writer-side selection keeps the top 30% by memory score, and maintenance occurs every Q = 30 tasks.
E More Results
On LifelongAgentBench DB and OS environments, OPD-Evolver’s slow-loop training improves both memory selection and writing. The resulting memory-score distributions shift upward, with fewer low-score memories selected or stored for future reuse.
- Additional LifelongAgentBench Results: OPD-Evolver shifts memory-score distributions upward versus the vanilla model on LifelongAgentBench DB and OS, improving selected and written memories.Figures 6 and 7 evaluate selection and writing distillation effects, respectively.
- Selection Distillation: OPD-Evolver selects higher-score memories than the base model on both DB and OS tasks, reducing low-score selections.The reduced low-score region indicates less injection of broad, noisy, or weakly related memories.
- Writing Distillation: OPD-Evolver’s writer produces higher and more concentrated calibrated memory scores than the vanilla writer on both DB and OS tasks.The reduction of low-score memories suggests less storage of generic or misleading advice.
F Case Study
The case studies show that slow-loop training makes OPD-Evolver more task-specific across execution, memory selection, and memory writing. Compared with vanilla models, it better respects environmental constraints, retrieves causally useful memories, and records narrower failure-targeted knowledge.
- Case Study: Across the three pipeline components, vanilla models often overgeneralize from prior experience or fail to localize the immediate cause of failure.The qualitative examples span InterCode-SQL, LifelongAgentBench-OS, and MiniHack.
- EXECUTOR: OPD-Evolver improves execution by respecting task-level constraints across InterCode-SQL, MiniHack, and LifelongAgentBench-OS.It separates SQL actions, navigates until the goal tile, and performs missing directory creation and permission-setting operations.
- SELECTOR: OPD-Evolver selects memories that directly support the current problem instead of injecting superficially related or empty memories.This contrasts with vanilla selection of minimum-count SQL, user/group setup, or empty injections after JSON parse failure.
- WRITER: OPD-Evolver writes narrower future-facing memories that target the root cause rather than broad advice or incorrect environment APIs.Examples include verifying column names, interpreting malformed schema queries, outputting bare filenames, and checking generated file lists.