From Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agents

1 Introduction

• Existing memory systems often retrieve factual or experiential records as passive context, leaving agents to reason over them again.This can repeat work, such as rediscovering repository files or reconstructing procedures from prior traces.Source: §1 · pp. 1–2
• Turning trajectories into reusable skills is difficult because traces contain failures and artifacts, feedback is sparse, and skills need applicability boundaries and verification.Direct distillation can overfit trajectories and lead to inappropriate skill invocation.Source: §1 · p. 2
• MSCE separates experience into grounded L1 traces, recurring L2 procedural policies, and L3 declarative environmental cognition.This hierarchy distinguishes evidence, procedure, and environmental knowledge while supporting verifiable, revisable skill deployment.Source: §1 · p. 2
• MSCE promotes L2 policies into skills only when they retain evidence, have positive estimated gain, and remain consistent with their trigger, procedure, and boundary.Reflection-weighted value backfilling propagates sparse terminal feedback through local self-reflections to produce trace values for memory and skill updates.Source: §1 · p. 2
• MSCE achieves state-of-the-art performance on EvoAgentBench and LoCoMo, with reported cross-domain transfer and lifelong-evolution gains.The framework is presented as effective for long-horizon and multi-domain environments.Source: §1 · p. 2

2.1 Memory for LLM Agents

• Factual and experiential memory systems improve recall or summarize trajectories, but retrieved memories often remain passive context requiring further agent reasoning.Factual systems can also burden context with episodic noise and lack higher-level abstraction.Source: §2.1 · pp. 2–3

2.2 Skills for LLM Agents

• Skill acquisition uses curated external expertise, trajectory distillation, or reinforcement learning, while MSCE inserts evidence-preserving memory before skill deployment.MSCE separates three-level memory from executable skills and links them through evidence anchors and value signals to govern promotion, applicability, and revision.Source: §2.2 · p. 3

3 Background

• An episode is MSCE’s feedback-and-update unit, and each episode contains step traces recording semantic context, action, observation, and self-reflection.Source: §3 · p. 3
• At each step, the agent checks whether the current state matches a skill trigger; a retrieved skill guides action generation, otherwise the agent acts from the current state.Terminal feedback may be verifier reward or textual user feedback converted to a scalar.Source: §3 · p. 3
• The objective is to maximize cumulative episode-level feedback, which may reflect task success, user satisfaction, process quality, safety, or efficiency.Source: §3 · p. 3

4.1 Overview

• MSCE separates auditable L1 traces, revisable L2 policies, L3 environmental knowledge, and downstream skills; it crystallizes evidence-backed positive-gain policies and backfills values from terminal feedback.Source: §4.1 · pp. 3–4

4.2 Three-level Memory

• L1 preserves links from higher-level memories and skills to supporting traces, while normalized, capped evidence limits persisted raw observations.Source: §4.2 · p. 4
• L2 policies encode triggers, procedures, verification or fallback criteria, boundaries, and supporting L1 evidence; valued traces update matching policies or enter candidate pools for induction.Induction requires evidence from at least nmin distinct episodes.Source: §4.2 · pp. 4–5
• Policy gain compares values for linked traces with other traces in the evaluation pool and serves as a heuristic governance signal, not a causal-effect estimate.Source: §4.2 · p. 5
• L3 abstracts declarative environmental facts, action-response regularities, and constraints from multiple active policies, excluding imperative procedures and reducing retrieval priority when later evidence weakens support.Source: §4.2 · p. 5

4.3 Skill Crystallization and Invocation

• A policy becomes a callable skill only when it retains evidence, has positive estimated gain, and remains consistent with its trigger, procedure, and applicability boundary.Skills add evidence anchors, decision guidance, and reliability estimates to inherited policy fields.Source: §4.3 · p. 5
• A deterministic verifier rejects drafts that fail schema or evidence-grounding checks; deployed skills gain or lose reliability with outcomes and may be active, probationary, or archived.Invocation uses L3 environmental priors when available and falls back to traces if no skill applies or a skill fails.Source: §4.3 · pp. 5–6

4.4 Online Update via Dual-Signal Value Backfilling

• After terminal feedback arrives, MSCE blends it with discounted next-step values using reflection weights, favoring locally informative steps; valued traces then govern memory and skill updates.Reflections may be noisy or unfaithful, while terminal feedback is more reliable but sparse and delayed.Source: §4.4 · p. 6

5.2 Main Results

• MSCE achieves the best or tied-best Pass@1 in all five EvoAgentBench domains, including a tie at 61.54% on Code while reducing cost from 3.9 to 2.0 turns.Against the strongest non-MSCE baseline, Pass@1 gains are 4.61 points on IR, 4.00 on Math, 15.39 on SE, and 5.17 on KW.Source: Table 1 · p. 7, §5.2 · p. 7

5.1 Experimental Settings

• The evaluation uses EvoAgentBench’s five domains and LoCoMo separately, compares three baseline categories, and gives methods identical tools, budgets, and task order.GPT-5.2 is the agent backbone; online methods accumulate memory and skills without updating the base model, and Pass@1 is the main EvoAgentBench effectiveness metric.Source: §5.1 · p. 7

5.2 Main Results

• On LoCoMo, MSCE beats SkillFlow-Evolve by 2.01 points in overall judge score and 1.18 points in overall F1, with highest scores on single-hop, multi-hop, and temporal questions.MSCE ranks second on open-domain questions, while EverOS scores highest there excluding Vanilla Agent.Source: Table 2 · p. 8, §5.2 · p. 8

5.4 Ablation Study

• Replacing MSCE with Flat Memory drops Pass@1 by 15.38, 16.00, and 19.23 points on IR, Math, and SE; component ablations also show broad performance or cost degradation.Removing skill crystallization reduces Pass@1 by 6.15–11.54 points across domains and raises cost in every domain; removing L3, reflection weighting, or value calibration also consistently hurts performance.Source: Table 3 · p. 8, §5.4 · p. 10

5.2 Main Results

• MSCE reduces cost against the strongest non-MSCE baseline on IR, Math, Code, and KW, but increases SE cost from 37.3 to 40.8 turns while gaining 15.39 Pass@1 points.The authors associate the SE increase with repository navigation, editing, and testing often required by successful solutions.Source: §5.2 · p. 8

5.3 Generalization Analysis

• Cross-domain transfer improves Pass@1 in all six transfer pairs by +2.56 to +5.13 points, averaging +3.93 points, while cost decreases in four pairs.Transfers into Code raise cost by +4.6% and +8.1% while also improving Pass@1; the authors characterize the results as transferable problem-solving structure.Source: §5.3 · pp. 8–9, Figure 2 · p. 9
• Under lifelong evolution, Pass@1 rises monotonically from p0 to p100, with gains of 17.00 points on Math, 15.39 on SE, and 13.84 on IR.Normalized cost initially increases at p25 and then falls toward p100; at p100 it is below p0 on Math and SE.Source: Figure 3 · p. 9, §5.3 · p. 9

• MSCE organizes agent experience into grounded traces, reusable policies, and environmental cognition, then crystallizes evidence-backed, positive-gain policies into skills.Source: §6 · p. 10
• Experiments validate MSCE on long-horizon self-evolution and conversational memory benchmarks, with reported cross-domain transfer and lifelong-evolution capabilities.Source: §6 · p. 10
• The paper presents MSCE as a foundation for connecting memory and skills in more efficient, autonomously evolving LLM agents.Source: §6 · p. 10

Limitations

• MSCE's heuristic value, policy-gain, and reliability estimates support ranking, filtering, and revision, but do not establish counterfactual causal effects.Source: Limitations · p. 10
• Prompted LLM update operators may remain noisy or sensitive to model and prompt choices despite evidence constraints and verification, while adding latency and cost.Source: Limitations · p. 10
• The implementation depends on a particular runtime, tool interface, and backbone, so absolute results may vary across models or environments.Source: Limitations · p. 10
• Truncation, deduplication, and redaction reduce but do not eliminate risks of retaining sensitive information, environment secrets, or unsafe procedural knowledge.The paper recommends stronger privacy filters, access control, and safety checks for generated skills.Source: Limitations · p. 10

A.1 Self-Evolution of LLM Agents

• MSCE couples sparse external feedback with internal reflection through reflection-weighted value backfilling to guide memory retrieval, policy induction, and skill crystallization.Source: §A.1 · p. 14

B.2 Episode Definition and Boundary Management

• An episode is MSCE's feedback and update unit, containing ordered response or tool-action traces; tasks may span multiple episodes, and boundaries distinguish corrections, follow-ups, and new tasks.Episodes scope shared terminal feedback, trace valuation, cross-episode policy evidence, and episode-level retrieval.Source: §B.2 · pp. 14–15

B.3 Reward Quantification from Textual Feedback

• When numerical rewards are unavailable, an LLM evaluator scores textual feedback for goal achievement, process quality, and user satisfaction to produce a normalized reward; a polarity heuristic is the fallback.Source: §B.3 · p. 15

B.4 Policy Gain for L2

• L2 policy gain compares values of linked and unlinked traces using a weighted with-policy aggregate and a conservative-baseline-shrunk without-policy aggregate; it remains a heuristic utility estimate, not a causal effect.Source: §B.4 · p. 16

B.5 L2 Association, Induction, and Revision

• Valued traces associate with matching L2 policies or enter candidate pools; induction requires evidence across distinct episodes, while sustained negative gain retires policies and procedural drift can trigger re-induction.Source: §B.5 · p. 16

B.6 L3 Abstraction and Confidence

• L3 abstraction groups active policies by domain, prompts for declarative environmental knowledge, and merges or adds validated cognitions; confidence changes with supporting or contradicting evidence.Low-confidence cognitions are excluded from default retrieval until further support appears.Source: §B.6 · p. 17

B.7 Online Update Operator U

• The online update operator finalizes traces, backfills values after feedback, updates eligible L2 policies, abstracts L3 knowledge, and verifies and deploys eligible skills.Source: §B.7 · p. 17, §B.9 · p. 18

B.8 Skill Lifecycle

• Crystallized skills move from probationary to active when reliability clears a threshold after sufficient use, and may be archived after repeated failures, rejection, or negative source-policy gain.Feedback can adjust reliability and boundary violations can shrink applicability boundaries.Source: §B.8 · p. 17

B.9 Hierarchical Retrieval

• Hierarchical retrieval prioritizes matching active skills; if none applies or one fails, it retrieves L1 traces, while structural uncertainty prompts L3 environmental priors that do not prescribe actions.Source: §B.9 · pp. 17–18

B.10 Decision Guidance and Repair

• MSCE creates decision guidance when similar contexts yield action patterns with substantially different values and the lower-valued pattern has failure or correction support.Guidance records preferred and avoided actions with evidence links and reliability, and is injected only when context matches a skill or policy trigger.Source: §B.10 · p. 18

B.11 Storage and Privacy Controls

• Truncation, deduplication, and rule-based redaction limit sensitive-content propagation, but do not eliminate privacy risks from stored traces and related memory objects.Source: §B.11 · p. 19

C Example: Environment Dependencies

• Across two dependency-resolution tasks, backfilled traces support induction of a container-specific missing-library policy that can be crystallized into a skill after stable positive gain.The example's policy trigger is pip failure from missing system libraries in a container; its boundary excludes native systems with pre-installed libraries.Source: C Example: Environment Dependencies · p. 19

D.1 Experimental Settings

• Benchmark-level Pass@1 is the unweighted mean of task-level Pass@1, with standard error computed across tasks.Source: §D.1 · p. 20
• Rewards are binary in IR, Math, SE, and Code, while KW uses rubric scoring with scores below 0.6 mapped to failure and success requiring r > 0.6.Source: §D.1 · p. 20
• The listed principal MSCE hyperparameters are implementation choices, not intrinsic values, and can be tuned for different environments.Source: §D.1 · p. 20, Table 6 · p. 20

D.2 Effect of Simulated Human Feedback

• Simulated feedback improved Pass@1 in four of five domains, with gains of +13.79 points on KW and +7.69 on SE; Math was unchanged.Code and IR improved by +2.56 and +1.54 points, respectively. C1 reduced cost on Code, IR, and SE but increased it on KW and Math; feedback was injected during training, not evaluation.Source: Table 7 · p. 21, §D.2 · p. 21

E Prompted Operators

• Five evidence-constrained LLM operators return structured fields and evidence identifiers; outputs with missing fields or unsupported evidence or tools are discarded.The operators cover reflection scoring, reward quantification, L2 policy induction, L3 environmental cognition abstraction, and skill drafting.Source: E Prompted Operators · p. 21

E.1 Reflection Scoring

• Reflection scoring rates faithfulness, causal insight, transferability, and concreteness, assigning empty or tautological reflections α_i,t = 0.When scoring is disabled, non-empty reflections receive α_i,t = 0.5; malformed judge outputs fall back to existing reflection metadata rather than updating the trace.Source: §E.1 · pp. 21–22

E.2 Reward quantification.

• When numerical rewards are unavailable, a reward evaluator converts textual feedback into terminal feedback by scoring goal achievement, process quality, and user satisfaction.Its inputs include the original query, final output, episode summary, and user feedback.Source: §E.2 · p. 22

E.3 L2 induction.

• L2 induction derives procedural policies from evidence traces, specifying a detectable trigger, action procedure, verification rule, applicability boundary, and supporting evidence.The prompt keeps declarative environmental facts in the separate L3 layer.Source: §E.3 · p. 23

E.4 L3 abstraction.

• L3 abstraction extracts declarative environmental knowledge as entities or structures, action–response regularities, and constraints, while forbidding imperative instructions.It uses active L2 policies and representative evidence, and separates environmental facts from L2 procedures.Source: §E.4 · p. 24
• L3 cognition is framed as environmental knowledge rather than procedure, with a schema for environment, inference, and constraint entries.The supplied example distinguishes an environmental description from a procedural instruction, which belongs at L2.Source: §E.4 · p. 25

E.5 Skill drafting.

• Skill drafting turns an L2 policy and supporting evidence into a callable procedure, accepting only tools and claims grounded in the evidence set.Inputs include successful traces, an observed-tool whitelist, optional counter-examples and repair hints, and existing skill names; the output includes preconditions, ordered steps, examples, and decision guidance.Source: §E.5 · p. 25