PaperlayerView exact revision on arXiv

Source-linked AI summary

Principles that Guide, Actions that Inform: Agent Evolution via Knowledge Abstraction

Bowen Ye, Yongchao Xu, Junkai Ma, Xiang Yin, Wenzhao Li

arXiv:2610.06964v1cs.AIcs.LG

TL;DR

LLM agents lack mechanisms for continual adaptation from interaction, and existing memory often remains tied to particular tasks and contexts. SAGA builds an evidence-linked hierarchy of reusable knowledge and uses task-conditioned principles to guide and refine actions. Experiments report improved performance on ScienceWorld and ALFWorld, with ablations highlighting complementary roles for contextual instantiation and action regulation.

  • Problem

    Existing memories often remain episodic or task-specific, leaving open how to derive reusable principles with applicability conditions for lifelong adaptation.

    Source: §1 · p. 2

  • Method

    SAGA abstracts trajectories into episodic descriptions, procedures, and principles linked to execution evidence, then instantiates principles as task guidance and uses checks to refine actions.

    Source: §1 · p. 2

  • Results

    SAGA improves performance on ScienceWorld and ALFWorld, while ablations identify contextual instantiation and action regulation as complementary mechanisms for principle-level knowledge reuse.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    SAGA connects experience abstraction to task-specific guidance and execution-time action refinement in a feedback loop updated by new trajectories.

    Source: §1 · p. 2

    Procedures that depend strongly on individual task instances have limited transferability across diverse tasks and environments.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Large language model (LLM) agents have demonstrated strong capabilities in interactive environments, yet their ability to continually evolve from experience remains limited. Although fine-tuning enables adaptation, its dependence on parameter access and high computational costs restrict its flexibility, especially for large-scale and closed-source LLMs. External memory offers an alternative by allowing agents to accumulate experience without modifying model parameters. However, existing methods mainly focus on experience representation and organization, while the acquired knowledge remains tightly coupled with specific tasks and contexts, limiting generalization. A key challenge is how to transform concrete interactions into abstract and reusable knowledge that guides future decisions beyond individual experiences. To address this challenge, we propose SAGA (\underline{\textbf{S}}elf-evolving \underline{\textbf{A}}gents through Experience-\underline{\textbf{G}}rounded \underline{\textbf{A}}bstraction), a framework for experience-grounded knowledge abstraction and utilization in LLM agents. SAGA progressively transforms interaction trajectories into episodic descriptions, reusable procedures, and principles with explicit applicability conditions, while maintaining links to execution evidence. Retrieved principles are instantiated into task-specific guidance and used to refine candidate actions through corrective feedback and resampling. This creates an execution--abstraction feedback loop, where accumulated knowledge guides future interactions and new experiences continuously update hierarchical memory. Experiments on ScienceWorld and ALFWorld demonstrate improved task performance, with ablation studies highlighting the importance of contextual instantiation and action regulation for leveraging principle-level knowledge.

1 INTRODUCTION

LLM agents need ways to learn from interaction after deployment, but existing memory often preserves task-specific experiences without extracting transferable principles. SAGA links evidence-grounded knowledge abstraction to task-conditioned guidance and action refinement, and reports improved task performance.

  • Fine-tuning and reinforcement learning can adapt agent behavior but require substantial computational resources and model-parameter access; external memory offers another route to adaptation.Source: §1 · p. 2
  • Existing memory methods largely store and retrieve episodic records, retrieved contexts, or task-specific procedures, leaving transferable principles and applicability conditions an open challenge.Source: §1 · p. 2
  • SAGA organizes experiences into trajectories, episodic descriptions, procedures, and principles with applicability conditions, then instantiates retrieved principles into task guidance and regulates candidate actions.Links between abstraction levels preserve supporting execution evidence, and new trajectories update the knowledge representation.Source: §1 · pp. 2–3
  • Experiments across ScienceWorld and ALFWorld show improved task performance, while ablations identify contextual instantiation and action regulation as complementary mechanisms for principle-level reuse.Source: §1 · p. 3

2 RELATED WORK

Prior work explores agent adaptation through reflection, skills, experience reuse, and external memory. SAGA emphasizes connecting multi-level experience abstraction to execution evidence, then turning retrieved principles into operational guidance and action checks.

  • Experience-driven agents use reflection, accumulated skills, retrieved experiences, or parameter updates to improve behavior and execution.Source: §2 · p. 3
  • External-memory research develops memory management, long-term personalization, graph retrieval, temporal architectures, and evolving context, while SAGA links abstraction levels to execution evidence.Source: §2 · p. 3
  • Prior abstraction methods include executable skills (Wang et al., 2023), cross-task insights (Zhao et al., 2024), causal abstractions, and conditional guidelines (Fu et al., 2024).Source: §2 · p. 3
  • Unlike AutoManual (Chen et al., 2024) and AutoGuide (Fu et al., 2024), SAGA couples multi-level abstraction with post-retrieval instantiation, action regulation, and evidence from new execution trajectories.Source: §2 · pp. 3–4

3 SAGA: EXPERIENCE-GROUNDED KNOWLEDGE EVOLUTION

SAGA forms a feedback loop in which interaction experience becomes hierarchical, evidence-linked knowledge that guides later execution and is refined by new trajectories. Retrieved knowledge is adapted to the current task and used to regulate candidate actions.

3.2 HIERARCHICAL KNOWLEDGE REPRESENTATION

  • SAGA organizes experience into four levels: raw trajectories, episodic descriptions, reusable procedures, and experience-grounded principles.Figure 3 uses blue, green, yellow, and red for L0 through L3, respectively.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.1 OVERVIEW

  • Raw trajectories preserve fine-grained execution evidence, but their dependence on individual task instances limits transfer across diverse tasks and environments.SAGA’s execution–abstraction loop turns such trajectories into reusable knowledge for future decision making.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.2 HIERARCHICAL KNOWLEDGE REPRESENTATION

  • Episodic knowledge compresses an individual trajectory, whereas procedures aggregate shared patterns across episodes into generalized strategies for related tasks.Procedural representations remain grounded in original trajectories, and multiple episodes may contribute to one procedure.Source: §3.2 · p. 5
  • Principles capture regularities across procedures and provide higher-level guidance while remaining traceable to their supporting experiences.The hierarchy preserves evidence links across abstraction levels.Source: §3.2 · pp. 4–5

3.3 KNOWLEDGE UTILIZATION AND SELF-EVOLVING LOOP

  • SAGA retrieves task-relevant knowledge, instantiates principles for the current task, and uses violated principles as feedback to resample candidate actions.The task description and a generated plan inform retrieval; principles regulate actions without replacing the generative policy.Source: §3.3 · p. 6
  • New trajectories are abstracted into knowledge and added to memory, while successful executions reinforce strategies and failures support knowledge refinement.This repeated retrieval, execution, and abstraction cycle is SAGA’s self-evolving loop.Source: §3.3 · p. 7

4 EXPERIMENTS

Experiments evaluate SAGA on ScienceWorld and ALFWorld under prompt-based and training-based adaptation, then ablate abstraction and utilization components. Results report improved performance, with ablations highlighting contextual instantiation and action regulation.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans prompt-based experience evolution with fixed parameters and training-based adaptation, comparing SAGA with reasoning, experience-augmentation, and training-based baselines.ScienceWorld reports average final task scores; ALFWorld reports task success rates.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • ScienceWorld GRPO reaches 66.9% at the selected checkpoint, improving final evaluation performance from the SFT initialization despite intermediate fluctuations.The reported trajectory concerns experience-guided parameter adaptation.Source: §4.2 · p. 8, Figure 4 · p. 8
  • 99.3% overall success rate with Qwen3.8-Max; ALFWorld memory augmentation raises Qwen2.5-7B-Instruct success from 45.5% to 57.5%.Training-based adaptation reaches 90.3% after SFT and 93.3% after GRPO; external test-time adaptation reaches 96.3%.Source: §4.2 · p. 8
  • SAGA maintains strong performance on unseen ALFWorld tasks, which are excluded from experience collection and parameter optimization in non-External configurations.The evaluation covers held-out Look and Pick2 task types.Source: §4.1 · p. 7, §4.2 · p. 9

4.3 ABLATION STUDIES

  • Raw trajectory retrieval improves scores over memory-free execution, while adding principle-level knowledge directly decreases scores relative to episodic and procedural knowledge.Replacing trajectories with episodic and procedural knowledge adds 0.8 and 0.2 points for the two backbones; the pattern indicates that abstraction alone does not ensure better execution.Source: §4.3 · p. 9
  • Task-conditioned instantiation raises scores from 63.9 to 66.2 with Qwen3.8-Max and from 54.5 to 55.8 with DeepSeek-v4-Flash, compared with uninstantiated principles.The instantiated configuration also exceeds episodic-plus-procedural knowledge by 1.7 and 0.8 points, respectively.Source: §4.3 · p. 9
  • The complete SAGA configuration exceeds instantiated guidance alone by 0.8 points with Qwen3.8-Max and 0.4 points with DeepSeek-v4-Flash.Action regulation also yields 3.3- and 1.4-point gains over raw trajectory retrieval, respectively.Source: §4.3 · p. 9

5 CONCLUSION

The conclusion presents SAGA as experience-grounded knowledge abstraction linked to execution. Experiments show improvements on ScienceWorld and ALFWorld, and ablations identify contextual instantiation and action regulation as important to using principle-level knowledge.

  • SAGA turns interaction trajectories into episodic descriptions, reusable procedures, and principles with explicit applicability conditions, while retaining links to execution evidence.Retrieved principles become task-specific guidance and principle-derived checks regulate later decisions.Source: §5 · p. 9

A HIERARCHICAL MEMORY CONSTRUCTION

The memory construction process incrementally converts training interactions into linked, evidence-grounded levels of knowledge. Selection, abstraction, validation, and updating preserve provenance while constraining how procedures become principles.

  • Memory links raw trajectories, episodic records, procedures, and principles through adjacent-level parent-child links, preserving traceability to source task metadata.Principles specify condition, effect, mechanism, scope, and exceptions, and are treated as evidence-grounded explanatory hypotheses rather than verified causal laws.Source: A HIERARCHICAL MEMORY CONSTRUCTION · p. 12
  • Construction starts from memory-free training interactions, then extracts episodic records, groups them into procedures, and lifts supported procedures into principles.Cross-task merging uses similarity 0.75 to generate candidate clusters, but a principle is admitted only with evidence from at least two task types.Source: A HIERARCHICAL MEMORY CONSTRUCTION · p. 12
  • Each update selects at most one trajectory per task variant by terminal score, peak score, and recency, then incrementally adds selected experiences to memory.ScienceWorld admits positive-peak trajectories; ALFWorld also admits executed failures, including zero-score episodes, as negative evidence.Source: A HIERARCHICAL MEMORY CONSTRUCTION · p. 12
  • Procedures matching an existing same-task principle add evidence without prompting a new principle; unmatched procedures may be lifted, deduplicated, validated, and committed.Rejected candidates enter an exclusion registry, evaluation trajectories are excluded from subsequent construction, and updated memory is frozen for evaluation.Source: A HIERARCHICAL MEMORY CONSTRUCTION · pp. 12–13
  • Benchmark-specific validation separates whether trajectories support a principle from whether deploying it has empirical utility.The ScienceWorld utility ledger applies only where utility screening is enabled; earlier batches are not retrospectively treated as ledger-validated.Source: A HIERARCHICAL MEMORY CONSTRUCTION · p. 13

B KNOWLEDGE RETRIEVAL AND UTILIZATION

SAGA retrieves experience at multiple abstraction levels, instantiates applicable principles as task-specific guidance, and uses harmful principles to regulate candidate actions through bounded resampling.

  • The task and a one-sentence plan form the retrieval query, while plan confidence sets the budget for injected principles.No principle is injected at confidence of at least 0.8; lower confidence permits up to one or three principles, depending on the confidence band.Source: B KNOWLEDGE RETRIEVAL AND UTILIZATION · p. 14
  • Retrieved principles are instantiated once per episode as operational constraints, with NOT APPLICABLE suppressing injection when a principle does not apply.The context presents similar-case evidence, a complete procedure, and the instantiated mechanism constraint; explicit task instructions take precedence over memory.Source: B KNOWLEDGE RETRIEVAL AND UTILIZATION · p. 14
  • For principles identified as harmful, a compiled predicate accepts an action or triggers at most three principle-conditioned resampling attempts.If no candidate satisfies the predicate, the original action is executed; successful compilation alone does not establish predicate validity.Source: B KNOWLEDGE RETRIEVAL AND UTILIZATION · p. 14

C EVALUATION PROTOCOLS AND OPTIMIZATION DETAILS

The evaluation uses shared hierarchical memory with benchmark-specific protocols and model roles. Training-based results combine supervised fine-tuning and GRPO, while the reported comparisons retain protocol and attribution boundaries.

C EVALUATION PROTOCOLS AND OPTIMIZATION DETAILS

  • Both benchmarks share hierarchical memory and selected extraction components but use benchmark-specific interaction protocols and evaluation metrics.Tables 5 and 6 define evaluation configurations; Table 7 identifies each setting’s actor and builder.Source: C EVALUATION PROTOCOLS AND… · p. 14, Table 6 · p. 15, Table 7 · p. 15
  • ScienceWorld comparisons use a strict evaluation configuration, while ALFWorld evaluates on unseen scenes and keeps Look and Pick2 zero-shot except in External.Across ALFWorld protocol families, comparisons are not treated as controlled estimates of model-only effects; the strict 7B baseline and evolution runs also differ in concurrency, a potential implementation-level confound.Source: C EVALUATION PROTOCOLS AND… · pp. 14–15
  • The reported training-based effects characterize the full actor-builder pipeline, not an isolated contribution from the builder’s pretrained knowledge.The authors identify a same-model builder-actor control as necessary to isolate that contribution.Source: C EVALUATION PROTOCOLS AND… · pp. 15–16
  • The SFT pipeline separates trajectory collection, offline abstraction, and policy optimization, using Qwen3.8-max-0902 as builder and Qwen2.5-7B-Instruct as execution policy.ScienceWorld’s corpus includes 8,788 full-score action samples, 4,882 zero-final-score prefixes, and 9,292 partial-score peak prefixes; score-based inclusion does not establish semantic optimality of every action.Source: C EVALUATION PROTOCOLS AND… · p. 16, Table 8 · p. 16
  • GRPO starts from the SFT checkpoint and optimizes a LoRA adapter, with the ALFWorld training pool restricted to 2,432 games from four seen task types.Look and Pick2 are absent from parameter optimization; Table 9 summarizes the optimization configurations.Source: C EVALUATION PROTOCOLS AND… · p. 16, Table 9 · p. 17
  • 66.9 on strata175 is the reported ScienceWorld average after GRPO; on ALFWorld, knowledge-conditioned SFT obtains 90.3%, SFT followed by GRPO obtains 93.3%, and the separate External variant obtains 96.3%.GRPO uses only the four seen task types; External permits test-time skill acquisition on Look and Pick2.Source: C EVALUATION PROTOCOLS AND… · p. 16

D EXTENDED RESULTS AND STATISTICAL ANALYSES

Extended results report gains from hierarchical memory and continued ALFWorld evolution, alongside supplementary analyses of knowledge use and execution behavior. The results also distinguish performance gains from validated principle use in the fixed 7B arm.

D EXTENDED RESULTS AND STATISTICAL ANALYSES

  • On ScienceWorld, complete SAGA scores 67.0 versus 63.7 with trajectory retrieval after seven evolution rounds using Qwen3.8-Max, and 56.2 versus 54.8 at the reported round-5 checkpoint in a six-round DeepSeek-flash batch.Source: D EXTENDED RESULTS AND STATISTICAL… · p. 17
  • Direct injection of L3 knowledge has a statistically significant negative effect, whereas contextual instantiation has a statistically significant positive effect in the paired utilization analysis.Under instantiation, the estimated principle-level gain is approximately three times the procedural gain (+0.0365 versus +0.0127); this supplementary analysis uses a distinct harness and more than 80% of episodes have zero terminal score.Source: D EXTENDED RESULTS AND STATISTICAL… · p. 17, Table 11 · p. 18
  • 99.3% at R3 versus 96.3% at R0 is the Qwen3.8-Max ALFWorld success rate, a net gain of four successful games among 134 evaluation games.All 99 seen-task games are solved at R3; the remaining failure is an unseen Pick2 instance where the context budget is exhausted.Source: D EXTENDED RESULTS AND STATISTICAL… · pp. 17–18, Table 12 · p. 18
  • For the fixed Qwen2.5-7B-Instruct actor, 57.5% is the R2 trajectory-only result, while 59.4% is the R3 complete-hierarchy result.No L3 principle passed validation in the complete 7B configuration during R1–R3, so its performance improvement does not establish a contribution from validated principle-level knowledge.Source: D EXTENDED RESULTS AND STATISTICAL… · p. 18, Table 12 · p. 18
  • 66.9 on strata175 is the ScienceWorld average for the dense-reward GRPO configuration starting from SFT initialization.Figure 4 shows the optimization trajectory.Source: D EXTENDED RESULTS AND STATISTICAL… · p. 18
  • Across seven Qwen3.8-Max rounds, hierarchical memory has estimated self-induced regression of 0.024 versus 0.045 under trajectory retrieval, with 41 versus 35 round-7 perfect-score episodes.The authors interpret these diagnostics as supporting reduced execution-time score regression, rather than an increased attainable score ceiling, as the principal observed benefit.Source: D EXTENDED RESULTS AND STATISTICAL… · p. 18

E CASE STUDIES, FAILURE ANALYSIS, AND LIMITATIONS

The case studies show how abstract principles can guide a task-specific action while concrete episodes and procedures retain execution evidence. Failure analysis and limitations bound what the results establish about directed behavior, principle validation, and generality.

E CASE STUDIES, FAILURE ANALYSIS, AND LIMITATIONS

  • In the pigment-mixing case, a principle captures the condition, effect, and mechanism of mixing two primary pigments, while episodes and procedures retain concrete execution evidence.For an orange target, the principle is instantiated to select red and yellow rather than replaying the green-producing trajectory; off-target candidate actions are revised through resampling.Source: E CASE STUDIES, FAILURE ANALYSIS, AND… · pp. 18–19
  • Three strong-backbone ALFWorld failures that exhausted the context budget in round 1 were solved in round 3 in 17, 15, and 13 steps, respectively.These reductions are consistent with more directed exploration, not an expanded interaction budget; the residual cross-round regression is an unseen Pick2 game lacking training trajectories.Source: E CASE STUDIES, FAILURE ANALYSIS, AND… · p. 19, Table 13 · p. 19
  • In the fixed 7B complete-hierarchy arm, performance rises from 45.5% at R0 to 59.4% at R3, but no principle passes validation in R1–R3.The reported gains are not attributed to validated L3 use; at R2, trajectory-only scores 57.5% versus 54.5% for complete hierarchy.Source: E CASE STUDIES, FAILURE ANALYSIS, AND… · p. 19
  • The training-based pipelines use different models for abstraction and execution, leaving the builder’s pretrained-knowledge contribution unisolated.Some residual ScienceWorld failures are execution-bound; extracted principles remain evidence-supported explanatory hypotheses, and their generality beyond ScienceWorld and ALFWorld is unverified.Source: E CASE STUDIES, FAILURE ANALYSIS, AND… · p. 19

F FUTURE WORK

Future work proposes personalizing hierarchical memory and adding active preference elicitation. These directions raise open problems around preference drift, validation, and auditable updates.

F FUTURE WORK

  • A personalized extension would condition hierarchical memory on user-specific preferences, constraints, and interaction habits alongside task knowledge.The proposed design adds user identifiers to node metadata and biases retrieval toward evidence from or validated against the current user.Source: F FUTURE WORK · pp. 19–20
  • Personalized memory would need to distinguish durable preferences from transient instructions, handle contradictions and drift, and limit stale evidence.The authors identify disentangling user-specific from task-general knowledge and per-user calibration of the injection budget as open problems.Source: F FUTURE WORK · p. 20
  • Active elicitation could query users when retrieved user-conditioned principles conflict or have low confidence, then record resolved preferences as episodic records.Long-horizon personalization also requires preference revision, forgetting, explicit overrides, time-decayed evidence, and auditable provenance.Source: F FUTURE WORK · p. 20
Loading 2610.06964v1…