PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory-Informed Memory Generation for Self-Improving Agent Systems

Gaodan Fang, Vatche Isahagian, K. R. Jayaram, Ritesh Kumar, Vinod Muthusamy, Punleuk Oum, Gegi Thomas

arXiv:2603.10600v1cs.AIcs.DBcs.IR

TL;DR

LLM agents lack systematic mechanisms to learn from execution experience, limiting their ability to reuse successful strategies or recover from recurring errors. This paper extracts structured, actionable learnings from trajectories and retrieves context-specific guidance, improving AppWorld performance, including held-out-task gains and larger benefits on complex tasks.

  • Problem

    LLM agents lack systematic mechanisms for learning from execution experiences, so they cannot automatically reuse successful strategies or recovery patterns.

  • Method

    The framework analyzes trajectories, attributes outcomes to decisions and reasoning steps, generates strategy, recovery, and optimization tips, and retrieves relevant tips using contextual dimensions.

  • Results

    The AppWorld evaluation shows consistent improvements, including up to 14.3 percentage point scenario goal completion gains on held-out tasks and 28.5 pp improvement on complex tasks.

  • Takeaways & Limitations

    Trajectory-informed structured memory can improve future agent performance beyond storing generic conversational facts by capturing execution patterns and retrieving task-relevant guidance.

  • Takeaways & Limitations

    Generic memory systems remain limited because they do not represent execution patterns, causal decision chains, or structured actionable learning categories.

Abstract

from arXiv · show

LLM-powered agents face a persistent challenge: learning from their execution experiences to improve future performance. While agents can successfully complete many tasks, they often repeat inefficient patterns, fail to recover from similar errors, and miss opportunities to apply successful strategies from past executions. We present a novel framework for automatically extracting actionable learnings from agent execution trajectories and utilizing them to improve future performance through contextual memory retrieval. Our approach comprises four components: (1) a Trajectory Intelligence Extractor that performs semantic analysis of agent reasoning patterns, (2) a Decision Attribution Analyzer that identifies which decisions and reasoning steps led to failures, recoveries, or inefficiencies, (3) a Contextual Learning Generator that produces three types of guidance -- strategy tips from successful patterns, recovery tips from failure handling, and optimization tips from inefficient but successful executions, and (4) an Adaptive Memory Retrieval System that injects relevant learnings into agent prompts based on multi-dimensional similarity. Unlike existing memory systems that store generic conversational facts, our framework understands execution patterns, extracts structured learnings with provenance, and retrieves guidance tailored to specific task contexts. Evaluation on the AppWorld benchmark demonstrates consistent improvements, with up to 14.3 percentage point gains in scenario goal completion on held-out tasks and particularly strong benefits on complex tasks (28.5~pp scenario goal improvement, a 149\% relative increase).

1 Introduction

The paper addresses agents’ inability to learn systematically from execution experiences by extracting actionable, provenance-aware guidance from trajectories and retrieving it for future tasks. Its framework analyzes reasoning and decisions to generate strategy, recovery, and optimization tips tailored to context.

  • LLM agents often repeat inefficient patterns, fail to reuse successful strategies, and struggle with recurring errors because most are stateless.
  • It generates strategy tips from clean successes, recovery tips from failure handling, and optimization tips from inefficient successes.
  • Existing rule-based, prompt-engineering, and generic-memory approaches require manual updates or lack execution-pattern and causal understanding.
  • The framework analyzes trajectories to identify why decisions succeeded, failed, recovered, or produced inefficiencies.
  • Adaptive retrieval injects relevant guidance into prompts using task, domain, and execution-pattern context.

2 Problem Statement

The paper frames agent learning as a problem of extracting actionable, causal, contextual, and traceable knowledge from diverse execution outcomes. Existing approaches do not comprehensively provide these capabilities.

  • Execution trajectories contain learnings from clean successes, inefficient successes, failure recoveries, and complete failures.
  • Raw logs obscure causality because the decision causing a later failure or recovery may occur much earlier in the trajectory.
  • Retrieval must match guidance to task type, domain, semantic request similarity, and execution patterns.
  • Useful learnings must specify concrete checks, API patterns, recovery sequences, or prerequisite verification steps.
  • Provenance links each learning to its source trajectory and outcome for validation, debugging, auditing, and trust.
  • Rule-based systems and prompt engineering depend on anticipated patterns or manual refinement rather than automatic learning from outcomes.
  • Generic memory systems store conversational facts but lack execution-pattern distinctions, causal decision analysis, and structured actionable guidance.

3 Approach

The proposed system transforms execution trajectories into reusable guidance through analysis, consolidation, storage, and contextual retrieval. It supports both task-level and subtask-level representations and injects selected tips into prompts before reasoning.

  • Phase 1 analyzes reasoning traces to extract causal decision chains and structured tips for strategies, recoveries, and optimization opportunities.
  • Tips are extracted at task level for holistic patterns and subtask level for reusable logical phases such as authentication and data retrieval.
  • The system abstracts, clusters, and consolidates tips to remove entity-specific details and produce nonredundant guidance.
  • Retrieved tips are injected into prompts before reasoning using either cosine similarity or LLM-guided selection.
  • The memory process forms a self-reinforcing cycle in which improved trajectories provide further patterns for learning.

3.1 Phase 1: Trajectory Analysis and Tips Extraction

Phase 1 converts raw trajectories into structured, actionable tips by recognizing reasoning patterns, inferring outcomes, attributing decisions, and generating categorized guidance. It supports strategy, recovery, and optimization learning while balancing task-level specificity against subtask-level transfer.

  • Trajectory intelligence extraction: Reasoning is categorized into analytical, planning, validation, and reflection thoughts, while semantic analysis identifies validation, reflection, and self-correction patterns beyond keyword matching.
  • Trajectory intelligence extraction: Outcome determination uses evaluation reports when available and self-reflective signals to infer success, failure, or recovery otherwise.
  • Trajectory intelligence extraction: The extraction phase produces structured intermediate representations containing thoughts, cognitive patterns, outcome intelligence, success patterns, and trajectory metadata.
  • Decision attribution: Decision attribution traces failures, recoveries, inefficiencies, and successes to specific reasoning steps and distinguishes immediate, proximate, and root causes.
  • Decision attribution: The analyzer generates actionable, specific, causal prevention or improvement steps for each attributed decision point.
  • Contextual learning generation: The learning generator creates strategy tips from clean successes, recovery tips from error handling, and optimization tips from successful but suboptimal executions.
  • Contextual learning generation: Domain-specific and generic tips from the same trajectory trade precision for coverage across matching and novel contexts.
  • Granularity tradeoffs: Subtask-level extraction supports cross-task transfer, better matching, and compositional learning, whereas task-level tips can be overly specific and bundle distinct phases.

3.2 Phase 2: Tip Storage and Management

Phase 2 reduces redundancy and inconsistency in trajectory-derived tips by generalizing subtask descriptions, clustering semantically similar tips, and consolidating them into canonical guidance with structured metadata and provenance.

  • Description Generalization: The system generalizes task-specific subtask descriptions by removing context while preserving the core operation, enabling semantic comparison across tasks.An LLM produces maximally abstract descriptions that serve as clustering inputs.
  • Semantic Clustering: Tips are clustered using cosine similarity between generalized-description embeddings followed by hierarchical agglomerative clustering with a similarity threshold.The approach distinguishes genuinely shared subtasks from merely related wording.
  • Semantic Clustering: Clusters combine tips across source trajectories, task contexts, and extraction granularities when they reflect the same underlying subtask pattern.For example, credential-retrieval tips from different applications can form one cluster.
  • Tip Consolidation: LLM-based consolidation deduplicates near-identical tips, resolves conflicts using outcome metadata, and synthesizes complementary guidance into coherent instructions.Successful-trajectory tips take precedence over failed-trajectory tips during conflict resolution.
  • Memory Representation: Each consolidated memory stores a re-embedded canonical cluster description alongside vector and structured metadata, including category, priority, application, task, source trajectories, and timestamp.Canonical descriptions support retrieval at both subtask and task granularity.

3.3 Phase 3: Runtime Retrieval

Phase 3 retrieves tips for a new task and injects them into the agent prompt, comparing fast embedding-based lookup with more expressive LLM-guided selection under different cost-accuracy trade-offs.

  • Runtime Retrieval: At runtime, the system retrieves relevant tips for task description d and injects them as guidelines before the agent begins reasoning.The retrieval strategy determines whether guidance is actionable or distracting.
  • Cosine Similarity Retrieval: Cosine similarity retrieval performs a fast, inexpensive vector lookup without an LLM call, selecting tips associated with the most similar stored descriptions.Similarity thresholds control eligibility, while top-k selection limits prompt growth.
  • LLM-Guided Selection: LLM-guided selection detects application context and task category, then combines semantic similarity with metadata filters and category-aware ranking.It can identify relevant relationships that embedding similarity alone may miss, but requires an additional LLM call.
  • Strategy Comparison: Cosine retrieval favors latency and cost efficiency, whereas LLM-guided selection provides richer contextual reasoning at higher runtime cost.Both strategies are evaluated empirically in the paper.
  • Prompt Integration: Retrieved tips are formatted as scannable guidelines containing priority, category, actionable content, purpose, steps, and trigger conditions.The guidelines appear after task context and before standard agent instructions.
  • Prompt Integration: Prompt integration creates a feedback loop in which relevant tips can help agents avoid failures, improve efficiency, and apply successful strategies.Higher-quality trajectories then reinforce the memory system.

4 Evaluation

The evaluation tests trajectory-informed memory on AppWorld across tip granularities, retrieval strategies, and benchmark partitions using task- and scenario-level completion measures.

  • Benchmark: AppWorld evaluates realistic LLM-agent tasks across e-commerce, email, calendar, file management, and other application domains through API interactions.Tasks are expressed as natural-language instructions executed against application APIs.
  • Metrics: Task Goal Completion measures the percentage of individual tasks passing all programmatic unit tests, including correct API usage, database changes, and expected end states.Tasks typically involve multiple API calls across an average of 1.8 apps and 9.5 APIs.
  • Metrics: Scenario Goal Completion requires every variant in a scenario to pass, making it stricter than task-level completion.This metric tests consistency across related task variants.
  • Difficulty: The study covers easy, medium, and hard tasks, with hard tasks requiring planning, prerequisite management, cross-domain coordination, and error recovery.Hard tasks can involve up to 26 APIs and more than 50 equivalent lines of code.
  • Agent Configuration: The experimental agent uses a single-agent ReAct-style loop, while the memory-enhanced variant receives retrieved tips before reasoning begins.The base agent receives only the task instruction and standard execution guidance.
  • Tip Configurations: Task-level tips capture holistic end-to-end strategies, whereas subtask-level tips focus on reusable patterns shared across logical subtasks.The evaluation compares both extraction granularities.
  • Retrieval Configurations: The study compares cosine similarity retrieval with LLM-guided selection, retrieving the top 5 tips for both strategies.Cosine retrieval is fast and inexpensive, while LLM-guided selection uses contextual and metadata reasoning.
  • Evaluation Protocol: Evaluation spans held-out test-normal tasks plus train and development partitions used during tip generation, with automated checks of explicit and implicit goals.Each task is run independently for up to 30 reasoning-action steps.

4.2 Held-Out Results (Test-Normal)

On held-out AppWorld tasks, trajectory-informed memory improves both individual task completion and consistency across scenario variants. Performance depends on tip granularity and retrieval strategy: subtask-level tips favor TGC, while LLM-guided selection favors SGC.

  • Overall held-out performance: 73.2% TGC versus 69.6% for the baseline (+3.6 pp), while SGC reaches 64.3% versus 50.0% (+14.3 pp).The larger SGC gain indicates improved consistency across task variants.
  • Performance by task complexity: Difficulty 3 tasks show the largest gains: +4.7 pp TGC and +28.5 pp SGC, increasing SGC from 19.1% to 47.6% (a 149% relative increase).These tasks require sophisticated planning and robust error recovery.
  • Task-level cosine retrieval: Top-3 retrieval under τ≥0.5 underperforms the baseline, reaching 66.7% TGC and 48.2% SGC, with Difficulty 3 TGC falling 8.0 pp.Restricting retrieval to three matched descriptions may exclude relevant guidance.
  • Task-level cosine retrieval: τ≥0.6 without top-k restriction is the strongest task-level cosine configuration, achieving 72.0% TGC (+2.4 pp) and 62.5% SGC (+12.5 pp).The threshold excludes unrelated tips while retaining semantically equivalent descriptions that differ lexically.
  • Tip granularity: Subtask-level tips produce higher TGC than task-level tips, reaching 73.8% with cosine retrieval and 73.2% with LLM-guided selection versus 72.0% for task-level cosine.On Difficulty 3, subtask-level tips reach 58.7% TGC versus 54.0% for task-level tips.
  • Retrieval strategy: LLM-guided selection produces higher SGC than cosine retrieval at the same subtask granularity: 64.3% versus 57.1% (+7.2 pp).Its advantage is consistent across difficulty levels and is attributed to contextual prioritization of tip categories and metadata filtering.

4.3 Source Partition Results (Train and Dev)

Source-partition evaluation measures self-improvement when agents revisit tasks used for tip generation. Gains are larger on these structurally similar tasks, though simple tasks can show interference when baseline performance is already optimal.

  • Aggregate source-partition results: +4.4 pp TGC and +10.0 pp SGC on train, while dev improves by +12.3 pp TGC and +26.3 pp SGC.These partitions evaluate recurring or structurally identical tasks augmented with tips from prior executions.
  • Contextual relevance: Tips are most contextually relevant when agents encounter tasks structurally similar to those that generated the tips.
  • Partition-specific effects: On train Difficulty 1 tasks, memory-enhanced performance falls to 94.4% TGC and 83.3% SGC from a 100% baseline.The result suggests minor interference on simple tasks where the baseline already performs optimally.
  • Partition-specific effects: Dev Difficulty 3 already reaches 100% TGC and 100% SGC at baseline, so aggregate gains come entirely from Difficulty 1 and 2.

4.4 Cross-Configuration Summary

Across configurations and partitions, contextual memory improves performance, with especially strong gains in scenario goal completion and on tasks benefiting from relevant guidance. Retrieval restrictions can instead hurt performance by excluding useful tips.

  • Cross-partition aggregate results: +3.6 TGC and +14.3 pp SGC on test-normal demonstrate generalization to unseen tasks.
  • Cross-partition aggregate results: SGC improvements exceed TGC improvements across all partitions, indicating stronger gains in consistency across task variants.Recovery and strategy tips encode prerequisite verification and error-handling patterns that reduce behavioral variance.

5 Related Work

Related work spans memory architectures, trajectory-based learning, and self-improvement. The paper distinguishes its structured, typed, contextually retrieved tips from systems that store facts, workflows, reasoning strategies, evolving playbooks, or raw trajectories.

  • Memory architectures: Existing agent-memory research is organized around memory sources, forms, and operations, including interactions, reasoning, feedback, embeddings, databases, and reflection.
  • Memory architectures: Semantic memory systems such as Mem0 and A-MEM primarily store conversational facts or structured notes rather than execution-pattern learnings.
  • Trajectory-based learning: Agent Workflow Memory extracts reusable workflows from successful trajectories but does not learn from failures, recoveries, or inefficient executions.
  • Reasoning and strategy extraction: ReasoningBank extracts meta-cognitive strategies from successful and failed experiences, whereas this framework focuses on concrete behavioral tips from execution patterns.
  • Context engineering: ACE evolves a text playbook, while this framework produces typed memory entries with metadata and selective retrieval.
  • Experience replay: Memento retrieves raw trajectories without abstracting them into transferable insights, leaving consolidation to in-context reasoning.

6 Conclusions

The paper presents a four-component framework that converts execution trajectories into structured tips and retrieves them to improve future agent performance. AppWorld results show gains on held-out and complex tasks, while broader model and multi-agent evaluation remain future work.

  • Framework: The four-component pipeline extracts trajectory intelligence, attributes decisions, generates contextual learning, and adaptively retrieves memory tips.
  • Results: Up to 14.3 percentage point gains in scenario goal completion are reported on held-out tasks.
  • Results: Complex multi-step tasks show a 28.5 pp SGC improvement, corresponding to a 149% relative increase.
  • Future work: The framework is described as extensible to multi-agent systems, while evaluation with additional models such as Qwen and GPT-OSS is left for future work.
Loading 2603.10600v1…