PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Tasks

Cheng Yang, Xuemeng Yang, Licheng Wen, Daocheng Fu, Jianbiao Mei, Rong Wu, Pinlong Cai, Yufan Shen, Nianchen Deng, Botian Shi, Yu Qiao, Haifeng Li

arXiv:2510.08002v1cs.CLcs.AI

TL;DR

Long-horizon productivity agents are often static at test time and cannot accumulate experience reliably. MUSE addresses this gap with a hierarchical, experience-driven memory loop that reflects after subtasks and stores structured knowledge. On TAC, it achieves new state-of-the-art performance while showing continuous improvement and generalization to new tasks.

  • Problem

    Existing agents are test-time static, unable to consolidate past successes or failures into knowledge for complex long-horizon tasks.

    Source: §1 · pp. 1–2

  • Method

    MUSE uses a hierarchical Memory Module and a closed-loop process that plans, executes, reflects on, and memorizes sub-task experience without fine-tuning the base LLM.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    MUSE achieves new SOTA performance on TAC with a 51.78% score, a 20% relative leap over the previous SOTA, while showing continuous improvement and experience generalization.

    Source: §1 · pp. 2–3, §5 · p. 11

  • Takeaways & Limitations

    Structured experience accumulation enables MUSE to continuously learn and self-evolve on long-horizon productivity tasks, with stored knowledge generalizing to novel tasks.

    Source: §1 · p. 2, §5 · p. 11

    MUSE’s memory architecture remains limited for tasks involving high-level planning or multi-hop search, and TAC includes ambiguous descriptions and rigid evaluation scripts.

    Source: §5 · pp. 10–11

Abstract

from arXiv · show

Large Language Models have demonstrated remarkable capabilities across diverse domains, yet significant challenges persist when deploying them as AI agents for real-world long-horizon tasks. Existing LLM agents suffer from a critical limitation: they are test-time static and cannot learn from experience, lacking the ability to accumulate knowledge and continuously improve on the job. To address this challenge, we propose MUSE, a novel agent framework that introduces an experience-driven, self-evolving system centered around a hierarchical Memory Module. MUSE organizes diverse levels of experience and leverages them to plan and execute long-horizon tasks across multiple applications. After each sub-task execution, the agent autonomously reflects on its trajectory, converting the raw trajectory into structured experience and integrating it back into the Memory Module. This mechanism enables the agent to evolve beyond its static pretrained parameters, fostering continuous learning and self-evolution. We evaluate MUSE on the long-horizon productivity benchmark TAC. It achieves new SOTA performance by a significant margin using only a lightweight Gemini-2.5 Flash model. Sufficient Experiments demonstrate that as the agent autonomously accumulates experience, it exhibits increasingly superior task completion capabilities, as well as robust continuous learning and self-evolution capabilities. Moreover, the accumulated experience from MUSE exhibits strong generalization properties, enabling zero-shot improvement on new tasks. MUSE establishes a new paradigm for AI agents capable of real-world productivity task automation.

1 Introduction

MUSE addresses the inability of static agents to learn from experience during complex, cross-application productivity tasks. Its experience-driven memory loop supports self-evolution and achieves leading TAC performance.

1 Introduction

  • Static agents cannot consolidate successes or failures from prior tasks, limiting reliable replication and improvement through practice.This one-off interaction model is especially problematic for tasks requiring more than 100 steps and cross-application interaction.Source: §1 · p. 2
  • MUSE uses a hierarchical Memory Module to plan and execute long-horizon tasks across multiple applications.The module organizes procedural knowledge, strategic patterns, and tool-use guidance.Source: §1 · p. 2
  • After each sub-task, MUSE reflects on the execution trajectory and converts raw actions into structured, reusable memory.This process aims to reduce redundant exploration and steadily improve agent performance without human intervention.Source: §1 · pp. 2–3
  • MUSE’s natural-language memory enables experience gained by one model to transfer to another LLM.The accumulated knowledge is therefore LLM-agnostic.Source: §1 · pp. 2–3
  • The framework accumulates experience autonomously on TAC and exhibits increasingly superior task completion, continuous learning, and self-evolution.MUSE achieves these results using only a lightweight Gemini-2.5 Flash model.Source: §1 · p. 2
  • 51.78% on TAC establishes a new SOTA, representing a 20% relative leap over the previous SOTA.Extended experiments also support continuous learning and self-evolution.Source: §1 · p. 3

2 Related Work

Related work studies self-evolving agents through optimization, skill accumulation, tool construction, and self-reflection. Separate memory research develops storage, retrieval, and procedural abstractions, but existing evaluations often underrepresent complex long-horizon tasks.

2.1 Self-evolving agent

  • Self-evolving-agent research explores prompt optimization, reusable skills and toolsets, and language-based self-reflection for iterative capability improvement.These approaches draw on black-box optimization and cognitive-science-inspired learning mechanisms.Source: §2.1 · p. 3

2.2 LLM Agent Memory Mechanisms

  • LLM-agent memory research separates short-term and long-term memory, improves memory management, and extracts reusable procedures from execution trajectories.Examples include explicit memory operations, summaries and tags, natural-language insights, workflows, and lifelong procedural memory.Source: §2.2 · p. 3
  • Existing memory mechanisms are often validated on text or web benchmarks whose limited complexity and long-term dependencies may not fully assess real-world long-horizon performance.Source: §2.2 · p. 3

3 Methodology

MUSE is an experience-driven framework that uses a hierarchical Memory Module and a Plan-Execute-Reflect-Memorize loop to support long-horizon productivity tasks. Its agents retrieve and update structured experience, adapt plans through verification and retries, and refine memory after task completion.

3.1 Framework Overview

  • MUSE combines a Memory Module, Planning-Execution Agent, and Reflect Agent in a test-time learning loop for productivity tasks without fine-tuning the base LLM.The framework operates across a multi-application interactive environment using a predefined toolset.Source: §3.1 · p. 4

3.2 Memory Module

  • Successful sub-task trajectories become Procedural Memory, while post-task analysis produces Strategic and Tool Memory that are refined and integrated for future reuse.Strategic Memory captures dilemma-resolution patterns, Procedural Memory stores application-indexed SOPs, and Tool Memory provides static descriptions plus dynamic next-action guidance.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Planning-Execution Agent

  • The Planning-Execution Agent decomposes tasks into ordered sub-task queues, updates them after each assessment, and performs memory-enhanced ReAct execution with bounded retries.A retry can omit Procedural Memory retrieval to encourage discovering novel methods when existing knowledge is unsuitable.Source: §3.3 · p. 6

3.4 Reflect Agent

  • The Reflect Agent independently verifies sub-task outcomes using trajectory references and active environment checks for truthfulness, deliverables, and data fidelity.It records success by creating a new SOP, or records failure causes and triggers replanning.Source: §3.4 · pp. 6–7
  • After the overall task, reflection extracts dilemma-resolution patterns, codifies effective tool use, and integrates all memory types by removing redundancy and generalizing common patterns.This full-task upgrade reinforces Strategic and Tool Memory while refining Procedural Memory within the shared module.Source: §3.4 · p. 7

4 Experiments

MUSE is evaluated on TAC through continual-learning, generalization, and full-benchmark experiments, showing progressively stronger performance and new state-of-the-art results. The experiments test long-horizon productivity across complex multi-application tasks.

4.1 Benchmark

  • TAC contains 175 long-horizon tasks averaging over 40 actions across multiple applications and six professional roles.The benchmark evaluates planning, cross-platform information integration, and real-world productivity capabilities.Source: §4.1 · p. 7

4.2 Experimental Setup

  • The evaluation computes Spartial from checkpoint completion and full-task completion, while Sckpt measures the aggregate proportion of completed checkpoints.Spartial = 0.5 · Completed_ckpt/Total_ckpt + 0.5 · Sfull.Source: §4.2 · p. 8

4.3.1 Continuous Learning Experiments

  • Across three continual-learning iterations, both Sckpt and average Spartial increased steadily and monotonically, with the final MUSE round exceeding the memory-less baseline by over 10%.The experiment used 18 tasks, five complete runs, and no human intervention while carrying accumulated knowledge forward.Source: §4.3.1 · p. 8, Figure 3 · p. 8

4.3.2 Generalization Experiments

  • On 12 difficult unseen tasks, MUSE improved average Spartial from 23.65% without memory to 33.41% with pre-learned memory.The comparison used memory accumulated through three continual-learning iterations on the separate Tcl subset.Source: §4.3.2 · pp. 8–9, Table 1 · p. 9

4.3.3 TAC Full Benchmark

  • 51.78% average Spartial on all 175 TAC tasks establishes new state-of-the-art performance, nearly 20% above the previous SOTA.The frozen memory was accumulated from only approximately 10% of available tasks.Source: §4.3.3 · p. 9, Table 2 · p. 9

4.4 Ablation Study

Ablations indicate that reflection and memory transfer materially support MUSE’s performance. Removing reflection hurts execution quality, while the architecture and accumulated memory remain effective across language models.

4.4.1 Ablation Study for Reflect Agent

  • Removing the Reflect Agent causes a substantial performance drop on the 18-task Tcl subset.The ablation identifies reflection as important for execution quality and for producing learning signals.Source: §4.4.1 · p. 10, Table 3 · p. 10

4.4.2 Ablation Study for Different Models

  • With DeepSeek-V3-250324, MUSE outperforms other open-source-model frameworks even without memory, and pre-accumulated memory provides a significant additional boost.The result supports transfer of accumulated knowledge across different LLMs.Source: §4.4.2 · p. 10, Table 4 · p. 10

5 Discussions and Conclusion

MUSE uses hierarchical memory to support experience-driven learning on long-horizon productivity tasks, while acknowledging limitations in high-level planning, multi-hop search, and benchmark evaluation. Its evaluation reports continuous improvement, experience generalization, and new state-of-the-art TAC performance.

Discussion

  • Memory modules target long-horizon productivity tasks because fine-tuning is computationally intractable and reinforcement-learning rewards are sparse and difficult to formulate.The approach focuses on enhancing agent memory for test-time learning.Source: §5 · p. 10

Limitations

  • The current memory architecture is limited for tasks requiring high-level planning or multi-hop search.Source: §5 · p. 10
  • TAC task descriptions may be ambiguous or inaccurate, while rigid evaluation scripts can underestimate valid strategies or penalize them incorrectly.Two case studies illustrate these benchmark issues.Source: §5 · p. 11

Future Directions

  • MUSE’s Memory Module supports human feedback through direct management and revision of stored experiences, enabling potential human-agent collaborative iteration.Users can add, delete, modify, and query stored experiences.Source: §5 · p. 11

Conclusion

  • MUSE achieves continuous performance improvement, self-evolution, and experience generalization on TAC, attaining new state-of-the-art performance by a large margin.The framework uses autonomous experience accumulation to improve task completion and transfer experience to novel tasks.Source: §5 · p. 11

A.1 Selected Task Splits

The appendix defines two TAC task splits with different difficulty targets while maintaining coverage across all six professional domains and manually checking task correctness.

Selected Task Splits

  • The continual-learning split Tcl contains moderately difficult tasks with generally non-zero official TAC scores, whereas Thard contains tasks where most models nearly fail completely.The two sets are designed respectively to test progressive improvement on repetitive tasks and stress-test agent limits.Source: §A.1 · p. 15
  • Both task sets cover all six TAC professional domains, and every task was manually inspected to exclude obvious errors.The task lists are provided in Table 5.Source: §A.1 · p. 15, Table 5 · p. 15

A.2 Case Studies

The case studies show MUSE adapting its plans to complex productivity tasks, including benchmark-invalid but successful strategies and long-horizon cross-platform execution.

Case Studies

  • The hr-collect-feedbacks case used a multi-person chat to query three colleagues simultaneously, completing the evaluation efficiently despite falling outside TAC’s standard protocol.The agent integrated the feedback into a comprehensive performance evaluation for Liu Qiang.Source: §A.2 · p. 16, Figure 4 · p. 17
  • The GitLab case orchestrated nearly 20 sub-tasks and over 100 actions while dynamically discovering contacts and creating a missing account to assign the issue.The agent adapted its sub-task queue as new information emerged and ultimately assigned the issue to Li Ming.Source: §A.2 · p. 16, Figure 5 · p. 18
  • Across the cases, dynamic sub-task planning, self-reflection, and trial-and-error optimization enabled autonomous adaptation beyond the original task objectives.Source: §A.2 · p. 16

A.3 Exemplary Demonstration of the Memory Module

MUSE organizes experience into strategic, procedural, and tool memories, illustrated through structured examples that encode principles, interaction procedures, and optimized tool guidance.

Exemplary Demonstration of the Memory Module

  • The examples are presented in the native structured format of the three memory types.Source: §A.3 · p. 19
  • Strategic Memory provides high-level principles for robust agent behavior.Source: Table 6 · p. 19
  • Procedural Memory records step-by-step guides for common application interactions.Source: Table 7 · p. 20
  • Tool Memory supplies optimized tool instructions and descriptions.Source: Table 8 · p. 21

A.4 Tool Set

MUSE uses a compact tool set combining browser, coding, shell, visual extraction, and memory retrieval capabilities. Table 9 provides the complete tool-set overview.

A.4 Tool Set

  • MUSE’s minimal tool set includes a browser operator, Python interpreter, shell, visual extractor, and memory retriever.The browser uses accessibility-tree and interactive-element observations, while the visual extractor uses GPT-4o.Source: §A.4 · p. 22
  • Table 9 presents the complete overview of MUSE’s tool set.Source: Table 9 · p. 22

A.5 Complete Task-level Results

The complete TAC results report task-level completion scores across 175 benchmark tasks. Performance varies substantially across administrative, data-science, finance, human-resources, project-management, research, and software-engineering tasks.

A.5 Complete Task-level Results

  • Table 10 reports detailed results and scores for all 175 tasks in the complete TAC benchmark.Source: Table 10 · p. 23, §A.5 · p. 23
  • Task-level scores range from 0.0 to 100.0 across administrative and data-science tasks.Examples include several tasks at 100.0 and others at 0.0, showing substantial variation within these domains.Source: §A.5 · p. 23
  • Task-level scores range from 0.0 to 100.0 across finance, human-resources, and machine-learning tasks.The listed results include both fully completed tasks and tasks with no completed subtasks.Source: §A.5 · p. 24
  • Task-level scores range from 0.0 to 100.0 across project-management, question-answering, research, and software-development tasks.The software-development entries include both perfect scores and substantially lower completion rates.Source: §A.5 · p. 25
  • Software-development results continue across repository, coding, infrastructure, issue-management, and community tasks, with scores spanning 0.0 to 100.0.Several tasks achieve 100.0, while implementation and setup tasks include low completion scores.Source: §A.5 · p. 26
  • The task-level table includes a checkpoint column labeled “Spartial (%)”.Source: §A.5 · p. 27

Related papers

Loading 2510.08002v1…