PaperlayerView exact revision on arXiv

Source-linked AI summary

ExpVoyager: Direct Experience Navigation for Dynamic Agent Skill Synthesis

Kwangwook Seo, Dongha Lee

arXiv:2609.32630v1cs.CL

TL;DR

Existing skill synthesis fixes procedural knowledge before future task demands are known, risking both lost useful knowledge and irrelevant details. EXPVOYAGER navigates raw experience on demand through varied views while tracking what is known and what remains to investigate. Experiments report consistent downstream improvements, scaling gains, and compatibility with existing skills under efficient access.

  • Problem

    Existing approaches abstract past experience into fixed skills before future demands are known, risking loss of later-critical knowledge and retention of irrelevant details.

    Source: §1 · p. 1

  • Method

    EXPVOYAGER has a skill curator actively navigate raw experience through different views and resolutions, updating its state to track reusable knowledge and remaining needs.

    Source: §1 · p. 2

  • Results

    Experiments report consistent downstream task improvements, continual gains as experience scales, and compatibility with existing skills under efficient experience access.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EXPVOYAGER provides a framework for turning accumulated experience into reusable capabilities through dynamic skill synthesis.

    Source: §6 · p. 9

Abstract

from arXiv · show

Learning from experience in LLM agents has become a key paradigm for developing self-evolving agents that continuously learn and expand their capabilities. Within this paradigm, synthesizing the agent skill has emerged as a promising solution for transforming accumulated experience into reusable procedural knowledge, serving as an important layer for the harness system that supplies agents at runtime. Despite its potential, existing approaches largely abstract past experience into fixed procedural knowledge before downstream demands are known, which risks discarding knowledge that later becomes critical while retaining instance-specific details irrelevant to future tasks. In this paper, we reframe agent skill synthesis as a dynamic navigation problem over past experience, where agents actively explore accumulated trajectories on demand for the current task with targeted and fine-grained access to experience knowledge. To this end, we propose ExpVoyager, a novel framework in which a skill curator navigates raw experience across different views and resolutions, continually identifying reusable procedural knowledge from what it observes while tracking remaining knowledge needs that guide where to navigate next. Extensive experiments demonstrate both the effectiveness and versatility of ExpVoyager, showing consistent improvements in downstream task performance, continual gains as the experience space scales, and practical compatibility with existing skills under efficient experience access.

1 INTRODUCTION

Existing skill synthesis abstracts experience before future task demands are known, risking the loss of useful knowledge and retention of irrelevant details. EXPVOYAGER instead navigates raw experience on demand, and experiments report consistent downstream improvements alongside scaling and skill compatibility.

1 INTRODUCTION

  • Future task demands are unknown when fixed skills are constructed, risking the loss of later-critical knowledge and retention of irrelevant instance-specific details.This motivates dynamically synthesizing skills for the current task from raw experience.Source: §1 · p. 1
  • EXPVOYAGER reframes skill synthesis as active navigation, letting agents choose where and at what resolution to inspect experience for current needs.Passive top-k retrieval may miss localized cues in dissimilar trajectories or include irrelevant knowledge from similar ones.Source: §1 · p. 2
  • A skill curator uses a Navigable Interface and Navigation State to access varied experience views and track reusable knowledge and remaining needs.The state is updated after observations to guide further investigation.Source: §1 · p. 2
  • Experiments report consistent downstream task improvements, gains as experience scales, online self-evolution without pre-collected experience, and compatibility with existing skills under efficient access.The evaluation covers household interaction, online shopping, and scientific reasoning.Source: §1 · p. 2

2 PRELIMINARY ANALYSIS

The preliminary analyses examine knowledge preservation in pre-constructed skills and target-relevant knowledge access through retrieval. They find that pre-construction loses much oracle knowledge and retrieval offers limited access, motivating targeted knowledge seeking.

2 PRELIMINARY ANALYSIS

  • The analyses test verified helpfulness using ALFWorld skills and source trajectories, then compare retained or retrieved knowledge with task-specific oracle knowledge.The study collected 80 verified oracle skills and sampled 300 training trajectories for annotation.Source: §2 · pp. 2–3

2.1 PRELIMINARY ANALYSIS I: PRE-CONSTRUCTED SKILLS LOSE THE MAJORITY OF PROCEDURAL KNOWLEDGE USEFUL FOR FUTURE TASKS.

  • Even the best-performing pre-constructed skill method fails to preserve most oracle knowledge found in raw experience.Figure 1 uses raw experience as the 100% preservation reference.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 PRELIMINARY ANALYSIS II: EXISTING EXPERIENCE RETRIEVAL PARADIGMS PROVIDE LIMITED ACCESS TO TARGET-RELEVANT KNOWLEDGE.

  • Existing retrieval methods achieve low knowledge recall, and increasing retrieval depth brings diminishing recall gains while precision drops rapidly.The analysis measures helpful-source recall and knowledge-level recall and precision against oracle knowledge.Source: §2.2 · p. 3, Figure 2 · p. 3

3 EXPVOYAGER FOR DYNAMIC SKILL SYNTHESIS

EXPVOYAGER frames skill synthesis as target-conditioned navigation over past experience, then turns the curator’s findings into guidance for a frozen executor. Its interface exposes experience at multiple resolutions, while a navigation state tracks established knowledge and remaining questions.

3.1 PROBLEM FORMULATION

  • For a target task, the curator ϕ produces a skill from experience E, which a frozen executor θ uses alongside its interaction history to select actions.The experience consists of trajectories containing task instructions, observations, actions, and outcomes.Source: §3.1 · p. 4

3.2 BUILDING NAVIGABLE INTERFACE OVER AGENT EXPERIENCE

  • The Navigable Interface exposes trajectory-level procedures and step-level decisions, letting the curator search records and inspect their source trajectories for broader context.Trajectory views summarize action sequences and outcomes; step views expose observations, reasoning, actions, and immediate results.Source: §3.2 · pp. 4–5

3.3 GUIDING EXPERIENCE NAVIGATION VIA STATE MANAGEMENT

  • Navigation State Z tracks target-relevant knowledge K_r and open questions Q_r, so new observations can update what is known and guide subsequent navigation.The curator distinguishes transferable procedural relations from source-specific conditions and can revise earlier interpretations as it accesses more records.Source: §3.3 · p. 5

3.4 SYNTHESIZING SKILL FOR FROZEN EXECUTOR

  • After navigation, the curator synthesizes the accumulated state into a task-specific skill for the frozen executor, retaining unresolved conditions as execution-time checks and failure lessons as cautions or recovery guidance.Navigation continues until further investigation is no longer useful or its budget is exhausted.Source: §3.4 · p. 5

4 EXPERIMENT

Across three interactive benchmarks, EXPVOYAGER improves task performance over existing skill-based approaches and the base agent. Experiments also examine experience scaling, online learning, skill combinations, access budgets, and the mechanisms behind navigation.

4.1 EXPERIMENTAL SETUP

  • Experiments cover household interaction, online shopping, and scientific reasoning, measuring success rate and steps with metrics varying by dataset.Baselines include base ReAct, pre-constructed skills, and test-time skill synthesis; backbone models are matched within each configuration.Source: §4.1 · p. 6

4.2 RESULTS: EFFECTIVENESS AND VERSATILITY OF EXPVOYAGER

  • EXPVOYAGER improves base-agent performance and outperforms existing approaches across ALFWorld, WebShop, and ScienceWorld on task success and benchmark-specific scores, while reducing execution steps.The evaluation compares against base ReAct, pre-constructed skill methods, and test-time skill synthesis using matched backbone models within each configuration.Source: §4.2 · p. 6
  • In online task streams, EXPVOYAGER can initially provide limited gains or degrade performance, but improves as experience accumulates while baseline methods plateau.Figure 4 plots cumulative success gain over the base agent against task-stream progress.Source: §4.2 · p. 7, Figure 4 · p. 7
  • With larger source-experience pools, EXPVOYAGER improves while baseline performance declines, widening its advantage as the shared pool grows.Figure 5 evaluates methods on the same nested subsets of source trajectories.Source: §4.2 · p. 7, Figure 5 · p. 7
  • Combining EXPVOYAGER with pre-constructed skills boosts the corresponding baselines; compared with standalone EXPVOYAGER, combined variants reduce task-time cost and usually improve task performance.The task-performance comparison has one exception; Figure 6 reports end-to-end tokens across navigation and execution.Source: Figure 6 · p. 7, §4.2 · p. 8
  • Under comparable experience-access budgets, EXPVOYAGER outperforms agentic search extensions; it benefits from more navigation up to 20 rounds, while further navigation reduces its performance.Increasing budget yields only marginal gains for baselines and can hurt their performance; Figure 7 presents the comparison.Source: §4.2 · p. 8, Figure 7 · p. 8

4.3 ANALYSIS: WHY AND HOW EXPVOYAGER WORKS

  • Removing the Navigable Interface lowers knowledge recall and increases irrelevant discoveries, while removing Navigation State slows recall growth through repeated discovery of duplicated knowledge.Table 3 reports performance ablations, and Figure 8 analyzes oracle knowledge recall and discovered knowledge types.Source: Table 3 · p. 8, Figure 8 · p. 8, §4.3 · p. 9
  • The curator adapts its access across trajectory and step views to seek different procedural knowledge, and explores knowledge distributed across multiple source trajectories.Figure 9 analyzes navigation actions, sought knowledge types, and source diversity.Source: §4.3 · p. 9, Figure 9 · p. 9

5 RELATED WORK

Prior work reuses experience by abstracting trajectories into procedural guidance or curating reusable knowledge, while task-time retrieval can miss procedural relevance. EXPVOYAGER instead navigates raw experience on demand and supports inspection of execution structure across views and trajectories.

Learning from Experience for Self-Evolving Agents

  • Prior experience-learning approaches abstract trajectories into reusable guidance or curate such guidance before future task demands are known, risking loss of knowledge later needed for a particular task.Task-time top-k retrieval is an alternative, but relevance similarity can miss localized cues in superficially dissimilar trajectories.Source: §5 · p. 9

Agentic Search and Corpus Interaction

  • Agentic search refines queries from intermediate results, and newer approaches enable finer corpus inspection; EXPVOYAGER adapts this exploration to experience’s context-decision-consequence structure.Its interface allows movement between complementary views within and across trajectories.Source: §5 · p. 9

6 CONCLUSION

ExpVoyager reframes skill synthesis as dynamic navigation over accumulated experience, and the paper reports improved task performance, scaling, and compatibility with existing skills.

6 CONCLUSION

  • ExpVoyager reframes agent skill synthesis as dynamic navigation over past experience for targeted, fine-grained access to procedural knowledge on demand.The paper presents this approach as a foundation for self-evolving agents that turn accumulated experience into reusable capabilities.Source: §6 · p. 9
  • Experiments show that ExpVoyager consistently improves downstream task performance and continues to scale as the experience space grows.The conclusion also reports compatibility with existing skills under efficient experience access.Source: §6 · p. 9
  • The authors describe ExpVoyager as a promising foundation for self-evolving agents that convert accumulated experience into reusable capabilities.Source: §6 · p. 9

AI USE STATEMENT

The authors disclose AI assistance with manuscript writing and experiment-code implementation, while reviewing the text and manually verifying the code. The related-work discussion positions ExpVoyager as adapting procedural knowledge for a fixed executor, and the experimental details specify default models and navigation settings.

AI USE STATEMENT

  • The authors used AI assistance for manuscript polishing and experiment-code implementation, then reviewed the text and manually verified the code.They take responsibility for the final content, including AI-assisted text and code.Source: AI USE STATEMENT · p. 10

A EXTENDED RELATED WORK

  • ExpVoyager navigates an agent’s own task-execution records to compose task-specific procedural guidance, complementing rather than redesigning the execution harness.It adapts knowledge for a fixed executor without changing its underlying harness configuration.Source: A EXTENDED RELATED WORK · p. 16

B EXPERIMENTAL DETAILS

  • The default setup uses Qwen3.5-9B as curator and frozen executor, with a 20-round navigation budget and results averaged over three random runs.Tables 1 and 2 vary the curator–executor configuration; other experiments use this default, with baselines sharing the backbone and source pool within each configuration.Source: B EXPERIMENTAL DETAILS · p. 16

B.1 DATASETS

The experimental details describe three interactive benchmarks, four experience-reuse baselines, and ExpVoyager’s experience-navigation and skill-synthesis implementation. The curator searches trajectory and step views, updates a structured knowledge state, and synthesizes the final skill from that state.

B.1 DATASETS

  • ALFWorld tests household tasks across six categories, WebShop evaluates instruction-constrained product purchases, and ScienceWorld evaluates interactive scientific reasoning.The evaluations use 140 ALFWorld test instances, 500 held-out WebShop instructions, and 90 ScienceWorld instances; the source pools contain 1,000, 1,000, and 500 trajectories or task–variation pairs, respectively.Source: §B.1 · pp. 16–17

B.2 BASELINES

  • The comparisons include workflow retrieval (AWM), reasoning-memory retrieval (ReasoningBank), global skill consolidation (Trace2Skill), and task-specific synthesis (SkillTTA).The methods use the same source pools and held-out tasks; retrieval settings include top-3 workflows for AWM, top-1 memory for ReasoningBank, and top-3 trajectories for SkillTTA.Source: §B.2 · pp. 17–18

B.3.1 DETAILS OF EXPVOYAGER

  • ExpVoyager stores chronological trajectories in separate trajectory-level and step-level views, each retaining a reference to its full source trajectory.Trajectory records contain action sequences and outcomes; step records include observations, reasoning, actions, and immediate results.Source: §B.3.1 · p. 18
  • The curator searches selected fields with a regular-expression pattern and can inspect a matching source’s chronological trajectory.Searches return complete stored records and source references; inspection supports line offsets, limits, and continuation information for long records.Source: §B.3.1 · p. 18
  • Navigation uses native function calls, one operation per request, and can finish early or continue until the 20-round default budget is exhausted.Figures 10 and 11 show an example state update and a generated skill with procedures and checks for a newspaper-placement task.Source: §B.3.1 · p. 18, Figure 10 · p. 19, Figure 11 · p. 19
  • ExpVoyager represents navigation state as knowledge items and open questions, updates them after observations, and synthesizes skill.md from the final state and target context.Knowledge items encode source-specific observations, transferable procedural relations, and conditions to verify during execution; the runtime validates updates and tracks supporting observations.Source: §B.3.1 · pp. 18, 20

B.3.2 PROMPTS

  • The prompts direct the curator to select open questions and experience views, assess whether observations resolve questions, and synthesize supported procedures with conditions and recovery checks.The prompt suite covers navigation-action selection, state updates, and final skill synthesis.Source: §B.3.2 · p. 20

B.4 EXPERIMENTAL DETAILS FOR PRELIMINARY ANALYSES

The preliminary analyses construct and verify target-specific oracle skills, annotate procedural knowledge in source trajectories, and evaluate how methods preserve or retrieve that knowledge. A separate comparison matches task-time experience-access budgets and examines the effect of longer navigation.

B.4 EXPERIMENTAL DETAILS FOR PRELIMINARY ANALYSES

  • An oracle skill is retained only after the frozen executor succeeds on the corresponding task in three independent runs, followed by manual review of the skill and trajectories.The process obtains at least four successful and four failed trajectories per target when collecting evidence, allows up to 20 candidate versions, and yields 80 reviewed oracle skills.Source: §B.4 · p. 20
  • For 80 oracle skills, annotators examine 300 ALFWorld training trajectories and mark sources helpful only when they contain evidence for actionable procedures, conditions, ordering, failure lessons, recovery, or checks.Both successful and failed sources can be helpful, and annotations and none labels receive manual review.Source: §B.4 · pp. 20–21
  • Claim-level matching decomposes artifacts into atomic procedural claims and scores semantic correspondence to oracle knowledge on a normalized five-point scale.Conditions, ordering, causal relations, and verification requirements inform matching; each item contributes only its highest match rather than summing matches.Source: §B.4 · p. 21
  • Knowledge preservation compares claims in pre-constructed AWM, ReasoningBank, and Trace2Skill artifacts with oracle knowledge from their contributing sources, then macro-averages target-level best-match scores.This measures how faithfully source knowledge remains in the pre-constructed artifacts.Source: §B.4 · p. 21
  • Source-level evaluation compares AWM workflows, ReasoningBank memories, and SkillTTA trajectories at retrieval depths 1, 3, 5, 10, and 20, using annotated helpful sources as the reference.Knowledge-level evaluation separately extracts claims from retrieved content, removes semantic duplicates, and computes recall and precision from best matching scores; retrieving a helpful source alone does not credit all its associated knowledge.Source: §B.4 · pp. 21–22

B.5 EXPERIMENTAL DETAILS FOR COMPARISON TO AGENTIC SEARCH

  • ExpVoyager consistently outperforms agentic-search extensions of retrieval baselines under approximately matched pre-execution token budgets.The budget includes retrieval or navigation, reasoning over accessed experience, and construction of guidance where applicable, but excludes downstream execution; performance declines beyond approximately 20–25 navigation rounds, possibly due to redundant or less relevant experience.Source: §B.5 · p. 22

B.6 EXPERIMENTAL DETAILS FOR ANALYSIS ON WHY EXPVOYAGER WORKS

The ablations isolate the roles of the Navigable Interface and Navigation State while retaining the rest of ExpVoyager’s iterative navigation setup.

B.6 EXPERIMENTAL DETAILS FOR ANALYSIS ON WHY EXPVOYAGER WORKS

  • Removing the Navigable Interface leaves the curator with chronological raw trajectories and general-purpose terminal search, without dedicated views for execution-specific structure.The same Navigation State updater receives matched trajectory portions and their surrounding context.Source: §B.6 · p. 22
  • Removing the explicit Navigation State preserves interface access and navigation history, but no longer consolidates observations into procedural knowledge and open questions revised across rounds.Earlier history is compressed into a running summary that informs subsequent decisions.Source: §B.6 · p. 23

B.7 EXPERIMENTAL DETAILS FOR ANALYSIS ON HOW EXPVOYAGER WORKS

The analysis examines ExpVoyager’s experience access across three benchmarks and describes how navigation actions, information needs, source diversity, and ALFWorld subtasks are measured.

B.7 EXPERIMENTAL DETAILS FOR ANALYSIS ON HOW EXPVOYAGER WORKS

  • The analysis uses three runs per task on ALFWorld, WebShop, and ScienceWorld, with Qwen3.5-9B for curator and executor and navigation capped at 20 rounds.It records navigation actions, open questions, curator reasoning, and accessed trajectories.Source: §B.7 · p. 23
  • Figure 9(a) action proportions aggregate calls across tasks and runs, separating trajectory-view search, step-view search, and trajectory inspection.The inner ring distinguishes search exp from inspect traj; the outer ring splits search exp by view.Source: §B.7 · p. 23
  • Figure 9(b) classifies each round’s primary information need as procedure overview, local decision, or cross-step context, then reports category proportions by selected action.GPT-5.6 Sol assigns labels using the open question and curator reasoning.Source: §B.7 · p. 23
  • Figure 9(c) measures source diversity by unique trajectories accessed per task and run, grouped into five count intervals and averaged over three runs.Repeated accesses and multiple records from one trajectory count as one source.Source: §B.7 · p. 23
  • Table 4 provides ALFWorld results broken down by subtask type, using Qwen3.5-9B as curator.The passage identifies the table’s scope but supplies no cell values.Source: Table 4 · p. 24
  • Table 5 details the ALFWorld and ScienceWorld experience-access budget comparison, reporting end-to-end tokens per task and navigation rounds alongside baseline settings.Budget groups B01–B06 use iterative retrieval for baselines; one-shot settings are top-3 for AWM and SkillTTA and top-1 for ReasoningBank.Source: Table 5 · p. 24

C.1 DETAILED RESULTS

The detailed-results section identifies three experiment breakdowns, while the case-study section examines successful and failed target executions across three benchmarks.

C.1 DETAILED RESULTS

  • The detailed results cover ALFWorld subtasks, experience-access budgets, and combinations of ExpVoyager with pre-constructed skills.Table 6 compares standalone ExpVoyager and combinations on ALFWorld success rate and WebShop score, alongside end-to-end tokens per task.Source: §C.1 · p. 24, Table 6 · p. 25

C.2 CASE STUDY

  • The case study uses selected successes and failures to examine navigation across experience views, evolving knowledge needs, and how observations become target-task skills.Figures present the target task, example navigation rounds, observations, and updated Navigation State.Source: §C.2 · p. 25

C.2.1 SUCCESS CASES

The cases show how ExpVoyager can turn varied trajectory observations into target-specific procedures and execution checks, while also exposing failures in preserving or operationalizing that knowledge.

C.2.1 SUCCESS CASES

  • The curator transfers procedures rather than source-specific locations or intermediate actions, using broader trajectory context to retain search-and-placement guidance and cooling before final placement.In the newspaper case, the skill checks the expected source and broadens the search if unavailable; in the plate case, it orders cooling before cabinet placement.Source: §C.2.1 · pp. 25–26
  • For cleaning and shopping tasks, synthesized skills preserve action conditions and execution-time checks, including verifying a cleaned object and distinguishing selectable options from descriptive attributes.The executor cleans before placement and verifies the result; for the requested product, it selects black and large before purchase.Source: §C.2.1 · pp. 25–26
  • When a product’s description suggests a requested attribute that its selectable options lack, the skill guides recovery by switching to a candidate with the required option.The first slipper listing mentions “Black-Grey,” but its selectable colors are pink and lake blue; the executor selects grey on another candidate.Source: §C.2.1 · p. 26
  • Combining a failed sandwich trajectory with reasoning from another recipe reveals that ingredients must share a container before mixing, which the skill translates into ingredient-transfer actions.The executor transfers the ingredients into a cup and produces the sandwich.Source: §C.2.1 · p. 26
  • Recorded reasoning and cross-step context separate reusable comparison procedure and action order from source-specific animals, guiding selection of the longest-lived candidate before the shortest-lived one.The source observation need not contain numerical lifespan information for its reasoning to clarify the comparison purpose.Source: §C.2.1 · pp. 26–27

C.2.2 FAILURE CASES

  • A synthesis failure reverses sequential acquisition-and-placement guidance, causing the executor to keep searching while holding one remote and never place it.The navigation observations had identified that the held remote should be placed before searching for another.Source: §C.2.2 · p. 27
  • A revisited WebShop product can lose previously selected options, so the executor’s earlier selections do not ensure the required state at purchase.In the failure case, the purchase record has an empty options map after the product is reopened.Source: §C.2.2 · p. 27
  • Identifying the shared-container precondition is insufficient when the synthesized skill proposes mixing instead of actions that transfer ingredients into the container.The executor repeatedly attempts to mix without establishing the required state.Source: §C.2.2 · p. 27
Loading 2609.32630v1…