PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks

Vishnu Sarukkai, Zhiqiang Xie, Kayvon Fatahalian

arXiv:2505.00234v3cs.LGcs.CL

TL;DR

Task-specific prompts, curated examples, and custom observation or action spaces often demand human knowledge engineering for sequential decision-making agents. This paper builds and curates databases of agents’ successful self-generated trajectories as in-context examples. Naive accumulation improves performance across three benchmarks, and curation raises ALFWorld success to 93%.

  • Problem

    Improving sequential decision-making agents commonly relies on task-specific prompts, human-crafted examples, or custom observation and action spaces.

    Source: §1 · p. 1

  • Method

    The method accumulates successful self-generated trajectories and curates the database of examples retrieved for future decisions.

    Source: §1 · p. 1

  • Results

    Trajectory accumulation improves performance across ALFWorld, Wordcraft, and InterCode-SQL, while curation raises ALFWorld success to 93%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings point to data quantity and quality as important dimensions for improving LLM agents through experience.

    Source: §7 · pp. 11–12

    The setup assumes an initial small set of human-generated trajectories, an action-space descriptor, and access to training tasks from the task distribution.

    Source: §3 · p. 4

Abstract

from arXiv · show

Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering--custom prompts, curated examples, and specialized observation/action spaces. We investigate a different approach where agents automatically improve by learning from their own successful experiences without human intervention. Our method constructs and refines a database of self-generated trajectories that serve as in-context examples for future tasks. Even naive accumulation of successful trajectories yields substantial performance gains across three diverse benchmarks: ALFWorld (73% to 89%), Wordcraft (55% to 64%), and InterCode-SQL (75% to 79%). These improvements exceed those achieved by upgrading from gpt-4o-mini to gpt-4o and match the performance of allowing multiple attempts per task. We further enhance this approach with two innovations: database-level curation using population-based training to propagate high-performing example collections, and exemplar-level curation that selectively retains trajectories based on their empirical utility as in-context examples. With these enhancements, our method achieves 93% success on ALFWorld--surpassing approaches that use more powerful LLMs and hand-crafted components. Our trajectory bootstrapping technique demonstrates that agents can autonomously improve through experience, offering a scalable alternative to labor-intensive knowledge engineering.

1 Introduction

The paper proposes autonomous agent improvement through successful self-generated trajectories, addressing the human effort involved in task-specific knowledge engineering. It studies how to collect and curate those examples for future decisions.

1 Introduction

  • Task-specific agent improvements commonly rely on prompt tuning, human-crafted examples, or custom observation and action spaces, scaling human effort.The paper frames trajectory bootstrapping as an alternative to that knowledge engineering.Source: §1 · p. 1
  • The method autonomously bootstraps agents by accumulating successful self-generated trajectories and selecting relevant, effective examples for each decision point.The approach targets both example quality and relevance to the current decision.Source: §1 · p. 1
  • The database construction problem is to identify trajectories that help or hinder new tasks and curate valuable examples for future retrieval.The assumed agent retrieves examples according to their relevance to the current situation.Source: §1 · p. 1
  • 73% to 89% on ALFWorld, 55% to 64% on Wordcraft, and 75% to 79% on InterCode-SQL with naive trajectory accumulation.Database-level and exemplar-level curation further raise ALFWorld success to 93%.Source: §1 · p. 2

2 Preliminaries

The paper frames its agents as acting over multiple steps with partial observations and sparse end-of-episode rewards. Its ReAct-style loop plans, reasons, acts, and retrieves situation-relevant examples while avoiding task-specific prompts and custom spaces.

2.1 Sequential Decision-Making Tasks

  • Agents face multi-step tasks with partial observations, and success is rewarded only at episode end.The paper models tasks as POMDPs and focuses on sparse-reward environments.Source: §2.1 · p. 2

2.2 ReAct-style Agent Loop

  • The ReAct-style agent plans before execution, then reasons and acts while retrieving different trajectory segments for each decision point.Retrieval selects examples by cosine distance to lookup keys; the database is the paper's focus for construction and refinement.Source: §2.2 · pp. 2–3
  • The agent avoids task-specific prompting and custom observation or action spaces, keeping task-specific knowledge in the trajectory database.The paper treats hierarchical learning as complementary rather than part of this database-construction focus.Source: §2.2 · p. 3

3 Problem Statement

The problem is to construct a trajectory database that maximizes expected agent performance across tasks. The paper focuses on generating trajectories and selecting which ones to retain for novel tasks.

3 Problem Statement

  • The objective is to construct a trajectory database that maximizes expected agent performance across a task distribution.Success is the binary outcome of an agent execution.Source: §3 · pp. 3–4
  • The setup assumes a small initial set of human-generated trajectories, an action-space descriptor, and training tasks drawn from the task distribution.The agent self-generates trajectories and chooses additions to improve expected performance on novel tasks.Source: §3 · p. 4
  • The paper asks whether more successful trajectories improve guidance and how to select the subset most useful for solving new tasks.These questions distinguish trajectory collection from trajectory curation.Source: §3 · p. 4

4 Related Work

The related work motivates in-context learning for agent improvement and distinguishes the paper's dynamic, trajectory-based retrieval from fixed exemplars and other self-improvement approaches.

4 Related Work

  • In-context learning is presented as portable across models, useful in low-sample regimes, and accessible when weight modification is difficult.Prior work suggests performance can scale with additional examples, while this paper emphasizes database quality as well as quantity.Source: §4 · p. 4
  • Unlike approaches selecting fixed exemplars for all instances, this method dynamically selects different in-context examples for each decision.Related work includes optimized instructions and curated examples, as well as self-generated reasoning examples.Source: §4 · p. 4
  • Prior agent self-improvement includes search, feedback from failures, abstraction-based transfer, task-specific designs, and hierarchical execution.The paper instead focuses on identifying trajectories that contribute to successful outcomes as in-context examples; hierarchical methods are described as complementary.Source: §4 · pp. 4–5

5 Methods

The methods build a database of successful self-generated trajectories, then curate it at both database and exemplar levels. They shift extra computation to training while keeping test-time costs identical across the three methods.

5.1 Traj-Bootstrap: Constructing a Database of Previously-Solved Tasks

  • Traj-Bootstrap grows a database from successful agent trajectories, storing only successes because failed trajectories require identifying which parts were good or bad.The initial human-provided exemplars can be minimal or empty; using failed trajectories to guide exploration is left for future work.Source: §5.1 · p. 5

5.2 +DB-Curation: Database-Level Data Curation

  • Five InterCode-SQL trials show variable test-time success; +DB-Curation periodically replaces the worst-performing database with a copy of the best.The method keeps N independently growing databases, evaluates recent task performance at doubling thresholds beginning at 10, and propagates the top performer.Source: Figure 1 · p. 5, §5.2 · p. 5
  • Database-level curation preserves collective properties such as coverage, diversity, and complementarity, which whole-database selection can retain but exemplar removal may discard.Early trajectories can guide later trajectories, creating cascading database-level effects.Source: §5.2 · p. 6, §5.3 · p. 6

5.3 +Exemplar-Curation: Exemplar-Level Data Curation

  • Exemplar-Curation selects the highest-value successful trajectory for each training task across database instances, forming a composite database of exemplars.Trajectory value reflects retrieval frequency and outcomes, prioritizing examples associated with successful completions and penalizing those associated with failures.Source: §5.3 · p. 6

5.4 Train-Time vs Test-Time LLM Costs

  • The curation methods use N times more LLM inference calls during training, require no additional training tasks, and have the same test-time computational cost as Traj-Bootstrap.This shifts computation to training rather than increasing LLM calls per test-time task.Source: §5.4 · p. 7

6 Experiments

Across three benchmarks, self-collected trajectory databases improve agent success, and combining database-level and exemplar-level curation yields the strongest results on ALFWorld and InterCode-SQL. The experiments also compare these methods with test-time attempts, model upgrades, task-specific and hierarchical approaches, and evaluate diagnostics and fine-tuning from collected trajectories.

6.1.1 Benchmark Tasks

  • The experiments use ALFWorld, InterCode-SQL, and Wordcraft, selected for train/test task pools, diverse sequential-reasoning challenges, and comparisons with prior work.The compared approaches include Fixed-DB, Traj-Bootstrap and its curation variants, Autoguide, and AutoManual.Source: §6.1.1 · p. 7, §6.1.2 · p. 8

6.2 Traj-Bootstrap Results

  • Traj-Bootstrap generally improves as training tasks and database size increase, with most gains in the first 25% of added tasks and substantial variation across and within trials.As the database grows, each new example is retrieved less often; some added trajectories can degrade performance.Source: §6.2 · p. 8, Figure 2 · p. 9
  • +DB-Curation improves final success only on ALFWorld and can dip at smaller database sizes when low-sample quality estimates add noise to curation.The figure comparison reports that database curation improves success at smaller database sizes across all benchmarks.Source: §6.2 · p. 9, Figure 3 · p. 10
  • +Exemplar-Curation improves final success on InterCode-SQL and Wordcraft; high-quality trajectories outperform low-quality ones across database sizes on ALFWorld and Wordcraft, and at smaller sizes on InterCode-SQL.The “best” trajectory curve is +Exemplar-Curation; the “worst” selects the bottom-ranked trajectory instead of the top-ranked one.Source: §6.2 · p. 9, Figure 4 · p. 10
  • 0.93 on ALFWorld and 0.82 on InterCode-SQL are the best final success rates, achieved by combining database-level and exemplar-level curation; exemplar curation alone leads on Wordcraft at 0.71.On Wordcraft, Traj-Bootstrap and +DB-Curation both score 0.64, while the combined method scores 0.67.Source: Table 1 · p. 8, §6.2 · p. 10

6.3 Contextualizing performance boosts from Traj-Bootstrap

  • Across all benchmarks, with one test-time attempt per task, Traj-Bootstrap achieves success rates between Fixed-DB pass@2 and pass@3; +DB+Exemplar-Curation reaches between pass@3 and pass@5.Source: Table 3 · p. 11
  • On ALFWorld, Traj-Bootstrap’s 20-point boost after 3,500 training tasks exceeds the 15-point improvement from upgrading Fixed-DB to a more powerful LLM.With GPT-4o-mini, the combined method scores 0.93, exceeding AutoManual’s 0.91 with GPT-4-turbo and GPT-4o-mini and hand-designed observation and action spaces [4]; with 100 training tasks, the best approach scores 0.86 versus Autoguide’s 0.79.Source: §6.3 · p. 11

6.4 Extending Traj-Bootstrap

  • On InterCode-SQL and Wordcraft, classifiers trained on task-goal and initial-observation embeddings reach AUROC 0.77 and 0.71, respectively, with the largest databases; their predicted probabilities closely match empirical success rates.Fine-tuned agents outperform the in-context approach on ALFWorld and Wordcraft, but perform worse on InterCode-SQL.Source: §6.4 · p. 11

7 Discussion

The discussion argues that gains come primarily from accumulating successful examples, with data quantity and quality rivaling architectural complexity. The agent uses task-agnostic prompts and retrieval at both trajectory and state levels to supply relevant experience during planning and action.

7 Discussion

  • Performance gains stem primarily from accumulating successful examples, while the quantity and quality of accessible data rival architectural complexity.The authors point to data-centric directions including strategic collection and refined filtering.Source: §7 · pp. 11–12

A Key Agent Details

  • Retrieval draws examples at trajectory level for planning and at state level for reasoning and action selection.State retrieval returns a window around a similar state; multi-key retrieval averages cosine similarities across the selected trajectory keys.Source: A Key Agent Details · p. 15, §B.1 · p. 16

B.1 Prompt Templates

  • The same minimalist, task-agnostic prompt templates are used across benchmarks, with retrieved successful examples supplied as context.The templates include examples suited to the relevant component, such as goal-plan examples for planning.Source: §B.1 · pp. 15–16

B.2 Retrieval Implementation

  • Retrieval uses FAISS exact nearest-neighbor search, averaging similarity across requested keys and including nearby states in a window of up to five steps.The surrounding states provide coherent episode fragments.Source: §B.2 · p. 16

B.3 Population-Based Training Details

Database-level curation replaces poorly performing databases with copies of high-performing ones as the collection grows. Exemplar-level curation scores trajectories by their retrieval-weighted association with task outcomes, with safeguards for sparse retrieval data.

B.3 Population-Based Training Details

  • Every database-size doubling triggers validation; the bottom 20% of databases are replaced with copies of the top 20%.The method maintains five database instances and evaluates performance on tasks attempted since the previous doubling.Source: §B.3 · pp. 16–17

B.4 Quality Metric Computation

  • Trajectory quality is computed from outcomes of tasks where the trajectory was retrieved, weighted by its retrieval frequency.Scores are computed after training; trajectories retrieved on fewer than three tasks receive the overall average task success rate instead.Source: §B.4 · p. 17

B.5 Note on planning step

  • The agent omits planning on short-trajectory Wordcraft and InterCode-SQL tasks but retains it for 30-step ALFWorld tasks.The passage describes planning as valuable for long-horizon coherence on ALFWorld.Source: §B.5 · p. 17

C Can Self-Collected Examples Improve a Fine-Tuned LLM Agent?

The paper tests whether trajectories collected for in-context use can also train a fine-tuned agent. ReAct-Finetune is competitive with the best in-context approach, with benchmark results varying by task.

C. Can Self-Collected Examples Improve a Fine-Tuned LLM Agent?

D Can We Predict Agent Success Rates

Success prediction improves as self-collected databases grow and is well calibrated on Wordcraft and InterCode-SQL, though this analysis excludes ALFWorld. On Wordcraft, human-provided starting examples outperform empty initialization; a separate InterCode-SQL comparison contextualizes the gains against a hand-crafted agent with schema access.

D Can We Predict Agent Success Rates

  • InterCode-SQL AUROC rises from 0.60 (100 tasks) to 0.77 (800 tasks), while Wordcraft rises from 0.50 (100 tasks) to 0.71 (4000 tasks).Predicted success probabilities closely match observed rates for the final classifiers; the analysis excludes ALFWorld because goal and initial observation do not reveal scene layout.Source: D Can We Predict Agent Success Rates · p. 18, Figure 6 · p. 19

E Is it Possible to Bootstrap an Agent Without Initial Hand-Crafted Examples?

  • On Wordcraft, initialization with five human-provided trajectories outperforms an empty database, which still scales but remains below that seeded performance after 4000 training tasks.The authors report that the initial trajectories shaped reasoning and action patterns in a way that boosted continual database construction on this task.Source: Figure 7 · p. 19, E Is it Possible to Bootstrap an Agent… · p. 19

F.1 How does Automanual Leverage Hand-Crafted Information

  • On InterCode-SQL, GameSQL with schema access scores 0.84, slightly above +DB+Exemplar-Curation at 0.82; without schema access, GameSQL scores 0.73 versus Fixed-DB at 0.74.The passage says the database-construction boost nearly matches the boost from privileged schema information.Source: Table 5 · p. 21

F.2 A note on training and test sets

The evaluation keeps database-building trajectories on designated training tasks and tests on separate, unseen tasks. It also compares the InterCode-SQL method with hand-crafted agents and describes the three benchmark setups.

F.2 A note on training and test sets

  • Trajectories for database construction come only from designated training tasks, while evaluation uses separate test tasks unseen during construction.The authors describe this separation as essential for validating generalization rather than memorization.Source: §F.2 · p. 21

G Comparison to Hand-Crafted InterCode-SQL Agent

  • 0.82 versus 0.84: +DB+Exemplar-Curation approaches GameSQL+Cheat, while Fixed-DB performs similarly to GameSQL at 0.74 versus 0.73.The authors say database construction lifts a generic ReAct-style agent nearly as much as schema-handicap access lifts the hand-crafted agent.Source: G Comparison to Hand-Crafted… · pp. 21–22

H.1 ALFWorld

  • ALFWorld evaluates text-based navigation and object manipulation across 3,500 training tasks and 134 out-of-distribution test tasks.The setup also searches over task categories and initializes databases with 18 successful trajectories, three per category.Source: §H.1 · p. 22

H.2 InterCode-SQL

  • InterCode-SQL evaluates SQL query generation on a Spider-based subset, split into 800 training tasks and 234 test tasks.Agents inspect the schema, receive syntax and execution feedback, and must submit a query within 10 steps.Source: §H.2 · p. 22

H.3 Wordcraft

  • Wordcraft tests multi-step element-combination tasks using 4,000 training tasks and 500 test tasks with disjoint goal elements.The selected tasks require up to two solution steps, allow four action steps, and start from four human-annotated trajectories.Source: §H.3 · p. 23

H.4 Note on Benchmark Selection

The appendix explains why some benchmarks were excluded, reports computational resources, and reproduces parts of the hand-crafted GameSQL prompt.

H.4 Note on Benchmark Selection

  • The benchmark suite covers text-based navigation and object manipulation, interactive code generation, and compositional reasoning.WebShop was excluded after the authors encountered bugs in generating achievable goals and identified incorrect rewards; QA benchmarks were excluded because retrieval quality and self-evaluation would confound the study.Source: §H.4 · p. 23

I Computational Resources

  • Database operations, including embedding computation, storage, and retrieval, accounted for less than 5% of total computation time.Most computation was spent on OpenAI API calls; the reported API usage cost was approximately $3,000 USD.Source: I Computational Resources · pp. 23–24

J GameSQL Prompt

  • The GameSQL prompt recommends first inspecting accessible tables, then writing SQL commands to answer the natural-language query.It describes a multi-turn environment that returns query output and a reward indicating how closely the result matches the correct answer.Source: J GameSQL Prompt · pp. 24–25

Related papers

Loading 2505.00234v3…