Source-linked AI summary
ExpeL: LLM Agents Are Experiential Learners
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, Gao Huang
TL;DR
Existing LLM-agent adaptation methods face costly finetuning, unavailable proprietary parameters, and limited cross-task memory. ExpeL instead gathers experiences, extracts natural-language insights, and recalls them with successful trajectories at evaluation. The agent consistently outperformed strong baselines across three domains, demonstrated forward transfer, and exhibited additional emergent abilities.
Problem
Finetuning LLM agents is computationally costly, requires access to proprietary parameters, and may reduce generalization, motivating parameter-free learning from agent experience.
Method
ExpeL autonomously gathers experiences from training tasks, extracts natural-language insights, and uses those insights plus successful trajectories during single-attempt evaluation.
Results
ExpeL consistently outperformed strong baselines across three domains, showed positive forward transfer from source to target tasks, and exhibited emergent abilities.
Takeaways & Limitations
ExpeL demonstrates that an LLM agent can improve from accumulated experience without gradient updates or access to model parameters.
Takeaways & Limitations
The study focuses on textual observations, closed-source API LLMs, deterministic environments, and insights that fit within the current context window.
Abstract
from arXiv · showhide
The recent surge in research interest in applying large language models (LLMs) to decision-making tasks has flourished by leveraging the extensive world knowledge embedded in LLMs. While there is a growing demand to tailor LLMs for custom decision-making tasks, finetuning them for specific tasks is resource-intensive and may diminish the model's generalization capabilities. Moreover, state-of-the-art language models like GPT-4 and Claude are primarily accessible through API calls, with their parametric weights remaining proprietary and unavailable to the public. This scenario emphasizes the growing need for new methodologies that allow learning from agent experiences without requiring parametric updates. To address these problems, we introduce the Experiential Learning (ExpeL) agent. Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions. Our empirical results highlight the robust learning efficacy of the ExpeL agent, indicating a consistent enhancement in its performance as it accumulates experiences. We further explore the emerging capabilities and transfer learning potential of the ExpeL agent through qualitative observations and additional experiments.
1 Introduction
LLM agents offer broad versatility through embedded world knowledge, but existing adaptation strategies either require costly parameter access or lack durable cross-task memory. ExpeL addresses this by learning natural-language insights from experience without parameter updates, improving across domains and supporting transfer.
- LLMs provide broad versatility across scenarios through extensive world knowledge.
- Finetuning requires substantial computation and parametric-weight access, while potentially restricting functionality and harming generalization.
- ExpeL gathers training-task experiences, derives natural-language insights, and recalls successful experiences during evaluation without updating parameters.
- Across three different domains, ExpeL consistently outperformed strong baselines and showed positive forward transfer from source tasks to target tasks.
- The paper introduces ExpeL, evaluates its learning abilities across diverse tasks, and demonstrates forward transferability.
2 Related Work
Related work includes prompt-based learning and methods that improve LLM reasoning through structured prompting or iterative reflection. ExpeL extends this broader direction toward experience-based adaptation.
- Prompt-based Learning: Prompt-based learning adapts model inputs rather than parameters, enabling task adaptation with few or no labeled examples.
- Prompt-based Learning: Prior prompting research includes LAMA, GPT-3, automatic reasoning chains, and in-context learning for reducing prompt-design difficulty.
3 Preliminaries
The paper frames LLM agents as autoregressive language models operating in interactive environments, with ReAct and Reflexion providing planning and reflective retry mechanisms. ExpeL’s overview organizes learning around collecting, abstracting, and applying experiences.
- Complex Interactive Tasks: The agent receives observations over time, uses its observation history to choose actions, and seeks to achieve a goal in deterministic environments.
- Large Language Models: A large language model is a statistical natural-language model, and the paper uses an autoregressive model that predicts the next token from preceding tokens.
- ExpeL Agent Overview: ExpeL’s pipeline collects success and failure experiences, extracts cross-task knowledge, and applies insights and recalled successes during evaluation.
- ExpeL Agent Overview: The overview depicts Reflexion enabling task reattempts after self-reflection on failures and insight updates through ADD, UPVOTE, DOWNVOTE, and EDIT operations.
- ReAct intertwines observations, actions, and thoughts, while Reflexion adds reflection before retrying the same task.
4 ExpeL: An Experiential Learning Agent
ExpeL equips an LLM planning agent to learn from inter-task experiences without parameter updates by extracting natural-language insights and recalling successful trajectories. It gathers and analyzes trial-and-error experiences, then augments evaluation prompts with insights and task-similar examples, while also supporting transfer across domains with limited target examples.
- 4 ExpeL: An Experiential Learning Agent: ExpeL improves planning agents through inter-task experience learning without updating model parameters.The agent gathers experiences during training, stores them in an experience pool, and uses them during evaluation of unseen tasks with a single try.
- 4.1 Gathering Experiences: Trial-and-error collection stores trajectories and uses reflections on failed attempts to generate diverse success and failure experiences.Training tasks can be retried at most Z times; successful trajectories enter the pool, while failures prompt additional self-reflection before retrying.
- 4.2 Extracting Insights: ExpeL learns in two complementary ways: recalling successful trajectories as examples and extracting high-level insights for generalization.The framework combines memory-like retrieval of specific trajectories with natural-language knowledge extracted from experience comparisons and successful-trajectory patterns.
- 4.2 Extracting Insights: Insight extraction compares same-task failure and success pairs, identifies patterns across L successful trajectories, and updates an insight set with ADD, EDIT, DOWNVOTE, and UPVOTE operations.The insight set is initialized empty and iteratively refined from sampled experience groups.
- 4.3 Task Inference: During evaluation, ExpeL concatenates extracted insights with task specifications and retrieves top-k successful trajectories by task similarity as few-shot examples.The prompt template adds both past insights and dynamically retrieved successful in-context examples to the base ReAct inputs; retrieval is proposed to help manage growing context.
- 4.4 Transfer Learning: ExpeL consistently outperformed strong baselines across three domains and showed positive forward transfer from source tasks to target tasks.The transfer setting uses source-domain insights and few-shot target examples to make the insights more applicable with limited target data.
5 Experiments
ExpeL was evaluated across diverse text-based decision-making benchmarks using success rate, consistently outperforming baselines and showing benefits from autonomous, diverse experience collection. Ablations and additional experiments indicate that insights, task-similar retrieval, transfer, and repeated attempts each contribute to performance or capability.
- 5.1 Experimental Setup: ExpeL was evaluated on HotpotQA, ALFWorld, and WebShop using four-fold validation and success rate as the primary metric.HotpotQA uses exact matching, ALFWorld requires timely task completion, and WebShop requires purchasing an item matching all attributes.
- 5.2 Main Results: ExpeL consistently outperformed ReAct and Act across all three domains, while insights-only and retrieval-only variants showed complementary strengths.Insights had greater influence on HotpotQA, retrieval was more useful for ALFWorld, and WebShop showed near-equilibrium performance between the two modes.
- 5.6 Ablation Studies: Autonomously collected experience improved performance beyond few-shot examples, and diverse success/failure pairs gathered with Reflexion outperformed ReAct-only experience collection.The few-shot-only agent had no advantage over ReAct, while agents with more experience performed significantly better.
- 5.6 Ablation Studies: Learned insights outperformed hand-crafted insights, while adding reflections could hurt insight extraction, possibly because hallucinated reflections misled the process.Using a better language model for insight generation was also associated with improved ExpeL performance.
- 5.6 Ablation Studies: Task-similarity retrieval performed best in ALFWorld, whereas reasoning-similarity retrieval slightly reduced performance and random sampling caused a significant drop.The results support selecting the most pertinent successful in-context example from the experience pool.
6 Conclusion and Limitations
The paper concludes that ExpeL learns from task experience without model-parameter access and transfers extracted knowledge across tasks. It also identifies boundaries involving observation modality, model availability, context length, and theoretical grounding.
- Limitations: The experiments used textual observations, limiting applicability to real-world scenarios that require image observations.
- Limitations: The evaluation relied on closed-source API LLMs, while lifelong use may require extra insight retrieval to keep the context window manageable.
- Limitations: Prompting techniques lack the theoretical underpinnings of reinforcement learning, which could affect the efficiency of resulting policies.
- Conclusion: ExpeL autonomously gathers experience from training tasks and improves evaluation-task performance without access to model parameters.
- Conclusion: ExpeL demonstrated forward transfer from source tasks to a target task and developed several unexpected abilities during training.
- Conclusion: The agent's memory combines successful in-context examples with extracted insights gathered from its own experience.
D.1 Evaluation Task Set
The evaluation uses four-fold validation with train/test halves exchanged across folds and follows established prompts where appropriate. WebShop was modified to make experiment instantiations deterministic.
- D.1 Evaluation Task Set: Four-fold validation trains on one half of each dataset and evaluates on the other half, then reports mean and standard error across folds.
- D.1 Evaluation Task Set: The experiments reuse ReAct and Reflexion few-shot examples and prompts, adding one WebShop few-shot example.
- D.1 Evaluation Task Set: WebShop item prices and price constraints use average values instead of uniformly sampled values to ensure deterministic instantiations.
D.4 WebShop Reward Function
WebShop evaluates purchased products by matching their attributes, options, price, and textual similarity to the target product. Its reward function maps this matching quality to a value from 0 to 1.
- D.4 WebShop Reward Function: WebShop's reward function converts similarity between expected and purchased product attributes into a value ranging from 0 to 1.
- D.4 WebShop Reward Function: The reward aggregates matched required attributes, matched optional attributes, and whether the purchased price satisfies the user's price constraint.
- D.4 WebShop Reward Function: The reward assigns values from 0 to 1 according to TextMatch, query and category matches, with 1 for the otherwise-matching case.
- D.4 WebShop Reward Function: TextMatch measures textual overlap of pronouns, nouns, and proper nouns between the selected and target product titles.
D.5 Base Language Model
The experiments use OpenAI API language models through Langchain, with model roles divided across experience gathering, insight extraction, and evaluation. The appendix also provides environment, retrieval, agent-parameter, and task-specific acting templates.
- D.5 Base Language Model: All experiments use Langchain to make OpenAI API calls, with different GPT-3.5 and GPT-4 variants assigned to experience gathering, insight extraction, and evaluation.
- D.5 Base Language Model: The policy and actor prompt templates are adapted from ReAct with minimal changes for extracted ExpeL insights.
- D.5 Base Language Model: Appendix Figure 7 presents the ExpeL HotpotQA acting template.
- D.5 Base Language Model: Appendix Figure 8 presents the ExpeL ALFWorld acting template.
- D.5 Base Language Model: Appendix Figure 9 presents the ExpeL WebShop acting template.
- D.5 Base Language Model: Appendix Figure 10 presents the ExpeL FEVER acting template.
G Example Insights
The paper presents example insights extracted from failed and successful trajectories by GPT-4 or humans. Purple highlighting marks particularly interesting insights, including emergent abilities discussed elsewhere.
- Example insights are extracted from both failed and successful trajectories.
- GPT-4-0613 and humans are both used as sources of extracted insights.
- Purple highlighting identifies selected interesting insights, including emergent abilities discussed in Section 5.3.
G.1 HotpotQA insights
The examples illustrate how ExpeL extracts, transfers, and uses insights across HotpotQA, ALFWorld, WebShop, and FEVER. Its trajectories show environment adaptation, query refinement, self-correction, and world-model updating relative to ReAct.
- HotpotQA insights: HotpotQA insights recommend decomposing complex questions into simpler queries, resembling Auto-GPT’s mechanism.
- HotpotQA insights: GPT-4 extracts insights that overlap with hand-crafted insights derived from successful and unsuccessful Reflexion trajectories.
- Cross-environment insights: ExpeL transfers extracted insights to FEVER and showcases extracted insights in ALFWorld and WebShop.
- Emergent abilities: In HotpotQA, ExpeL reevaluates trajectories and observations to formulate educated guesses and answer questions instead of conceding.
- Emergent abilities: In ALFWorld, experience updates a pan’s likely location from a countertop to a stoveburner, enabling successful task completion.
- Emergent abilities: ExpeL self-corrects mistaken actions in ALFWorld, whereas ReAct recognized some errors but never remedied them by discarding incorrect items.
J Additional Quantitative Results
The paper supplements its main evaluation with environment-specific scores, outcome breakdowns, and trajectory statistics. These results cover ALFWorld, WebShop, HotpotQA, and additional step and token metrics.
- Additional quantitative results include ALFWorld average success rates and WebShop environment-specific average rewards.
- Figures 25–27 break down task outcomes into success, failure, and halt categories for HotpotQA, ALFWorld, and WebShop.
- Table 6 reports average step and token statistics per trajectory for each benchmark.