PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, Kai Yu

arXiv:2306.07929v2cs.CLcs.AI

TL;DR

LLMs face challenges learning from interaction experiences: fine-tuning is costly, while embedding trajectories in prompts is limited by input length. REMEMBERER combines an LLM with RLEM, which updates persistent experience memory through RL; it reports gains over prior SOTA on both evaluated task sets.

  • Problem

    LLMs struggle to learn effectively from interaction experiences, while fine-tuning is costly and embedding trajectories in prompts is limited by input length.

    Source: §1 · p. 1

  • Method

    REMEMBERER combines an LLM with RLEM, which updates persistent experience memory through RL rather than modifying the model parameters.

    Source: §1 · p. 2

  • Results

    REMEMBERER reports average improvements over SOTA models of 2 points on WebShop and 4 points on WikiHow.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The framework is evaluated by training on a few tasks and testing on other tasks to assess use of experience on unseen episodes.

    Source: §1 · p. 2

    Performance may saturate quickly in training, which the authors suggest may be due to the limited number of active exemplars.

    Source: §6 · p. 10

Abstract

from arXiv · show

Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as REMEMBERER. By equipping the LLM with a long-term experience memory, REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed REMEMBERER constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of REMEMBERER.

1 Introduction

The paper proposes equipping LLM agents with persistent experience memory so they can learn from interaction histories without fine-tuning model parameters. REMEMBERER combines this memory with RLEM and reports gains over prior state of the art on WebShop and WikiHow.

  • The authors motivate using past episodes by drawing on the role of episodic memory in human decision-making and learning from success and failure.Source: §1 · p. 1
  • The paper identifies costly LLM fine-tuning and input-length limits on embedded trajectories as challenges to learning from interaction experiences.RLEM updates external experience memory through RL rather than modifying model parameters.Source: §1 · pp. 1–2
  • Unlike task-goal-bound working memory and one-time immediate failure feedback, persistent experience memory can retain histories across episodes and task goals.The paper presents this as an opportunity to discover late failures and learn from prior episodes.Source: §1 · p. 2
  • REMEMBERER combines an LLM with RLEM and selectively uses stored experiences from past episodes to support decisions on unseen tasks.The framework updates its experience memory through an RL process without fine-tuning the LLM parameters.Source: §1 · p. 2
  • +2 points on WebShop and +4 points on WikiHow over SOTA models are reported for REMEMBERER.The reported averages are for the two task sets.Source: §1 · pp. 2–3

2 Related work

The related work contrasts REMEMBERER's persistent, environment-grounded experience memory with external knowledge resources, task-bound working memory, and methods that use immediate failure feedback. It also situates the work among LLM-based decision-making agents by its combination of RL and experience learning.

LLM with external information

  • REMEMBERER uses structured, environment-grounded persistent memory to reuse successful and failed experiences across task goals, without human intervention.The paper contrasts this with plain-text successful-trajectory records and task-dedicated working memory.Source: §2 · p. 3

LLM learning from failure

  • REMEMBERER applies RL to learn from late and immediate environment-reward failures without human feedback, and reuses those experiences across different task goals.The related work contrasts this with one-time immediate feedback, task-bound reflections, and failure corrections that rely on human feedback.Source: §2 · p. 3

LLM for decision-making

  • The authors describe their work as the first to combine an LLM-based agent with RL to learn from interaction experiences and self-evolve.The paper presents REMEMBERER as equipping the LLM with external memory for successful and failed experiences.Source: §2 · p. 3

3 Method

REMEMBERER couples an LLM decision maker with persistent experience memory, retrieving relevant records for each observation and updating memory from interaction transitions. It turns stored experiences into dynamic prompt examples, including advice based on estimated action values.

3.1 RLEM pipeline

  • At each decision step, the LLM retrieves related experiences using the current observation, chooses an action using those records and interaction feedback, then updates memory with the resulting transition.Source: §3.1 · p. 4

3.2 Experience memory of REMEMBERER

  • The persistent memory stores task information, observations, actions, and Q estimates; initialized records demonstrate the input/output format, while new experiences let the system evolve.Source: §3.2 · p. 4
  • New transitions drive off-policy updates: unseen records receive estimates, existing records use Q-learning, and n-step bootstrapping addresses sparse sampling [Mnih et al., 2016].Source: §3.2 · p. 5

3.3 Usage of the experiences

  • A similarity function compares the task goal and current observation with stored records, and the m most similar records become prompt exemplars.Source: §3.3 · p. 5
  • Advice promotes high-Q actions and discourages poor-Q ones; when records lack useful candidates, REMEMBERER favors randomly sampled advice.Source: §3.3 · pp. 5–6

4 Experiments & results

Experiments evaluate REMEMBERER on WebShop and WikiHow, comparing it with existing agent baselines and varying memory-training and retrieval choices. The reported results favor REMEMBERER, while ablations examine which memory-update and advice components matter.

4.1 Experiment setup & implementation details

  • WebShop observation similarity uses page patterns, whereas WikiHow computes it from the longest common sequence of HTML elements.For WebShop, the observation similarity lookup uses four page patterns; WikiHow represents screens as HTML element sequences.Source: §4.1 · pp. 6–7

4.2 Results on WebShop

  • REMEMBERER outperforms ReAct, RL, IL, and IL+RL baselines on both reported metrics on WebShop.Table 1 reports WebShop results; ReAct uses the authors’ public implementation, while RL, IL, and IL+RL results are retrieved from Yao et al. [2022a].Source: Table 1 · p. 7, §4.2 · p. 8
  • REMEMBERER performs better and more stably than ReAct across different WebShop initializations and training sets.The experiments vary initial experience combinations and training sets.Source: Table 3 · p. 7, §4.2 · p. 8
  • REMEMBERER trains on 10 tasks for 74 steps, while RL and IL are expected to need about 100 thousand steps for acceptable performance.The comparison also reports that REMEMBERER requires few annotated samples to initialize its memory, unlike IL.Source: §4.2 · p. 8, Table 5 · p. 8, Table 4 · p. 8

4.3 Results on WikiHow

  • On WikiHow, REMEMBERER surpasses the fixed-exemplar baseline and prior result; annotated-memory evaluation demonstrates use of expert experiences.The annotated-memory evaluation is run without further training, and the authors note that annotated examples may not offset the particular LLM’s shortage.Source: Table 2 · p. 7, §4.3 · pp. 8–9

4.4 Ablation study

  • Without bootstrapping, reward-estimation error is far greater and WikiHow performance degrades, while WebShop shows no apparent final-performance disparity; random advice has minor superiority.The error compares memory-based average reward estimates with real training rewards; advice strategies are compared in Table 8.Source: §4.4 · pp. 9–10, Table 7 · p. 9, Table 8 · p. 9
  • Removing discouraged actions produces much poorer performance; these actions are especially useful when retrieved experiences offer no proper action to encourage..Source: Table 8 · p. 9, §4.4 · p. 10
  • Removing observation similarity causes serious degradation, whereas removing task similarity has no remarkable effect in these experiments.The authors suggest WikiHow’s pattern-based task similarity may be too coarse to distinguish different observations under similar instructions.Source: Table 8 · p. 9, §4.4 · p. 10

5 Conclusion

The conclusion presents RLEM and REMEMBERER as a way for LLM agents to learn from past interactions through persistent experience memory, reporting improvement over a baseline.

  • REMEMBERER combines an LLM with persistent experience memory that is updated using an RL algorithm.Source: §5 · p. 10
  • RLEM is introduced to help LLMs learn from interaction experiences in decision-making tasks.Source: §5 · p. 10
  • The experiments report significant improvement over the baseline and describe the results as demonstrating REMEMBERER’s superiority.Source: §5 · p. 10

6 Limitations

The limitations include open questions about applying REMEMBERER to more demanding environments and sustaining training gains; the appendix describes the observation formats used for WebShop and WikiHow.

6 Limitations

  • The tested benchmarks do not resolve how REMEMBERER applies to longer episodes or more extensive or visual-rich observations; training performance also saturates quickly.The authors suggest limited active exemplars may contribute to saturation and identify more complicated RL techniques as future work.Source: §6 · p. 10

A Details about the observation formats

B Lookup table of the pattern-based similarity functions

The appendix describes pattern categories and lookup tables for page similarity in WebShop and instruction similarity in WikiHow.

B.1 Lookup table of the page similarity function of WebShop

B.2 Lookup table of the instruction similarity function of WikiHow

  • WikiHow step instructions were categorized into six patterns, and a lookup table defines their similarity function.Source: §B.2 · pp. 14–15

C Hyper-parameters

The hyperparameters set undiscounted reward accumulation, an update rate inversely proportional to update count, and equal weighting of the two similarity components.

D Capability evolving of REMEMBERER

REMEMBERER’s performance improves during training, with success-rate saturation preceding average-reward saturation; the pattern differs between WebShop and WikiHow.

D Capability evolving of REMEMBERER

  • Performance on the 20-task subset improves across epochs, though it appears to saturate, possibly because active exemplars and training tasks are limited.Average reward saturates later than success rate, suggesting hard tasks can still gain rewards after success rate levels off.Source: D Capability evolving of REMEMBERER · p. 15, Figure 6 · p. 15
  • WebShop reaches saturation earlier than WikiHow, alongside fewer new experiences entering memory in later epochs.The authors leave whether this difference reflects the training set or task-domain characteristics for future investigation.Source: D Capability evolving of REMEMBERER · p. 16

E Q function fitting ability of REMEMBERER

The Q-function analysis finds that Double Q-Learning suppresses over-estimation but introduces under-estimation, while action ranking can remain useful without precise Q-value prediction. The section also provides WebShop and WikiHow exemplar examples.

E Q function fitting ability of REMEMBERER

  • The section reports that n-step bootstrapping improves the precision of learned memory Q-values.This is identified as a result of the main paper’s ablation study.Source: E Q function fitting ability of… · p. 16
  • Double Q-Learning suppresses over-estimation but introduces serious under-estimation, and its estimation error does not improve over plain Q-Learning in few-step circumstances.The authors attribute this to the additional convergence steps required by iteratively updating two Q-value lookups.Source: E Q function fitting ability of… · p. 16, Table 13 · p. 16
  • The LLM’s average absolute Q-value prediction error on WebShop is 0.417, yet it can still select appropriate actions by distinguishing valuable candidates.The authors state that relative relations among candidate-action values matter more than absolute precision during testing.Source: E Q function fitting ability of… · p. 17

G Case study

The case studies illustrate how discouraged actions and observation similarity affect experience-guided decisions. Omitting either can impair the agent’s ability to avoid repeated failures or retrieve appropriate experiences.

G Case study

  • Without discouraged actions in advice from an experience lacking encouraged actions, the LLM may repeat the same failure pattern.In the illustrated case, a random action is generated when no valuable action is available to encourage.Source: G Case study · p. 17, Figure 10 · p. 20
  • Omitting observation similarity makes retrieval depend only on the instruction, preventing selection from adapting to the particular observation and leading to poorer performance.Figure 11 illustrates the similarity-function ablation on WikiHow.Source: G Case study · p. 17, Figure 11 · p. 20
  • The examples include a WebShop page representation and a WikiHow exemplar; YAML markup in WebShop separates observation keywords from colon-ended page titles.The WebShop example includes a search for a specific deodorant, while the WikiHow example concerns hiding gauges.Source: Figure 8 · p. 18, G Case study · pp. 18–19, Figure 9 · p. 19
Loading 2306.07929v2…