Source-linked AI summary
Contextual Experience Replay for Self-Improvement of Language Agents
Yitao Liu, Chenglei Si, Karthik Narasimhan, Shunyu Yao
TL;DR
Language agents often lack environment-specific experience and do not continually learn from past tasks, limiting their preparation for complex web navigation. CER distills and replays past experiences through a dynamic memory buffer, and achieves competitive results on WebArena and VisualWebArena.
Problem
Language agents often lack environment-specific knowledge and efficient continual learning, so they may explore complex web environments from scratch for each task.
Source: §1 · p. 1Method
CER distills environment dynamics and decision-making patterns from trajectories, stores them in a dynamic buffer, and retrieves and replays relevant experiences in context.
Source: §1 · p. 2Results
CER achieves competitive results on WebArena and VisualWebArena, including a 36.7% WebArena success rate and 51.0% relative improvement over the GPT-4o baseline.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that learning from past experiences supports language agents’ adaptation to different environments and autonomous evolution.
Source: §6 · p. 9CER requires goal-oriented trajectories for high-quality experiences, performs worse with random-exploration trajectories, and still has difficulty avoiding dependence on human annotators.
Source: §7 · pp. 9–10
Abstract
from arXiv · showhide
Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks. Moreover, current LLM agents are not designed to continually learn from past experiences during inference time, which could be crucial for them to gain these environment-specific experiences. To address this, we propose Contextual Experience Replay (CER), a training-free framework to enable efficient self-improvement for language agents in their context window. Specifically, CER accumulates and synthesizes past experiences into a dynamic memory buffer. These experiences encompass environment dynamics and common decision-making patterns, allowing the agents to retrieve and augment themselves with relevant knowledge in new tasks, enhancing their adaptability in complex environments. We evaluate CER on the challenging WebArena and VisualWebArena benchmarks. On VisualWebArena, CER achieves a competitive performance of 31.9%. On WebArena, CER also gets a competitive average success rate of 36.7%, relatively improving the success rate of the GPT-4o agent baseline by 51.0%. We also conduct a comprehensive analysis on it to prove its efficiency, validity and understand it better.
1 Introduction
CER targets the lack of environment-specific experience and continual learning in language agents by distilling and replaying past-task knowledge. It reports competitive results on WebArena and VisualWebArena and analyzes its self-improvement behavior.
1 Introduction
- Language agents often lack environment-specific knowledge and efficient continual learning, leading them to explore complex web environments from scratch for each task.The introduction notes that frontier methods achieve around or below 20% success without human involvement on WebArena and VisualWebArena.Source: §1 · p. 1
- WebArena results show a 36.7% overall success rate and a 51.0% relative improvement over the GPT-4o baseline; VisualWebArena success is 31.9%.On VisualWebArena, CER also outperforms the tree search-based method by 20.8% in relative performance with dozens of times fewer token costs.Source: §1 · p. 2
- CER is presented as a self-improvement system with analyses of cross-template success, stability, and plasticity, and compatibility with sampling-based methods.The paper also characterizes CER as showing stability and plasticity and synergy with other off-the-shelf methods.Source: §1 · p. 2
- CER distills fine-grained skills and environment dynamics from successful and failed trajectories, and supports offline, online, and hybrid settings.The agent retrieves relevant experiences from a dynamic buffer and replays them in context when solving new tasks.Source: §1 · p. 2
- The authors report a comprehensive analysis intended to validate CER and clarify its improvements.Source: §1 · p. 2
2 Related work
Related work covers LLM agents, web-agent environments, and systems that reuse memories or past trajectories. CER is positioned as a self-improvement framework for challenging web environments that distills both dynamics and decision-making patterns.
LLM Agents
- LLM agents use methods such as ReAct to interleave action and reasoning, while planning and search methods support revision and deliberate action selection.Source: §2 · p. 3
Web Agent Environments
- Web-agent benchmarks range from e-commerce tasks to realistic navigation and multimodal tasks; WebArena and VisualWebArena provide interactive environments for self-improvement research.Source: §2 · p. 3
Learning from Memory or Past Experiences
- Prior systems store and retrieve agent memories or trajectories, but some use raw observation-action pairs, simpler environments, or require ground-truth rewards.CER instead distills environment dynamics and decision-making patterns and is evaluated across different learning paradigms and agent methods.Source: §2 · p. 3
3 CER: Contextual Experience Replay
CER uses separate modules to distill and retrieve experiences, then integrates selected knowledge into the agent’s context. It supports offline, online, and combined learning from trajectories.
3 CER: Contextual Experience Replay
- CER comprises a distillation module, a retrieval module, a dynamic experience buffer, and a base decision-making agent, implemented by prompting a visual language model.Source: §3 · p. 3
3.1 Distill experiences from trajectories
- Distillation extracts environment dynamics and useful decision-making patterns, or skills, from trajectories and generalizes them into reusable experiences.Separate modules distill dynamics and skills; skills include high-level summaries and step-by-step guidelines, and existing buffer experiences help avoid repetitive distillation.Source: §3.1 · pp. 3–4
3.2 Retrieve experiences from buffer
- Retrieval modules use the current task goal and website descriptions to select the top-k useful dynamics and skills from the buffer.Source: §3.2 · pp. 4–5
3.3 Decision-making with Contextual experience replay
- Selected experiences are mapped into natural-language descriptions and integrated into the model context to inform the agent’s actions.Figure 2 contrasts the ReAct baseline with ReAct plus CER, where dynamics and skills are replayed in context.Source: Figure 2 · p. 4, §3.3 · p. 5
3.4 Combination of offline and online learning
- Online CER distills each completed task’s trajectory for retrieval on the next task; offline CER learns from a fixed trajectory set, and the two can be combined.The combined setting learns from a fixed training set before self-evolving with self-generated data in the environment.Source: §3.4 · p. 5
4 Experiments
The experiments evaluate CER on WebArena and VisualWebArena, using offline, online, and hybrid settings on WebArena and an online setting on VisualWebArena. Results compare CER with a reproduced GPT-4o BrowserGym baseline and other methods.
4 Experiments
- WebArena includes 812 tasks across five websites; CER is evaluated offline, online, and in hybrid settings there, but only online on VisualWebArena for cost consideration.VisualWebArena has 910 tasks, with many requiring visual input as part of the task goal.Source: §4 · p. 5
4.1.1 WebArena
- For WebArena, offline CER uses human-annotated, goal-oriented training trajectories filtered to avoid overlap with test tasks, with GPT-4o and ReAct as the agent setup.The retrieval limits are five dynamics and five skills; the study also evaluates online and hybrid settings.Source: §4.1.1 · pp. 5–6
4.1.2 VisualWebArena
- For VisualWebArena, the authors implemented a visual environment on BrowserGym and used Set-of-Marks screenshots alongside text observations to ground interactable elements.Table 2 reports success rates and marks the best result in each domain; CERonline is the only CER setting evaluated there.Source: §4.1.2 · p. 6, Table 2 · p. 6
4.2 Results
- 51.0% relative improvement over the GPT-4o BrowserGym baseline for CERhybrid on WebArena and 21.8% for CERonline on VisualWebArena.The comparisons are summarized in Tables 1 and 2; VisualWebArena results also compare CER with a GPT-4o tree-search method at much lower token cost.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
5 Analysis
CER’s analysis examines its token efficiency, cross-template generalization, stability and plasticity, performance on a weaker model, compatibility with sampling, robustness to noisy trajectories, and the roles of dynamics and skills.
5.1 Token efficiency
- CER improves performance by 52.5% with only minor additional token costs, and the analysis counts input tokens because output tokens change little.The comparison covers the ReAct baseline and CER in three settings.Source: §5.1 · p. 7, Table 3 · p. 8
5.2 Investigating improvements of CER
- CER’s cross-template success-rate improvement suggests it generalizes through fine-grained experiences rather than memorizing whole task trajectories.The measure counts templates with at least one solved task; goal-oriented experiences highlight promising states, actions, and decision patterns, though some can be noisy or misleading.Source: §5.2 · p. 7
5.3 Stability and Plasticity
- With most original abilities retained, CER solves 41% of new problem types on the Forum split, supporting its stability and plasticity as a self-improvement system.Stability measures preservation of baseline-solved tasks; plasticity measures additional tasks solved compared with the baseline.Source: §5.3 · pp. 7–8, Table 4 · p. 8
5.4 Validity on open-source models
- On WebArena’s GitLab split, CER improves Llama-3.1-70B’s baseline success rate by 26.53%, but less than it does with stronger models.The authors attribute the smaller improvement partly to weaker output formatting on challenging tasks with larger action spaces, which affects robust distillation of useful skills.Source: §5.4 · p. 8, Table 5 · p. 8
5.5 Synergy with performant methods
- With trajectory sampling and reranking, CER improves CER w/ ReAct performance by a relative success-rate increase of 39.5%.The authors suggest that higher-quality experiences from more precise methods can support more robust decisions with environment-specific knowledge.Source: §5.5 · p. 8, Table 6 · p. 9
5.6 Access to ground truth rewards
- CER performs better with ground-truth rewards, while its significant baseline improvements and acceptable gap indicate robustness to noisy trajectories.The authors suggest successful trajectories are more informative, while failed ones may provide misleading action sequences.Source: §5.6 · pp. 8–9, Table 7 · p. 9
5.7 Division of Dynamics and skills
- Both environment dynamics and skills matter: dynamics provide page information and URLs, while skills offer promising actions and decision heuristics.The authors describe the two experience types as providing complementary state and action heuristics.Source: §5.7 · p. 9, Table 8 · p. 9
6 Conclusions
The paper presents CER as a training-free framework for language-agent self-improvement in complex web environments, using past experiences during inference and evaluating improvements through stability and plasticity.
6 Conclusions
- CER is a training-free framework for efficient and effective language-agent self-improvement in complex web environments, learning from past experiences and replaying them during inference.The paper also analyzes its improvements and evaluates its effectiveness as a self-improvement system through stability and plasticity.Source: §6 · p. 9
7 Limitations
CER’s applicability depends on having goal-oriented trajectories for high-quality experience distillation, while its learning setup can require trajectory data before tasks begin. Offline data may depend on human annotation, and online learning may not inform the initial decision.
7 Limitations
- CER requires goal-oriented trajectories to distill high-quality experiences; performance is limited when training data comes from random exploration.The authors identify more fine-grained use of low-quality trajectories as future work.Source: §7 · p. 9
- Online CER may lack learned trajectory experience for the initial decision, while offline learning depends on pre-collected, human-annotated trajectories.The authors say hybrid learning can hardly avoid dependence on human annotators.Source: §7 · pp. 9–10
- More efficient ways to generate trajectories are an open research direction for reducing dependence on human annotation.Source: §7 · pp. 9–10
8 Ethical and Broader Impacts
The paper discusses potential accessibility benefits and automation impacts of autonomous agents, the need to assess bias and safety, and the research-only scope of the presented prototypes.
8.1 Real World Impacts
- Autonomous agents could improve access to computer-based tasks for people with disabilities or limited technical skills and automate routine computer work, with possible employment implications if deployed.Source: §8.1 · p. 10
8.2 Bias and Safety
- Deployment requires further analysis of unintended bias and safeguards against harmful behavior, which agents could cause more than regular LLMs.Source: §8.2 · p. 10
8.3 Intended Uses
- CER is evaluated on research benchmarks, and its models and methods are research prototypes not intended for practical deployment in their current state, especially in high-risk domains.Source: §8.3 · p. 10
A.1 CER prompts
The appendix identifies separate system prompts for CER’s distillation and retrieval modules.
A.1 CER prompts
- Figures 3–4 show prompts for distillation, while Figures 5–6 show prompts for retrieval.Source: §A.1 · p. 13
A.2 Experience examples
The appendix provides experience examples and task materials, then compares human demonstrations with self-guided explorations as offline data sources. Both improve offline performance over baseline, but only human-annotation-based offline-plus-online learning is reported to surpass online-only learning.
A.2 Experience examples
- The appendix presents natural-language experience examples for WebArena and VisualWebArena, transformed from structured buffer entries for addition to the model’s system prompt.Source: §A.2 · p. 13
A.3 Self-guided exploration prompt
- The self-guided exploration prompt samples 10 trajectories and limits each to 30 steps while prompting a language model to propose diverse actions.Source: §A.3 · p. 13, Figure 9 · p. 20
A.4 Human annotated tasks for WEBARENA
- WebArena offline training tasks were designed by people unfamiliar with the benchmark, with five to ten tasks per website, about 5% of the test set.Source: §A.4 · p. 13
A.4.1 Shopping Admin / A.4.2 Shopping
- Shopping and Shopping Admin examples cover information lookup and account or order changes, alongside product, review, and order questions.Source: §A.4.1 · p. 13, §A.4.2 · p. 14
A.4.3 Gitlab
- GitLab examples include repository and issue queries, project creation, issue assignment, and direct typing during filtering.Source: §A.4.3 · p. 14
A.4.4 Map / A.4.5 Forum
- Map and Forum examples ask for route and place information or actions and queries involving posts, comments, and user profiles.Source: §A.4.4 · p. 14, §A.4.5 · p. 14
A.5 Offline data sources
- Both offline data sources improve over baseline, and five human-annotated training tasks let offline-plus-online learning surpass online-only learning on Forum tasks.The comparison uses the WebArena Forum split; a task example describes retrieved forum-page dynamics and sorting skills supporting successful task completion.Source: §A.5 · p. 15, Table 9 · p. 15
- Self-guided exploration data makes offline-plus-online performance worse than online-only performance because its trajectories are noisy and unstructured and may mislead the agent.Source: §A.5 · p. 15