PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization

Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, Peter Clark

arXiv:2310.10134v1cs.CLcs.AIcs.LG

TL;DR

Efficient learning from trial-and-error remains challenging for language agents, while prior reflection methods help agents retry tasks. CLIN uses persistent memory of causal abstractions, continually improving on repeated trials and transferring learning to new tasks and environments.

  • Problem

    Efficient learning from trial-and-error remains challenging for language agents because reinforcement learning requires extensive training samples and expensive model fine-tuning.

    Source: §1 · p. 1

  • Method

    CLIN maintains persistent memory of causal abstractions, using a frozen LLM to create or update insights after each trial.

    Source: §3.2 · p. 5, §5 · p. 10

  • Results

    CLIN continually improves on repeated trials in the same task and environment, outperforming reflective agents like Reflexion by 23 points, and transfers learning to new tasks and environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CLIN presents a nonparametric learning paradigm that may give frozen language agents never-ending learning abilities.

    Source: §5 · p. 10

    CLIN’s learning depends on past experience; unobserved locations or actions cannot yield insights, making exploration important.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Language agents have shown some ability to interact with an external environment, e.g., a virtual world such as ScienceWorld, to perform complex tasks, e.g., growing a plant, without the startup costs of reinforcement learning. However, despite their zero-shot capabilities, these agents to date do not continually improve over time beyond performance refinement on a specific task. Here we present CLIN, the first language-based agent to achieve this, so that it continually improves over multiple trials, including when both the environment and task are varied, and without requiring parameter updates. Our approach is to use a persistent, dynamic, textual memory centered on causal abstractions (rather than general "helpful hints") that is regularly updated after each trial so that the agent gradually learns useful knowledge for new trials. In the ScienceWorld benchmark, CLIN is able to continually improve on repeated trials on the same task and environment, outperforming state-of-the-art reflective language agents like Reflexion by 23 absolute points. CLIN can also transfer its learning to new environments (or new tasks), improving its zero-shot performance by 4 points (13 for new tasks) and can further improve performance there through continual memory updates, enhancing performance by an additional 17 points (7 for new tasks). This suggests a new architecture for agents built on frozen models that can still continually and rapidly improve over time.

1 INTRODUCTION

Language agents can act in simulated environments, but task-specific reflections may transfer poorly. CLIN instead maintains causal abstractions in dynamic memory and improves across repeated trials and varied tasks and environments.

1 INTRODUCTION

  • Existing language agents can interact with environments, but reinforcement learning is costly and specific reflective advice may not generalize to a different task or environment.Source: §1 · p. 1
  • CLIN stores causal abstractions about action effects in a continually updated memory, allowing useful knowledge to persist and unhelpful knowledge to be dropped across tasks and environments.Source: §1 · p. 2
  • On repeated same-task trials, CLIN outperforms Reflexion by 23 points; transfer adds 4 points for new environments (13 for new tasks), with continual updates adding 17 (7 for new tasks).Source: §1 · p. 2
  • The paper presents CLIN as a nonparametric learning architecture that learns faster than prior systems and generalizes better to new tasks and environments.Source: §1 · p. 2
  • The authors conclude that dynamically maintained causal memory is a promising approach for frozen-model agents to improve continually over time.Source: §1 · p. 2

2 RELATED WORK

Prior work spans trial-intensive reinforcement learning, prompted language agents that do not adapt from experience, and frozen-model systems that learn from task-specific reflections or other memory.

2 RELATED WORK

  • Traditional RL agents learn tasks over repeated trials but typically need many trials and adapt poorly to unexpected test-environment changes.Examples include DRRN, KG-A2C, and CALM.Source: §2 · p. 2
  • Prompted LLM agents such as SayCan, ReAct, and SwiftSage perform reasonably with little training but do not adapt from experience.Source: §2 · p. 3
  • Voyager grows a code-based skill library from failures, while Reflexion (Shinn et al., 2023) uses task- and environment-specific reflections to retry; CLIN instead maintains persistent memory across trials and settings.Source: §2 · p. 3
  • Prior work uses learned memory to improve frozen LLMs in QA (Tandon et al., 2022) and social-behavior modeling (Park et al., 2023), while action-model learning is formalized in planning.CLIN applies the action-model idea as linguistic causal abstractions for LLM agents.Source: §2 · p. 3

3 APPROACH

CLIN studies how an agent can update beliefs through interaction and use past experience to solve unseen task variations. Its approach combines a persistent memory of causal abstractions with trial-by-trial and cross-episode updates.

3 APPROACH

  • The setup treats tasks as POMDPs across environment configurations; agents retain memory across repeated trials and episodes while acting from observations and rewards.A trial contains action-observation-reward steps, and the environment resets when a task ends.Source: §3 · p. 3

3.1 CLIN: A GENERATIVE AGENT CONTINUALLY LEARNING FROM INTERACTIONS

  • CLIN retains a dynamic collection of natural-language causal abstractions from experience, including actions that failed to contribute, to guide future decisions.The memory can reduce action search by preserving insights such as opening a fridge to access apple juice.Source: §3.1 · pp. 3–4
  • The controller uses the task, retrieved memories, and trial history to propose goals; the executor turns goals into valid actions, and the simulator returns observations and rewards.CLIN uses no gold data to identify goals or memories.Source: §3.1 · p. 4

3.2 MEMORY: A COLLECTION OF ADAPTIVE CAUSAL ABSTRACTIONS

  • The memory generator encodes helpful transitions as “X is NECESSARY to Y” and unsuitable ones as “X DOES NOT CONTRIBUTE to Y,” with wording that marks uncertainty.“X may ...” signals moderate-to-high uncertainty, while “X should ...” signals low uncertainty; uncertainty can change with experience and fitness to the task.Source: §3.2 · p. 5
  • After each trial, CLIN updates memory using the latest trial, reward, and memories from the three most recent trials, then prunes insights by saliency and success.The generated memory is observed to contain fewer abstractions than the trial contains actions.Source: §3.2 · pp. 5–6

3.3 META-MEMORY FOR BETTER GENERALIZATION

  • For generalization, CLIN selects memories from the most successful trial in each of up to ten archived episodes and combines them into task- or environment-specific meta-memory.The generator uses target-task descriptions and associated trial rewards to combine insights and assign uncertainty for unseen tasks or environments.Source: §3.3 · p. 6

4 RESULTS AND ANALYSIS

In ScienceWorld, CLIN is evaluated on repeated-trial adaptation and transfer across unseen environments and related new tasks. Results show gains from persistent memory, while discussion identifies memory-structure, controller, exploration, and retrieval factors that qualify its performance.

4.1 CLIN EXHIBITS RAPID TASK ADAPTATION

  • CLIN improves across repeated trials, with stronger relative gains than Reflexion, especially on longer tasks.It can also become more efficient in later trials by using fewer steps; Figure 4 shows example trends and comparisons.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 CLIN OUTPERFORMS SOTA, GENERALIZING TO NOVEL ENVIRONMENTS AND TASKS

  • In generalization across unseen environments, CLIN shows a positive GEN-ENV effect, averaging a 4-point gain over BASE; in G+A, it improves by 16 additional points and beats the SOTA reflective agent by 23.Source: §4.2 · p. 8, Table 1 · p. 8
  • 13-point improvement over BASE in GEN-TASK; CLIN scores higher 38.8% of the time and solves transferred tasks in fewer steps.The transfer setup uses related tasks that share task-critical objects or locations.Source: §4.2 · p. 8

4.3 DISCUSSION

  • Unstructured free-form memory lowers average reward by 6 points in 10% of cases compared with CLIN, supporting the usefulness of causal abstractions.The ablation removes constraints on the format of generated advice.Source: §4.3 · pp. 8–9
  • A controller ablation drops BASE performance in 44% of cases and average reward by 18 points, making CLIN equivalent to ReAct.The controller generates a goal before the next action; Figure 5d shows the ablation.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Across environments, CLIN refines specific trial insights into meta-memory abstractions, such as generalizing from a lighter and stove to heat sources for containers.In the example, this lets CLIN use a lighter in an unseen environment with a broken stove.Source: §4.3 · p. 9
  • CLIN cannot generate insights about actions or locations it has never explored, limiting performance when tasks depend on that unobserved knowledge.The paper’s examples include an unseen art studio and unsuccessful temperature measurement in boil or freeze tasks.Source: §4.3 · p. 9
  • Poor retrieval can repeatedly favor an unsuitable insight, and varied initial conditions make this problem worse during early generalization.The paper leaves improved memory representation as future work.Source: §4.3 · pp. 9–10

5 CONCLUSION

The conclusion presents CLIN as a nonparametric language-agent architecture that continually refines causal-abstraction memory to support adaptation and generalization.

5 CONCLUSION

  • CLIN builds and refines a persistent memory of causal abstractions to improve performance on future tasks.Source: §5 · p. 10
  • The architecture targets continual improvement through retries on a task and generalization to novel tasks and environments.Source: §5 · p. 10
  • The paper evaluates a nonparametric learning paradigm for frozen language agents.Source: §5 · p. 10

A CLIN PROMPTS

Figure 6 provides the complete prompts for CLIN’s controller and executor, which together generate the next action.

A CLIN PROMPTS

B EXAMPLE MEMORIES

The appendices show example generated memories and memory-generator prompts for CLIN’s ADAPT, GEN-ENV, and GEN-TASK setups, alongside a table captioned as adaptation results.

B EXAMPLE MEMORIES

C MORE RESULTS

The additional-results section specifies the ScienceWorld evaluation’s task and variant coverage, defines its short- and long-task split, and maps abbreviated task names.

C MORE RESULTS

  • The ScienceWorld evaluation excludes electricity tasks and uses the first 10 test variants for each of 18 selected tasks.The selected tasks and the number of variants used for each are listed in the passage.Source: C MORE RESULTS · p. 20
  • Short tasks have oracle lengths below 37 steps, while long tasks have oracle lengths of at least 37 steps.Source: C MORE RESULTS · p. 20
  • The section provides abbreviated names for task groups, including Temp, Pick&Place, Chemistry, and Lifespan.Source: C MORE RESULTS · p. 20
  • The task-name map also lists Biology, Boil, Freeze, Grow Plant, Grow Fruit, Force, Friction, and Genetics.Source: C MORE RESULTS · p. 20

Related papers

Loading 2310.10134v1…