PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-RL Induces Exploration in Language Agents

Yulun Jiang, Liangze Jiang, Damien Teney, Michael Moor, Maria Brbic

arXiv:2512.16848v2cs.LGcs.AI

TL;DR

LLM agents trained with standard RL often struggle with active exploration and efficient adaptation from trial-and-error feedback. LAMER uses cross-episode Meta-RL and in-context reflection to adapt policies, outperforming RL baselines and generalizing better to harder tasks.

  • Problem

    RL-trained LLM agents often struggle with active exploration and efficient adaptation from trial-and-error experiences in long-horizon tasks.

  • Method

    LAMER combines cross-episode Meta-RL, long-term reward optimization, and in-context policy adaptation through reflections from previous episodes.

  • Results

    LAMER consistently outperforms prompting and RL baselines across environments, with 11%, 14%, and 19% absolute gains over RL on Sokoban, MineSweeper, and Webshop.

  • Takeaways & Limitations

    Meta-RL induces exploration strategies that support rapid in-context adaptation and better generalization to harder or out-of-distribution tasks.

  • Takeaways & Limitations

    LAMER requires sequential episode rollouts, producing around twice the training time cost of RL in the reported implementation.

Abstract

from arXiv · show

Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.

1 INTRODUCTION

LLM agents often struggle to explore actively and adapt from feedback, motivating LAMER’s cross-episode Meta-RL framework for learning exploration strategies and improving downstream exploitation.

  • 1 INTRODUCTION: Existing approaches either target single-turn reasoning or rely on offline data, limiting their fit for active exploration in interactive environments.The introduction contrasts these methods with learning directly from environment interaction.
  • 1 INTRODUCTION: LAMER is designed to make agents actively explore, gather environment feedback, and exploit that experience more effectively.The approach targets multi-turn tasks with sparse success signals by treating episodes as units of exploration and exploitation.
  • 1 INTRODUCTION: Figure 1 reports that Meta-RL retains higher sample diversity while achieving better MineSweeper success rates than standard RL.Trajectory diversity is quantified using entropy of the empirical trajectory distribution.
  • 1 INTRODUCTION: LAMER trains agents across multiple episodes so early exploration and feedback inform later policy adaptation while optimizing long-term cross-episode rewards.The framework uses trial-and-error experiences and reflections to guide subsequent episodes.
  • 1 INTRODUCTION: 11%, 14%, and 19% absolute performance gains over RL are reported on Sokoban, MineSweeper, and Webshop, respectively.LAMER consistently outperforms prompting and RL baselines across the evaluated environments.

2 RELATED WORK

Related work spans prompting, reflection, RL, supervised trajectory training, and Meta-RL, with LAMER positioned as a way to learn exploration and test-time adaptation across tasks.

  • 2 RELATED WORK: LLM-agent research includes prompting methods such as ReAct and Reflexion, RL algorithms, and supervised training on interaction trajectories.Reflexion extends prompting to multi-episode reflection and memory.
  • 2 RELATED WORK: Prior exploration methods use offline trajectories or focus on non-agentic reasoning, whereas LAMER targets active environment interaction.The introduction identifies offline data as limiting direct adaptation from environment feedback.
  • 2 RELATED WORK: Meta-RL trains across task distributions so outer-loop learning improves an inner-loop adaptation strategy for rapidly solving new environments.The framework is described as learning to reinforcement learn through learned exploration strategies.
  • 2 RELATED WORK: LAMER relates Meta-RL to test-time compute by training multi-episode adaptation that balances exploration and exploitation for faster test-time adjustment.The paper reports matched training compute between RL and Meta-RL baselines when evaluating this framing.

3 PRELIMINARIES

The paper formulates LLM interaction as a discounted-return MDP and contrasts standard RL’s fixed policies with Meta-RL’s learned exploration strategies for rapid adaptation.

  • 3 PRELIMINARIES: An LLM agent acts in an MDP by mapping observed states to actions, receiving rewards, and transitioning through an environment over each trajectory.A trajectory contains states, actions, and rewards across an episode.
  • 3 PRELIMINARIES: Standard RL maximizes expected discounted return, but trained agents often learn fixed policies that struggle to explore and adapt at test time.The discount factor is denoted γ_step and lies in [0, 1].
  • 3 PRELIMINARIES: Meta-RL trains across task distributions so agents learn general exploration–exploitation strategies that support rapid adaptation to new tasks.The LLM parameters serve as the meta-parameters in this formulation.

4 LAMER: A META-RL FRAMEWORK FOR LLM AGENTS

LAMER trains LLM agents across sequential episodes, using cross-episode credit assignment and in-context self-reflection to explore early and exploit later. Its Meta-RL objective extends learning across episodes while adapting the policy from accumulated trajectories and reflections.

  • Cross-episode training framework: LAMER generates sequential episodes within each trial, updating later-episode policies from accumulated history and terminating when feedback indicates success or the episode budget is exhausted.This cross-episode structure supports trial-and-error learning from repeated attempts at the same initial task state.
  • Cross-episode training framework: LAMER’s cross-episode return assigns credit across episodes to enhance exploration and maximize long-term reward.The framework defines a discounted return across the trial rather than treating each episode independently.
  • Cross-episode training framework: Larger γtraj emphasizes long-horizon return and encourages more exploration early, whereas smaller γtraj favors rapid exploitation.γtraj controls the exploration–exploitation trade-off across episodes.
  • In-context policy adaptation with self-reflection: LAMER adapts its in-context policy after each episode by generating textual reflections that provide feedback and plans for the next episode.The policy is updated by modifying context rather than by gradient descent, using history and reflections stored in inter-episode memory.
  • Comparison to RL training: Unlike RL’s independent rollouts, Meta-RL conditions each episode on preceding rollouts and uses cross-episode credit assignment.LAMER can optimize the resulting objective with standard policy-gradient methods, including PPO, GRPO, and GiGPO.

5 EXPERIMENTS

LAMER is evaluated across diverse long-horizon environments, with experiments testing overall performance, test-time scaling, exploration behavior, and generalization to harder and unseen tasks.

  • Overall performance: LAMER consistently outperforms prompting and RL baselines across Sokoban, MineSweeper, and Webshop on final pass@3 success rates.It achieves 55.9% on Sokoban, 74.4% on MineSweeper, and a 14% advantage over RL-trained methods on Webshop.
  • Test-time scaling: LAMER gains 13.5% from pass@1 to pass@3 on Sokoban, exceeding the less-than-5% gains of RL-trained and prompting baselines.Although it starts slightly below GiGPO on MineSweeper and Webshop, it surpasses all baselines by pass@2 and pass@3.
  • Exploration behavior: LAMER preserves more trajectory diversity than RL-trained agents, supporting greater test-time exploration while balancing exploration and exploitation.RL reduces diversity toward deterministic behavior, whereas the base model has the highest entropy without correspondingly higher success.
  • Generalization to harder tasks: On harder Sokoban and MineSweeper settings, Meta-RL remains ahead of RL, with 10% and 5% performance gaps at the most difficult levels.Difficulty increases through more boxes in Sokoban and more mines in MineSweeper.
  • Generalization to unseen tasks: The evaluation also tests out-of-distribution generalization in ALFWorld by training on four task categories and evaluating on two unseen categories.The unseen categories are Cool and Pick2, while Pick, Look, Clean, and Heat are used as in-distribution tasks.

6 ANALYSIS

The analysis examines how trajectory discounting, inter-episode memory, and training-budget differences affect LAMER. Results show environment-dependent exploration–exploitation trade-offs, benefits from reflection, and higher training-time cost despite matched trajectory usage.

  • 6.1 INFLUENCE OF TRAJECTORY DISCOUNT FACTOR: γtraj controls reward propagation across a trial and therefore provides a practical way to balance exploration and exploitation during training.Higher γtraj encourages more exploration, but larger values do not necessarily improve final pass@3 performance.
  • 6.1 INFLUENCE OF TRAJECTORY DISCOUNT FACTOR: Intermediate γtraj values such as 0.6 perform best on Sokoban and Webshop, whereas MineSweeper benefits from a larger value such as 0.9.The optimal setting varies by environment rather than increasing monotonically with exploration.
  • 6.2 ABLATION ON THE INTER-EPISODE MEMORY: 21.6% improvement on Sokoban, 11.0% on MineSweeper, and 3.5% on Webshop show that self-reflection benefits LAMER across environments.Reflection-only memory also outperforms the default trajectory-and-reflection configuration across all environments.
  • 6.3 TRAINING BUDGET: LAMER and standard RL consume the same total number of trajectories under the matched-budget comparison.The standard RL group size is set three times larger so both methods use the same number of trajectories per gradient update.
  • 6.3 TRAINING BUDGET: LAMER takes around twice the training time of RL baselines because episodes within a trial must be generated sequentially.The authors suggest asynchronous rollout as a possible efficiency improvement.

7 CONCLUSION

The conclusion presents LAMER as a Meta-RL framework that uses cross-episode returns and training-time exploration to support in-context adaptation. Across diverse environments, it outperforms RL methods, generalizes to harder environments, and scales with more test-time episodes.

  • 7 CONCLUSION: LAMER maximizes discounted cross-episode returns rather than single-episode returns, balancing exploration and exploitation for long-term performance.Training-time exploration teaches strategies that support rapid in-context adaptation at test time.
  • 7 CONCLUSION: Across diverse environments, LAMER outperforms RL methods, generalizes to harder environments, and scales better with more test-time episodes.These are the paper’s reported conclusion-level outcomes.
  • 7 CONCLUSION: LAMER trained on easier environments can generalize to harder environments of the same kind or relatively similar domains.The authors identify adaptation to completely novel environments as a direction for future work.

A TASK DESCRIPTION AND DETAILS

The appendix describes four interactive environments and the prompts used to operate and reflect within them. The tasks span fully and partially observable settings, including games, shopping, and household activities.

  • Sokoban: Sokoban is a fully observable box-pushing game where the agent must place every box on a target without trapping boxes.Actions are up, down, left, and right; boxes cannot be pulled.
  • MineSweeper: MineSweeper is partially observable: the agent reveals safe cells using neighboring mine counts while avoiding hidden mines.A revealed mine ends the task in failure, and numbered cells indicate adjacent mine counts.
  • Webshop: Webshop is a partially observable text-based shopping environment requiring product search, navigation, and attribute matching.The instruction specifies a product and attributes such as price, color, size, and reviews.
  • ALFWorld: ALFWorld is a partially observable interactive-fiction environment involving navigation and multi-step household tasks.Agents may need to find, pick up, clean, heat, or combine objects.
  • Prompt Design: The prompts separate standard action generation from reflection on past experience, with history, reflections, and task-specific variables inserted at runtime.The first episode has no past-experience reflection, and actions are marked with <action> tags while reflections use <remark> tags.
  • Sokoban Prompt Details: Sokoban reflection prompts require identifying mistakes, devising an improved plan from the original state, and then selecting admissible actions.The action prompt emphasizes planning around box positions, targets, walls, and traps.

C TRAINING DETAILS

The training setup uses GiGPO by default and matches sampled experience between RL and Meta-RL for fair comparison. Meta-RL samples three episodes per example with a group size of eight.

  • C TRAINING DETAILS: LAMER is compatible with standard policy-gradient algorithms and uses GiGPO as its default optimization algorithm.The training setup matches total sampled experiences between RL and Meta-RL.
  • C TRAINING DETAILS: Meta-RL samples N = 3 episodes per example and uses group size 8, while RL uses group size 24 for experience matching.The larger RL group size preserves equal total experience usage for comparison.

D.1 EXPERIMENTS ON DIFFERENT ARCHITECTURE

LAMER remains effective when evaluated with Llama3.1-8B-Instruct, outperforming RL baselines across Sokoban and MineSweeper. Inter-episode memory improves some RL baselines but does not uniformly improve performance.

  • D.1 EXPERIMENTS ON DIFFERENT ARCHITECTURE: Table 4 reports success rates under one, two, and three attempts for Sokoban and MineSweeper using Llama3.1-8B-Instruct.
  • D.1 EXPERIMENTS ON DIFFERENT ARCHITECTURE: LAMER outperforms RL baselines across Sokoban and MineSweeper with Llama3.1-8B-Instruct, supporting applicability across model architectures and sizes.The comparison is against GiGPO, identified as the strongest baseline in the supplied passage.

E EXAMPLES

The MineSweeper example shows LAMER agent trajectories as sequences of board clicks paired with reflections. Because mines are hidden and clicking one causes failure, the trajectories illustrate interaction under uncertainty.

  • E EXAMPLES: MineSweeper trajectories are represented as sequences of numbered-cell clicks, paired with corresponding agent reflections.
  • E EXAMPLES: Hidden mines create a failure risk: clicking a mine ends the game.
Loading 2512.16848v2…