PaperlayerView exact revision on arXiv

Source-linked AI summary

Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

Xucong Wang, Ziyu Ma, Shidong Yang, Tongwen Huang, Pengkun Wang, Yong Wang, Xiangxiang Chu

arXiv:2606.10917v2cs.AI

TL;DR

LLM agents face limited interaction feedback and static training environments, motivating Role-Agent’s single-LLM bootstrapped co-evolution of agent and environment. Across benchmarks, it consistently improves performance, including average success-rate gains of 78.0% on ALFWorld and 59.1% on WebShop over prompt-based approaches.

  • Problem

    Existing agents typically evolve against fixed environments that do not expose hidden weaknesses or provide feedback targeted to current failure modes.

  • Method

    Role-Agent uses one LLM as both agent and environment through WIA predictive rewards and AIW failure-mode analysis with targeted task retrieval.

  • Results

    78.0% on ALFWorld and 59.1% on WebShop in success rate are Role-Agent’s average gains over prompt-based approaches.

  • Takeaways & Limitations

    The experiments demonstrate practical gains from using a single LLM to support bootstrapped agent-environment co-evolution in text-based interactive environments.

  • Takeaways & Limitations

    Evaluation is confined to text-based environments, so extending Role-Agent to multimodal or real-time embodied settings remains future work.

Abstract

from arXiv · show

Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization. To address these limitations, this paper introduces Role-Agent, \textcolor{black}{a framework} that harnesses a single LLM to function concurrently as both the agent and the environment, enabling a bootstrapped co-evolution. Role-Agent comprises two synergistic components: World-In-Agent (WIA) and Agent-In-World (AIW). In WIA, the LLM acts as the agent and predicts future states after each action; the alignment between predicted and actual states is then used as a process reward, encouraging environment-aware reasoning. In AIW, the LLM analyzes failure modes from failed trajectories and retrieves tasks with similar failure patterns, thereby reshaping the training data distribution for targeted practice. Experiments on multiple benchmarks show that Role-Agent consistently improves performance, yielding an average gain of over 4\% over strong baselines.

1 Introduction

Role-Agent addresses the limitations of fixed environments by enabling bootstrapped agent-environment co-evolution with a single LLM serving dual roles. Its World-In-Agent and Agent-In-World modules respectively improve environment-aware decision-making and adapt training toward the agent’s failure modes.

  • Motivation: Existing self-evolving agents typically improve only the agent while treating the environment as a fixed source of tasks, observations, and rewards.Such environments may fail to expose hidden weaknesses or provide feedback targeted to current failure modes.
  • Motivation: Adaptive synthetic environments could diagnose deficiencies and present targeted challenges, but often require additional environment models, task generators, or scheduling mechanisms.
  • Role-Agent: Role-Agent uses a single LLM as a dual-role entity to enable bootstrapped agent-environment co-evolution without human supervision.The framework differs from agent-side self-improvement and state-grouped RL methods.
  • World-In-Agent: World-In-Agent predicts future observations after actions and uses prediction gaps against actual states as reward and credit-assignment signals.This incorporates environment priors into rollouts and encourages reliable decisions when action consequences are uncertain.
  • Agent-In-World: Agent-In-World analyzes failed trajectories to identify failure modes, retrieves similar tasks, and redistributes training toward historical deficiencies.The module prioritizes difficult and easily overlooked tasks through adaptive data distribution.
  • Results: Extensive experiments show that RoleAgent achieves substantial improvements over strong baselines across diverse benchmarks and practical gains in text-based interactive environments.

2 Related Work

Prior work has developed LLM agents through tool use, reflection, memory, and related methods for autonomous interaction. Self-evolving agents instead emphasize capability iteration in dynamically changing environments through experience distillation and automated agent-design exploration.

  • Large Language Model (LLM) Agents: LLM agents have been enhanced with tool use, reflection, and memory to transform language-model backbones into autonomous, interactive systems.The passage situates these developments across a wide range of application domains.
  • Self-Evolving Agents: Self-evolving agents pursue autonomous capability iteration within dynamically evolving open environments rather than fixed data distributions and tasks.This line of work includes EvolveR’s self-contained lifecycle for distilling experiences into principles and evolving the agent policy.
  • Self-Evolving Agents: Other self-evolving-agent research explores agent design automatically, complementing approaches centered on experience-based policy evolution.The passage identifies automated design exploration as a distinct focus alongside EvolveR’s experience distillation.

3 Methodology

Role-Agent enables bootstrapped agent-environment co-evolution by using one LLM as both an action-producing agent and an adaptive environment. Its WIA component rewards accurate future-state prediction, while AIW analyzes failures and retrieves similarly challenging tasks.

  • Role-Agent Overview: Role-Agent integrates World-In-Agent (WIA) and Agent-In-World (AIW) to achieve bootstrapped agent-environment co-evolution.A single LLM switches between agent and environment roles in a closed-loop process.
  • World-In-Agent (WIA): In WIA, the agent predicts states up to H steps after each action, and Longest Matching Subsequence alignment with ground-truth states produces predictive rewards.Each predictive reward lies in [0, 1] and quantifies foresight h steps ahead.
  • World-In-Agent (WIA): Predictive rewards multiply task rewards, preserving credit for accurately predicted actions while preventing plausible predictions from independently rewarding failed trajectories.This modulation weakens advantage signals when predictions are inaccurate and applies only to actions with non-zero task reward.
  • World-In-Agent (WIA): WIA groups actions occurring under identical states and computes state-level advantages to reduce within-trajectory redundancy and improve reward attribution.The resulting state-level advantages revise trajectory-level GRPO policy optimization.
  • Agent-In-World (AIW): AIW analyzes failed trajectories to identify failure patterns, stores their reflections, and retrieves tasks sharing similar failure modes for targeted data-distribution adjustment.On ALFWorld, the retrieval library contains 11 unique failure modes across training, with negligible storage or retrieval costs.

4 Experiments

Role-Agent is evaluated across ALFWorld, WebShop, and search-augmented QA against prompt-based, RL-based, closed-source, and search-based baselines. It consistently improves performance, with gains especially evident on complex multi-step tasks, while ablations and sensitivity analyses support both co-evolution components and default settings.

  • Benchmarks and Implementation: Experiments cover household decision-making in ALFWorld, simulated e-commerce interaction in WebShop, and search-augmented question answering.The backbone models are Qwen2.5-1.5/3B/7B-Instruct, with E5 retrieval for search tasks and training on a single node with 8 NVIDIA H20 GPUs.
  • ALFWorld and WebShop: Role-Agent outperforms prompt-based methods by an average of 78.0% on ALFWorld and 59.1% on WebShop in success rate.ReAct and Reflexion improve over zero-shot models but remain below RL-based methods, while Gemini reaches 92.8% on Pick-ALFWorld yet has lower average performance.
  • ALFWorld and WebShop: 4.2% / 6.9% relative gains over GiGPO on ALFWorld / WebShop show Role-Agent mostly surpasses RL-based competitors across two datasets.GRPO achieves 72.8% / 75.8% on ALFWorld / WebShop, while GiGPO achieves 86.7% / 83.1% with Qwen2.5-1.5B-Instruct.
  • Search-Augmented QA: 45.8% average performance makes Role-Agent best on search-augmented QA, exceeding GiGPO by 3.7%.Improvements are +8.2% on 2Wiki and +5.2% on MuSiQue, with larger gains on multi-hop than single-hop QA.
  • Ablation Study: Removing AIW decreases WebShop performance by 5.0%, while removing either AIW or predictive reward reduces overall performance.The results attribute AIW’s stronger effect to targeted environment feedback and dynamically accumulated failure modes that guide revisiting historical deficiencies.

5 Conclusion

Role-Agent is introduced as a bootstrapped framework for agent–environment co-evolution that addresses undirected, non-specific feedback in static environments. It uses one LLM as both agent and environment through World-In-Agent (WIA) and Agent-In-World (AIW).

  • Role-Agent targets undirected and non-specific feedback challenges arising in static environments.
  • A single LLM serves concurrently as the agent and the environment in Role-Agent.
  • WIA improves planning and reasoning by enabling the agent to predict future states from its actions.

Limitations

Role-Agent’s effectiveness is accompanied by limitations involving AIW’s dependence on stronger frozen environment models, similarity-threshold state grouping, and evaluation restricted to text-based environments.

  • Model dependence and fairness: A stronger frozen environment LLM may improve AIW but introduces external knowledge and complicates fair comparison with same-backbone baselines.The limitation concerns both added knowledge and changed comparison fairness.
  • State grouping: State grouping uses a similarity threshold adopted from previous studies, limiting cross-task generalization.The threshold-based grouping mechanism is identified as a constraint on transfer across tasks.
  • Evaluation scope: Current evaluation is confined to text-based environments, leaving extensions to multi-modal or real-time embodied settings unresolved.The passage identifies broader environment modalities as future extensions.

A Dataset Details · A.1 ALFWorld

ALFWorld connects text-based environments with physically embodied simulations through parallel representations of the same world. This dual-modality framework supports semantic priors, language-based reasoning, stronger generalization, and greater training efficiency.

  • A.1 ALFWorld: ALFWorld bridges text-based environments and physically embodied simulations.It provides an interactive framework spanning both modalities.
  • A.1 ALFWorld: Agents learn high-level policies in TextWorld and apply them within the visual AL-FRED benchmark.The framework links policy learning in text to execution in visual simulation.
  • A.1 ALFWorld: Parallel representations encode the same world across ALFWorld’s text-based and visual environments.This shared-world design connects the two modalities.
  • A.1 ALFWorld: ALFWorld enables agents to leverage semantic priors for task generalization.The parallel representations allow knowledge from one modality to inform reasoning in another.
  • A.1 ALFWorld: Language-based reasoning helps agents generalize more effectively to new tasks.ALFWorld uses its text-based representation to support reasoning beyond the original tasks.
  • A.1 ALFWorld: The dual-modality design promotes stronger generalization and greater training efficiency than vision-only approaches.Its text-and-vision structure is presented as an advantage over vision-only training.

A.2 WebShop

WebShop is a large-scale simulated e-commerce benchmark for grounded language agents, containing over 1.18 million products and 12,087 crowd-sourced natural language instructions. Agents fulfill requirements through search and click actions under an automatically computable product-attribute reward that supports sim-to-real transfer.

  • Benchmark setting: WebShop is a large-scale simulated e-commerce environment with over 1.18 million real-world products and 12,087 crowd-sourced natural language instructions.The benchmark is designed for training grounded language agents.
  • Interaction protocol: Agents use search[query] and click[element] to fulfill complex user requirements.These are the benchmark’s two available actions.
  • Evaluation: WebShop provides an automatically computable reward based on product attributes and shows sim-to-real transfer capabilities on shopping websites such as Amazon and eBay.The reward function connects simulated evaluation with product attributes observed in real shopping contexts.

A.3 Search-QA Tasks · B More Studies · B.1 Standard Deviations

The paper evaluates search-augmented QA across datasets spanning single-hop retrieval, long-tail knowledge, multi-hop reasoning, controlled inference, distractor filtering, and compositional generalization. Reported results use means and standard deviations computed over three runs.

  • A.3 Search-QA Tasks: Natural Questions benchmarks locating and extracting answers from a single passage in real Google search queries.Each example pairs a user question with an answer extracted from Wikipedia.
  • A.3 Search-QA Tasks: TriviaQA tests retrieval and synthesis of facts from unstructured text involving complex entity relationships.Its question-answer-evidence triples come from Wikipedia and news articles, with evidence collected through distant supervision.
  • A.3 Search-QA Tasks: Over 14,000 PopQA questions probe retrieval of obscure, long-tail entities rather than reliance on parametric memory.The dataset focuses on less frequently mentioned entities.
  • A.3 Search-QA Tasks: HotpotQA evaluates multi-hop reasoning across two or more Wikipedia paragraphs by requiring chains of evidence.The task tests following interconnected evidence instead of locating a single fact.
  • A.3 Search-QA Tasks: 2WikiMultihopQA provides controlled multi-step inference through predefined reasoning paths categorized by logical type.Categories include comparison, temporal, and compositional questions.
  • A.3 Search-QA Tasks: MuSiQue tests dependency-chain reasoning while filtering irrelevant information through composed questions and unanswerable distractors.Its questions are programmatically composed from datasets including SQuAD and TriviaQA, with strict connectivity between reasoning steps.
  • A.3 Search-QA Tasks: Bamboogle requires decomposition and sequential retrieval across multiple documents, making it unsolvable by parametric models alone.It evaluates compositional generalization by requiring facts from different sources to be combined in unseen ways.
  • B.1 Standard Deviations: Table 7 reports means and standard deviations over three runs.The standard deviations summarize variability across the repeated evaluations.

B.2 Relation between predictive reward and outcome reward

Predictive reward is positively associated with outcome reward and improves during training, indicating increasingly accurate state prediction. On ALFWorld rollouts with Qwen2.5-3B-Instruct, its correlation with outcome reward is statistically significant.

  • Relation between predictive reward and outcome reward: 0.41 point-biserial correlation (p < 0.01) links predictive and outcome rewards across 200 ALFWorld rollouts with Qwen2.5-3B-Instruct.Predictive reward rises from about 0.60 at initialization to the mid-to-high 0.70 range near convergence, indicating improved state prediction quality.

C Implementation Details … F The Use of Large Language Models

The paper details Role-Agent’s training setup, prompts, and algorithmic workflow, including dual-role prediction and failure-driven task retrieval. It also states that LLMs assisted only with language polishing and lightweight coding, while authors produced and verified the scientific results.

  • C Implementation Details: Role-Agent uses the VeRL framework, Qwen2.5-1.5/3/7B-Instruct backbones, and 8× NVIDIA H20 GPUs with tensor parallel equals 1.Hyper-parameters and failure modes are listed in Tables 5 and 6.
  • C Implementation Details: Its failure-mode library covers ALFWorld, WebShop, and search-augmented QA tasks.These failure modes are presented as the data used by Agent-In-World.
  • D Prompts: The prompts provide interaction history, search queries, and results, then ask the LLM to search the website or answer the question.Other prompts use task context and failed trajectories to generate failure categories, core lessons, and summaries before retrieving similar tasks.
  • F The Use of Large Language Models: LLMs were used to improve grammar and spelling without changing scientific content and to provide lightweight coding assistance, while authors produced and verified all reported results and claims.The paper explicitly assigns method design, experiments, analyses, and verification to the authors.
  • E Algorithm: Algorithm 1 samples task batches, uses the same LLM to predict future states, computes predictive and task rewards, and modulates the reward.The algorithm also groups identical states and computes state-level and final advantages.
  • E Algorithm: The environment role analyzes failure causes, generates failure modes and reflections, retrieves similar tasks, and updates the task distribution toward difficult and overlooked tasks.These steps operationalize failure-driven practice within Role-Agent training.
  • F The Use of Large Language Models: A case analysis identifies MISSING_PRECONDITION when an agent places a dirty soapbar without cleaning it, then retrieves tasks involving similarly unverified object-state requirements.The retrieved examples include cleaning a cloth, sponge, or dishsponge before placement.
  • F The Use of Large Language Models: A second case identifies WRONG_TARGET_LOCATION when an agent searches shelves and drawers but fails to check the desk for a desklamp, then retrieves tasks with the same search-ordering failure.The reflections recommend checking the desk first for desk-related items.
Loading 2606.10917v2…