Source-linked AI summary
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, Lihong Li
TL;DR
Training effective web agents for multi-turn interaction remains difficult because web environments require long-horizon decisions and adapt to changing states. WebAgent-R1 addresses this with an end-to-end multi-turn RL framework built around online interaction, M-GRPO, and efficiency mechanisms. On WebArena-Lite, it raises Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, while analyses highlight the importance of behavior-cloning initialization and increased interaction depth.
Problem
Multi-turn web-agent training remains underexplored because dynamic web environments require long-horizon decision-making beyond predominantly single-turn RL tasks.
Method
WebAgent-R1 performs end-to-end multi-turn RL from a behavior-cloned policy using online interactions, M-GRPO, dynamic context compression, and parallel trajectory rollouts.
Results
33.9% success rate is achieved by Qwen-2.5-3B versus 6.1%, and 44.8% by Llama-3.1-8B versus 8.5% on WebArena-Lite.
Takeaways & Limitations
Behavior cloning provides a crucial RL foundation, while thinking-based prompting and increased interaction depth improve web-agent performance.
Takeaways & Limitations
The approach uses only textual inputs, relies on rule-based outcome rewards, and employs a fixed action set that limits flexibility with unseen operations.
Abstract
from arXiv · showhide
While reinforcement learning (RL) has demonstrated remarkable success in enhancing large language models (LLMs), it has primarily focused on single-turn tasks such as solving math problems. Training effective web agents for multi-turn interactions remains challenging due to the complexity of long-horizon decision-making across dynamic web interfaces. In this work, we present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework for training web agents. It learns directly from online interactions with web environments by asynchronously generating diverse trajectories, entirely guided by binary rewards depending on task success. Experiments on the WebArena-Lite benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, significantly outperforming existing state-of-the-art methods and strong proprietary models such as OpenAI o3. In-depth analyses reveal the effectiveness of the thinking-based prompting strategy and test-time scaling through increased interactions for web tasks. We further investigate different RL initialization policies by introducing two variants, namely WebAgent-R1-Zero and WebAgent-R1-CoT, which highlight the importance of the warm-up training stage (i.e., behavior cloning) and provide insights on incorporating long chain-of-thought (CoT) reasoning in web agents.
1 Introduction
WebAgent-R1 addresses the underexplored challenge of training web agents for long-horizon, multi-turn interaction in dynamic environments. Its end-to-end RL framework combines multi-turn optimization with efficiency mechanisms and achieves strong WebArena-Lite performance.
- Motivation: Multi-turn web-agent RL remains underexplored compared with single-turn mathematical reasoning, despite requiring long-horizon decisions in dynamic environments.
- Motivation: Prompting and behavior cloning have shown initial success but cannot explore diverse strategies or learn effectively through trial and error.
- Motivation: Dynamic web-state changes make online, adaptive learning important because earlier actions can alter later task possibilities.
- Approach: WebAgent-R1 combines dynamic context compression, M-GRPO, and parallel trajectory rollouts for efficient end-to-end multi-turn RL.
- Results: 33.9% success rate is achieved by Qwen-2.5-3B versus 6.1%, while Llama-3.1-8B reaches 44.8% versus 8.5% on WebArena-Lite.
- Results: Analyses and ablations identify behavior cloning, thinking-based prompting, and test-time scaling as important factors in web-agent performance.
2 WebAgent-R1
WebAgent-R1 formulates web tasks as partially observable, multi-turn interactions and trains agents with behavior-cloning initialization followed by online M-GRPO optimization. Dynamic context compression, parallel rollouts, and binary rule-based rewards support scalable training in WebArena.
- Problem Formulation: Web tasks are modeled as a POMDP where text-only HTML observations lead to predefined web actions and binary success rewards.Interaction ends when the task succeeds or the maximum number of steps is reached.
- Behavior Cloning: Behavior cloning first trains the policy on expert demonstrations, providing basic web interaction skills and a foundation for reinforcement learning.The policy imitates expert actions conditioned on the full interaction history.
- Evaluation: Compared with other web-agent methods, the framework combines trial-and-error learning, on-policy updates, replay-buffer-free training, and no external supervision.WebArena-Lite evaluation reports task success rates across multiple websites using rule-based rubrics.
- End-to-End Multi-Turn Reinforcement Learning: The framework combines online rule-based outcome rewards with M-GRPO, which samples trajectory groups and optimizes the policy using group-relative advantages.M-GRPO extends GRPO to multi-turn web interactions.
- End-to-End Multi-Turn Reinforcement Learning: Dynamic context compression replaces earlier observations with simplified versions while preserving action history, reducing accumulated context and memory usage.Loss masks are updated so optimization remains focused on action tokens.
- End-to-End Multi-Turn Reinforcement Learning: Parallel browser instances independently generate diverse trajectories from the same starting page, improving sampling efficiency for M-GRPO.Each instance maintains its own context, such as cookies.
3 Experiments
Experiments evaluate WebAgent-R1 in realistic web environments and show that end-to-end multi-turn RL improves performance, with behavior cloning, thinking-based prompting, and deeper interaction all contributing to effectiveness.
- Experimental Setup: WebArena-Lite evaluates 165 verified tasks, while 647 remaining tasks provide RL training data and 9,460 public trajectories support behavior cloning.Task success rate uses WebArena-Lite’s built-in rule-based rubrics.
- Ablation Study: Behavior cloning is essential: WebAgent-R1-Zero starts at 6.1% without SFT and deteriorates slightly after RL because incomplete actions rarely receive positive rewards.The authors connect this initialization failure to ineffective exploration and learning.
- Ablation Study: 24.5% versus 20% success rate shows that long-CoT behavior-cloning data improves the pre-RL SFT model’s web-task performance.Long-CoT traces are generated with a strong reasoning model before SFT.
- Ablation Study: 20% to 33.9% versus 24.5% to 30.3% shows that RL gains are smaller after long-CoT SFT than after standard SFT.The authors hypothesize that deterministic long-CoT patterns constrain exploration during RL.
- Prompting and Test-Time Scaling: Thinking-format prompting raises o4-mini’s success rate from 15.9% to 36.9% and increases Qwen2.5-3B interactions from 6 to 17 while response length stays similar.The reported Qwen2.5-3B response length changes from 139 to 142 tokens.
- Prompting and Test-Time Scaling: More interaction turns consistently improve success rates across prompting-based, SFT, and RL-based methods while producing longer trajectories.The authors associate this scaling strategy with deeper exploration and iterative action refinement.
4 Related Works
Prior LLM-agent RL research has concentrated on single-turn reasoning or simplified multi-turn API interactions, leaving realistic dynamic web environments comparatively underexplored.
- Prior RL for LLMs: Most prior RL work targets single-turn tasks such as mathematical problems, with limited exploration of multi-turn settings.Recent efforts include repeated search-engine use but generally do not involve full environment interaction.
- Multi-Turn Agent RL: Some concurrent methods apply RL to multi-turn simulated games and coding environments, while on-policy learning uses trajectories generated by the current agent.On-policy updates avoid replay-buffer and stale-trajectory overheads associated with off-policy RL.
5 Conclusion
WebAgent-R1 extends GRPO to multi-turn web interaction and combines efficient training mechanisms with analyses of initialization, prompting, and interaction depth.
- Conclusion: M-GRPO, dynamic context compression, and parallel trajectory rollouts form WebAgent-R1’s end-to-end multi-turn RL framework.The framework is designed for efficient training of web agents.
- Conclusion: WebAgent-R1 achieves state-of-the-art WebArena-Lite results and shows that behavior cloning provides a strong foundation for effective RL.The conclusion also reports consistent benefits from increased interaction depth.
- Conclusion: Future work targets multimodal inputs and broader GUI-based tasks beyond web environments, including computer use.These directions define the stated scope boundary of the current work.
Limitations and Potential Risks
WebAgent-R1 is limited to textual inputs, rule-based outcome rewards, and a fixed predefined action set. These constraints restrict applicability to settings with ambiguous goals, unavailable verifiable rewards, or unseen interactive operations.
- Text-only web tasks exclude visual information such as layout and colors that may help navigation and decision-making.
- Rule-based outcome rewards may be unavailable in open-ended scenarios where goals are ambiguous and no clear reference or verifiable outcome exists.
- A fixed set of predefined actions can limit flexibility when interactive elements require unseen operations.
- Dynamic adaptation to new operations remains an open challenge for web agents.
- Real-world deployment, especially with administrative privileges, risks destructive actions affecting sensitive data.Suggested safeguards include permission controls, verification prompts, and protections against high-impact or irreversible actions.
A Web Environment
WebArena-Lite is a self-hostable web environment and curated benchmark spanning diverse real-world web tasks. Agents use predefined browser actions, while success is measured by rule-based checks tailored to each task objective.
- WebArena-Lite selects 165 representative tasks for evaluation and uses 647 remaining tasks for training.The benchmark also provides 9,460 automatically annotated trajectories for behavior cloning.
- The environment covers social forums, collaborative coding, e-commerce content management, mapping, and online shopping.
- Agents interact through predefined actions including clicking, typing, searching, hovering, scrolling, pressing Enter, and switching tabs.
- Success Rate records whether the final task goal is achieved rather than evaluating a closed-form solution.
- String matching, URL matching, and program execution verify answers, destinations, or final webpage states.
B Implementation Details
The implementation uses Qwen2.5-3B and Llama3.1-8B backbones with behavior-cloning initialization and distributed full-parameter training. It also compares standard WebAgent-R1 with a long-CoT variant that produces more detailed reasoning.
- Qwen2.5-3B and Llama3.1-8B serve as the backbone models for prompting and fine-tuning.The default models are instruction-tuned, and RL initialization comes from supervised fine-tuning checkpoints obtained via behavior cloning.
- Training uses one node with 8 NVIDIA A100 80GB GPUs and full-parameter fine-tuning.DeepSpeed ZeRO-3 offload, gradient checkpointing, FlashAttention-2, and bf16 mixed precision improve computation efficiency.
- WEBAGENT-R1-COT shows more detailed thinking than WEBAGENT-R1 on successful trajectories for the same task.
- RL training uses a constant learning rate of 1e-6, batch size 16, maximum context length 16,384, and 1,024 new tokens.
C Data Augmentation for Behavior Cloning with long-CoT Traces
The long-CoT variant augments behavior-cloning data with reasoning traces generated by QwQ-32B, then applies supervised fine-tuning and reinforcement learning. This produces WEBAGENT-R1-COT, which generates more detailed thinking than WEBAGENT-R1.
- Behavior-cloning data is augmented with long-CoT traces generated by the QwQ-32B reasoning model.
- Supervised fine-tuning on the augmented data precedes RL training to produce WEBAGENT-R1-COT.
- WEBAGENT-R1-COT generates more detailed thinking than WEBAGENT-R1.
D Prompt Templates
The data-augmentation prompt defines the web-agent action space in the system prompt and uses a thinking-format version by default.
- The system prompt specifies the web-agent action space.
- The default prompt uses the version with the thinking format.
E Qualitative Examples
Figures 6–10 show successful WebAgent-R1 trajectories across all five WebArena websites, covering CMS, Map, Shopping, GitLab, and Reddit tasks.
- Successful trajectories are presented for each of the five WebArena websites.The examples span CMS, Map, Shopping, GitLab, and Reddit.
- CMS: The CMS example asks for the top-3 best-selling products in January 2023.
- Map: The Map example asks for the estimated driving time from Homewood Suites Southpointe to PPG Paints Arena.
- Shopping: The Shopping example asks for living-room furniture products listed by descending price.
- GitLab: The GitLab example asks the agent to retrieve the user's RSS feed token.
- Reddit: The Reddit example asks the agent to edit a Star Trek: Starfleet Academy post by adding a specified sentence to its body.