PaperlayerView exact revision on arXiv

Source-linked AI summary

WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, Yuxiao Dong

arXiv:2411.02337v3cs.CL

TL;DR

Existing web agents often depend on expensive proprietary APIs, while open LLMs lack strong web-agent decision-making capabilities. WebRL addresses this gap with self-evolving online curriculum reinforcement learning, robust outcome supervision, and adaptive updates, achieving substantial gains across open LLMs on WebArena-Lite.

  • Problem

    Existing high-performing web agents rely mainly on expensive proprietary APIs, while open LLMs lack sufficient decision-centric training for web interaction.

  • Method

    WebRL combines self-evolving task generation, an outcome-supervised reward model, and adaptive reinforcement learning to train open LLM web agents online.

  • Results

    42.4% success rate: WebRL raises Llama-3.1-8B from 4.8% on WebArena-Lite, while the framework achieves 49.1% with Llama-3.1-70B.

  • Takeaways & Limitations

    WebRL enables open-source LLMs to become capable web agents and substantially surpasses reported proprietary-API and open-agent baselines on WebArena-Lite.

Abstract

from arXiv · show

Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents heavily rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WebRL, a self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. WebRL addresses three key challenges in building LLM web agents, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4 models into proficient web agents. On WebArena-Lite, WebRL improves the success rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B. These open models significantly surpass the performance of GPT-4-Turbo (17.6%) and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's effectiveness in bridging the gap between open and proprietary LLM-based web agents, paving the way for more accessible and powerful autonomous web interaction systems.

1 INTRODUCTION

WebRL is introduced as a self-evolving online curriculum reinforcement learning framework for training capable web agents with open LLMs. It addresses limited training tasks, sparse feedback, and policy distribution drift while producing strong WebArena-Lite results and continual improvement.

  • Challenges: WebRL addresses insufficient training tasks, sparse feedback signals, and policy distribution drift in online learning.Its design combines curriculum generation, outcome-supervised rewards, and adaptive learning strategies to handle these challenges.
  • The WEBRL Framework: WebRL trains LLM-based web agents with a self-evolving online curriculum reinforcement learning framework.The framework is designed to enable reinforcement learning for LLM web agents in online environments such as WebArena.
  • The WEBRL Framework: 42.4% success rate: Llama-3.1-8B improves from 4.8% on WebArena-Lite after WebRL training.The result spans a diverse set of five websites.
  • The WEBRL Framework: 49.1% success rate: Llama-3.1-70B surpasses GPT-4-Turbo at 17.6% and AutoWebGLM at 18.2%.These comparisons are reported on WebArena-Lite.
  • The WEBRL Framework: WebRL generates novel tasks from unsuccessful attempts in preceding phases, supporting progressive learning through self-evolution.The curriculum uses exploration and an outcome-supervised reward model to drive task generation and learning.
  • The WEBRL Framework: Moderately difficult replayed experiences improve performance, while a larger KL-divergence constraint improves policy updates using past experience.These findings support confidence-filtered experience replay and adaptive reinforcement learning strategies.

2 WEBRL: SELF-EVOLVING ONLINE CURRICULUM RL

WebRL trains web agents online by combining self-evolving task generation, outcome-supervised rewards, and reinforcement-learning mechanisms that limit policy drift and support learning from sparse feedback.

  • Problem formulation: The web-task process is modeled as a finite-horizon MDP whose state combines current-page HTML with the history of previous actions.A trajectory ends when the task is completed or the maximum interaction horizon is reached.
  • Outcome-supervised reward: The outcome-supervised reward model evaluates whether a rollout completes its instruction and supplies a binary success or failure reward.The model outputs a task-success judgment, with 1 for success and 0 for failure.
  • Self-evolving curriculum: WebRL continuously generates increasingly difficult training instructions from tasks the agent previously failed, tailoring curriculum difficulty to current capability.The process uses generation and filtering to produce tasks that are incrementally more challenging while remaining suitable for the agent.
  • Policy-drift control: The current policy is constrained relative to the previous policy with KL divergence while entropy regularization remains part of the optimization objective.β controls the KL constraint strength, and H(πθ) denotes the current policy’s entropy.
  • Sparse-feedback learning: Advantage estimates use a value network and generalized advantage estimation despite receiving reward only at the terminal step.The method focuses on next-step and final-step advantage estimators because intermediate rewards are absent.
  • Policy-drift control: Successful trajectories are replayed selectively using actor-confidence filtering to reduce knowledge forgetting across curriculum phases.The replay buffer stores successful trajectories, and the prior-phase actor scores buffered actions by perplexity before reuse.

3 EXPERIMENTS

Experiments evaluate WEBRL on WebArena using open and proprietary baselines, finding strong performance across model scales, error types, task lengths, and instruction complexity. Ablations support the contributions of replay, KL-constrained updates, curriculum learning, and outcome supervision.

  • Main results: 42.4% average accuracy is achieved by WEBRL-trained Llama3.1-8B on WebArena-Lite, surpassing prompting and training baselines.WEBRL also reaches 46.7% on GitLab and 54.3% on CMS.
  • Scaling effect: 49.1% overall accuracy is achieved by WEBRL-trained Llama3.1-70B, a 26.1% improvement over SFT.The reported gains increase as model scale grows from Llama3.1-8B to Llama3.1-70B.
  • Error analysis: WEBRL reduces “Get Stuck Midway” errors and improves recovery from failures compared with baseline methods.The paper attributes these improvements to long-term action optimization and curriculum-based adaptation after failures.
  • Varying step requirements: Performance of SFT and Filtered BC declines as task length increases, whereas WEBRL improves handling of tasks requiring long-term planning.The analysis estimates required steps using the shortest successful trajectory after excluding tasks completed by no model.
  • Varying complexity: WEBRL performs well across instruction-complexity levels and particularly excels on more complex instructions.Complexity is measured by the number of requirements in each task.
  • Ablation study: Removing the replay buffer, KL-constrained update, or self-evolving curriculum worsens performance or slows progress relative to WEBRL.The replay buffer preserves earlier experience, KL updates balance past and new knowledge, and curriculum generation raises the performance ceiling.
  • Replay and policy-update analysis: Very low-perplexity replay data deteriorates performance, while moderate-difficulty experiences improve performance and larger KL constraints perform better when replaying past experience.These findings motivate selective replay and adaptive policy updates.
  • Reward model: Approximately 80% accuracy is achieved by the proposed ORM, compared with slightly above 70% for baseline output-supervised methods.The evaluation concerns output-supervised methods without accessing proprietary GPT-4.

4 RELATED WORKS

Prior work studies LLM agents for device control through training-free prompting and training-based methods, including reinforcement learning. Existing approaches can struggle with complex, long-horizon tasks or require costly demonstrations and pairwise data.

  • Training-free methods augment pretrained LLMs with prompts or demonstrations, but their accuracy remains insufficient for complex tasks.
  • Many existing methods optimize individual actions without adequately modeling long-term effects, limiting generalization.
  • Reinforcement learning offers an alternative for multi-step device-control tasks by training models to select actions based on device state.
  • LLM agent research increasingly targets device control, where models select actions through sequential interactions with device states.

5 CONCLUSION

The paper concludes that WEBRL enables continual improvement of open-source LLM web agents in online environments. It reports substantial gains over existing web agents and proprietary LLM APIs.

  • WEBRL addresses training-task scarcity, sparse feedback, and policy distribution drift through self-evolving curriculum reinforcement learning.
  • WEBRL enables continual and consistent improvement of agent performance in online environments such as WebArena.
  • WEBRL substantially surpasses existing state-of-the-art web agents and proprietary LLM APIs.

A.1 DERIVATION

The derivation defines an off-policy policy-learning objective based on an estimated advantage and examines why mean squared error is preferred to KL divergence. It also incorporates discounting into advantage estimation.

  • The policy loss trains πθ to match the target policy π∗ using experience from an unrestricted distribution ν, enabling off-policy learning.
  • Mean squared error is used to measure distance between policy distributions instead of KL divergence.
  • KL-based optimization imposes stronger training-data restrictions because data must conform to the reference-policy distribution.
  • Eq. 13 cannot decrease the probability of actions with negative advantage.
  • With discounting, the advantage uses immediate reward plus the discounted next-state value minus the current-state value.

A.2 PROOF

The proof argues that training with a value function estimated from mixed current and historical experience can improve the policy. It establishes this through an improved intermediate policy and a close approximation by the trained policy.

  • The proof defines a behavioral policy πµ from current rollouts and replay-buffer experiences, then optimizes πθ against value-based targets.
  • The constructed policy πnew improves over πµ, and an accurately trained πθ inherits that improvement when it closely matches πnew.
  • Using only correct historical trajectories ensures that πµ does not perform worse than the old policy πold.
  • The policy-improvement argument iteratively applies policy evaluation and improvement to move πθ toward the optimal policy π∗.
  • An experiment compares critics trained with rollout data alone versus rollout data combined with replay-buffer experiences.

A.3 COMPARISON

WEBRL is compared with DPO, PPO, and AWR through differences in data requirements, policy constraints, and sampling efficiency. The section also points to an algorithm comparison summarized in Table 4.

  • Table 4 compares WEBRL with different algorithms using various criteria.
  • DPO uses paired data to construct a learning target without explicitly estimating the reward, whereas WEBRL explicitly fits the value function.
  • State backtracking is difficult on web pages, making it challenging to collect DPO’s paired outcomes from the same page state.
  • PPO is on-policy, has low sampling efficiency, and requires large amounts of data for stable improvement; WEBRL operates off-policy.
  • WEBRL explicitly minimizes KL divergence between the reference policy and current policy, while AWR constrains divergence between the current and behavioral policies.

B TRAINING DETAILS

The WEBRL agent observes the user instruction, action history, and current webpage HTML, then outputs an action for the webpage. Training begins with supervised fine-tuning and continues through curriculum reinforcement learning.

  • The agent observation consists of the user instruction, action history, and HTML content of the current webpage.
  • HTML is simplified and clickable elements receive distinct IDs so the model can identify which element to manipulate.
  • The action space includes clicking, hovering, typing, searching, key presses, scrolling, dropdown selection, and tab operations.
  • Action comments identify the operated element and quote webpage information relevant to completing the instruction.
  • WEBRL first applies supervised fine-tuning, initializes replay and failure sets, and then selects 500 filtered GPT-4o-generated instructions in each curriculum phase.

B.2 DETAILS OF BASELINES

The baseline comparison uses shared interaction data for most RL baselines, while DigiRL is reproduced with its value functions, replay buffer, and online training setup. Figure 11 describes the common agent interface.

  • Except for DigiRL, RL baselines use WEBRL’s first-phase interaction data and therefore operate as offline reinforcement-learning methods.
  • DigiRL is reproduced in WebArena with AWR, instruction-level and step-level value functions, and a replay buffer, alongside format and hyperparameter adjustments.
  • DigiRL conducts 8 rounds of interaction and training.
  • The baseline task asks the agent to list canceled orders for Jennifer White and sort them by decreasing total value.
  • Figure 11 presents task instruction, action history, and current-page HTML as input, with the selected webpage action as output.

B.3 DETAILS OF ORM

The outcome reward model is trained on augmented WebArena-Lite trajectories labeled by task success. WEBRL’s training algorithm initializes successful trajectories in replay and failures in a separate set before curriculum reinforcement learning.

  • WebArena-Lite provides 1,186 training samples, and augmentation plus collected rollouts produces a 12,200-sample dataset for training MORM.
  • The WebArena-Lite reward function labels trajectories according to whether they successfully complete their tasks.
  • MORM outputs “YES” or “NO” to indicate whether a trajectory successfully completes a task.
  • Algorithm 1 takes the WebArena-Lite training set, instruction set, base model, replay buffer, failure set, reward model, phase count, and instructions-per-phase as inputs.
  • Algorithm 1 fine-tunes the base model, evaluates rollouts, stores successful trajectories in replay buffer B, and places failed instructions in failure set F.

C OTHER QUANTITATIVE EXPERIMENTS

WEBRL generally improves performance across websites and remains above baselines after training, while curriculum learning and replay-buffer choices materially affect results. Performance is robust across random seeds and task-generation prompts, although website-specific trade-offs and variability remain.

  • Performance trends: Most websites show an upward performance trend under WEBRL, while Map eventually declines and GitLab slightly decreases, hypothesized to reflect OSS and CMS trade-offs.The Map decline follows an initial improvement; the authors associate it with increased OSS and CMS implementation.
  • Training stability: Across four random seeds, WEBRL maintains an overall upward accuracy trend and surpasses every baseline after the final training phase.The authors note some accuracy fluctuations despite the consistent aggregate improvement.
  • Curriculum ablation: Adding curriculum learning significantly improves DigiRL, while removing it reduces WEBRL performance; WEBRL still slightly exceeds curriculum-enabled DigiRL without curriculum learning.The comparison evaluates both WEBRL and DigiRL with and without self-evolving curriculum learning.
  • Variability: WEBRL consistently outperforms baselines despite higher variability associated with randomness in curriculum-generated tasks.DigiRL with curriculum learning also shows higher standard deviation than ordinary baselines.
  • Replay buffer: Including failed trajectories in the replay buffer produces slower and more volatile improvement than retaining only successful trajectories.The comparison attributes this issue to estimating historical-action advantages from final trajectory rewards.
  • Prompt robustness: WEBRL remains strong after eight training phases across three task-generation prompts, indicating stability to prompt variation.The prompts are alternative versions used to generate diverse tasks from failed trajectories or related examples.
Loading 2411.02337v3…