PaperlayerView exact revision on arXiv

Source-linked AI summary

WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, Yuxiao Dong

arXiv:2411.02337v3cs.CL

TL;DR

High-performing web agents depend on expensive proprietary APIs, while open LLMs lack proficient web-agent decision-making. WebRL trains open LLMs through self-evolving online curriculum reinforcement learning, raising Llama-3.1-8B’s WebArena-Lite success rate from 4.8% to 42.4%.

  • Problem

    High-performing web agents rely on expensive proprietary APIs, while open LLMs lack proficient web-agent decision-making capabilities.

    Source: §1 · p. 2

  • Method

    WebRL trains open LLM web agents with self-evolving online curriculum reinforcement learning, generating progressively harder tasks and using an outcome-supervised reward model with adaptive learning strategies.

    Source: §1 · p. 3, §2.1 · p. 4

  • Results

    42.4% success rate, up from 4.8%, for Llama-3.1-8B on WebArena-Lite.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings demonstrate WebRL’s effectiveness in advancing open-source LLM capabilities for web-based tasks.

    Source: §5 · p. 12

Abstract

from arXiv · show

Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents heavily rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WebRL, a self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. WebRL addresses three key challenges in building LLM web agents, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4 models into proficient web agents. On WebArena-Lite, WebRL improves the success rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B. These open models significantly surpass the performance of GPT-4-Turbo (17.6%) and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's effectiveness in bridging the gap between open and proprietary LLM-based web agents, paving the way for more accessible and powerful autonomous web interaction systems.

1 INTRODUCTION

WebRL trains open LLM web agents through self-evolving online curriculum reinforcement learning, addressing scarce training tasks, sparse feedback, and policy drift. It substantially improves WebArena-Lite success rates and uses replay and policy constraints to support learning.

Introduction

  • WebRL addresses limited online training tasks, sparse and costly feedback on long-horizon tasks, and policy drift that can cause forgetting and degradation.It generates new tasks from unsuccessful attempts using an outcome-supervised reward model, and constrains updates with a KL-divergence term between reference and actor policies.Source: §1 · p. 2
  • 49.1% SR on WebArena-Lite for Llama-3.1-70B exceeds GPT-4-Turbo’s 17.6% and AutoWebGLM’s 18.2%; Llama-3.1-8B rises from 4.8% to 42.4%.The paper also reports state-of-the-art WebArena-Lite results and improvements over AWR and DigiRL.Source: §1 · pp. 2–3
  • Replay improves performance when selected experiences are moderately difficult, while a larger KL-divergence constraint performs better when past experience is incorporated.An actor-confidence filtering strategy is used to preserve replayed-experience fidelity and prevent overfitting to previously acquired knowledge.Source: §1 · pp. 2–3
  • WebRL provides infrastructure for reinforcement learning in WebArena with an outcome-supervised reward model, enabling open LLMs to become capable web agents.The introduction describes existing high-performing web agents as relying on expensive proprietary APIs, while open-source models lack decision-centric training data and imitation learning has not ensured continual improvement.Source: §1 · pp. 2–3

2 WEBRL: SELF-EVOLVING ONLINE CURRICULUM RL

WebRL trains web agents through ongoing interaction with WebArena, combining tasks adapted to agent proficiency, outcome-based rewards, KL-constrained policy updates, and replay to support incremental learning.

2 WEBRL: SELF-EVOLVING ONLINE CURRICULUM RL

  • WebRL frames web-task completion as a finite-horizon MDP, with page HTML and action history as state and binary reward for task success.The framework continually collects interaction trajectories and uses curriculum-generated tasks, an outcome-supervised reward model, KL-constrained updates, and replay to support incremental learning.Source: §2 · p. 3
  • The outcome-supervised reward model judges trajectory success as binary reward, comparing the probabilities of “YES” and “NO” given the instruction, action history, and final-page HTML.Only the final-state HTML is included to accommodate LLM context limits; a higher “YES” probability yields reward 1, otherwise 0.Source: §2 · p. 4

2.1 SELF-EVOLVING NEW INSTRUCTION FOR CURRICULUM LEARNING

  • Failed instructions seed increasingly complex tasks, filtered by critic scores of 0.05–0.75 and GPT-4o screening for WebArena feasibility.This two-step generation-and-filtering process aims to make tasks progressively harder while matching the agent’s current capability.Source: §2.1 · p. 4

2.2 REINFORCEMENT LEARNING FOR LLMS IN ONLINE WEB ENVIRONMENTS

  • To limit policy drift across phases, the update constrains the current policy relative to the previous phase’s policy while optimizing new-task performance.The KL constraint’s strength is controlled by β, and the update adjusts action probabilities according to estimated advantage while limiting divergence from the reference policy.Source: §2.2 · pp. 4–5
  • A value network and GAE estimate advantages from binary terminal rewards, with λ set to 0.5 to balance bias and variance.There are no intermediate rewards, so the method focuses on next-step and final-step advantage estimators and trains the value network with cross-entropy.Source: §2.2 · p. 6
  • The replay buffer stores successful trajectories and filters their actions by actor perplexity from 1/0.95 to 1/0.5 before reusing them in the current phase.This excludes data that is over-familiar or still too challenging for the previous-phase actor, while avoiding unreliable intermediate-state estimates from failed trajectories.Source: §2.2 · p. 6

3 EXPERIMENTS

The experiments evaluate WEBRL against prompting and training baselines on WebArena-Lite, then analyze its scaling, error patterns, task-length and complexity performance, component ablations, ORM accuracy, and curriculum-generated tasks.

3.2 MAIN RESULTS

  • 42.4% average accuracy was achieved by WEBRL-trained Llama3.1-8B, surpassing all baselines on WebArena-Lite.WEBRL also achieved 46.7% on Gitlab and 54.3% on CMS. The authors attribute imitation-learning overrepetition to weak long-term performance, while RL uses a critic to value steps; WEBRL’s self-evolving curriculum also addresses DigiRL’s fixed-task limitation. Similar benefits were observed for GLM-4-9B.Source: Table 1 · p. 7, §3.2 · pp. 7–8

3.3 SCALING EFFECT OF WEBRL

  • 49.1% overall accuracy was achieved by WEBRL-trained Llama3.1-70B, a 26.1% improvement over SFT accuracy.The authors report that WEBRL’s performance gains over SFT increase as model scale grows from Llama3.1-8B to Llama3.1-70B.Source: Table 1 · p. 7, §3.3 · p. 8

3.4 DISTRIBUTION ANALYSIS OF ERROR TYPES

  • WEBRL reduced “Get Stuck Midway” errors versus SFT and Filtered BC and had the lowest rates of “Stop at Wrong Page” and “Fail to Make Reasonable Attempt” errors.The authors also report greater robustness to “Fail to Recover”; they attribute recovery to curriculum learning that teaches agents to adapt after failures.Source: §3.4 · p. 8, Figure 3 · p. 8

3.5 PERFORMANCE ON TASKS WITH VARYING STEP REQUIREMENTS

  • WEBRL improved performance on long-horizon tasks, while SFT and Filtered BC declined as task length increased and DigiRL struggled beyond 10 steps.The authors attribute WEBRL’s long-sequence gains to curriculum learning that progressively increases task difficulty.Source: §3.5 · pp. 8–9, Figure 4 · p. 9

3.6 PERFORMANCE ON TASKS WITH VARYING COMPLEXITY

  • WEBRL performed well across instruction-complexity levels, particularly on more complex instructions, whereas DigiRL struggled as complexity increased.The authors connect this difference to WEBRL’s self-evolving curriculum, which adapts task complexity to the model’s capabilities.Source: §3.6 · p. 9, Figure 6 · p. 9

3.7 ABLATION STUDY

  • Removing the replay buffer caused performance to worsen over time, while WEBRL consistently outperformed its no-KL variant.The replay buffer retains earlier experiences; the KL-constrained update better preserves past knowledge, while REINFORCE without KL quickly overfits current-phase data. Without curriculum learning, progress was slower and reached a lower ceiling because tasks remained fixed rather than adapting to capabilities.Source: §3.7 · pp. 9–10, Figure 5 · p. 9
  • Training on replay-buffer data with very low perplexity ([1, 1/0.95]) led to performance deterioration.The passage reports that multiple perplexity thresholds were tested during the first learning phase.Source: §3.7 · p. 10, Table 2 · p. 10

3.8 EVALUATION OF ORM

  • Approximately 80% accuracy was achieved by WEBRL’s ORM, exceeding the baselines’ slightly-above-70% accuracy.The evaluation used WebArena-Lite test data and 100 manually labeled sampled rollouts, comparing ORM with GPT-4-Turbo, Captioner + GPT-4-Turbo, and GPT-4V.Source: §3.8 · pp. 10–11, Table 3 · p. 10

4 RELATED WORKS

Related work spans training-free and training-based LLM agents and RL device-control agents; WebRL targets their limits with evolving tasks and online-update mechanisms.

Adopting LLMs as Agent

  • Prior agent methods face base-model constraints, costly demonstrations, and weak long-horizon action modeling (Chen et al., 2023; Ghosh et al., 2021); WebRL dynamically generates tasks and updates policy.Static task sets can hinder continued improvement as agent capabilities evolve.Source: §4 · pp. 11–12

Reinforcement Learning for LLMs

5 CONCLUSION · A DETAILS OF POLICY UPDATE ALGORITHM IN WEBRL

WEBRL is a self-evolving online curriculum reinforcement learning framework that addresses key obstacles to training LLM-based web agents and enables continual, consistent improvement in online environments.

5 CONCLUSION

  • WEBRL introduces a self-evolving online curriculum reinforcement learning framework for training LLM-based web agents.Source: §5 · p. 12
  • The framework addresses the scarcity of training tasks for web agents.Source: §5 · p. 12
  • WEBRL addresses sparse feedback signals in online web-agent training.Source: §5 · p. 12
  • The framework also tackles policy distribution drift during online learning.Source: §5 · p. 12
  • WEBRL enables continual and consistent improvement in agent performance in online environments such as WebArena.Source: §5 · p. 12
  • The authors report substantial performance gains that surpass existing state-of-the-art web agents and proprietary LLM APIs, advancing open-source LLM capabilities for web tasks.Source: §5 · p. 12

A.1 DERIVATION

The derivation trains an approximating policy to match a target policy using estimated advantages, allowing off-policy experience. It motivates the loss choice, the training-data mixture, and the discount-factor adjustment.

A.1 DERIVATION

  • The policy loss matches πθ to target policy π* using estimated advantages, with no restriction on experience distribution ν and thus allowing off-policy training.A* is estimated using eqs. 7 and 8; critic training is also off-policy.Source: §A.1 · pp. 17–18
  • For stable improvement, ν typically mixes data from the current policy with prior successful experiences to reinforce task-completing actions and address policy deficiencies.Source: §A.1 · p. 17
  • Eq. 11 is preferred to KL-based eq. 13 because it permits replay data, avoids increasing negative-advantage actions, and is experimentally reported as superior.Eq. 13 requires data sampled from πref, which may omit past successes; Figure 9 compares the two targets.Source: §A.1 · pp. 17–18, Figure 9 · p. 18
  • With discounting, γ modifies the advantage estimate but leaves the policy-loss derivation and critic loss unchanged; experiments use γ = 0.9.Source: §A.1 · p. 18

A.2 PROOF

The proof argues that training against a value-based advantage estimate from current and replay-buffer data can improve the policy when the learned policy accurately matches an improved target. An experiment supports using both data sources to train the critic, while the convergence argument assumes historical data contains only correct trajectories.

A.2 PROOF

  • Minimizing the stated objective on trajectories from the current policy and replay buffer is presented as a route to policy improvement using the behavioral policy’s value and action-value functions.The objective matches the policy’s log probability to an exponential function of the behavioral policy’s action-value minus value, scaled by β.Source: §A.2 · p. 18
  • If πnew improves on πµ and πθ closely matches πnew, then πθ also improves on πµ, provided the approximation is accurate.The proof invokes the soft Bellman equation and the bound in Eq. 18.Source: §A.2 · p. 19
  • Assuming historical data includes only correct trajectories, πµ is no worse than πold; iterative policy evaluation and improvement can then move πθ toward π∗.Without replay-buffer data, the same theoretical process applies using Vπold and Qπold instead of Vπµ and Qπµ.Source: §A.2 · p. 19
  • The Figure 10 model performs better when its critic uses both replay-buffer experiences and current rollout trajectories than when it uses rollout trajectories alone.The result highlights the value of including prior successful trajectories in critic training.Source: §A.2 · p. 19

A.3 COMPARISON

The comparison distinguishes WebRL’s value-function-guided, off-policy optimization from DPO, PPO, and AWR, emphasizing its fit for costly web interactions and its handling of unsuccessful traces.

Comparison with DPO

  • WebRL explicitly fits a value function to guide policy optimization, whereas DPO uses contrastive paired data to avoid estimating rewards (Rafailov et al., 2024b).Pairwise data are difficult to obtain for web tasks because backtracking web pages to collect outcomes for different actions in the same state is challenging.Source: §A.3 · pp. 19–20

Comparison with PPO

  • WebRL is off-policy, unlike PPO (Schulman et al., 2017), which the paper characterizes as sampling-inefficient and data-intensive for stable improvement in costly web environments.The passage identifies WebArena and real-world websites as environments where interaction is expensive and inefficient.Source: §A.3 · p. 20

Comparison with AWR

  • Unlike AWR (Peng et al., 2019), WebRL minimizes KL divergence from a reference policy and directly reduces incorrect-action probability, making use of unsuccessful traces without increasing that probability.AWR instead constrains divergence from the behavioral policy derived from training data; the passage notes unsuccessful traces are common in web-task samples.Source: §A.3 · p. 20

B TRAINING DETAILS · B.1 DETAILS OF WEBRL

B.1 specifies WebRL’s browser-agent observations and actions, then describes the supervised initialization and data-mixing process used during self-evolving curriculum reinforcement learning.

B.1 DETAILS OF WEBRL

  • The agent observation combines the user instruction, action history, and current webpage HTML.Source: §B.1 · p. 20
  • Webpage HTML is simplified and clickable elements receive distinct IDs so the model can identify targets for manipulation.Source: §B.1 · p. 20
  • The action set follows WebArena-Lite with additions, including click, hover, type, search, keyboard, scrolling, dropdown, tab, navigation, and exit actions.Source: §B.1 · p. 20
  • Actions include “# Element:” comments describing the operated element and “# Note:” comments quoting webpage information relevant to the instruction.Source: §B.1 · p. 20
  • Training begins with supervised fine-tuning on WebArena-Lite training data, then initializes a replay buffer and failure set using the SFT model on corresponding instructions.Source: §B.1 · p. 21
  • Each curriculum phase selects 500 filtered GPT-4o-generated instructions and trains actor and critic on their new interactions plus perplexity-filtered replay data capped at twice the interaction-data size.Historical samples have perplexity between 1/0.95 and 1/0.5.Source: §B.1 · p. 21

B.2 DETAILS OF BASELINES

This section distinguishes offline RL baselines from DigiRL’s online setup and summarizes how DigiRL was reproduced in WebArena.

B.2 DETAILS OF BASELINES

  • Except for DigiRL, RL baselines use first-phase WEBRL interaction data and therefore fall under offline reinforcement learning; DigiRL trains online on first-phase instructions.DigiRL is the exception to the offline setup.Source: §B.2 · p. 21
  • DigiRL is reproduced in WebArena with its AWR method, instruction- and step-level value functions, and replay buffer, adapting data format and hyperparameters for WEBRL.DigiRL also conducts 8 interaction-and-training rounds; baseline hyperparameters are presented in Table 5.Source: §B.2 · p. 21

B.3 DETAILS OF ORM

The ORM is trained on labeled WebArena-Lite trajectories and used to judge success on newly generated curriculum tasks without predefined reward functions.

B.3 DETAILS OF ORM

  • 12,200 samples form the MORM training set, combining original WebArena-Lite samples with rollouts from baseline methods on rewritten instructions.The original training set contains 1,186 samples; the reward function labels trajectories according to task completion.Source: §B.3 · pp. 21–22
  • MORM outputs “YES” or “NO”; the higher assigned probability determines whether a trajectory is judged successful.Its training trajectories are labeled using their associated WebArena-Lite reward functions.Source: §B.3 · p. 22
  • MORM judges rollout success during self-evolving curriculum RL, where newly generated tasks lack predefined reward functions.The algorithm uses MORM to evaluate rollouts, then adds successful trajectories to the replay buffer and failures to the failure set.Source: §B.3 · p. 22

C OTHER QUANTITATIVE EXPERIMENTS · D PROMPTS EMPLOYED IN WEBRL · E QUALITATIVE EXAMPLES

The appendix reports additional WebRL experiments, describes the prompts used in training and evaluation, and provides qualitative examples from all five WebArena sites.

C OTHER QUANTITATIVE EXPERIMENTS

  • Across three task-generation prompts, WEBRL delivers strong performance after eight training phases, indicating stability to prompt variation.The results are reported in Table 7.Source: Table 7 · p. 24, C OTHER QUANTITATIVE EXPERIMENTS · p. 24
  • WEBRL shows upward performance trends on every site except Map, where performance first rises and then declines, alongside a slight GitLab decline.The paper hypothesizes that increased OSS and CMS implementation creates a trade-off contributing to the Map decline and slight GitLab decline.Source: C OTHER QUANTITATIVE EXPERIMENTS · p. 22, Figure 12 · p. 24

D PROMPTS EMPLOYED IN WEBRL

  • WebRL uses prompts for MORM completion judgments, generating and filtering instructions, and defining feasible actions and behavioral requirements for baseline models.MORM outputs YES or NO using only the final state’s HTML; failed tasks seed generation of varied tasks at similar difficulty, while the filter applies feasibility rules across five WebArena sites.Source: D PROMPTS EMPLOYED IN WEBRL · pp. 24–25

C OTHER QUANTITATIVE EXPERIMENTS

E QUALITATIVE EXAMPLES

Related papers

Loading 2411.02337v3…