PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents

Rui Yang, Qianhui Wu, Yuxi Chen, Hao Bai, Wenlin Yao, Hao Cheng, Baolin Peng, Huan Zhang, Tong Zhang, Jianfeng Gao

arXiv:2606.02031v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Open visual web agents still face limited scalable training data and open, reproducible methods for learning on dynamic websites. OpenWebRL addresses this with online multi-turn RL on live websites, and OpenWebRL-4B achieves a 68.4% average success rate across three live-web benchmarks while outperforming existing open-source agents.

  • Problem

    Open visual web agents lack scalable training frameworks and rely on expensive, static demonstrations while lagging proprietary systems on long-horizon planning and dynamic websites.

  • Method

    OpenWebRL trains visual web agents with online multi-turn RL using live-browser rollouts, a 0.4K-trajectory supervised warm start, multimodal context management, and trajectory-level judging.

  • Results

    68.4% average success across three live-web benchmarks, substantially outperforming existing open-source agents with a 4B backbone and a smaller training set.

  • Takeaways & Limitations

    OpenWebRL provides an actionable and reproducible foundation for open, cost-efficient, and scalable visual web-agent training.

  • Takeaways & Limitations

    Training uses GPT-4.1 as the default trajectory judge, introducing reliance on a proprietary judge model.

Abstract

from arXiv · show

Building capable visual web agents requires long-horizon reasoning, precise grounding, and robust interaction with dynamic real-world websites. Despite rapid progress, the strongest systems remain largely proprietary, while open agents still depend heavily on supervised post-training over large collections of curated web trajectories. This dependence creates a major scalability bottleneck: high-quality demonstrations are expensive to collect, and static datasets offer limited coverage of the diverse, ever-changing open web. Although online RL has shown promise for text-based agents, its potential for training visual web agents directly on live websites remains largely underexplored. In this paper, we introduce OpenWebRL, an open framework for training visual web agents with online multi-turn RL on real websites. OpenWebRL covers the full training pipeline, including scalable live-browser infrastructure, supervised initialization, multimodal context management, trajectory-level success judging, and efficient multi-turn policy optimization. Using this framework, we train OpenWebRL-4B, which establishes a new open-source state of the art on challenging live-web benchmarks. With only 0.4K initialization trajectories and 2.2K open-ended RL training tasks, OpenWebRL-4B achieves 67.0% success on Online-Mind2Web and 64.0% on DeepShop, outperforming prior open agents of similar or larger scale and remaining competitive with proprietary systems including OpenAI CUA and Gemini CUA. Beyond strong benchmark performance, we systematically study the key design choices that make online RL effective for visual web agents, and analyze how RL improves agentic reasoning. Overall, our work offers a practical path toward building more capable, reproducible, and cost-efficient open web agents. We will release our training data, models, and code to support future research.

1 Introduction

OpenWebRL addresses the gap between proprietary and open visual web agents by providing a fully open framework for online multi-turn RL directly on live websites. It replaces costly dependence on large static demonstration datasets with scalable interaction-based training and a practical recipe for compact models.

  • Motivation: Open agents lag behind proprietary systems in long-horizon planning and adaptation to dynamic websites, limiting reproducible open research.Closed models, training data, pipelines, and API-based evaluation create substantial barriers for open research.
  • Motivation: 278K curated trajectories illustrate the costly, static-data bottleneck facing supervised post-training for open web agents.Online reinforcement learning instead enables agents to improve through interaction with real websites and task outcomes.
  • OpenWebRL: OpenWebRL introduces a fully open framework for end-to-end online RL of visual web agents through direct interaction with live websites.The framework is designed to expose and systematically study the factors behind effective open-web learning.
  • OpenWebRL: 0.4K trajectories provide the supervised warm start, placing the policy in a productive exploration regime before online training.The framework uses robust browser infrastructure for large-scale parallel rollout collection.
  • OpenWebRL: OpenWebRL combines robust browser infrastructure, trajectory-level judging, and efficient multimodal context management for practical multi-turn RL.This recipe is intended to make open-web training effective for compact models.
  • Contributions: The authors release a strong 4B-scale open agent together with a detailed empirical study of the ingredients that make online RL effective.The study examines the key design choices behind online learning for visual web agents.

2 Related Work

Prior web-agent research has advanced through stronger models, agent frameworks, web-specific post-training, and curated supervision, while multimodal-agent RL is increasingly moving toward long-horizon interactive decision-making. Recent work commonly follows an SFT-then-RL paradigm, extending outcome-based reinforcement learning from static visual tasks toward multi-turn agents.

  • LLM/VLM-based Web Agents: Web-agent advances span stronger foundation models, agent frameworks, and web- or GUI-specific post-training.Systems also use curated demonstrations, synthetic data, and supervised fine-tuning for realistic browser interaction.
  • Reinforcement Learning for Multimodal Agents: Multimodal agents require long-horizon decision-making beyond static visual understanding, motivating reinforcement-learning approaches.This shift targets visually grounded environments and interactive agent behavior.
  • Reinforcement Learning for Multimodal Agents: Recent research increasingly adopts a two-stage SFT-then-RL paradigm for training multimodal and interactive agents.Outcome-based RL for language reasoning has encouraged extensions of reinforcement learning with verifiable rewards to VLMs and interactive agents.
  • Reinforcement Learning for Multimodal Agents: Early multimodal-agent RL work focused on static perception, grounding, and visual question answering tasks, while newer methods target multi-turn interaction.This progression reflects the broader move from static understanding toward interactive decision-making.

3 Preliminaries

The paper formulates multimodal web-agent training as a POMDP in which policies produce reasoning and structured browser actions from textual browser information, screenshots, instructions, and interaction history. It trains with multi-turn GRPO, sampling full trajectories, assigning trajectory-level rewards, and propagating group-relative signals across actions and generated tokens.

  • Problem Formulation: Multimodal web-agent training is formulated as a POMDP M = (S, O, A, T, R), with each task specifying a start URL and instruction.At each step, the agent observes textual browser information and a screenshot, then generates reasoning and a structured browser action from the interaction history.
  • Problem Formulation: An episode terminates when the agent calls done, exhausts the step budget, or encounters an environment failure.Task success is evaluated only after the full interaction using rule-based checks or a judge model.
  • Multi-turn GRPO: Multi-turn GRPO replaces a learned value function with group-relative advantages and samples full trajectories rather than individual responses.For each task, the policy samples a group of trajectories, each receiving a trajectory-level reward.
  • Multi-turn GRPO: Multi-turn GRPO propagates each trajectory-level signal to the action tokens generated across all turns.The objective is defined over the k-th response token yi,t,k at turn t of trajectory τi.

4 OpenWebRL: An Online RL Training Framework for Visual Web Agents

OpenWebRL trains visual web agents through an end-to-end online RL pipeline designed for scalable live-web interaction and systematic study of its key components. The framework combines supervised warm-starting, fault-tolerant browser rollouts, multimodal context management, structured tool use, success judging, and trajectory-level optimization.

  • Framework overview: OpenWebRL warm-starts small multimodal policies with supervised finetuning before iterative online RL on open-web tasks.The experiments focus on small VLMs such as Qwen3-VL-4B because online RL is computationally expensive and general-purpose models often lack web-specific knowledge.
  • Browser environment: A fault-tolerant live-browser environment handles retries, timeouts, and structured failure attribution amid dynamic website noise.The infrastructure separates unstable website behavior from model behavior, making failures diagnosable during large-scale rollouts.
  • Agent interface: The agent receives multimodal browser observations and uses 13 atomic tools spanning pointer, keyboard, navigation, tab-management, and termination actions.The done(response) tool is the sole mechanism for successfully ending an episode and emitting the final user-facing answer.
  • Context management: Recent screenshots are retained for perception while older history is compressed into textual feedback and reasoning traces to control multimodal context cost.A 30-step trajectory can exceed the context budget even for 64k-token models, while environment feedback remains preserved after screenshots are discarded.
  • Reward and judging: The reward combines protocol-format correctness with VLM-judged task success, using GPT-4.1 by default and distilling an 8B judge to reduce evaluation cost.A typical training run requires 43.2K judge API calls costing approximately $545.5; the distilled judge learns from 12.5K online rollouts and GPT-4.1 labels.
  • Policy optimization: The policy objective omits trajectory-length normalization so longer, harder tasks retain stronger learning signals.The optimization uses token-level importance ratios and masks non-assistant tokens.

5 Experiments

OpenWebRL achieves strong live-web performance with compact training and evaluation budgets, while online RL, SFT initialization, and targeted context design drive the gains. Experiments further show that rollout curricula, textual feedback, historical reasoning, and a distilled judge are important for stable and effective training.

  • Overall Performance: 68.4% average success rate: OpenWebRL-4B establishes open-source state of the art with a 4B backbone, 30-step evaluation budget, and much less training data than prior open systems.It outperforms FARA-7B by +32.9 and +37.8 points and MolmoWeb-8B by +31.7 and +21.7 points on the two benchmarks.
  • Overall Performance: 69.2% average success rate: OpenWebRL-8B reaches this score with 50-step evaluation, compared with 68.7% using 30 steps, but higher computation and wall-clock cost favor 30 steps by default.Both OpenWebRL-4B and OpenWebRL-8B surpass several proprietary agents on Online-Mind2Web and DeepShop.
  • Training Effects: 68.4% average success rate: MM-GRPO raises OpenWebRL-4B from 52.0% after SFT, versus 39.3% for the base model, yielding +16.4 points over SFT and +29.1 points over the base model.SFT provides a strong warm start, while subsequent online RL contributes the larger improvement.
  • Training Effects: ∼10% evaluation-success advantage: SFT-initialized MM-GRPO maintains this lead over base-model initialization, with the largest hard-task gain reaching +22.3 points versus +2.3 points from the base model.The results support warm-starting online RL from an SFT model rather than training directly from the base model.
  • Judge and Reasoning Analysis: 68.3 overall average score: OpenWebRL-Judge-8B nearly matches the GPT-4.1-Judge variant at 68.4 while producing reliable training signals; Qwen3-VL-8B instead triggers reward hacking.The distilled judge scores 67.3 on Online-Mind2Web and 68.7 on DeepShop and outperforms several strong judge baselines in overall reliability.
  • Ablations: 14.6, 23.7, and 8.6 points: removing historical reasoning causes the largest ablation degradation relative to the 15-step rollout baseline.Historical reasoning functions as compact memory of plans, observations, actions, and accumulated context during long-horizon navigation.

6 Additional Analysis

Additional analyses show that OpenWebRL-4B’s gains are especially informative on harder benchmarks, while failures are driven mainly by access and environment issues, followed by reasoning and interaction limitations.

  • Benchmark difficulty: Online-Mind2Web and DeepShop require longer trajectories and have lower success rates than WebVoyager, making OpenWebRL-4B’s gains especially informative.Appendix C.1 analyzes benchmark difficulty.
  • Failure analysis: 51% of 100 failed OpenWebRL-4B trajectories stem from access and environment issues, followed by 27% reasoning and constraint-tracking limitations.Visual grounding and interaction errors account for 13%, while task definition or judge issues account for 9%.
  • Failure analysis: Future progress requires stronger policies and more robust open-web agent infrastructure.This conclusion follows from the distribution of failure causes in the analyzed trajectories.

7 Conclusion … A.5 Training Details

OpenWebRL presents an open framework for training visual web agents with online multi-turn RL on live websites, combining robust infrastructure, multimodal feedback, structured actions, gated rewards, and training procedures. OpenWebRL-4B demonstrates strong live-web benchmark performance from relatively modest supervised initialization and scalable online interaction.

  • 7 Conclusion: OpenWebRL introduces an open framework for training visual web agents with online multi-turn RL on live websites.The framework is designed to support capable, reproducible, and cost-efficient web-agent training.
  • A.1 Robust Web Environment Infra: The live-web environment handles layout changes, asynchronous loading, pop-ups, redirects, anti-automation defenses, network errors, and non-deterministic page states.It uses Playwright with Chromium, isolated Kubernetes sandboxes, retries, separate timeouts, and explicit failure diagnostics.
  • A.2 Textual Environment Feedback: Textual feedback compares browser states around each action and reports execution status plus observable changes such as URLs, tabs, focus, DOM targets, and scrolling.This compact feedback helps detect failed clicks, blocked navigation, unintended tab changes, rejected inputs, and boundary scrolling.
  • A.3 Action Space: 13 atomic tools cover pointer control, keyboard input, navigation, tab management, and task termination through structured, parseable tool calls.The action space is designed for core open-web interaction operations.
  • A.4 Reward Design: The hybrid reward combines deterministic format and status checks with VLM-as-a-judge evaluation for completed trajectories that provide valid final answers.The judge uses the task, final answer, recent screenshots, and action history, while malformed or incomplete trajectories are filtered or penalized.
  • A.4 Reward Design: A trajectory receives reward 1 only when it follows the required browser-action format and is judged successful, while repeated format errors receive −1.Evaluation failures from judge timeouts or infrastructure failures are removed from optimization by zeroing their loss masks.
  • A Implementation Details: SFT trains Qwen3-VL-4B for 3 epochs with peak learning rate 10−5, while the 8B model adds 500 InSTA-v3 trajectories for 912 total.The setup uses cosine scheduling, 10% linear warmup, gradient accumulation, and global batch size 128 across 8 workers.
  • A.5 Training Details: MM-GRPO initializes from the SFT checkpoint and trains interactively on live-browser tasks using screenshots, recent context, browser feedback, and gated judge-based rewards.The rollout state contains the current observation, recent screenshot context, and environment feedback in browser tool-call format.

A.6 Evaluation Details

Evaluation uses interactive browser trajectories capped at 30 steps, with SGLang serving and Browser-Use Stealth Browsers for official success-rate judging. The protocol uses stochastic decoding by default, while evaluations without aborted tasks use deterministic decoding.

  • Execution and judging: Each evaluation task runs as an interactive browser trajectory with a maximum of 30 steps, served through SGLang.The setup uses SGLang to serve the model during evaluation.
  • Execution and judging: Official success-rate evaluation follows prior-work judging protocols and uses Browser-Use Stealth Browsers.This specifies the browser environment used for official success-rate evaluation.
  • Decoding settings: Stochastic decoding uses temperature 0.6, top-p 0.95, top-k 20, maximum response length 4096 tokens, and repetition penalty 1.0.These settings are adopted based on prior findings that stochastic sampling outperforms deterministic sampling for web agents.
  • Decoding settings: Evaluations without aborted tasks use deterministic decoding with temperature 0.0.This is distinct from the stochastic decoding used for the default success-rate evaluation.

B Benchmark Details · C Additional Experiment Results · C.1 Task Difficulty Understanding

The paper evaluates visual web agents on three live-web benchmarks with distinct domains, horizons, and judging protocols. Difficulty analysis shows that Online-Mind2Web and DeepShop require more steps and yield lower success than WebVoyager.

  • B Benchmark Details: Three online benchmarks are used for evaluation.The benchmarks are WebVoyager, Online-Mind2Web, and DeepShop.
  • B Benchmark Details: WebVoyager contains 595 curated tasks across 15 popular websites.The benchmark originates from a 643-task open-domain set; FARA curation removes infeasible or outdated tasks.
  • B Benchmark Details: WebVoyager evaluation follows the canonical protocol with GPT-4o as judge.This setup is consistent with FARA.
  • B Benchmark Details: Online-Mind2Web comprises 300 longer-horizon tasks from 136 popular websites across diverse domains.Its tasks generally require more extended multi-step reasoning and interaction sequences than WebVoyager, and evaluation uses the OSU AgentTrek protocol with o4-mini as judge.
  • B Benchmark Details: DeepShop contains 150 online-shopping tasks requiring product search, comparison, and selection under realistic constraints.Success uses Molmo-Web’s structured-output protocol with GPT-4o programmatically verifying the final response.
  • C.1 Task Difficulty Understanding: Task-difficulty analysis examines Qwen3-VL-235B-Thinking across all three online benchmarks.The analysis characterizes both task success and the number of steps required for completion.
  • C.1 Task Difficulty Understanding: 11.5 and 14.2 average steps are required on Online-Mind2Web and DeepShop, respectively, versus 8.0 on WebVoyager.Online-Mind2Web and DeepShop also exhibit lower success rates; step counts average all trajectories, including failed ones.

C.2 Results on AgentRewardBench · C.3 Comparison with Online Filtered Behavior Cloning Baseline

OpenWebRL-Judge generalizes to AgentRewardBench despite distribution shift and incomplete feedback, while MM-GRPO outperforms online filtered behavior cloning as training progresses. The results highlight strong judge performance and the importance of MM-GRPO’s dynamic online training design.

  • C.2 Results on AgentRewardBench: AgentRewardBench evaluates judges on 1,302 trajectories spanning WebArena, VisualWebArena, AssistantBench, WorkArena, and WorkArena++.The benchmark aggregates trajectories from five widely used web-agent benchmarks.
  • C.2 Results on AgentRewardBench: OpenWebRL-Judge-8B achieves the best overall F1 and Recall while maintaining competitive Precision across AgentRewardBench benchmarks.These results support the judge’s generalizability and effectiveness on a different evaluation collection.
  • C.2 Results on AgentRewardBench: OpenWebRL-Judge-4B achieves the best Precision on WebArena and WorkArena.The 4B judge performs strongly despite being evaluated across multiple benchmark domains.
  • C.2 Results on AgentRewardBench: AgentRewardBench introduces distribution shift through different action representations, unavailable action types, approximate conversions, and weaker environment feedback.Its trajectories lack the judges’ per-step feedback and instead provide partial signals such as URL changes, propagated errors, and accessibility-tree fragments.
  • C.3 Comparison with Online Filtered Behavior Cloning Baseline: Online filtered BC shares MM-GRPO’s online setup and SFT checkpoint but retains only successful trajectories, uses fixed rollout query batches, and omits dynamic sampling.The baseline performs supervised updates only on successful trajectories.
  • C.3 Comparison with Online Filtered Behavior Cloning Baseline: Although online filtered BC initially achieves competitive training rewards, its evaluation success rate without aborted tasks steadily declines throughout training.The reported decline eventually continues to the point described in Figure 9, whose excerpt is truncated here.

D Proxy-Based Analysis of Response Length

The analysis uses four regex-based lexical proxies to explain response-length growth, finding that later training amplifies early reasoning patterns rather than introducing wholly new modes. History summarization and retry-plan loops are linked to longer responses, while blocker diagnosis and condition proof capture distinct failure-handling and requirement-verification behavior.

  • Proxy methodology: Four lightweight lexical proxies identify interpretable reasoning patterns in assistant responses before the first tool call.The proxies are automatically computed with case-insensitive regular-expression families and are not intended as exhaustive semantic labels.
  • Proxy methodology: The no-proxy baseline shows smaller length growth than proxy-bearing responses, indicating that the four patterns explain a meaningful portion of per-step verbosity increases.A step is proxy-bearing when it matches any of the four proxy families; responses matching none are tracked separately.
  • Training effects: Later iterations amplify proxy patterns already present early, with history summarization and retry-plan loops associated with longer responses and heavier response-length tails.This suggests training strengthens recurring reasoning patterns rather than introducing entirely new reasoning modes.
  • Training effects: Blocker diagnosis captures CAPTCHA or access-restriction failures, while condition proof captures explanations that valid or successful outcomes satisfy requirements.The proxies distinguish failure-state diagnosis from explicit requirement-verification behavior.
  • Limitations: Regex-based proxies may miss paraphrases or create mild false positives, especially because the broad retry-plan proxy can count benign planning statements.The condition-proof proxy is more conservative and requires explicit requirement-satisfaction patterns or concrete constraints with proof.

E Error Analysis · F Prompt Templates · G Broader Impacts

The paper identifies dominant visual web-agent failures in live-web access, long-horizon reasoning, grounding, and evaluation, then specifies rollout and judging prompt templates. It also presents OpenWebRL’s release as broadly enabling while acknowledging misuse risks and describing mitigation measures.

  • E Error Analysis: 51% of 100 inspected failures involved access and environment issues, including loading failures, access restrictions, and CAPTCHA blockers.Agents often followed reasonable strategies, but environmental obstacles prevented verifiable terminal states.
  • E Error Analysis: 27% of failures reflected reasoning and knowledge limitations, especially missed constraints in long-horizon shopping or search tasks.Examples include overlooking price, color, rating, size, or product-type requirements.
  • E Error Analysis: 13% of failures were visual grounding and interaction errors, while 9% arose from task-definition ambiguity or judge errors.Interaction failures included clicking nearby elements, missing controls, and failing to detect unapplied filters; the paper calls for stronger monitoring and recovery.
  • F.1 Agent Input and Output: The rollout template supplies action history, the user request, and observations containing tab information, screenshots, and an optional A11y Tree.The agent must emit one <think> block and one or more <tool call> blocks in the specified serialized format.
  • F.2 OpenWebRL-Judge Input and Output: The judge evaluates the task instruction, action history, screenshots, and result response, treating screenshots as the strongest evidence when sources conflict.It must explain its reasoning and return either SUCCESS or NOT SUCCESS.
  • G Broader Impacts: The released framework, training data, and 4B-parameter OpenWebRL-4B lower barriers for researchers studying online RL for visual agents.The paper emphasizes access to a previously proprietary research area and reduced reliance on expensive human demonstrations.
  • G Broader Impacts: More capable visual web agents could enable prohibited scraping, bot-detection circumvention, inauthentic engagement, and harmful long-horizon state-changing actions.The paper mitigates these risks through benchmark environments, rate limiting, bounded release scale, documentation, and guidance to respect site terms and applicable law.

H Additional Case Studies

The case studies demonstrate that OpenWebRL handles long-horizon shopping tasks with multiple constraints through extended web interaction. The trajectories show concrete subgoal decomposition and reflective monitoring during execution.

  • Representative Tasks: The agent completes long-horizon shopping tasks requiring multiple constraints, including product type, color, size, price, seating capacity, material, and rating.The examples target a top-selling brown men’s Birkenstock clog in size 10–10.5 and a cheapest black leather sofa with at least three seats and at least four stars.
  • Agent Capabilities: The agent decomposes high-level shopping requests into subgoals such as category navigation, filtering, candidate inspection, option selection, and locating the purchase control.
  • Agent Capabilities: The agent monitors whether actions achieve the intended state and revises its strategy when they do not.
Loading 2606.02031v2…