Source-linked AI summary
WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks
Hao Bai, Alexey Taymanov, Tong Zhang, Aviral Kumar, Spencer Whitehead
TL;DR
Visual web-agent training needs larger, more varied task sets with meaningful evaluation signals, while rollout sampling remains a major efficiency bottleneck. WebGym addresses these needs with a large open-source environment, procedural rubric-based task construction, and faster rollout-oriented training infrastructure, with experiments reporting improved sample efficiency and generalization to unseen websites.
Problem
Visual web-agent training requires substantially larger and more varied task sets with meaningful learning signals, while rollouts can take longer than learning updates.
Method
WebGym combines a large open-source visual-web training environment with procedural task construction, structured evaluation rubrics, and an efficiency-focused rollout system for scaling agent training.
Results
WebGym experiments find that explicit memory supports long-horizon tasks, penalizing repeated ineffective actions improves sample efficiency, and domain diversity improves generalization to entirely unseen websites.
Takeaways & Limitations
Scaling task breadth and difficulty alongside efficient rollouts supports training visual web agents that generalize beyond websites encountered during training.
Takeaways & Limitations
Rubric-based evaluations can be overly strict, reducing sample efficiency, and this regression is difficult to eliminate with LLM-based rubrics.
Abstract
from arXiv · showhide
We present WebGym, the largest-to-date open-source environment for training realistic visual web agents. Real websites are non-stationary and diverse, making artificial or small-scale task sets insufficient for robust policy learning. WebGym contains nearly 300,000 tasks with rubric-based evaluations across diverse, real-world websites and difficulty levels. We train agents with a simple reinforcement learning (RL) recipe, which trains on the agent's own interaction traces (rollouts), using task rewards as feedback to guide learning. To enable scaling RL, we speed up sampling of trajectories in WebGym by developing a high-throughput asynchronous rollout system, designed specifically for web agents. Our system achieves a 4-5x rollout speedup compared to naive implementations. Second, we scale the task set breadth, depth, and size, which results in continued performance improvement. Fine-tuning a strong base vision-language model, Qwen-3-VL-8B-Instruct, on WebGym results in an improvement in success rate on an out-of-distribution test set from 26.2% to 42.9%, significantly outperforming agents based on proprietary models such as GPT-4o and GPT-5-Thinking that achieve 27.1% and 29.8%, respectively. This improvement is substantial because our test set consists only of tasks on websites never seen during training, unlike many other prior works on training visual web agents.
1 Introduction
WebGym addresses the difficulty of scaling reinforcement learning for visual web agents by combining a large, diverse task set with efficient rollout collection. Training on WebGym improves a strong vision-language model across easy to long-horizon tasks in strongly out-of-domain settings.
- Motivation: Visual web-agent RL is difficult because rendered interfaces require fine-grained reasoning, success signals are often ambiguous, and rollouts are slow and inefficient.These challenges especially inhibit scaling to diverse and long-horizon tasks.
- WebGym: WebGym provides nearly 300,000 tasks spanning diverse domains, difficulty levels, and evaluation criteria for training visual web agents.The task set is designed to support generalization across challenging scenarios.
- WebGym: The asynchronous rollout system collects 1,800 trajectories averaging 13.2 steps in 30 minutes using 128 CPUs and 24 NVIDIA H100 GPUs.This system achieves a 4-5× speedup over traditional synchronous rollout systems.
- Results: Scaling task-set breadth, depth, and size produces significant improvements across the studied dimensions.The paper evaluates scaling along all three task-set dimensions.
- Results: A simple REINFORCE-like algorithm brings Qwen3-VL-Instruct-8B to state-of-the-art performance on its scale across easy to long-horizon, strongly out-of-domain tasks.The reported gains span a wide spectrum of task horizons and difficulty levels.
2 Related Work
Prior work includes visual web-agent benchmarks, online RL for language and vision-language models, and emerging training environments. WebGym combines large-scale task construction, rubric-based evaluation, and asynchronous rollouts to address limitations in task diversity, hardness, and speed.
- Training environments: Existing realistic-web training environments provide large-scale training infrastructure, but their task sets remain limited in diversity and hardness and their rollout systems are not speed-optimized.WebGym is positioned as addressing both limitations.
- Online reinforcement learning: Online RL has improved language-model and vision-language-model agents across software engineering, mathematics, deep research, and general reasoning.Prior results include methods as simple as REINFORCE.
- WebGym: WebGym reports 292,092 tasks across 127,645 websites, exceeding the listed prior task sets in scale.The task set is built from 10 widely used benchmarks and environments.
- WebGym: WebGym constructs a large, diverse task set and trains agents with on-policy RL from collected rollouts.Its task construction expands diversity through rubric-derived fact groups and controlled decomposition into atomic and compositional variants.
3 Tasks in WebGym and Evaluation Protocol
WebGym constructs a large, diverse training task set from rubric-based fact groups and evaluates trajectories against explicit criteria. Its decomposition and rubric design provide varied difficulty levels and denser, more consistent reward signals, though overly strict rubrics can reduce recall and sample efficiency.
- Motivation: Online RL requires diverse websites, varied task difficulties, and verifiable evaluators, but existing web-agent resources are often small, static, or limited in coverage.These constraints motivate WebGym’s task-set and evaluation design.
- Task construction: WebGym builds tasks by annotating seed tasks with fact-group rubrics and generating decomposed subtasks from valid fact-group combinations.Construction begins with sourced seed tasks, adds evaluation criteria, and uses controlled decomposition to expand breadth and depth.
- Task decomposition: Decomposition requires at least 2 fact groups and one group with at least 3 facts, producing strictly easier tasks that remain consistent with the original objectives.Proper subsets are used, while the full fact-group set is excluded.
- Train-test split: All OOD tasks come from entirely unseen websites relative to training, supporting evaluation of website-level generalization.The train-test split is constructed at the website level rather than by holding out an entire source benchmark.
- Task statistics: 7.8, 9.9, and 11.9 average steps are required for easy, medium, and hard tasks, respectively, while KDE modes rise from 3.7 to 4.9 to 5.5 steps.The trajectory-length statistics align increasing difficulty with longer answered trajectories.
- Evaluation: Rubric-guided evaluation improves accuracy and precision across tested evaluators, but stricter rubrics can reduce recall and sample efficiency.The evaluator produces reward only when all criteria are satisfied, and some LLM-generated rubrics are overly strict.
4 Rollout System in WebGym
WebGym replaces synchronized rollout collection with an asynchronous server-client system that streams work as resources become available. This design reduces burst-idle inefficiency and delivers 4–5× faster collection, with scaling eventually limited by GPU inference throughput.
- Motivation: Slow rollout generation is a primary bottleneck for scaling reinforcement learning with visual web agents, especially when simulation resources are limited.WebGym targets this bottleneck rather than changing the RL algorithm.
- System design: The asynchronous system replaces synchronized batch collection with process-isolated browser rollouts and immediate inference when resources become available.A server hosts simulation environments, while a client organizes the rollout loop without global step or episode barriers.
- Synchronous baseline: Synchronous barriers force faster sessions to wait for slower ones, creating burst-idle CPU and GPU utilization across variable-length web tasks.This behavior arises because sessions advance together and wait for the slowest session at synchronization points.
- Benchmarking: 4–5× speedup is achieved over the synchronous framework; with 64 CPUs, WebGym collects 1.8k rollouts in 48.6 minutes versus 264 minutes synchronously.The benchmark collects 1,800 trajectories averaging 13.2 steps under fixed GPU resources.
- Resource scaling: Rollout time decreases as CPUs increase until about 128 CPUs, after which inference throughput plateaus because three H100 GPU nodes become the primary bottleneck.Additional CPUs provide minimal benefit in this GPU-limited regime.
- Resource scaling: GPU throughput scales sub-linearly but close to the desired linear speed-up as the number of GPU nodes increases with sufficient CPU resources.The framework is benchmarked with 768 CPUs while varying H100 GPU nodes.
5 Training Web Agents with WebGym
WebGym training improves visual web-agent performance through explicit memory, repetition penalties, and carefully scaled task composition. Broad domain coverage and balanced difficulty sampling support stronger generalization, while aggressive emphasis on harder tasks can cause overfitting and regression.
- Design Choices for Agent Training: Explicit memory mechanisms improve performance after RL on long-horizon tasks requiring information retention across steps.The memory prompt addresses limitations in retaining and reusing information gathered earlier in a trajectory.
- Design Choices for Agent Training: Filtering repeated actions that produce identical screenshots makes RL substantially more sample efficient.Standard RL reduces ineffective repetition but does not fully eliminate it, motivating the repetition penalty.
- Scaling the Task Set: Domain breadth matters because excluding domains slows improvement and lowers final success, limiting generalization to unseen websites and subdomains.This degradation is not attributed to oversampling, since collected trajectories remain far fewer than the remaining training tasks.
- Scaling the Task Set: Training exclusively on easy tasks is more stable than overemphasizing medium and hard tasks, which can produce plateaus or mild regression after about 36k trajectories.The authors connect this pattern to the larger and more diverse pool of easy tasks, which reduces repeated exposure to narrow domains.
- Scaling the Task Set: Uniform sampling yields the highest overall performance by adding medium- and hard-difficulty training signals without substantially changing hard-task test performance.It improves medium-difficulty results over easy-only training, while the two settings show no significant difference on hard tasks.
- Scaling Agent Performance: 42.9% success rate surpasses the previous best checkpoint’s 38.2% after tightening the step budget.The authors hypothesize that tighter budgets regularize interaction by discouraging low-yield late exploration.
6 Discussion and Conclusion
WebGym combines a large open-source visual-web training environment with asynchronous rollouts and design choices that improve efficiency and generalization. A simple REINFORCE algorithm achieves strong generalization to unseen websites, while rubric strictness remains a limitation.
- WebGym uses procedural task construction to provide domain breadth and difficulty depth while keeping decomposed tasks semantically coherent.
- An asynchronous rollout system removes synchronization barriers and accelerates data collection for on-policy reinforcement learning.
- Explicit memory, repetition penalties, domain diversity, uniform difficulty sampling, and shorter horizons improve long-horizon retention, sample efficiency, generalization, or final performance.
- A simple REINFORCE algorithm enables strong generalization to entirely unseen websites and outperforms proprietary models.
- LLM-generated rubrics can be overly strict, slightly reducing sample efficiency during training.
A Raw Experimental Results
The appendix reports raw experimental values and defines the ablation table’s key configuration labels for interpreting the trained-agent results.
- The paper provides raw values for all experiments presented in Section 5.
- Table 4 reports raw ablations and performance results for trained agents on the out-of-distribution test set.
- Table 4 labels training horizon, difficulty sampling, domain coverage, memory prompting, and repetition penalties as experimental factors.
B.1 Rollout System: Operation-specific Local Request Queue
WebGym’s rollout system models web-agent trajectories as interleaved CPU- and GPU-bound operations and uses operation-specific local queues to avoid resource starvation. Under extreme CPU load, the local queue significantly reduces crashes.
- WebGym assigns different operation types to local queues without prioritizing one operation type over another.
- The queue system balances CPU and GPU usage through operation-specific request handling.
- Trajectory rollout includes navigation, screenshots, action proposal, action execution, and reward evaluation across CPU- and GPU-bound stages.
- A global FIFO queue makes later screenshot and execution requests wait behind earlier operation types, starving the GPU during multi-step rollouts.
- The local queue prevents GPU starvation by propagating balanced request processing through the rollout pipeline.
- Under extreme CPU load with 64 CPUs and 256 environments, the local queue significantly decreases the crash percentage.
B.2 Rollout Sampler: Anti-blocking
WebGym detects websites that block frequent requests and maintains a list of blocked sites to support rollout anti-blocking.
- WebGym uses an LLM to detect websites that block the agent after frequent requests from the same server.
- Detected blocked websites are maintained in a list for the rollout system.
C Modeling the Markov Decision Process On Long-Horizon Web Navigation Tasks
Web navigation is modeled as a partially observable decision problem because agents receive incomplete observations and must use history to infer appropriate actions. Windowed histories fit VLM context limits but can cause repetitive behavior on long-horizon tasks, motivating memory-based context management.
- POMDP formulation: A POMDP represents web tasks with latent states, actions, transitions, rewards, observations, an observation model, and a discount factor.Because the agent cannot directly observe the latent state, its policy maps belief states to actions.
- Windowed history: Recent web-agent models approximate the true state with a windowed history of the last n observations, including screenshots and webpage metadata.This representation is designed to fit within VLM context limits.
- Windowed history: For long-horizon tasks, a small observation window can oversimplify the problem because policies may revisit the same state without access to their full history.The supplied passage states that the window size n is usually set to 3.
- Failure mode: Limited history can produce repetitive action sequences because the policy does not know what it has already completed.This failure motivates richer state representations for sequence-level reinforcement learning.
- Memory-based context: Full-history observations exceed practical VLM context limits, so prior work uses memory tokens that recursively compress task-relevant information across steps.The memory is updated from the current observation, previous memory, and task instruction, then included in the next observation.
- Evaluation context: Rubric examples show a correlation between task-instruction length and the number of evaluation criteria.The passage reports this relationship without quantifying its strength.
D.2 Comparing Rubric-based Evaluation with Task-based Evaluation
Rubric-based evaluation decomposes task success into explicit criteria, enabling more comprehensive checks than evaluating only the final task response. The examples also show that screenshot evidence can support some criteria while leaving others unverifiable.
- Rubric-based evaluation: Rubric-based evaluation checks details for each criterion, producing many wins over evaluation based only on the task response.The supplied passages contrast criterion-level checking with task-description-based evaluation.
- Evaluation comparison: Figure 12 illustrates a trajectory where human evaluation labels the example NOT SUCCESS, while the evaluation comparison concerns vanilla and rubric-based procedures.The figure is explicitly described as a trajectory comparison.
- Evaluation comparison: Task-based evaluation asks whether the response identifies the eliminated groups, whereas rubric evaluation additionally checks the reasons for each group.The task and its criteria are stated explicitly in the Whole30 example.
- Evidence grounding: Some response claims about Whole30 motivations are corroborated by screenshots, showing that rubric judgments can distinguish supported from unsupported details.The supplied passage cites identifying food sensitivities and improving the relationship with food as corroborated motivations.
- Example comparison: The same example shows that a response can correctly identify food groups while failing to provide distinct, screenshot-grounded reasons for eliminating each group.The evaluation distinguishes factual identification from evidence-supported explanation.
- Example comparison: In the Whole30 example, rubric evaluation verifies the eliminated food groups but marks the task unsuccessful because group-specific reasons are not supported by screenshots.The final verdict is NOT SUCCESS.
E Prompts
The prompts define how agents generate browser actions and how tasks, trajectories, and evaluation criteria are represented. They specify structured outputs, available actions, evaluator decisions, and atomic rubric facts.
- E.1 Agent Action Prompt: Supported browser actions include clicking, typing, scrolling, waiting, going back, navigating to HTTPS URLs, and answering.Typing automatically clicks, enters text, and presses Enter; CAPTCHA screens trigger navigation elsewhere.
- E.1 Agent Action Prompt: Action arguments specify coordinates, text, scroll direction, wait time, or navigation URL according to the selected action.Coordinates use a 0–1000 range and are required for clicking and typing.
- E.1 Agent Action Prompt: The agent-action prompt requires structured outputs containing memory, progress, intention, action, and one tool-call block when memory is enabled.A no-memory variant instead requires only an action sentence and one tool-call block.
- E.1 Agent Action Prompt: The prompt presents screenshots, task instructions, and initial websites, then has the agent generate the next UI action and tool call.The examples show navigating National Rail by clicking and typing station information.
- E.2 Evaluator Prompt: Keypoint Detection: The evaluator labels an image YES when it contains any task-related content and NO only when it is completely irrelevant.The evaluator must provide one sentence of reasoning followed by a YES or NO decision.
- E.5 Task Set Generation Prompt: Evaluation Criteria Proposal: Task rubrics group all required facts into atomic verification units, use AND logic by default, and mark a trajectory correct only when every fact is verified.Overall difficulty equals the total number of facts across groups, and unspecified facts must not be invented.
F Hyper-parameters
WebGym’s system uses four components—environment management, rollout data collection, model training, and the agent model—with hyperparameters tuned for performance, scalability, and sample efficiency.
- System components: WebGym’s system comprises environment management, data collection, model training, and the agent model.The paper states that hyperparameters for these components are tuned to balance performance, scalability, and sample efficiency.
- Hyperparameter reporting: Table 6 reports the final hyperparameters for all experiments and bolds the experimentally optimal option.The supplied caption identifies the table’s scope and formatting convention.