PaperlayerView exact revision on arXiv

Source-linked AI summary

AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents

Hao Bai, Rui Yang, Chenlu Ye, Spencer Whitehead, Aviral Kumar, Tong Zhang

arXiv:2606.05597v3cs.LG

TL;DR

Multi-step visual web-agent RL wastes compute through synchronous GPU idleness and unnecessarily long trajectories. AsyncWebRL combines asynchronous rollout infrastructure with decoupled off-policy correction and constant trajectory normalization, achieving faster training and stronger WebGym OOD performance. The method reaches 45.4% versus 42.9% prior SOTA while contracting trajectories at matched reward.

  • Problem

    Multi-step visual web-agent RL is compute-intensive because synchronous execution idles GPUs and trajectory-length normalization contributes to excessive trajectory and token usage.

  • Method

    AsyncWebRL overlaps rollout, gradient updates, and policy refresh with an everlasting rollout pool and lightweight screenshot handling, while replacing 1/|τ_i| with 1/k and decoupling importance-sampling factors.

  • Results

    45.4% versus 42.9% on WebGym’s OOD split, with a 2.4–2.9× end-to-end speedup and a further 1.80× per-step speedup at matched reward.

  • Takeaways & Limitations

    The combined system and algorithm establish a new open state of the art for visual web-agent RL while improving training efficiency and trajectory compactness.

  • Takeaways & Limitations

    Evaluation reports single-seed results throughout, relying on WebGym’s analysis that final test performance is sufficiently stable across runs.

Abstract

from arXiv · show

Training vision-language web agents with multi-step RL is compute-intensive, with two dominant forms of inefficiency: idle GPUs in synchronous RL, and trajectories that use more steps and tokens than necessary. We present AsyncWebRL, which addresses both. On the system side, an asynchronous design overlaps rollout, gradient update, and policy refresh across iterations, paired with two web-agent-specific adaptations, namely an everlasting rollout pool and lightweight screenshot handling, that together deliver up to a $2.9\times$ end-to-end training-throughput speedup over the previously fastest open synchronous pipeline (WebGym). On the algorithmic side, we identify the per-trajectory normalizer $1/|τ_i|$ in multi-step GRPO as the root cause of trajectory-level and token-level inefficiency: because failures are systematically longer than successes, it down-weights the negative gradient on failed tokens, so the policy keeps producing verbose memory schemas. Replacing $1/|τ_i|$ with a constant $1/k$ breaks this coupling, contracting trajectories while preserving aggregate success. Together, these contributions set a new open-source state of the art on the WebGym out-of-distribution test split (+5.8% relative over the 42.9% prior best), with the largest gains on the harder slices (+42% relative on Medium, +48% relative on Hard).

1 Introduction

AsyncWebRL targets both GPU idleness and excessive trajectory/token usage in multi-step visual web-agent RL. It combines an asynchronous system with a constant-normalization algorithmic fix, improving throughput and WebGym OOD performance.

  • Motivation: Multi-step visual web-agent RL is compute-intensive, making trajectories consumed per wall-clock hour central to fixed-budget performance.Training uses hundreds of GPU-hours and concurrent browser sessions.
  • Motivation: Existing systems do not jointly provide visual, multi-step, and fully asynchronous RL, while screenshot traffic can overwhelm shared data stores.The visual rollout payload may contain tens of high-resolution screenshots per trajectory shared across workers.
  • System contribution: AsyncWebRL addresses system inefficiency with lightweight screenshot handling and an everlasting rollout pool that keeps workers alive across iteration boundaries.Image tensors remain outside the shared store, while lightweight references are routed between workers and the trainer.
  • Algorithmic contribution: The framework uses decoupled importance sampling to separate rollout staleness from current-update movement and center clipping on a proximal policy.This roughly halves clip-trigger rates and substantially speeds reward improvement.
  • Algorithmic contribution: Failures average 12.5 steps versus 5.1 for successes, causing 1/|τ_i| to attenuate failed-token gradients by approximately 2.4×.Replacing 1/|τ_i| with constant 1/k targets trajectory- and token-level inefficiency while preserving aggregate success.
  • Results: 2.4 to 2.9× end-to-end speedup and +5.8% relative OOD improvement over WebGym’s 42.9% prior best establish the reported headline gains.The largest harder-slice gains are +42% relative on Medium and +48% relative on Hard.

2 Related Work

AsyncWebRL bridges separate lines of work in asynchronous RL and multi-step visual-agent training. Its algorithmic analysis extends length-normalization concerns from single-turn responses to multi-step trajectories.

  • System: visual, multi-step, and async RL: Open-source systems largely split between asynchronous text or single-turn visual RL and synchronous multi-step visual-agent RL.Single-turn systems typically handle at most one image per training example, unlike multi-turn web rollouts.
  • System: visual, multi-step, and async RL: Tens of high-resolution screenshots per trajectory and hundreds of concurrent rollouts can exhaust shared data-store budgets and induce synchronization bubbles.AsyncWebRL targets sustained screenshot traffic and iteration-level synchronization in high-cadence web training.
  • Algorithm: loss shape under length asymmetry: Dr. GRPO removes single-turn token-number normalization 1/|y_i| to correct length bias between long and short responses.AsyncWebRL uses this diagnosis as a reference for its multi-step extension.
  • Algorithm: loss shape under length asymmetry: Multi-step GRPO instead uses 1/|τ_i|, which couples per-token gradient scale to trajectory length in web-agent training.Horizon-dominated failures and growing additive-memory responses create a self-reinforcing failure mode.
  • Algorithm: loss shape under length asymmetry: AsyncWebRL applies the corresponding step-level fix to break the length-coupled loop at its source.The paper frames this as extending the diagnosis one level up from single-turn token granularity.

3 AsyncWebRL

AsyncWebRL overlaps rollout, optimization, and policy refresh while reducing visual data movement and length-coupled loss attenuation. Its core algorithm decouples off-policy correction and replaces trajectory-length normalization with a constant factor.

  • System design: Fully asynchronous execution keeps rollout workers alive across iteration boundaries and overlaps rollout, gradient updates, and policy refresh.Episodes restart immediately on the same worker, while updated weights are broadcast during ongoing rollout.
  • System design: Lightweight screenshot handling keeps image tensors out of the shared store and routes only references, preventing disk spill from concurrent high-resolution rollouts.Together with persistent workers, this removes waiting for the slowest trajectory and rollout-pool rebuild warm-up.
  • Decoupled off-policy correction: Decoupled correction splits πθ/π_behave into rollout-staleness π_prox/π_behave and current-update πθ/π_prox factors.PPO clipping is centered on π_prox rather than a single ratio confounded by both movements.
  • Removing trajectory-length normalization: Replacing 1/|τ_i| with constant 1/k gives each rollout weight |τ_i|/k, restoring full per-token gradient weight on long failures.Here k is the Easy-difficulty horizon, fixed at 10 throughout the paper.
  • Loss: The loss shares one trajectory-level advantage across all tokens, combining constant normalization and decoupled clipping in the standard GRPO surrogate.With G rollouts per task, each rollout has |τ_i| steps and each step has |τ_i,j| tokens.
  • Baseline: RAFT++ retains 1/|τ_i| but filters for successful trajectories, effectively behavior-cloning positives without contrastive signal from below-average rollouts.It serves as a contrasting off-policy baseline.

4 Experiments

Experiments evaluate AsyncWebRL on WebGym’s held-out OOD split using two Qwen3-VL-8B variants and compare synchronous WebGym with async RAFT++ and the full method. The full system improves harder-slice success while substantially increasing throughput and reducing trajectory-related computation.

  • Setup: WebGym provides roughly 290k training tasks across 128k websites and a 1,167-task OOD test split with Easy, Medium, and Hard levels.The OOD websites do not appear in training, and experiments use 10, 20, and 30-step per-difficulty horizons.
  • Setup: Experiments compare the prior synchronous WebGym pipeline, AsyncWebRL-RAFT++, and AsyncWebRL (full) across Instruct and Thinking Qwen3-VL-8B variants.The full method combines multi-step GRPO, decoupled importance sampling, and the constant 1/k normalizer.
  • Test performance: 45.4% average success on Instruct is achieved by AsyncWebRL (full), versus 42.9% for WebGym, a +5.8% relative improvement.The relative gains are +42% on Medium and +48% on Hard, while Easy improves by +2.9% relative.
  • Efficiency: 2.4 to 2.9× end-to-end speedup is obtained over synchronous WebGym, with AsyncWebRL producing about 3,100 trajectories per hour versus approximately 1,300 and 1,050.The comparison includes the distributed inference server and in-memory weight broadcast.
  • Efficiency: Mean per-token off-policy gap stays near 1.5 and the maximum near 2.0 with maximum staleness η = 2 throughout training.The reported values remain below the cap while GPUs stay busy.
  • Efficiency: The constant 1/k normalizer preserves matched test reward while shortening trajectories and reducing total per-step wall-clock time by 18–19%, versus 4–5% for 1/|τ_i|.The constant-normalizer run also uses fewer tokens per step; standard normalization reaches around 240 tokens per step.

5 Dynamics Analysis

The analysis attributes trajectory and token inefficiency to length-coupled normalization in multi-step GRPO, and shows that replacing 1/|τ_i| with 1/k removes verbose-memory dynamics while preserving reward. Ablations support the loss factor as the driver rather than the prompt or surrounding algorithm.

  • Mechanism: Failures average 12.5 steps versus 5.1 for successes, so 1/|τ_i| attenuates failed-token gradients by approximately 2.4×.The resulting length asymmetry pulls early training toward longer rollouts.
  • Main dynamics: At matched test reward, constant 1/k produces fewer steps per trajectory and fewer tokens per step than 1/|τ_i|.The standard normalizer also yields lower per-token entropy and longer trajectories in both Qwen3-VL-8B variants.
  • Memory dynamics: Under 1/|τ_i|, Memory accumulates generic keys and changes continually, whereas constant 1/k preserves task-anchored schemas close to the Base model.The table compares first-to-last key-set equality, consecutive-step stability, and generic-placeholder frequency.
  • Consequences: The constant 1/k fix restores full per-token weight on long failures, eliminating memory bloat, longer responses, lower entropy, and generic schemas.The observed response-length increase under 1/|τ_i| is approximately 33%.
  • Ablations: RAFT++ shows the same length-coupled Memory drift under 1/|τ_i|, though more mildly, despite differing in other algorithmic components.Its success-conditioned filter mutes the effect on failures because failed rollouts provide no direct gradient.
  • Ablations: A compressive prompt leaves Add/Del/Mod rates elevated with 1/|τ_i|, while doubling horizons from 10/20/30 to 20/40/60 amplifies the predicted Memory-growth effect.These interventions respectively show that prompt rewriting does not reach the cause and longer failures worsen the symptoms.

6 Conclusion

AsyncWebRL is a fully asynchronous multi-step RL framework for visual web agents. Its system design improves end-to-end speed, while constant 1/k contracts trajectories at matched reward and decoupled importance sampling achieves the reported WebGym OOD result.

  • Conclusion: 2.4–2.9× end-to-end speedup over the fastest prior open pipeline comes from the everlasting rollout pool and lightweight screenshot handling.The framework is fully asynchronous for visual web-agent training.
  • Conclusion: 1.80× per-step speedup follows from replacing 1/|τ_i| with constant 1/k while maintaining matched reward.The replacement contracts trajectories rather than changing what the policy can solve.
  • Conclusion: 45.4% versus 42.9% establishes the reported open state of the art on WebGym’s out-of-distribution split.The conclusion attributes the result to the decoupled importance-sampling correction.

A.2 Why GRPO Outperforms RAFT++

GRPO outperforms RAFT++ because it directly downweights below-average trajectories, while RAFT++ filters for successful rollouts and lacks that contrastive signal. Memory edit rates further indicate that replacing 1/|τ_i| with 1/k removes the loss-driven tendency toward longer schemas.

  • Algorithmic explanation: GRPO’s group-relative advantage pushes probability down on below-average trajectories, whereas RAFT++’s r > 0 filter only reinforces successful trajectories.This provides GRPO with a contrastive signal that RAFT++ lacks.
  • Algorithmic explanation: RAFT++’s entropy collapses from 0.057 to 0.010, while GRPO’s settles near 0.025, with both runs near 0.3% ϵ-clip rates.The similar clipping rates indicate that clipping mechanics do not explain the entropy gap.
  • Memory edit operations: GRPO with 1/|τ_i| edits Memory more aggressively than RAFT++, while a compressive prompt reduces but does not remove the elevated edit rates.The comparisons are based on per-step Add/Del/Mod operations in Table 3.
  • Memory edit operations: Replacing 1/|τ_i| with 1/k returns Memory edit rates to roughly Base levels, and below Base on successful trajectories.This result supports loss aggregation, rather than prompt wording alone, as the root cause of the memory drift.

A.4 Batch Size Sensitivity

Batch size changes the early optimization path but not the full-horizon test reward or the qualitative length-coupled memory-drift behavior. Batch size 128 therefore remains a sensible default under the tested configuration.

  • Experimental setting: Batch size 128 is the canonical Instruct GRPO setting used throughout the paper.The sensitivity rerun used batch size 32 as the comparison condition.
  • Test-reward sensitivity: Batch size 32 reaches ≈0.38 test reward several hours before batch size 128, then intersects it around 10 hours at ≈0.40.All other hyperparameters were held fixed.
  • Test-reward sensitivity: Over the full training horizon, batch size is essentially neutral on test reward.The early advantage from more frequent updates does not persist through the longer run.
  • Variance: Batch size 32 produces visibly higher per-iteration variance in training reward, entropy, and steps per trajectory than batch size 128.The wider variation appears throughout wall-clock training.
  • Qualitative behavior: At batch size 32, entropy collapse, falling steps per trajectory, and rising tokens per step still reproduce.These behaviors therefore remain associated with the loss shape under this batch-size change.

A.5 Within-Run Per-Step Time Contraction

The constant 1/k loss causes per-step computation to contract within a single run, unlike the nearly flat timing under 1/|τ_i|. This contraction accompanies shorter trajectories and fewer tokens per step.

  • Measurement: The reported ratio divides average time per training step over the last 20 steps by the average over the first 20.Total-time measurements include rollout collection, evaluation, and checkpointing, with setup-affected initial steps excluded.
  • Timing result: 0.81–0.89 is the within-run per-step-time ratio under 1/k, an 11–19% reduction from the start to the end of training.The ratio compares the last 20 steps with the first 20 on the same hardware.
  • Timing result: 0.95–0.97 is the corresponding per-step-time ratio under the standard 1/|τ_i| loss, indicating essentially flat timing.Both gradient-update-only and total-time measurements agree.
  • Mechanism: The 1/k policy emits shorter trajectories and fewer tokens per step as training proceeds.This trajectory contraction explains why each later gradient update consumes less compute.

B Implementation Details

AsyncWebRL’s implementation combines asynchronous rollout infrastructure with memory-efficient screenshot transport and stabilized GRPO optimization. The appendix also documents the measurements, prompts, and loss variants used to analyze trajectory contraction and memory behavior.

  • System: AsyncWebRL stores screenshots in a dedicated in-memory actor and routes lightweight references instead of raw image tensors through the shared RPC store.This avoids the object-store and disk-spill bottleneck created by concurrent high-resolution rollouts.
  • Memory analysis: Table 3 reports per-step Add, Del, and Mod operations on Memory across the first 30 steps, split by trajectory outcome.The operations count inserted, removed, and value-changed keys between consecutive steps.
  • Sensitivity analysis: Figure 10 compares batch sizes 128 and 32 against wall-clock hours using test reward, training reward, entropy, steps per trajectory, and response tokens per step.Batch size 128 is the canonical setting, with other hyperparameters held fixed in the comparison.
  • Optimization: Dual-clipped GRPO caps the surrogate contribution from below when the advantage is negative, preventing rare low-probability tokens from producing unbounded negative terms.The appendix describes this as an extension of PPO-style clipping with an absolute lower bound.
  • Optimization assumption: The appendix treats horizon-exhausting trajectories as failures, accepting the loss of some potentially useful signal because their actions are considered suboptimal when another group trajectory succeeds.This setting motivates penalizing unnecessary long-horizon behavior rather than exempting it.
  • Trajectory analysis: Sample responses compare 1/|τ_i| with 1/k and show that extra tokens under 1/|τ_i| occur in low-entropy boilerplate rather than decision-bearing positions.The comparison connects memory-schema drift with token-level entropy behavior.

C.1 Instruct Last-Step Response

The instruct model often completes tasks successfully while producing either concise responses or substantially longer memory and progress traces. Removing length normalization is associated with a much shorter successful response in the leather-jacket example.

  • C.1 Instruct Last-Step Response: 178 tokens were produced by the base model for a successful leather-jacket task, ending with a $395.00 answer.The response included memory entries marking both task components finished before issuing the final answer.
  • C.1 Instruct Last-Step Response: 174 tokens were produced by RAFT++ for a successful romantic-dinner dessert search, which identified a fruit tart recipe.The rollout marked the website search and task completion as finished.
  • C.1 Instruct Last-Step Response: 150 tokens were produced by GRPO without length normalization for a successful leather-jacket task, compared with 589 tokens for the length-normalized medicine-search example.The shorter rollout recorded cookies accepted, navigation completed, and the jacket price found.
  • C.1 Instruct Last-Step Response: 589 tokens were produced by GRPO with length norm for a successful medicine-search task, whose trace remained incomplete and encountered an error page.The rollout recorded an in-progress review and an unfinished summary despite obtaining an NIH article summary.

C.2 Thinking Last-Step Response

Thinking-model traces can be highly verbose even when they eventually complete tasks, with long reasoning and memory structures documenting navigation, failures, and intermediate findings. The examples also show that successful completion may coexist with unresolved subtasks or inaccessible sources.

  • C.2 Thinking Last-Step Response: 387 tokens were produced by the base model while documenting a completed Food Network soup search with ingredients and a 12-minute cooking time.The trace included an extended reasoning block and structured memory describing the selected Simple Egg Drop Soup recipe.
  • C.2 Thinking Last-Step Response: $3,095 was identified as the sale price for ThinkTokyo.com on HugeDomains, with “Buy now” and “Start payment plan” options.The progress state marked status, price, and listing site as finished while the summary subtask remained unfinished.
  • C.2 Thinking Last-Step Response: 612 tokens were produced by GRPO without length normalization for a Food Network soup task whose answer cited ingredients and total cooking time.The trace described the current page and prior successful navigation before providing the final response.
  • C.2 Thinking Last-Step Response: 911 tokens were produced by GRPO with length norm while attempting to identify Central Park’s most common tree species after access problems.The trace switched from the NYC Tree Map to Wikipedia and treated American elms as the closest available identification, although the evidence did not explicitly state “most common.”

C.3 Token-level Entropy

Token-level entropy annotations expose how the policy’s confidence changes within a rollout. The reported example shows mean entropy collapsing from the first turn to the last, while each high-entropy token records its sampled-token probability and alternatives.

  • C.3 Token-level Entropy: Psampled denotes the probability assigned to the token actually sampled by the policy, while bracketed values list top-k alternative tokens and their probabilities.The annotation is applied to each high-entropy token in the qualitative visualization.
  • C.3 Token-level Entropy: 0.0297 nats at the first turn fell to 0.0077 nats at the last turn in one rollout trained with the 1/|τ_i| normalizer.The visualization uses token-background opacity for entropy and annotates high-entropy tokens with the chosen token’s probability and top alternatives.
  • C.3 Token-level Entropy: The default prompt uses additive memory, whereas the ablation uses compressive memory that updates the memory representation at each step.The two variants share the same agent prompt template except for the rule controlling memory updates.
  • C.3 Token-level Entropy: The prompt requires Memory, Progress, Intention, Action, and one tool call in every response, with Memory and Progress formatted as JSON.The displayed template specifies the roles of these fields for each interaction step.
Loading 2606.05597v3…