PaperlayerView exact revision on arXiv

Source-linked AI summary

ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis

Guohong Liu, Jialei Ye, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, Yuanchun Li

arXiv:2605.25160v2cs.AI

TL;DR

GUI-agent research needs realistic, reproducible environments with reliable rewards, but real-world apps make verification and state resetting difficult. ScaleWoB uses coding agents to synthesize backend-free, URL-served environments and executable validators across GUI platforms. Its mobile benchmark exposes substantial headroom in current agents, and evaluations generalize to real-app tasks.

  • Problem

    Real-world GUI environments complicate reliable reward verification and reproducible state management, while existing reproducible settings often have narrow task coverage.

  • Method

    ScaleWoB uses coding agents to generate and refine backend-free webpages from task descriptions, then injects tasks and synthesizes executable reward validators.

  • Results

    The mobile benchmark covers 63 simulated applications and 120 tasks; five agents average 27.92% success overall and 17.82% on long-horizon tasks, versus 92.08% for humans.

  • Takeaways & Limitations

    Synthetic-environment assessments generalize to real apps, while experiments reveal substantial headroom for current mobile GUI agents, especially on long-horizon tasks.

  • Takeaways & Limitations

    The released benchmark currently exposes visual observations and webpage affordances, focuses on single-app workflows, and quantitatively instantiates only mobile environments.

Abstract

from arXiv · show

GUI agents powered by large language models are advancing rapidly, creating urgent needs for evaluation and training based on realistic environments. However, directly doing so in real-world environments introduces some challenges that cannot be overlooked. Real-world environments are complex and uncontrollable, making it difficult to construct verifiable rewards and to save or reset states. Existing works prioritize reproducibility but are often limited to open-source apps or file-operation tasks for reliable reward building, leaving a persistent gap from real-world usage. Furthermore, relying on virtual machines or docker images demand high resource requirements and suffer from slow response speeds, which limit the efficiency. We present \sys, a framework that could produce high-fidelity synthesized interactive environments for GUI agents across platforms with verifiable rewards. These environments behave as backend-free webpages accessible via URL, requiring near-zero setup and low resource cost, making the approach suitable for both large-scale evaluation and downstream agent training. We support multiple GUI platforms including mobile, desktop, and automotive/in-vehicle interfaces based on the same pipeline, covering 100+ environments and 1000+ verifiable tasks. Among them, 120 challenging tasks across 63 simulated mobile applications are released as a fully synthesized mobile GUI agent benchmark. Experiment results on five state-of-the-art mobile GUI agents reveal substantial headroom -- the average success rate is only 27.92\%, dropping to 17.82\% on long-horizon subset -- while humans reach 92.08\%. A comparison against real-world sample tasks shows that assessments made in our synthetic environments generalize to real apps. The project website is at https://scalewob.github.io.

1. Introduction

ScaleWoB addresses the difficulty of evaluating and training GUI agents in realistic environments by synthesizing platform-agnostic webpages with verifiable rewards. Its mobile benchmark spans challenging tasks and reveals substantial performance gaps while maintaining generalization to real-app assessments.

  • Motivation: Real-world GUI environments make task completion difficult to verify because application-internal states are often inaccessible.Existing approaches use LLM-based judging, open-source applications, file-operation tasks, or manually engineered execution checks.
  • Motivation: Real-world environments also hinder reproducibility because dynamic information and irreversible operations make state resetting nearly impossible.This constrains stable evaluation and training of GUI agents.
  • ScaleWoB: ScaleWoB uses coding agents to iteratively generate backend-free webpages that simulate applications, validate task solvability, and emit executable reward functions.The framework takes natural-language task descriptions and optional screenshots as generation inputs.
  • ScaleWoB: The framework has produced over 100 environments and over 1000 verifiable tasks across mobile, desktop, and automotive interfaces.Its platform-agnostic pipeline supports multiple GUI interaction styles.
  • Benchmark: The released mobile benchmark covers 63 simulated applications and 120 tasks, with half of the tasks requiring more than 20 interaction steps.Tasks span simple, long-horizon, and math-related categories, with the hardest exceeding 50 steps.
  • Benchmark: 27.92% average success across tasks and 17.82% on long-horizon tasks contrast with 92.08% for humans, while synthetic-environment assessments generalize to real apps.Five state-of-the-art mobile GUI agents were evaluated.

2. Related Work

GUI-agent benchmarks range from scalable static datasets to interactive environments, but reliable scoring and realistic, complex workflows remain difficult to scale. ScaleWoB treats environment construction itself as code generation, combining executable rewards, platform-neutral webpages, and emphasis on demanding tasks.

  • Benchmark Types: Static GUI-agent datasets support scalable offline evaluation but generally omit closed-loop interaction dynamics and stateful multi-step dependencies.They do not fully capture recovery from mistakes or delayed feedback.
  • Benchmark Types: Interactive benchmarks often rely on accessible internal state or extensive manual rule engineering, limiting realistic workflows with complex, unobservable task states.Many existing tasks also remain short-horizon or structurally simple, reducing discriminative power.
  • Prior Synthesis Approaches: Controllable simulators such as AppWorld and τ-bench provide reproducibility through state-based checks but are manually implemented and confined to a few domains.They expose API-level simulated worlds rather than broad platform-neutral GUI environments.
  • ScaleWoB’s Position: ScaleWoB makes environment construction a code-generation problem, producing backend-free URL-accessible environments with executable rewards instead of hand-building a fixed suite.This positions the framework on the supply side of environment creation.
  • ScaleWoB’s Position: The platform-neutral webpage design targets mobile, desktop, and automotive interfaces while combining realistic app simulation with fast benchmarking throughput.Near-zero-cost resets also support downstream agent training, although the released evaluation setting is mobile.

3. ScaleWoB

ScaleWoB uses a two-stage coding-agent pipeline to synthesize realistic, task-executable GUI environments with verifiable rewards, then validates and repairs them before release. Its mobile benchmark derives user needs into diverse tasks across apps, languages, return formats, and difficulty levels, while the same URL-served framework extends across platforms.

  • 3.1. Environment Generation: ScaleWoB synthesizes interactive environments and reward validators through a two-stage pipeline that first builds app logic, then injects tasks and verification criteria.Stage 1 constructs a minimal working environment; Stage 2 edits it for task executability and creates executable task-specific validators.
  • 3.1. Environment Generation: The generation workflow iteratively refines UI pages, data entities, and interaction logic from app metadata and optional screenshots before task-specific editing.The pipeline first drafts a product requirements document and then generates code from it.
  • 3.1. Environment Generation: Validation agents execute bundled tasks, route failures to human experts for diagnosis and repair, and subject passing bundles to random quality audits and manual verification.Environment-side issues return to the repair loop, while the final benchmark environments and tasks are manually verified.
  • 3.2. Tasks & Environments: The benchmark converts user-study requests into executable tasks by deduplicating intents, screening infeasible requirements, specifying completion criteria, and balancing language, domains, and difficulty.The study collected 260 valid requests from 86 participants across 77 industries.
  • 3.2. Tasks & Environments: ScaleWoB releases 120 tasks across 63 virtual apps and 20 of 33 Google Play categories, with interaction lengths ranging from about 10 to over 50 steps.The broader pipeline has produced over 100 environments and over 1000 verifiable tasks across mobile, desktop, and automotive profiles.
  • 3.2. Tasks & Environments: Tasks span Chinese and English, explicit return-value requirements, and multiple difficulty categories, while environments are URL-served and share a common coordinate-action evaluation interface across platforms.Return-required tasks are verified using schema-compliant JSON outputs.

4. Experiments

ScaleWoB evaluations show substantial performance gaps across mobile GUI agents, especially on return-value, long-horizon, vague-description, and fine-grained-control tasks. Synthetic-environment rankings remain consistent with real-app sample-task rankings.

  • 4.1. Settings: Only screenshot-based agents are evaluated, with success rate as the primary metric and trajectories capped at 100 steps.Local models are additionally evaluated only on tasks without return-value requirements because they failed to produce schema-valid JSON outputs under prompting.
  • 4.3. Analysis: 50.00% versus 30.43% success rate shows seed-1.8 performs better on tasks without return values than on tasks requiring them.The corresponding UI-TARS-1.5 scores are 39.86% and 13.04%, respectively.
  • 4.2. Experimental Results: 92.08% average score for humans contrasts with the lower agent performance, indicating a substantial human–agent gap.The comparison underscores the remaining headroom in the evaluated mobile GUI agents.
  • 4.2. Experimental Results: Relative model-performance rankings on ScaleWoB sample tasks are largely consistent with rankings on real-app sample tasks.The authors report this consistency as evidence that synthetic-environment evaluations can generalize to real-world apps.
  • 4.2. Experimental Results: 42.50% overall success rate is achieved by the best model, while the aggregate average is 27.92% and the weakest model reaches 9.17%.ScaleWoB scores are substantially below AndroidWorld's 69.38% average, and the best–worst spread reaches 33.33 points.
  • 4.3. Analysis: 17.82% average success rate on long-horizon tasks reflects a clear drop from simple tasks across all evaluated agents.A representative comparison is 56.48% versus 29.09%; the associated failure case involves correct UI operations but failure to preserve intermediate information.
  • 4.3. Analysis: Vague descriptions and inconspicuous entry points degrade performance when agents fail to explore proactively after an initial strategy fails.The case study describes an agent searching the wrong view for a calendar event and not systematically switching strategies.
  • 4.3. Analysis: 5/11 fine-grained-control tasks are completed by Gemini 3 Pro, while other agents complete only about 0.5 to 1.5 tasks on average.These tasks include sliders, date/time pickers, drag confirmations, and long-press context menus; failures are attributed to insufficient action precision and limited interaction-strategy knowledge.

5. Limitations

The released benchmark has scope boundaries in observation modalities, cross-app workflows, platform coverage, and empirical validation of training benefits.

  • Observation space: Web-based environments expose visual observations and page interaction affordances rather than richer native-platform signals such as accessibility trees.This limits the observation space available in the released benchmark.
  • Cross-app task support: ScaleWoB currently focuses on single-app workflows and does not cover tasks requiring coordinated operations across multiple applications.Cross-app task support remains outside the present benchmark scope.
  • Platform coverage: The quantitative released benchmark is instantiated on mobile, while desktop and automotive task suites have only demonstration-level coverage.Scaling those additional platform suites remains future work.
  • Training validation: The release empirically validates evaluation but leaves downstream training gains for future work.The environments expose resettable, reward-bearing interfaces suitable for online reinforcement learning, but training benefits are not demonstrated here.

6. Conclusion

ScaleWoB uses coding agents to synthesize interactive environments for GUI-agent evaluation and training. Its mobile release spans 63 simulated applications and 120 tasks, while experiments expose substantial headroom, especially on long-horizon tasks.

  • ScaleWoB automatically constructs backend-free, URL-served environments and task-specific reward validators from natural-language task descriptions.
  • The same webpage-based pipeline spans mobile, desktop, and automotive interfaces, with the mobile release covering 63 simulated applications and 120 tasks across three categories.
  • Experiments show substantial performance headroom for current mobile GUI agents, particularly on long-horizon tasks, and identify weaknesses in proactive exploration and fine-grained action control.

A. ScaleWoB Environment Synthesizing

ScaleWoB synthesizes simulated applications through iterative specification, code generation, review, and refinement. Its environments package workflows, task data, reset functions, task schemas, and direct state-based evaluation into lightweight webpages.

  • The pipeline first generates a detailed PRD describing workflow, UI transitions, data entities, and styles, then produces application code from that specification.
  • The iQIYI example organizes simulated video entertainment around browsing, playback, membership, discovery, and related interface flows.
  • The generated applications are iteratively reviewed and refined, with updates spanning user state handling, routing, interactive views, episodes, collections, and connected settings flows.
  • After 15 iterations, the application becomes a minimal working example before enrichment with data and task requirements.
  • Three DOM-based APIs bundle environments with task execution: reset restores initial state, getTasks returns task descriptions and schemas, and evaluateTask checks target state.
  • Task evaluation directly accesses relevant application state through local storage, avoiding additional manually written pattern matching.

B. Comprehensive Experimental Results

The experiments evaluate mobile GUI agents through standardized interfaces, parallel execution, and task-category analysis, while real-world samples test whether synthetic-environment assessments reflect app usage.

  • All GUI agents are evaluated using resources from their official implementations with eight parallel workers.
  • Results are organized by agent and by simple, long-horizon, and math-related task categories.
  • ScaleWoB exposes screenshots for observation and maps actions such as click, swipe, long press, typing, clearing text, entering, and waiting to simulator operations.
  • A real-mobile validation set contains 20 tasks across 17 apps, with manually verified results covering the same three task categories.
  • The real-world sample tasks include search, liking, favorites, navigation, settings, music, shopping, and multi-day flight comparisons.

C. Broader Impacts

ScaleWoB offers a controlled, privacy-preserving testbed for diagnosing mobile GUI-agent failures, but stronger agents may also enable harmful automation. The authors therefore distinguish benchmark progress from readiness for unsupervised deployment.

  • Synthetic, backend-free environments support fast, reproducible evaluation without exposing real accounts, private records, payments, messages, or other sensitive app data.
  • The benchmark can help identify failures in long-horizon reasoning, memory, exploration, and fine-grained control before deployment in real applications.
  • The authors caution that improved GUI agents could facilitate unauthorized operations, spam, fraud, scraping, or privacy-invasive workflows without safeguards.
  • ScaleWoB should be treated as a controlled research benchmark rather than evidence of readiness for unsupervised real-world deployment.

D. User Study

A volunteer user study gathered realistic mobile-agent requests about daily app tasks and their pain points. The requests were filtered, normalized, and converted into synthetic benchmark tasks without including real accounts or personal records.

  • Volunteers described daily mobile-app tasks they wanted an agent to perform, along with brief background pain points.
  • Collected requests were manually filtered, normalized, and transformed into synthetic benchmark tasks.
  • The study excluded account credentials, private messages, payment records, and other sensitive personal data, and ScaleWoB contains no real user accounts or personal records.
  • The study materials include volunteer questions and examples linking user requests to the actual tasks included in ScaleWoB.
Loading 2605.25160v2…