PaperlayerView exact revision on arXiv

Source-linked AI summary

WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering

Yifei He, Pranit Chawla, Yaser Souri, Subhojit Som, Xia Song

arXiv:2512.10962v3cs.LGcs.AI

TL;DR

CUA training faces costly GUI interaction and a scarcity of high-quality training data, while offline trajectory data is limited. The pipeline augments teacher rollouts with intermediate thoughts and grades individual actions, retaining high-quality steps for training. On live web tasks such as WebVoyager, our 7B model trained with step-level filtering surpasses the SoTA general-purpose CUA of similar size (UI-TARS-1.5-7B) by more than 15%.

  • Problem

    CUA training faces costly GUI interaction and a scarcity of high-quality training data, while offline trajectory data is limited.

    Source: §1 · p. 1

  • Method

    The pipeline augments teacher rollouts with intermediate thoughts and grades individual actions, retaining high-quality steps for training.

    Source: §1 · p. 2

  • Results

    On live web tasks such as WebVoyager, our 7B model trained with step-level filtering surpasses the SoTA general-purpose CUA of similar size (UI-TARS-1.5-7B) by more than 15%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Step-level filtering consistently outperforms trajectory-level filtering, even under simple SFT.

    Source: §1 · p. 3

    The approach relies exclusively on SFT, which cannot fully address long-horizon credit assignment or exploration in unseen environments.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Computer use agents (CUAs) can operate real-world digital interfaces but remain difficult to train due to the high cost of graphical user interface (GUI) interaction and the scarcity of high-quality trajectory data. Existing datasets rely on human demonstrations, limiting scalability. A natural alternative is to synthesize data from strong CUAs, yet their rollouts are highly noisy, with incorrect or suboptimal actions consisting a large proportion of the steps, making naive imitation ineffective. To tackle this challenge, we introduce a scalable data synthesis pipeline that transforms noisy rollouts into reliable supervision without human annotation. The core idea is step-level filtering, which evaluates actions individually to retain only correct steps, complemented by reasoning augmentation for improved planning. Using this pipeline, we construct WebSTAR, a dataset of 13.3K trajectories and 267K graded, reasoning-rich steps synthesized from OpenAI's computer-use-preview model. We train Qwen-2.5-VL-Instruct models (7B and 32B) on WebSTAR. On WebVoyager, our 7B model surpasses SoTA open-source CUA model UI-TARS-1.5-7B by more than 15% with only supervised finetuning. Building on step-level grading, we further create WebSCORE, a dataset of graded step-level actions, and train StepRM, a 7B multimodal process reward model distilled from o4-mini, which matches its grading quality while being far more efficient to deploy at scale. Our results establish step-level filtering as a key principle for scalable CUA training and construct two new datasets (WebSTAR, WebSCORE) and a lightweight process reward model (StepRM) as practical tools to advance robust and efficient CUAs.

1 Introduction

CUA training is constrained by costly GUI interaction and scarce high-quality data, while teacher-generated trajectories are noisy. The paper introduces a pipeline that augments and grades steps, then filters them for training; its 7B model exceeds UI-TARS-1.5-7B by more than 15% on WebVoyager with SFT.

1 Introduction

  • CUA training is difficult because GUI interaction is costly and high-quality training data is scarce.Offline data is limited by the expense of manual labeling and by VLM-curated data that may rely on auxiliary information unavailable in general deployment.Source: §1 · p. 1
  • The three-stage pipeline collects teacher rollouts, augments steps with intermediate thoughts, then grades and retains high-scoring steps.The grading model scores each step from 0 to 10; retained steps provide high-quality supervision.Source: Figure 1 · p. 2, §1 · p. 2
  • Teacher CUA rollouts are noisy: incorrect or suboptimal actions make up more than half of their steps, motivating filtering and augmentation.Direct training on noisy rollouts degrades student performance.Source: §1 · p. 2
  • WebSTAR contains 13.3K trajectories and 267K reasoning-enriched, step-scored steps; WebSCORE provides a complementary dataset for process reward modeling.The authors also train a lightweight 7B StepRM on WebSCORE, reporting grading quality comparable to o4-mini at a fraction of the cost.Source: §1 · pp. 2–3
  • On live web tasks such as WebVoyager, the 7B model trained with step-level filtering surpasses the similar-sized SoTA general-purpose CUA UI-TARS-1.5-7B by more than 15%.Step-level filtering consistently outperforms trajectory-level filtering even under simple SFT.Source: §1 · p. 3

2 Preliminaries

A CUA observes an interface screenshot and produces actions in response to a task instruction, with a bounded interaction horizon. The paper frames training as predicting each step’s thought and action from the instruction and recent visual context.

2 Preliminaries

  • At each step, the agent observes a screenshot and executes an action; a maximum horizon T bounds the interaction.The setup adds an intermediate thought to guide each action and excludes auxiliary inputs such as accessibility trees and specialized annotations.Source: §2 · p. 3
  • The agent is trained to predict the next thought–action pair from the task instruction and a truncated window of recent screenshots.Only image inputs are truncated because of their high token cost.Source: §2 · p. 3
  • The SFT objective minimizes the negative log-likelihood of the ground-truth thought–action pairs at each step.This step-level objective aligns training with the recursive nature of CUA execution rather than treating full trajectories as atomic units.Source: §2 · p. 3

3 Method

The method first collects and filters teacher rollouts, then grades individual actions in successful trajectories. Step-level grading targets flawed intermediate decisions that trajectory-level filtering leaves in place.

3.1 Teacher CUA Rollout

  • 13,338 successful trajectories remain after an initial trajectory-level success judgment of 21K rollouts generated from 1,326 queries.The collection uses 16 rollouts per query, with a maximum horizon of 100 steps.Source: §3.1 · pp. 3–4

3.2 Step-Level Filtering

  • Successful trajectories can still contain incorrect intermediate steps, which can bias SFT toward flawed decisions and degrade downstream performance.Trajectory-level filtering does not match the step-level thought–action training objective.Source: §3.2 · p. 4
  • The grading model assigns each action a quality score based on its expected contribution to task success, using task and action context plus screenshots and a zoomed crop of the action target.The authors use o4-mini for its reasoning capability and relatively lightweight inference cost.Source: §3.2 · p. 4
  • The grader reviews whether a proposed action meaningfully advances the goal, assigning lower scores to misaligned or unhelpful actions such as misclicks.Its screenshot inputs include a crop focused on the relevant web element.Source: §3.2 · p. 4

3. Alternative analysis: The model proposes

Step-level grading evaluates alternative actions and scores each proposed step for correctness and optimality; SFT then uses only high-quality steps for its loss. The section also examines how thought augmentation affects performance, especially across multiple sampled rollouts.

3. Alternative analysis: The model proposes

  • A grading model compares the proposed action with three alternatives and penalizes it if any alternative is strictly more effective.It assigns a 0–10 score based on correctness and optimality; 0 signals irreversible error or guaranteed failure, 5 a partially correct or suboptimal action, and 10 a helpful step with no superior alternative.Source: §3 · p. 4
  • The SFT objective retains full trajectories as context but computes loss only for steps scored above 5, leaving incorrect actions out of the optimization signal.This preserves realistic histories that may include suboptimal decisions while allowing learning from correct steps.Source: §3 · p. 4
  • Step-level filtering uses roughly half the data of trajectory-level filtering yet consistently improves performance across tasks.The comparison trains on correct steps within successful trajectories rather than every step in successful trajectories.Source: Table 2 · p. 5, §3 · p. 5

3.3 Thought Augmentation

  • Thought augmentation adds situation descriptions, action rationales, and actionable instructions to bridge perception and action.The thoughts are generated post hoc because the raw teacher trajectories contain observations and actions without explicit reasoning traces.Source: §3.3 · p. 5
  • Thought augmentation yields consistently higher Pass@4 across benchmarks and model sizes, while Pass@1 scores remain close with and without thought traces.Pass@4 reflects success across multiple sampled rollouts, whereas a single error can derail a long-horizon trajectory and make Pass@1 brittle.Source: §3.3 · p. 5

4 Experiments

Step-level filtering improves CUA performance over trajectory-level filtering across benchmarks and scales, with gains that persist as training data grows. The experiments also assess grading reliability and show that StepRM can substitute for o4-mini at lower deployment cost.

4.2 Main Results

  • 47.0% average success rate on WebVoyager for the 7B step-level model, versus 30.0% for UI-TARS-1.5-7B, despite using only SFT.Step-level filtering also outperforms trajectory-level filtering across almost all WebVoyager domains.Source: §4.2 · p. 6, Figure 2 · p. 6
  • Step-level filtering substantially improves success rates on Mind2Web-Live and OnlineMind2Web for both model sizes, with larger gains for 32B models.The table reports Pass@1 and Pass@4; the supplied passage does not provide their individual values.Source: §4.2 · p. 6, Table 4 · p. 6
  • With more data, both 7B and 32B step-level models continue gaining, while trajectory-level filtering plateaus early, particularly for 7B.Step-level filtering already has substantially higher success rates in the low-data regime of around 25K examples.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Ablations

  • On 100 manually annotated steps, o4-mini grading agrees with human judgments 73% of the time; disagreements mainly involve exploratory-but-not-wrong actions.The grading prompt is deliberately strict to favor clearly goal-directed actions.Source: §4.3 · p. 7
  • Across five repeated grades of 230 sampled steps, o4-mini scores vary by 0.81 points on average, with a median standard deviation of 0.49.The median coefficient of variation is below 6%.Source: §4.3 · p. 7
  • A score cutoff of 5 balances data quality and data efficiency; higher thresholds retain fewer steps initially and require more rollouts to reach 100K steps.The threshold ablation compares cutoffs {2, 4, 6, 8} at a fixed final dataset size of 100K steps.Source: §4.3 · pp. 7–8, Table 6 · p. 7

4.4 StepRM

  • StepRM, a Qwen2.5-VL-7B model trained on 200K WebSCORE steps, matches o4-mini filtering quality across multiple benchmarks while being more efficient to deploy.In a controlled comparison, both graders filtered the same independently collected 200K-step pool before SFT.Source: §4.4 · p. 8, Table 7 · p. 8
  • StepRM could provide dense step-level feedback as an RL critic, but its integration into reinforcement-learning pipelines is left for future work.The proposed role is to help address sparse rewards and credit assignment in CUA training.Source: §4.4 · p. 8

5 Related Works

Prior CUA datasets vary in scale, reasoning traces, grading, and interface grounding, while synthetic rollouts make filtering important because successful trajectories can still contain poor intermediate actions. This work presents step-level filtering with a separate grader as a scalable alternative aligned with step-wise training.

Computer use agent datasets

  • Human-annotated CUA data is costly and limited in scale, while several datasets lack reasoning traces or step-level quality control.Some earlier datasets rely on Set-of-Marks or accessibility trees rather than screenshot-only interaction settings.Source: §5 · pp. 8–9
  • WebSTAR combines scale, long-horizon trajectories, structured reasoning traces, and step-level grading without human annotation or auxiliary signals.The comparison is summarized in Table 8.Source: §5 · p. 9, Table 8 · p. 9

Data filtering

  • Trajectory-level filtering keeps successful tasks but can retain incorrect or suboptimal intermediate actions, misaligning supervision with step-wise CUA training.CUA step-level filtering is challenging because rewards are sparse and Monte Carlo credit assignment is high-variance.Source: §5 · p. 9
  • The proposed separate-grader approach filters actions individually and is described as stable and cost-effective relative to actor-critic methods.The authors characterize it as aligning supervision with the step-level nature of CUA training.Source: §5 · p. 9

6 Conclusions

The paper concludes that step-level filtering and reasoning augmentation improve CUA training from noisy rollouts, and introduces WebSTAR, WebSCORE, and StepRM as supporting resources. It presents StepRM as a lower-cost alternative to o4-mini and a potential tool for future reinforcement-learning optimization.

6 Conclusions

  • The pipeline combines step-level filtering and thought augmentation to improve performance across multiple benchmarks and surpass strong baselines with SFT.The conclusion frames step-level filtering as better aligned with the supervision objective because it retains correct actions from imperfect trajectories.Source: §6 · p. 9
  • WebSTAR provides trajectories enriched with reasoning and step-level scores, while WebSCORE supplies graded steps for training process reward models.The paper presents both as complementary datasets for CUA research.Source: §6 · p. 9
  • StepRM matches o4-mini grading quality at lower cost and is presented as a practical tool for future reinforcement-learning optimization.The conclusion describes efficient reward modeling as part of robust and scalable CUA training.Source: §6 · p. 9

Limitations

The pipeline’s scalability is constrained by teacher quality and by reliance on supervised finetuning alone. The authors identify reinforcement learning as a possible way to address some remaining challenges.

Limitations

  • Teacher CUA quality constrains scalability and limits the diversity of high-quality steps that can be synthesized.The authors expect this bottleneck to diminish as CUAs improve.Source: Limitations · p. 10
  • Exclusive reliance on SFT leaves long-horizon credit assignment and exploration in unseen environments unaddressed.The authors identify RL as a potential way to improve these aspects and StepRM as a pathway toward integrating step-level filtering with RL.Source: Limitations · p. 10

A More Results

Across WebVoyager results, step-level filtering outperforms trajectory-level filtering, while OpenAI CUA remains strongest. The appendix also examines confidence intervals, score distributions, and a challenging domain where the student often reaches its step limit.

HuggingFace analysis

  • 65% of student HuggingFace tasks exhaust the 100-step limit, versus 28% for OAI CUA; failures are dominated by running out of turns.The queries often require deep technical knowledge, multi-hop navigation, and precise extraction from code-heavy documentation.Source: A More Results · p. 12

Statistical significance

  • The main 7B results include 95% confidence intervals based on running each query four times, and Pass@4 is reported as a more stable metric for long-horizon tasks.The authors state that improvements remain statistically significant under reasonable uncertainty estimates.Source: A More Results · p. 12

Full numerical results

  • Step-level filtering outperforms trajectory-level filtering under both model sizes, and the 7B model outperforms UI-TARS-1.5-7B across domains while OAI CUA remains strongest.The 32B model significantly narrows the gap to OAI CUA.Source: A More Results · p. 12, Table 9 · p. 13, Table 10 · p. 13

Score distribution

  • Over half of actions in successful trajectories score below 5, so trajectory-level filtering can retain many flawed intermediate steps.Step-level filtering keeps high-quality actions and discards the majority of low-quality ones.Source: A More Results · pp. 12–13, Figure 4 · p. 13

B Evaluation Details

Evaluation excludes tasks affected by website access problems and Google-related human verification requirements.

Evaluation details

  • Tasks involving websites with anti-scraping, anti-robot, or website-bug access problems are excluded from training and testing.The listed access problems reflect testing in August 2025 and may later be fixed.Source: B Evaluation Details · p. 13
  • Google-related tasks are omitted from WebVoyager evaluation because they have strict human verification requirements.Source: B Evaluation Details · p. 13
  • The access-problem list reflects conditions at the time of testing in August 2025, so some listed tasks may later become valid again.Source: B Evaluation Details · p. 13

C Data Examples

The appendix describes WebSTAR datapoints and provides supplementary tables and trajectory samples.

Data examples

  • Each WebSTAR datapoint includes an agent action, an o4-mini-augmented thought, a step-level score and rationale, and a screenshot of the current state.Source: C Data Examples · p. 13
  • Screenshots have no annotations because the dataset uses no auxiliary source of information such as a set of marks.Source: C Data Examples · p. 13
  • The appendix includes 7B Pass@1 results with 95% confidence intervals and a table of websites with access problems in Mind2Web queries.Additional full-trajectory samples are available in the attached data folders.Source: Table 11 · p. 14, Table 12 · p. 14, C Data Examples · p. 14

E Licenses

This section lists dataset licenses and provides prompts for augmenting agent reasoning and grading proposed actions.

E Licenses

  • WebVoyager and OpenWebVoyager use Apache-2.0, while Online-Mind2Web and Mind2Web-Live use CC-BY-4.0.Source: E Licenses · p. 14

F Prompts for Thought Augmentation and Grading

  • The prompts use o4-mini to generate both thought-augmentation and grading answers.Source: F Prompts for Thought Augmentation and… · p. 14
  • The thought-augmentation prompt asks for screenshot-grounded situation analysis, reasoning aligned with the action, and a matching actionable instruction.It cautions against using the proposed action or screenshot annotation as hindsight and says not to invent a different action.Source: F Prompts for Thought Augmentation and… · p. 16
  • The grading prompt scores a proposed action from 0 to 10 after assessing task progress, action value, likely outcomes, and available alternatives.It instructs graders to evaluate the action before execution using the latest screenshot and to score at most 6 if an alternative is strictly better.Source: F Prompts for Thought Augmentation and… · pp. 17–18

Related papers

Loading 2512.10962v3…