PaperlayerView exact revision on arXiv

Source-linked AI summary

Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, Hongtao Duan, Lu Jiang, Lulu Hu, Bin Yang, Minying Zhang

arXiv:2605.29447v1cs.CVcs.CL

TL;DR

GUI agents often fail to recover from their own policy-induced errors, while existing evaluation and training underrepresent realistic, long-horizon failures. The paper introduces GUI-RobustEval and RoTS, a tree-based synthesis framework for diverse recovery trajectories. RoTS-32B reaches 47.4% success and 33.8% All-Pass@4 on OSWorld.

  • Problem

    GUI agents frequently make policy-induced errors, but existing evaluation and training underrepresent realistic compositional and long-horizon failures relevant to robust deployment.

  • Method

    The paper introduces GUI-RobustEval to measure error awareness and recovery, and RoTS to synthesize diverse long-horizon failure-recovery trajectories through tree-based exploration and recovery.

  • Results

    RoTS consistently improves robustness and overall performance, with RoTS-32B achieving 47.4% success and 33.8% All-Pass@4 on OSWorld.

  • Takeaways & Limitations

    The results underscore the value of long-horizon reflection for reliable GUI agents across robustness and overall task-performance evaluations.

  • Takeaways & Limitations

    The study focuses on desktop computer-use tasks; mobile and edge-device evaluation remains future work.

Abstract

from arXiv · show

While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains $1,216$ executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates $800k$ high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a $47.4\%$ success rate and a $33.8\%$ All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.

1. Introduction

GUI agents remain vulnerable to policy-induced errors, while existing evaluation and training underrepresent realistic, compositional, long-horizon failures. The paper introduces GUI-RobustEval and RoTS to measure and address these gaps.

  • Policy-induced errors arise from agents’ own actions, such as incorrect grounding, screen-state misinterpretation, or wrong subgoals, and can trap agents in erroneous states.
  • Existing benchmarks and training underemphasize policy-induced errors, especially compositional failures that emerge after multiple steps and require long-horizon recovery.The paper identifies both error-coverage mismatch and error-horizon mismatch.
  • GUI-RobustEval contains 1,216 test cases for measuring how GUI agents detect and recover from policy-induced errors.The benchmark reports Error-Awareness Rate and Post-Error Success Rate across error types and depths.
  • RoTS is a tree-based online synthesis framework that discovers diverse failure modes and generates long-horizon failure-recovery trajectories.It explores successful branches from fragile states and synthesizes recovery rollouts from failed branches.
  • An 800k-sample dataset and fine-tuned QWEN2.5-VL improve robustness on GUI-RobustEval and task success on OSWorld and WindowsAgentArena.

2. Benchmark for Policy-Induced Errors

The benchmark analyzes policy-induced errors by their root-cause types and detection horizons, then constructs controlled test cases to measure awareness and recovery. Its distributions expose coverage and horizon mismatches in existing data and show that deeper errors are harder to recover from.

  • Policy-induced error type is defined by the root-cause action, while error horizon is the minimum steps needed for that error to become identifiable.
  • The analysis compares error-type and error-horizon distributions from 1.5k trajectories with three representative training datasets.A VLM annotates error types, while experienced annotators identify the earliest recognizable step on 300 failed trajectories.
  • Existing training data concentrates on low-level or immediately identifiable errors, whereas real failures more often involve compositional perception, planning, and delayed detection.
  • Evaluation replays corrected prefixes plus a root-cause action and d subsequent steps before measuring Error-Awareness Rate and Post-Error Success Rate.The benchmark uses controllable depths d ∈ {0, 1, 3, 5}.
  • GUI-RobustEval contains 1,216 test cases spanning 11 error types and focuses on realistic policy-induced errors rather than synthetic errors or external perturbations.
  • Planning and progress-perception errors are harder to recover than low-level execution errors, and recovery performance decreases as error depth increases.Recovery difficulty is defined as 1−post-error success rate, averaged over five SOTA agents.

3. Robustness-driven Trajectory Synthesis

RoTS builds a replayable trajectory-tree pipeline that jointly explores fragile successful states and generates recovery rollouts from failed states. It then filters noisy steps and supervises models on selected actions to improve training-data quality.

  • Explore–Recovery Co-Expansion: RoTS organizes rollouts into a replayable trajectory tree and repeatedly expands both successful and failed subtrees.Successful and failed branches are partitioned using a reward model before separate exploration and recovery expansion.
  • Fragility-Driven Exploration: Fragility-driven exploration selects high-fragility nodes in successful trajectories with a progress critic and UCB-style rule, then replays their prefixes for further rollout.The progress critic estimates step correctness, while the selection rule encourages breadth by exploring less-visited nodes.
  • Experience-Informed Recovery: Experience-informed recovery localizes candidate error steps in failed trajectories, derives neighboring-branch guidance, and prioritizes advice-conditioned recovery rollouts.Candidate steps receive natural-language recovery guidance and expansion priorities before recovery-node selection.
  • Dataset Construction and Training: Post-processing filters inconsistent state transitions and removes incorrect steps from successful and unsuccessful trajectories using progress and action critics.The pipeline reuses critic outputs from tree expansion and performs additional checks only when needed.
  • Dataset Construction and Training: Training instances contain the instruction, history, observation, and action, but supervision is applied only to action tokens to limit noise propagation from imperfect rollouts.The action consists of a ReAct-style chain of thought followed by an executable action.

4. Experiment

The experiments evaluate RoTS on robustness and end-to-end GUI benchmarks, then analyze its rollout strategies, data mixtures, reflection ratio, scalability, and limitations. RoTS improves error recovery and task success, with policy-induced reflection and co-expansion providing the strongest results.

  • Main Results: RoTS-7B and RoTS-32B achieve the highest open-source GUI-RobustEval scores, with awareness rates of 51.9% and 58.8% and average success rates of 34.2% and 40.3%.At error depth 5, their success rates remain 26.7% and 33.2%, respectively, with the lowest performance drop.
  • Main Results: RoTS-7B and RoTS-32B reach 36.3% and 47.4% OSWorld success at maximum step 50, while RoTS-32B achieves 33.8% All-Pass@4.Increasing the maximum step budget from 15 to 50 adds 4.6% for RoTS-7B.
  • Rollout Strategies: Adding EIR raises All-Pass@4 to 12.1, while combining EIR and FDE reaches 14.1 and OSWorld success of 21.4 under the ablation protocol.FDE alone improves OSWorld success from 18.1 to 19.6 but yields a smaller robustness increase from 8.6 to 9.1.
  • Data Quality: Replacing human reflection samples with policy-induced reflections improves OSWorld All-Pass@4 from 8.4 to 11.6 and success rate from 16.1 to 18.8.Fully policy-induced data achieves 14.1 All-Pass@4 and 21.4 success rate.
  • Reflection Ratio: A reflective-data ratio of λref=0.1 achieves the best reported results, reaching 21.4 OSWorld success and 14.1% All-Pass@4.The dataset size remains fixed at 100k while reflective data progressively replaces reflection-agnostic data.
  • Scalability and Limitations: Increasing expansion iterations from 0 to 32 raises success rate from 15.8 to 21.4 under a 100k dataset, while performance may saturate under the current N=4, 32-round setting.The authors suggest larger branching and more expansion rounds may generate more diverse and effective trajectories.
  • Scalability and Limitations: RoTS occasionally over-reflects and wastes inference budget, although the data-synthesis method is reported as cost-effective and easy to scale.The over-reflection behavior is identified in OSWorld failure cases.

5. Related Work

Prior GUI-agent benchmarks emphasize grounding, perception, planning, single-step accuracy, environmental noise, and overall task success. Training commonly uses demonstrations, videos, synthetic trajectories, or offline reflection data, but policy-induced execution errors remain insufficiently addressed.

  • Benchmarks for GUI Agents: Existing GUI benchmarks primarily measure grounding and perception, single-step accuracy, planning, environmental-noise robustness, or overall interactive task success.These evaluation settings provide limited direct coverage of policy-induced error recovery.
  • Data for Training Robust GUI Agents: GUI-agent training commonly relies on videos, human demonstrations, synthetic trajectories, offline reflection datasets, or online reinforcement learning.The paper positions RoTS as a synthesis approach focused on policy-induced errors during execution.

6. Conclusion

The paper identifies under-covered planning-level, long-horizon policy-induced failures as a source of GUI-agent fragility and introduces benchmark and synthesis solutions. It reports consistent improvements in robustness and overall performance, while limiting evaluation to desktop computer-use tasks.

  • Conclusion: Existing training data under-covers planning-level, long-horizon failures that commonly occur during real GUI-agent execution.This under-coverage is presented as a reason current agents remain fragile to policy-induced errors.
  • Conclusion: GUI-RobustEval measures robustness to policy-induced errors, while RoTS explores diverse error modes and generates corresponding reflection data.The benchmark targets error awareness and recovery, and the synthesis framework targets training coverage.
  • Limitations: The evaluation focuses on desktop computer-use tasks, leaving mobile and edge-device evaluation for future work.GUI-RobustEval also requires cross-format conversion when injecting prefix histories into heterogeneous agents.

Impact Statement

The paper frames improved GUI-agent error recovery as beneficial for safer autonomous computer-use deployment. It nevertheless emphasizes continued human judgment for high-stakes decisions and verification of alignment with user intent.

  • Impact: Improved robustness helps GUI agents detect and recover from their own mistakes instead of blindly continuing erroneous actions.The statement connects this capability to safer deployment of autonomous computer-use systems.
  • Impact: Human judgment remains essential for high-stakes decisions and for verifying that agent behavior aligns with user intent.The paper places this requirement within increasingly important daily-life and productivity uses of computer-use agents.

A.1.1. DATA TYPES

The paper categorizes GUI-agent errors and illustrates representative failures ranging from incorrect parameters and missed steps to incorrect UI elements and compositional errors. It also describes the evaluation infrastructure supporting large-scale, distributed rollouts.

  • 11 error types are annotated in GUI-RobustEval, with multi-label assignment because one mistake can have several root causes.
  • 1,216 test cases span four error depths, while annotations are performed at the base-trajectory level.
  • Error examples include an incorrect output path, a skipped filename step, mis-targeted line-spacing controls, and cascading failures during VSIX installation.
  • The rollout infrastructure uses cloud-hosted Ubuntu and Windows systems with asynchronous distributed sampling for evaluation and synthesis.

B.2. Training Task Preparation

Training preparation combines curated GUI tasks, structured reward judging, and critic modules that filter trajectory steps and provide signals for reflection and recovery. The resulting system decomposes task progress, state transitions, and action quality into reusable supervision.

  • Training Task Preparation: 20k high-quality tasks are curated from AgentNet selections and LLM-generated everyday computer-use scenarios refined by annotators.
  • LLM-as-Judge Reward Models: WebJudge achieves 90.00% human-consistency accuracy and a 77.42% F1 Score, motivating its use as the online rollout reward model.
  • WebJudge for Experience Outputs: The adapted judge extracts task milestones, step-wise state-transition summaries, binary rewards, and structured rationales describing procedure completion.
  • Progress Critic and Action Critic: The progress critic assesses action feasibility and local task progress, while the action critic checks execution correctness from consecutive screenshots.
  • Progress Critic and Action Critic: Both critics reach at least 88% agreement with expert human judgments, supporting their use for downstream data filtering.

C.5.1. SELF-REFLECTION DATA EXAMPLE FOR SHORT ERROR

The examples show self-reflection correcting short-horizon GUI errors by identifying the immediate cause and selecting a concrete next action. They include both text-entry correction and recovery from commands entered into the wrong application.

  • Short Error Recovery: After typing 5.5 appends to the existing Hue value, the agent recognizes the mistake and plans to select all text before replacement.
  • Short Error Recovery: The GIMP example then uses Ctrl+A in the focused Hue field to prepare the value for exact replacement.
  • Short Error Recovery: After commands are entered into the Files search box, the agent identifies that the terminal is unfocused and closes the Files window to resume execution.
  • Short Error Recovery: The planned terminal recovery changes to ./Desktop, writes filtered last-command output to word, and then inspects the file’s first five lines.
  • Evaluation Context: GUI-RobustEval reports Error-Awareness Rate and Post-Error Success Rate across 1,216 cases, four depths, and 11 error types.

D.2.2. BASELINE METHODS ON OSWORLD

The OSWorld comparison evaluates RoTS against proprietary, open-weights, agentic, and general-purpose multimodal GUI-agent baselines. The experiments use both API-accessed proprietary models and locally deployed open models under reported evaluation settings.

  • Baseline Coverage: RoTS is compared with OpenAI CUA, UI-TARS-1.5, Claude Sonnet, Doubao-1.5-Thinking, Qwen3-VL, OpenCUA, and other GUI-agent baselines on OSWorld.
  • Baseline Coverage: The comparison spans proprietary APIs, open-weights agentic models, and general-purpose multimodal LLMs used as GUI agents.
  • Evaluation Setup: Open-sourced agents and the paper’s models run on a server with 32 NVIDIA A100 GPUs, while proprietary models are accessed through provider APIs.
  • Baseline Coverage: WindowsAgentArena comparisons include Claude 3.7 Sonnet, Qwen2.5-VL-72B, UI-TARS variants, OpenCUA, ScaleCUA, and Jedi-7B w/ GPT-4o.

E.1. Dataset Synthesis

RoTS synthesizes diverse GUI trajectories through multi-policy tree expansion, producing filtered reflection-related and reflection-agnostic training data. Fine-tuned RoTS models achieve competitive success on WindowsAgentArena.

  • Trajectory collection: 20k online tasks are expanded with three policy models, using 4 initial rollouts and 32 co-expansion rounds to yield 68 trajectories per task.The policies are UI-TARS-1.5-7B, OpenCUA-7B, and Qwen3-VL-Plus.
  • Data processing: Progress and action critics remove incorrect steps, while reflection validation separates the remaining data into reflection-agnostic and reflection-related subsets.Posterior filtering first discards trajectories with inconsistent state transitions caused by environment stochasticity.
  • Dataset statistics: Pass@68 measures whether at least one of 68 sampled trajectories succeeds, while |Dagn| and |Dref| count trainable steps in the two resulting datasets.The average reports the ratio of correct trajectories to total trajectories.
  • Model training: QWEN2.5-VL-7B and QWEN2.5-32B are fine-tuned for one epoch with global batch size 512 on 64 and 128 NVIDIA A100 GPUs, respectively.Both settings use DeepSpeed ZeRO-3, bfloat16 precision, FlashAttention, and a 32,768-token maximum sequence length.
  • Benchmark outcome: 28.2 and 39.1 success rates are achieved by RoTS-7B and RoTS-32B on WindowsAgentArena at maximum step 50.The comparison is reported in the benchmark results table for WindowsAgentArena.

F.2. Per-Error-Type Analysis on GUI-RobustEval

The analysis shows broad but uneven recovery behavior: RoTS-32B improves across error types, while long-horizon progress failures and excessive self-correction remain challenges. Ablations identify experience-informed recovery and moderate-depth data as important contributors.

  • Per-error-type results: RoTS-32B improves across all error types, with the largest gain on Ineffective Action at +11.3.Fail to Terminate remains at 17.4%, and Misunderstand Task Objective remains at 24.1%.
  • Trajectory case study: In a startup-settings case, RoTS identifies a Fail-to-Terminate deviation, returns to the correct settings page, and verifies that the target website was removed.OpenCUA instead continues harmful, task-irrelevant operations such as removing Bing from the search-engine list.
  • Exploration and recovery: RoTS completes an OSWorld FAQ task after failed navigation attempts and a typing error by switching to direct search and re-entering corrected keywords.The recovery uses Ctrl+A before typing the correct search terms.
  • Failure case: Over-reflection causes two unnecessary go-back operations when RoTS mistakes an incompletely rendered webpage for an incorrect page.The task could have progressed from the current page without those back-navigation steps.
  • Cost analysis: Benchmark construction costs $400, while data synthesis costs about $48,100 and takes about 16 days under the reported configuration.The synthesis estimate covers GPU servers, API calls, and cloud environment deployment.
  • EIR ablation: Full EIR reaches a 46.1 post-error success rate, while adding trajectory-derived experience raises error awareness from 62.6 to 67.1.The ablation compares reflector variants and an advice-conditioned recovery actor.
  • Recovery-depth ablation: Recovery data capped at depth ≤5 improves post-error success from 12.1 to 20.7, a gain of +8.6 under the 100k/7B setting.Gains beyond depth 7 are described as relatively small.
Loading 2605.29447v1…