PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, Haiwei Wang

arXiv:2604.05477v1cs.CL

TL;DR

GUI agents often assume actions execute successfully, despite delays and interruptions that can produce undetected failures and repeated ineffective behavior. VeriGUI addresses this with TVAE outcome verification, synthetic-failure Robust SFT, and offline GRPO with asymmetric verification rewards; on its robustness benchmark, both model sizes surpass tested open-source baselines in recovery success.

  • Problem

    Existing GUI agents assume issued actions execute as intended, while offline datasets lack explicit feedback about failures and recovery.

  • Method

    VeriGUI combines a TVAE closed-loop framework with Robust SFT on synthetic failure trajectories and GRPO using offline, idempotency-based feedback.

  • Results

    Recovery Success Rate reaches 51.1% for VeriGUI-3B and 52.5% for VeriGUI-7B, surpassing all tested open-source baselines.

  • Takeaways & Limitations

    The results support explicit execution-outcome verification and recovery as a practical direction for robust GUI-agent design.

  • Takeaways & Limitations

    The benchmark assumes failed actions leave screens unchanged, and performance degrades with task length without hierarchical planning.

Abstract

from arXiv · show

Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.

1 Introduction

Existing GUI agents often assume actions succeed, so delays and interruptions can leave them repeating ineffective actions without recognizing failure. VeriGUI adds explicit verification and recovery, trained through synthetic failures and offline feedback, and evaluates robustness with failure-aware metrics.

  • Network latency, rendering delays, and system interruptions can invalidate GUI actions while agents continue operating under incorrect progress assumptions.
  • Current agents may repeat unchanged, ineffective actions because failure scenarios are rarely represented during training.
  • Human interaction uses verification, diagnosis, and correction after each action, a loop largely absent from current GUI agents.
  • Online GUI reinforcement learning is costly and unstable, while offline datasets lack explicit signals for action failure and recovery.
  • VeriGUI’s TVAE framework closes the loop through outcome verification, failure diagnosis, and expected-effect prediction.
  • Recovery Success Rate reaches 51.1% for VeriGUI-3B and 52.5% for VeriGUI-7B, surpassing all tested open-source baselines.

2 Related Work

GUI-agent research has progressed from structured interface representations to visual grounding and reinforcement learning, but most systems still assume intended action execution. VeriGUI instead models action–effect consistency explicitly for step-level failure detection and recovery.

  • Earlier GUI agents depended on HTML trees or accessibility metadata, limiting robustness across heterogeneous platforms.
  • Recent VLM-based agents use raw screenshots for UI localization and action prediction without structured interface inputs.
  • Existing approaches typically assume selected actions execute as intended and handle failures implicitly through replanning.
  • VeriGUI explicitly verifies whether each action produces its intended visual outcome, enabling principled step-level failure detection and recovery.
  • GUI-agent reinforcement learning has expanded from trajectory optimization to preference-based and self-generated supervision, with GRPO supporting scalable long-horizon reasoning.

3 Methodology

VeriGUI combines a closed-loop TVAE inference cycle with Robust SFT and GRPO training to verify action outcomes and support self-correction. Offline data and GUI idempotency provide implicit feedback, while composite rewards couple action correctness, expected effects, and verification.

  • VeriGUI jointly predicts Thinking, Verification, Action, and Expected Effect instead of assuming deterministic execution.
  • TVAE links each predicted effect to the next step’s verification hypothesis, preserving temporal consistency across actions.
  • The inference cycle distinguishes successful outcomes from unchanged screens and switches from grounding to diagnosis and recovery when verification fails.
  • Robust SFT mixes successful trajectories with synthetic no-change failures, teaching agents to diagnose failure and generate corrective actions.
  • GRPO uses offline trajectories and GUI error idempotency to simulate environment feedback without live interaction.
  • The composite reward combines action correctness, expected-effect consistency, and asymmetric verification penalties to couple execution with outcome awareness.

4 Experiments

VeriGUI is evaluated across offline, pseudo-online, online, and robustness benchmarks, with ablations testing its training stages and reward design. It improves action selection, task completion under simulated failures, failure recovery, and transfer to live environments.

  • Offline benchmark: VeriGUI-3B reaches 72.2% Type Match and VeriGUI-7B reaches 74.2% on AndroidControl-High, with both leading open-source models at their evaluated scales.The 3B model also surpasses several 7B-scale baselines on Type Match.
  • Cross-distribution and online evaluation: VeriGUI-7B achieves 72.7 TM and 52.3 SR on GUI Odyssey, while VeriGUI-3B surpasses same-scale baselines and records higher Sim-TSR than both 3B baselines on AITW-Gen.The online results indicate transfer beyond the idempotency-based failures modeled during training.
  • Robustness benchmark: VeriGUI-3B reaches 51.1% RSR and VeriGUI-7B 52.5% on the Robustness Benchmark, while their Loop Rates are 24.3% and 15.6%, respectively.The 7B model sets the reported open-source best RSR, and its Loop Rate approaches UI-TARS-7B’s result.
  • Training-stage ablation: Robust SFT with 30% synthetic failures raises RSR from 29.7% to 45.5%, and GRPO reduces ASO from 2.25 to 1.25.Standard SFT improves step accuracy but does not provide a verification benefit, whereas GRPO further refines recovery efficiency.
  • Data and reward ablations: RSR peaks at 45.5% with a 70:30 positive-to-negative sample ratio, while asymmetric verification rewards raise RSR to 48.1% and full effect prediction gives the best overall performance.Excessive negative samples degrade Step Accuracy while yielding diminishing robustness gains.

5 Analysis

Figure 3 shows that training progressively improves failure recognition and recovery, while TVAE adds modest per-step output overhead that is amortized by breaking failure loops.

  • Failure handling: Without verification, the base model repeats failed actions after observing an unchanged screen, producing infinite loops.Robust SFT enables occasional verification, while GRPO supports reliable diagnosis and recovery after coordinate errors.
  • Failure handling: After GRPO, the model detects unchanged screens, diagnoses coordinate errors, and generates successful recovery actions.
  • Efficiency: TVAE increases per-step output length by approximately 45% for both VeriGUI-3B and VeriGUI-7B relative to same-scale baselines.The additional tokens support verification, reasoning, and effect prediction.
  • Efficiency: VeriGUI-3B’s trajectory-level time rises by only ∼26% relative to Qwen2.5-VL-3B, while Tok/Trajectory increases by just 6%.
  • Efficiency: VeriGUI-7B uses 712 Tok/Trajectory versus 847 for Qwen2.5-VL-3B, as stronger recovery eliminates more failure loops than added reasoning tokens cost.Inference runs cloud-side, while the mobile client handles rendering and execution.

6 Conclusion

VeriGUI addresses blind action selection with a Thinking–Verification–Action–Expectation framework and a two-stage training pipeline. Both model sizes show gains across accuracy, completion, robustness, and online evaluation, with transfer across application domains.

  • Contribution: VeriGUI addresses the blind-action limitation through a Thinking–Verification–Action–Expectation framework.
  • Contribution: Its two-stage training pipeline combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards.
  • Findings: VeriGUI-3B and VeriGUI-7B show consistent gains across step-level accuracy, task completion, robustness to execution failures, and online benchmarks.
  • Findings: Cross-distribution evaluation demonstrates reasonable transfer across application domains and dynamic real-world environments beyond controlled training.

Limitations

The evaluation assumes failed actions leave the screen unchanged, and performance degrades as tasks become longer. The analysis also details TVAE’s verification cycle and its benefits for first-step and recovery performance.

  • Limitations: The robustness evaluation does not cover unintended navigations, partial transitions, or application crashes because it assumes failed actions leave the screen unchanged.
  • Limitations: Step-level verification addresses local execution errors but does not substitute for hierarchical planning in long-horizon settings.
  • Task-length analysis: AndroidControl-High analyses stratify tasks into short, medium, and long trajectories, with performance degrading as task length increases.Long tasks exceeding nine steps have fewer than 6% of trajectories completed by all models.
  • Task-length analysis: VeriGUI-3B achieves 33.3% Sim-TSR on short tasks and shows its largest relative advantage on medium-length tasks.
  • TVAE inference cycle: The TVAE inference cycle compares the current screen with the previous expected effect, then selects success or no-change verification and generates grounding or recovery reasoning.
  • First-step analysis: VeriGUI-3B has one of the smallest first-to-all accuracy gaps, with 4.0 points for TM and 5.1 for SR.The analysis associates this pattern with structured reasoning and expected-effect prediction without action history.

A.4 Detailed Robustness Analysis

The detailed robustness analysis attributes VeriGUI-3B’s recovery advantage to improvements in both recovery-action type selection and grounding.

  • Recovery analysis: VeriGUI-3B achieves a 79.1% type match rate on recovery actions, compared with 72.5% for UI-TARS-7B.
  • Recovery analysis: Grounding accuracy for correctly-typed recovery actions reaches 64.6% for VeriGUI-3B versus 62.8% for UI-TARS-7B.These improvements compound to produce a 5.6 percentage point RSR advantage over UI-TARS-7B.

B Algorithm

The evaluation and inference procedures assess GUI agents under simulated unchanged-state failures, measuring whether agents repeat erroneous actions or recover correctly. Task-level performance is also examined across trajectory lengths and action types.

  • Action-type analysis: Click actions dominate the test set, scroll actions have uniformly high accuracy, and open_app accuracy is consistently low because models often predict click instead.Click actions comprise approximately 62% of the test set.
  • Pseudo-online simulation: Task-level metrics are computed in pseudo-online simulation, where correct actions preserve the simulated state and evaluation terminates upon completion or after 2 × TGT steps.The protocol exploits failure idempotency to model unchanged-state outcomes.
  • Robustness evaluation: The robustness benchmark injects unchanged-screen failures and measures whether agents repeat the erroneous action or produce the correct recovery action.Failure slices are constructed from AndroidControl-High trajectories using the prior screen, erroneous action, and incorrect expected effect.
  • Task-level evaluation: Performance degrades substantially as trajectory length increases, consistent with the multiplicative effect of sequential step-level errors.This comparison is reported for AndroidControl-High task-level metrics.

D Implementation Details

VeriGUI is implemented with 3B and 7B Qwen2.5-VL base models and evaluated using separate first-step, all-steps, and robustness metrics on AndroidControl-High. Inference uses greedy decoding on a single A100 GPU.

  • Models and hardware: VeriGUI-3B and VeriGUI-7B use Qwen2.5-VL-3B and Qwen2.5-VL-7B as their respective base models.Training uses 8× NVIDIA A100-80GB GPUs.
  • Evaluation setup: The implementation reports first-step and all-steps performance separately, with first-step metrics lower because task initialization lacks history context.The comparison is reported on AndroidControl-High for VeriGUI-3B.
  • Robustness metrics: Robustness evaluation reports type matching, grounding conditional on correct type matching, and recovery success requiring correct type and parameters.These metrics are defined as TMrec, GRrec, and RSR.
  • Inference and training: Inference uses greedy decoding with temperature = 0 on a single A100 GPU.Stage 1 training takes approximately 12 hours and Stage 2 approximately 20 hours on the reported multi-GPU setup.

E SFT Dataset Construction

The SFT construction combines standard success trajectories with synthetic failure-recovery trajectories, training agents to verify outcomes, diagnose failures, and execute corrected actions. Examples cover unchanged screens, coordinate errors, and network latency.

  • Dataset composition: The Robust SFT dataset contains success trajectories and failure-recovery trajectories to teach standard execution and error recovery.Type A accounts for 70% of samples, while Type B accounts for 30%.
  • Success trajectories: Success samples provide the current screen, instruction, and history, then follow the Think, Verification, Action, and Expected Effect sequence.The output follows the SUCCESS path through the four TVAE components.
  • Failure recovery trajectories: Failure samples simulate a failed prior step so the agent can recognize an unchanged state and recover instead of repeating the ineffective action.The NO_CHANGE path includes verification, diagnosis, corrected grounding, and a recovery action.
  • Two-stage training: Stage 1 trains on success and synthetic-failure samples with supervised learning, while Stage 2 samples multiple outputs and applies action, expected-effect, and verification rewards.The two-stage pipeline is specified in Algorithm 2.
  • Synthetic failure generation: Synthetic failures model realistic patterns including coordinate offsets and semantically related action-type errors, with sampling weights intended to approximate deployment failures.The reported generation pipeline also creates mismatched expected effects and structured reasoning for SUCCESS and NO_CHANGE paths.
  • Error-recovery example: The error-recovery example replaces an ineffective wait with text input after verifying that the screen and track remained unchanged.The recovery searches for “Slipping into Relaxed Sleep” and expects playback controls to appear.
  • Recovery cases: A coordinate-misalignment case recovers by adjusting the click location and succeeds on the second attempt, while a latency case waits two seconds before retrying Compose successfully.These cases illustrate different corrective actions for missed clicks and delayed rendering.
Loading 2604.05477v1…