Source-linked AI summary
VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie
TL;DR
Autonomous GUI agents struggle to verify completion and escape repetitive failures. VLAA-GUI combines mandatory visual completion verification and loop recovery with on-demand search and specialist agents, achieving strong results across Linux and Windows benchmarks, while retaining limitations in planning and memory.
Problem
GUI agents often stop without verifiable evidence and repeat failing actions, while unfamiliar workflows create additional capability challenges.
Method
VLAA-GUI uses a mandatory Completeness Verifier and Loop Breaker after each action, plus on-demand Search, Coding, and Grounding Agents.
Results
VLAA-GUI reaches 77.5% on OSWorld and 61.0% on WindowsAgentArena, with three of five backbones surpassing the 72.4% human OSWorld performance in one pass.
Takeaways & Limitations
The framework’s components consistently improve a strong backbone, while weaker backbones benefit more when the step budget is sufficient.
Takeaways & Limitations
The system uses relatively simple memory and planning without long-horizon decomposition or cross-task knowledge transfer.
Abstract
from arXiv · showhide
Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. We present VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search. First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step -- with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies show that all three proposed components consistently improve a strong backbone, while a weaker backbone benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models.
1 Introduction
VLAA-GUI targets premature task completion and repetitive action loops with mandatory verification and recovery mechanisms, while adding on-demand search and other specialized tools. Across GUI benchmarks and backbones, it reports strong performance and broad gains.
- Core challenges: VLAA-GUI addresses premature completion and repetitive loops through a Completeness Verifier and Loop Breaker applied throughout execution.The Verifier checks UI-observable success criteria, while the Loop Breaker escalates from interaction changes to strategy changes and reflection-driven intervention.
- Modular tools: The framework adds an on-demand Search Agent for unfamiliar workflows, alongside Coding and Grounding Agents for specialized actions.Search results are returned as plain text to the Manager, while coding and grounding are invoked when required.
- Evaluation: 77.5% on OSWorld-Verified is reported for VLAA-GUI with Opus 4.6, surpassing human performance in one pass.The figure caption attributes this result to the system with Opus 4.6 and identifies mitigation of early stopping and repetitive looping as key advantages.
- Evaluation: Three of five backbones surpass human performance of 72.4% in a single pass, while VLAA-GUI reaches 61.0% on WindowsAgentArena.The reported OSWorld backbone results include Opus 4.6 at 77.5%, Opus 4.5 at 74.9%, and Gemini 3.1 Pro at 72.5%.
- Analysis: Ablations show consistent improvements from all three components on a strong backbone, with weaker backbones benefiting more when action budgets are relaxed.The passage reports gains of up to +3.1% each with Sonnet 4.6 and attributes weaker-backbone sensitivity to tool-use costs under tight budgets.
2 Related Work
GUI-agent research spans standardized desktop benchmarks, end-to-end and screen-based agents, modular tool frameworks, and reliability-oriented methods. VLAA-GUI is presented within this broader landscape of planning, search, coding, grounding, reflection, and verification approaches.
- Benchmarks: OSWorld and WindowsAgentArena established standardized desktop evaluations that exposed a large gap between agent and human performance.OSWorld contains 369 Ubuntu Linux tasks, while WindowsAgentArena provides a complementary Windows-only suite.
- Benchmarks: Related benchmarks extend GUI evaluation across domains and platforms, including data science, visual grounding, macOS, mobile, and web settings.The cited examples include Spider2-V, ScreenSpot, macOSWorld, and multiple mobile and web benchmarks.
- Agent architectures: End-to-end and screen-based agents pursue GUI interaction directly from visual inputs without relying on HTML or accessibility trees.The related systems include UI-TARS, AGUVIS, ShowUI, CogAgent, OS-Atlas, and other screen-based agents.
- Modular frameworks: Modular frameworks combine multimodal models with planning, memory, search, coding, synthetic experience, or platform-specific control.Examples include Agent S, OS-Symphony, GTA1, CoAct, EvoCUA, UFO, and AutoGLM.
- Reliability: Prior reliability work uses mid-trajectory correction, explicit feedback, structured reflection, verifier-based training, and step-wise checking.These approaches include ReAct, Tree of Thoughts, Reflexion, Self-Refine, and verification methods for reasoning and digital agents.
- VLAA-GUI positioning: VLAA-GUI organizes its system around mandatory Loop Breaker and Completeness Verifier tools plus on-demand Searcher, Coder, and Grounder modules.The overview figure distinguishes mandatory post-action reliability tools from flexible tools deployed by the Manager.
3 The VLAA-GUI System
VLAA-GUI uses a Manager Agent with two mandatory post-action reliability checks and three on-demand specialist agents. The Completeness Verifier gates termination using visual evidence, while the Loop Breaker escalates responses to repeated failures and states.
- 3.1 Overview: At each step, the Manager receives a belief state containing the instruction, trajectory history, observation, and tool results before issuing a UI action or invoking a specialist.The specialist agents are Searcher, Coder, and Grounder, and the Manager controls when they are deployed.
- 3.1 Overview: The Loop Breaker and Completeness Verifier run after every action, whereas Searcher, Coder, and Grounder remain optional.The mandatory checks maintain execution reliability by detecting stagnation and cross-examining completion claims.
- 3.2 Completeness Verifier: The Completeness Verifier derives UI-observable success criteria and requires self-verification before the Manager can declare completion.Its Completion Gate operates at every step and represents termination as conditional on passing all criteria and UI stability.
- 3.2 Completeness Verifier: An independent MLLM judge checks completion claims using the task, current observation, and recent trajectory, returning an accept or reject decision.Termination requires agreement between the checking modules, and rejection reasons are added to the trajectory for subsequent actions.
- 3.3 Loop Breaker: The Loop Breaker uses action and screen-state repetition counters to detect local failures, navigation dead-ends, and broader looping patterns.Its three tiers respectively address local action failures, navigation dead-ends, and patterns that local Manager checks may miss.
- 3.3 Loop Breaker: Repeated unchanged actions trigger a modality switch, repeated screen states trigger a strategy change, and an external judge can force a hard strategy switch.The enforced alternatives include keyboard, menus, command-line actions, programmatic editing, and coding-agent invocation.
- On-demand agents: The Search Agent returns targeted textual search results for unfamiliar workflows, while the Coding Agent and Grounding Agent support programmatic edits and precise coordinate generation.The Coding Agent uses an independent Python/Bash loop, and the Grounding Agent combines visual cues with semantic context.
4 Experiments
VLAA-GUI is evaluated on Linux and Windows desktop benchmarks across step budgets, backbones, and component ablations. It achieves strong cross-platform performance, while analyses show that verification, search, and loop breaking contribute differently depending on model strength and available steps.
- Benchmarks: VLAA-GUI is evaluated on 361 OSWorld tasks and 154 WindowsAgentArena tasks across Ubuntu and Windows environments.OSWorld spans Web, Office, File, Multimedia, and Daily domains; WindowsAgentArena covers productivity, file management, and system configuration.
- Main Results: 77.45% with Opus 4.6 and 74.89% with Opus 4.5 are achieved on OSWorld at 100 steps, surpassing reported human performance of 72.4%.Three of five backbones surpass human performance at 100 steps, including Gemini 3.1 Pro at 72.47%.
- Main Results: 61.0% on WindowsAgentArena at 100 steps exceeds Agent S3 w/ GPT-5 by at least 4.4% and GTA1 w/ o3 by 9.2%.VLAA-GUI reaches 60.4% at 50 steps and 61.0% at 100 steps.
- Step Efficiency: 64.75% with Opus 4.6 and 64.13% with Sonnet 4.6 at 15 steps surpass the best reported 50-step system, while Opus 4.6 reaches 73.85% at 50 steps.Sonnet 4.6 scores 71.1% at 50 steps, exceeding OS-Symphony w/ GPT-5 by 7.5%.
- Ablations: All three modules improve the full system in OSWorld ablations, with the Completeness Verifier contributing −3.1% for Sonnet 4.6 and Search Agent −1.9%.On WAA at 50 steps, removing Search Agent, Completeness Verifier, and Loop Breaker drops accuracy by 11.0%, 9.1%, and 7.8%, respectively.
- Analysis: The Loop Breaker nearly halves wasted steps for Gemini 3 Flash, reducing them from 4.9% to 2.8%, while tool benefits depend on step budget.Sonnet 4.6 benefits consistently, whereas Gemini 3 Flash can be harmed at 15 steps by verifier and search overhead.
5 Conclusion
VLAA-GUI addresses premature task completion and repetitive action loops with three targeted components, achieving strong results across OSWorld and WindowsAgentArena while exposing remaining limitations in planning and memory.
- VLAA-GUI combines a Completeness Verifier, Loop Breaker, and Search Agent to address premature completion and repetitive action loops.The framework enforces UI-observable success criteria, escalates recovery strategies, and retrieves procedural knowledge on demand.
- 77.5% on OSWorld with Opus 4.6 surpasses human-level performance, while WindowsAgentArena performance reaches 61.0%.The conclusion reports top-tier results on both benchmarks.
- Ablations show that each proposed component contributes meaningfully, with benefits depending on backbone strength and step budget.
- False completion remains the dominant failure mode across backbones, motivating continued research on completion verification.
- The current system uses simple memory and planning without long-horizon task decomposition or cross-task knowledge transfer.More sophisticated memory and planning strategies may improve complex multi-step tasks.
A Appendix
The appendix supplies implementation details, the full Manager action space, system prompts, and additional ablation results.
- The supplementary material provides implementation details, prompt templates, the full action space, and additional ablation results.
- Section A.1: Section A.1 covers implementation details, including model configurations and benchmark categories.
- Section A.2: Section A.2 presents the full action space exposed to the Manager Agent.
- Sections A.3–A.4: Section A.3 gives system prompts for different agent roles, and Section A.4 reports additional ablation results.
Configurations.
The experimental configurations assign model backbones and specialized models across the Manager, Reflection, Completeness Verifier, Coding, Search, and grounding components.
- Table 5 summarizes model assignments for each component across the experimental configurations.
- OSWorld configurations: Three OSWorld configurations use Claude Opus 4.5, Claude Sonnet 4.6, or Gemini 3 Flash as shared backbones for four components.These components are the Manager, Reflection Agent, Completeness Verifier, and Coding Agent.
- OSWorld configurations: The three shared-backbone configurations use Gemini 3 Pro as the Search Agent.
- Additional configurations: Additional configurations vary the grounding model or use Gemini 3.1 Pro as the Search Agent or across nearly all components.
Hyperparameters.
VLAA-GUI uses iterative planning with experiment-dependent action budgets, conservative verification, separate Coding and Search budgets, and screenshot-based observations in hosted Linux and Windows environments.
- All agents operate in iterative planning mode without hierarchical subtask decomposition.
- Per-task budgets are 15, 50, or 100 actions depending on the experiment, while the Coding Agent has 20 code execution steps.
- The Completeness Verifier uses temperature T=0.2, whereas all other components use T=1.0.
- The VLM Search Agent has 20 browsing steps, while the LLM variant performs a single query.
- Experiments run in official Ubuntu Docker containers and Windows environments hosted on Amazon Web Services, with 1920×1080 screenshots as the primary observation modality.
Reported Categories.
The benchmarks organize GUI tasks into functional categories, with OSWorld spanning operating-system, office, daily-use, professional, and cross-application workflow tasks.
- OSWorld groups tasks into OS, Office, Daily, Professional, and Workflow categories.Office covers LibreOffice Calc, Impress, and Writer; Daily covers Chrome, VLC Player, and Thunderbird; Professional covers VS Code and GIMP.
A.2 Action Space
The full-system configuration exposes model-judged reflection and completion-verification actions that require evidence-based decisions about progress, feasibility, looping, strategy, and task completion. Ablations show that verifier and Loop Breaker effects depend on backbone and step budget.
- Reflection and control: The Reflection Agent analyzes each Manager action and can force a strategy change when its Strategy signal is SWITCH.It produces structured progress, outcome, loop, feasibility, termination, strategy, and verdict signals.
- Reflection and control: The reflection protocol uses DONE only when all success criteria have specific visible evidence, FAIL when infeasibility is demonstrated, and CONTINUE otherwise.A LOOP signal or uncertain/impossible feasibility usually triggers SWITCH.
- Completion verification: The Completeness Verifier accepts completion only when every requirement has direct, unambiguous evidence, preferring false negatives over false positives.It checks the current screenshot and recent history, requires visible confirmation for side effects, and favors stable UI states.
- Completion verification: Exact-value tasks require the specified value to be readable on screen or confirmed in trajectory text; otherwise the result is marked incomplete.The verifier also checks semantic equivalence for workarounds, required file formats, and explicit mode or state indicators.
- Ablation findings: 3–4%: Sonnet 4.6’s DONE accuracy improvement with the Completeness Verifier at every step budget, alongside consistently reduced false DONE rates.For Gemini 3 Flash, false DONE falls from 80.2% to 52.6% at 50 steps and from 38.7% to 13.7% at 15 steps.
- Ablation findings: +2.78%: Gemini 3 Flash’s Loop Breaker gain at 50 steps, while its effect is +1.82% at 100 steps and −6.15% at 15 steps.Sonnet 4.6 gains +1.4% at 15 and 50 steps, but only +0.04% at 100 steps; short budgets can make escalation overhead costly.