Source-linked AI summary
Computer-Use Agents as Judges for Generative User Interface
Kevin Qinghong Lin, Siyuan Hu, Linjie Li, Zhengyuan Yang, Lijuan Wang, Philip Torr, Mike Zheng Shou
TL;DR
Existing GUIs are optimized for humans, leaving a gap in agent-native interface design and evaluation. The paper introduces AUI-Gym, verifiers, and a Coder–CUA collaboration loop in which Coders design while CUAs judge and guide revisions. Across the benchmark, task solvability is foundational, navigation is the main bottleneck, and feedback-driven redesigns improve execution success and robustness.
Problem
Most GUIs prioritize human aesthetics and usability, forcing CUAs to adopt human-oriented behaviors instead of efficient agent-native interaction.
Source: §1 · p. 1Method
AUI-Gym combines 52 applications, 1560 validated tasks, programmatic verifiers, and a Coder–CUA loop with dashboard-compressed navigation feedback.
Source: §1 · p. 3, §3.3 · p. 5, §4 · p. 6Results
Task solvability is foundational while CUA navigation is the main bottleneck; feedback-driven redesigns substantially improve execution success and robustness.
Source: §1 · p. 3, §6 · p. 11Takeaways & Limitations
Positioning agents as both designers and judges supports interface design oriented toward agent-native efficiency and reliability.
Source: §1 · p. 2, §6 · p. 11
Abstract
from arXiv · showhide
Computer-Use Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI). Yet, most GUI remain designed primarily for humans--prioritizing aesthetics and usability--forcing agents to adopt human-oriented behaviors that are unnecessary for efficient task execution. At the same time, rapid advances in coding-oriented language models (Coder) have transformed automatic GUI design. This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design? To investigate, we introduce AUI-Gym, a benchmark for Automatic GUI development spanning 52 applications across diverse domains. Using language models, we synthesize 1560 tasks that simulate real-world scenarios. To ensure task reliability, we further develop a verifier that programmatically checks whether each task is executable within its environment. Building on this, we propose a Coder-CUA in Collaboration framework: the Coder acts as Designer, generating and revising websites, while the CUA serves as Judge, evaluating functionality and refining designs. Success is measured not by visual appearance, but by task solvability and CUA navigation success rate. To turn CUA feedback into usable guidance, we design a CUA Dashboard that compresses multi-step navigation histories into concise visual summaries, offering interpretable guidance for iterative redesign. By positioning agents as both designers and judges, our framework shifts interface design toward agent-native efficiency and reliability. Our work takes a step toward shifting agents from passive use toward active participation in digital environments. Our code and dataset are available at https://github.com/showlab/AUI.
1 Introduction
The paper reframes GUI design as an agent-native collaboration: a Coder creates and revises interfaces while a CUA judges task solvability and navigation. It introduces AUI-Gym and a dashboard-based feedback loop, finding that functional completeness is foundational while navigation remains the main bottleneck.
Motivation
- Human-centric GUIs force CUAs to imitate human-oriented behaviors and prioritize stylistic details that are redundant for efficient task completion.The paper motivates designing environments for agents themselves rather than optimizing only for human aesthetics and usability.Source: §1 · pp. 1–2
Coder–CUA Collaboration
- The Coder–CUA framework treats the Coder as Designer and the CUA as Judge, using navigation outcomes and task solvability as complementary feedback.The Coder initializes and revises UIs, while the CUA executes tasks and evaluates functionality.Source: §1 · p. 2
AUI-Gym
- AUI-Gym covers 52 applications across six domains with 1560 GPT-5–proposed, human-validated tasks and per-task rule-based checkers.The benchmark supports human-free automatic GUI creation and testing.Source: §1 · p. 3
CUA Dashboard
- A single-image 1K CUA Dashboard compresses task goals, actions, intermediate states, and outcomes, reducing visual tokens by 76.2% while preserving redesign cues.It highlights key interactive regions with adaptive scaling to make multi-step feedback concise and interpretable.Source: §1 · p. 3
Evaluation Insights
- Task solvability is foundational and improves through failure-driven functional summarization, whereas CUA navigation remains the primary bottleneck.Feedback-driven redesigns including de-stylization, higher contrast, and simplified layouts substantially improve execution success and robustness.Source: §1 · pp. 2–3
2 Related Works
Related work establishes CUAs for GUI automation and automatic interface generation, but existing benchmarks generally evaluate agents in fixed human-oriented environments. This paper instead targets automatically generated, agent-evaluated interfaces.
GUI Automation
- GUI automation agents commonly learn to complete tasks by imitating human trajectories through in-context examples or multimodal foundation models.This line of work supports autonomous interaction but remains tied to human demonstrations.Source: §2.1 · p. 3
Automatic Interface Design
- Automatic interface design and software-automation research provides semantic components and benchmarks, but existing interaction benchmarks assume agents operate within fixed environments.Examples include accessibility layers, ARIA tags, declarative frameworks, WebArena, and GAIA.Source: §2.2 · p. 3
3 AUI-Gym Benchmark
AUI-Gym benchmarks automatic GUI development with structured application requirements, diverse executable tasks, human quality control, and test-time verification. It evaluates whether generated websites support functional tasks through rule-based checkers and agent interaction.
3.1 Task Definitions
- Structured queries specify each website’s name, goal, functional features, and UI theme, while outputs must be fully functional application-level websites rather than static pages.The websites must support navigation, transitions, button interactions, and completion of functional goals.Source: §3.1 · p. 4
3.2 Task Creation
- GPT-5 proposes practical candidate tasks, humans filter them using domain-specific principles, and each application receives 30 curated tasks.Human review removes trivial, nonsensical, ambiguous, or out-of-scope tasks while preserving domain-relevant interaction demands.Source: §3.2 · p. 4
3 AUI-Gym Benchmark
- AUI-Gym covers 52 web applications across six domains, with 30 tasks per application for 1,560 total tasks spanning varied interaction paradigms and functional complexities.The domains are App, Landing, Game, Interactive, Tool, and Utility.Source: §3.2 · pp. 4–5
3.3 Evaluation with Verifiers
- A test-time Verifier analyzes each generated website and task, retaining solvable tasks and producing task-specific JavaScript checkers that encode success conditions.The verifier reasons over available elements, states, properties, and interaction paths; unsolvable tasks are discarded.Source: §3.3 · p. 5
- Evaluation combines CUA Success Rate, which counts completed tasks and checker failures, with Function Completeness, which measures whether functional checkers exist independently of CUA navigation.These metrics separate agent execution success from basic task validity.Source: §3.3 · pp. 5–6, Figure 4 · p. 6
4 CUA–Coder in Collaboration
The framework treats the Coder as a designer that iteratively updates UIs and the CUA as a judge that evaluates them through task solvability and navigation feedback. A CUA Dashboard compresses interaction evidence into compact visual guidance for redesign.
Overview
- The framework reverses the conventional setup: the CUA evaluates the current UI, while the Coder integrates feedback into successive design updates.The UI evolves deterministically from E_t to E_t+1 using feedback produced by CUA interaction.Source: §4 · p. 6
Coder as Designer
- The Coder first generates a complete HTML-rendered environment from a detailed user query, then revises it iteratively from language-form feedback.The revision rule is E_t+1 ← (E_t, R_t), where R_t summarizes feedback from the preceding evaluation.Source: §4.1 · p. 7
CUA as Judge
- CUA feedback combines task solvability checks that identify missing functionality with navigation trials that record actions, observations, and outcomes.Unsolvable tasks are filtered before navigation and returned as functionality failures; solvable tasks proceed through interaction trajectories.Source: §4.2 · pp. 7–8
Evaluation
- Evaluation reports function completeness and CUA success rate as the central outcomes of the collaboration process.The main-results table is organized by coder with function completeness above CUA success rate.Source: Table 3 · p. 8
CUA Dashboard
- The CUA Dashboard converts long navigation histories into one fixed-resolution image by tiling cropped interactive regions and preserving their temporal order.This representation reduces visual content by 76.2% while retaining cues needed for UI refinement.Source: Table 2 · p. 7, §4.2 · p. 8
5 Experiments
Experiments show that integrated task-solvability and navigation feedback improves functionality and CUA execution, while dashboard design and repeated revisions affect gains differently across coders. Qualitative cases show functional revisions adding missing controls and CUA-driven revisions simplifying interfaces for agent accessibility.
5.2 Main Results
- 81.5% function completeness after integrated GPT-5 revision, up from 67.9%, with maximum domain gains of 31.4%.Integrated revision improves all domains and coders, whereas either feedback source alone does not guarantee completeness gains.Source: §5.2 · p. 9
- 6.8% average improvement in CUA success rate from the framework, with weak coders gaining as much as 11.7%.Open-source CUAs initially average 13.5% overall success, and navigation feedback optimizes interfaces for agent-centric execution.Source: §5.2 · p. 9
5.3 Ablation Studies
- The Dashboard raises GPT-5 function completeness from 62.1% to 70.8% and CUA success rate from 18.7% to 25.7% versus action-only feedback.Screenshot-only feedback mostly performs worst, indicating that visual and textual evidence together better support refinement.Source: §5.3 · p. 9
- Repeated revision consistently improves function completeness, but CUA success may saturate for GPT-5 while continuing to improve for Qwen3-Coder-30B and GPT-4o.The pattern suggests stronger coders may reach diminishing revision gains, whereas weaker coders benefit more from iteration.Source: §5.3 · p. 9
5.4 Qualitative Comparison
- Function-test revisions add unsupported functionality, whereas CUA-test revisions make interfaces more accessible through clearer controls, higher information visibility, and reduced scrolling.Examples include adding a Running indicator or delimiter button, alongside clearer button boundaries and visual hints.Source: §5.4 · pp. 9–10
6 Conclusion
The paper introduces AUI-Gym and a Coder–CUA loop that designs interfaces for agent-native interaction. Its conclusion identifies task solvability as foundational, CUA navigation as the main bottleneck, and feedback-driven redesign as a route to higher execution success and robustness.
6 Conclusion
- AUI-Gym contains 52 applications and 1560 tasks with programmatic checkers, supporting automatic GUI-development evaluation.The benchmark frames UI design as an agent-native loop with the Coder as Designer and CUA as Judge.Source: §6 · p. 11
- Task solvability is foundational and readily improved through failure-driven functional summarization, whereas CUA navigation remains the primary bottleneck.The conclusion distinguishes missing functionality from difficulty executing multi-step interaction.Source: §6 · p. 11
- Feedback-driven redesigns such as de-stylization, higher contrast, and simplified layouts raise execution success and robustness.These changes reflect designing interfaces for agents rather than merely adapting human-centric interfaces.Source: §6 · p. 11
A Additional Results
Additional results compare CUA policies, dashboard variants, and verifier strategies for revising generated UIs. The findings identify policy-dependent navigation gains, dashboard efficiency, and the unreliability of final-screenshot VLM judging.
CUA policy comparison
- UI-TARS and Operator yield comparable functional-completeness gains, while Operator gains more CUA Success Rate with GPT-5 and UI-TARS slightly outperforms on Qwen3-Coder-30B.UI-TARS tends to expose richer failure cases, whereas Operator’s navigation strengths are most useful on more complex UIs.Source: A Additional Results · p. 13, Figure 7 · p. 13
- Operator produces consistent gains in CUA Success Rate and Function Completeness Rate during integrated revision, with stronger navigation in hard game and app domains.The supplied table caption defines each cell as CUA Success Rate with Function Completeness Rate in parentheses.Source: A Additional Results · p. 13, Table 4 · p. 13
Dashboard ablations
- 70.4% average token usage reduction makes the proposed Dashboard more efficient than screenshot-only commentary while retaining the strongest performance among dashboard variants.The Dashboard combines visual screenshots and textual interaction histories into refinement cues, though effective use requires strong visual perception.Source: A Additional Results · p. 13, Table 5 · p. 14
Verifier analysis
- Final-screenshot-only VLM evaluation is unreliable, showing near-0.55 balanced accuracy and low recall, F1, and Cohen’s κ against rule-based oracle evaluation.The results indicate that reliable judging may require more screenshots across the trajectory, increasing computational cost.Source: A Additional Results · p. 14, Table 6 · p. 14
B Prompts Usage
The prompts operationalize AUI-Gym task generation, initial website creation, failure-driven revision, and UI analysis. They emphasize realistic workflows, observable outcomes, complete functionality, and targeted fixes while preserving working behavior.
Task generation
- Task prompts request 30 diverse, realistic scenarios categorized as core functions, user workflows, or edge cases with precise observable terminal outcomes.Domain-specific prompts cover utilities, interactive experiences, landing pages, apps, games, and tools without external-file dependencies.Source: B Prompts Usage · pp. 14–16
Initial Website Generation
- Initial website prompts require complete, self-contained HTML applications with embedded CSS and JavaScript, semantic structure, responsiveness, unique automation IDs, and 1280x720 compatibility.The generated page must be fully functional and interactive using modern HTML5, CSS3, and vanilla JavaScript.Source: B Prompts Usage · pp. 16, 17, 20
Revision from CUA Failures
- Revision prompts use automated failure analysis to identify missing elements, functionality, timing, event-handling, state-management, and feedback problems.They require preserving existing successful functionality, adding unique IDs, implementing complete handlers and state updates, and providing immediate UI feedback.Source: B Prompts Usage · p. 17
- Revision requirements distinguish basic usage, workflows, advanced features, and edge cases, while insisting that fixes remain permissive and do not block task inputs.The prompts also require functional logic to update underlying state before exposing proxies or visible outputs.Source: B Prompts Usage · p. 17
C Full Statistics and Examples
AUI-Gym covers six application domains with generated websites, realistic task sets, and rule-based verification. The statistics and examples provide the benchmark’s domain-level composition and evaluation structure.
Full Statistics and Examples
- AUI-Gym organizes evaluation across six domains, pairing each GPT-5-generated website with 30 simulated real-world tasks and a rule-based verifier.The supplied table description presents the distribution and examples for each domain.Source: C Full Statistics and Examples · p. 21, Table 7 · p. 22