Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents

1 Introduction

• General CUAs struggle to generalize to unseen interfaces, and their performance depends partly on how interfaces represent states, expose actions, and organize interaction semantics.The paper frames interface design as an under-explored complement to improving agent architectures.Source: §1 · p. 1
• The paper distinguishes heuristics that transfer to agents from those needing adaptation to support stable perception, reduce reasoning burden, and ground actions reliably.The latter remain valuable for human usability but need additional design considerations when automation is an explicit objective.Source: §1 · p. 2
• The framework offers two general design suggestions and four agent-specific heuristics: make states perceivable, keep layouts and controls consistent, provide step-wise signals, and expose agent-friendly documentation and skills.The broader suggestions are to consider CUA compatibility early and use a human-centered, agent-aware perspective.Source: §1 · pp. 2–3
• UI-Verse compares functionally similar environments with differing interface heuristics and evaluates CUAs on task success, efficiency, and interaction failures.The paper also uses human studies to assess whether augmentations disrupt workflows or degrade usability.Source: §1 · pp. 2–3
• The proposed heuristics improve CUA reliability and efficiency while preserving human usability when designed appropriately.When combined, they yield peak gains of up to 0.46 absolute gain on related tasks.Source: §1 · pp. 2–3

User-Centric Interface Design

• Human-centered design research emphasizes visibility, consistency, error prevention, recovery, and predictable direct manipulation, but does not systematically address reliable operation by general UI-centered agents.Related work on AI-mediated interfaces likewise primarily optimizes for human users.Source: §2 · p. 3

General Computer-Use Agents

• Computer-use-agent research advances benchmarks, native GUI agents, reasoning, memory, sampling, and skill acquisition, but generally treats the UI as fixed and adaptation as the agent’s burden.In contrast, this paper treats interface design as an optimization target for human-agent collaboration.Source: §2 · p. 3

3.1 Revisiting Nielsen’s ten Usability Heuristics for Interface Design

• Consistency, efficient workflows, explicit recognition cues, and error prevention or recovery transfer naturally because they support agent knowledge reuse, reduce steps, and guide reliable action.These principles are broadly beneficial for CUA performance.Source: §3.1 · p. 4
• Implicit cues, metaphor-based controls, flexible paths, and documentation-heavy support can hinder agents, whose perception, memory, grounding, and recovery differ from humans’.The principles remain valuable for people; their original formulations do not fully address agent requirements.Source: §3.1 · pp. 4–5

3.2 General Suggestions for Human-Centered, Agent-Compatible Interface Design

• Consider agent compatibility early by checking whether important states are observable, actions easy to ground, and workflows stable enough for reliable execution.Source: §3.2 · p. 5
• Keep interfaces readable and trustworthy for humans while exposing enough structure for agents, so people can understand, supervise, and intervene in workflows.Source: §3.2 · p. 5

3.3 Operationalizing Agent-Compatible Design: CUA-Specific Heuristics

• Make visual states explicitly visible, stable, and semantically clear throughout task execution, rather than relying on subtle or transient cues.Source: §3.3 · p. 5
• Keep layouts and control semantics predictable across similar contexts so agents can transfer learned action mappings instead of repeatedly rediscovering them.Source: §3.3 · p. 5
• Provide explicit, attainable step-wise controls for progress and recovery; a single clear path can reduce ambiguity and help agents verify their actions.Source: §3.3 · p. 6
• Expose frequently used procedures as operable skills, with simple documentation access, so agents can retrieve, execute, and verify reusable steps without unnecessary exploration.Source: §3.3 · p. 6

4.1 Experimental Setup

• UI-Verse compares baseline interfaces implementing Nielsen’s heuristics with revised interfaces adding one or more proposed heuristics on tasks targeting the affected UI regions.The study compares task success, efficiency, and trajectory error types across interface variants.Source: §4.1 · p. 6, Figure 2 · p. 7

4.2 Main Results

• Combined heuristics also bring large execution-time reductions and lower grounding and reasoning errors, with more modest decreases in perception errors.Source: §4.2 · p. 7, Table 2 · p. 8
• Different heuristics reduce different failure types: A most consistently improves grounding, B and C mainly reduce reasoning errors, and D gives the strongest reasoning-error reduction.D also improves grounding in several cases.Source: §4.2 · p. 7
• Across evaluated settings, each single heuristic consistently raises task success, and combining all four produces further gains over baseline UIs.Table 2 reports agent performance and trajectory error attribution; the largest single-heuristic absolute gain is 35% for Qwen3-VL under Heuristic D.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Human Studies

• Most human judgments were ties across aesthetics, interaction fluency, layout coherence, and overall usability, suggesting the revisions generally preserve human usability.Annotators sometimes preferred the revised UI, especially under Heuristic A.Source: §4.3 · pp. 8–9, Figure 3 · p. 9
• The automatic trajectory analyzer’s error categories broadly align with human judgments across all four heuristics, including similar distributions of error-type shifts.Source: §4.3 · p. 9

• The paper treats agent-compatible interface design as complementary to model improvements and reports that augmented interfaces can improve task success and often efficiency without clear usability regressions.Source: §5 · p. 9
• UI-Verse provides a controlled benchmark for examining how interface design affects multiple computer-use agents on interfaces with and without augmented heuristics.Source: §5 · p. 9
• The authors describe the work as an initial step toward digital environments usable by people and dependable for general computer-use agents.Source: §5 · p. 9

Reproducibility Statement

• The authors state that they will publicly release the codebase and data upon acceptance to support replication and future extensions.Source: Reproducibility Statement · p. 10

A UI-Verse with Augmented Usability Heuristics

• UI-Verse pairs functionally similar interfaces with different heuristic sets, enabling comparisons on comparable task goals.Source: A UI-Verse with Augmented Usability… · p. 13

A.1 List of Functionalities Implemented

• MiniWord implements 63 document-processing unit functionalities, while broader and more complex functionality coverage is left to future work.The environments preserve the same functionality set while varying interface realization.Source: §A.1 · p. 13

A.2 Augmented Heuristics in Interface Design

• Figures 4–6 provide representative baseline–revised interface pairs, with red boxes and captions identifying the changes and their agent benefits.Source: §A.2 · p. 13
• Revised buttons avoid hover-only descriptions by keeping functionality labels visible, reducing agent confusion about button meanings.Source: Figure 5 · p. 14
• The combined revision uses larger buttons, visible text descriptions, consistent sidebars, consistent controls, shortcuts to return to prior states, and a documented skill set.These changes extend beyond visible interface modifications to environment support for agents.Source: Figure 6 · p. 14

B.1 Experimental Settings

• Four volunteer researchers with computer science and AI backgrounds participated, using separate annotation interfaces for the two studies.Figure 7 shows the first study’s annotation interface.Source: §B.1 · p. 14, Figure 7 · p. 15
• In the first study, annotators compared six baseline and six revised screenshots across human-centered UI quality dimensions after learning their definitions.Source: §B.1 · p. 14
• In the second study, annotators reviewed full CUA trajectories and assigned probabilities to perception, grounding, and reasoning errors.Trajectories included intermediate plans, reasoning, actions, and resulting UI changes.Source: §B.1 · p. 14
• Human annotators and the automatic evaluator showed strong agreement in overall error patterns, including similar heuristic-related shifts across perception, grounding, and reasoning.Source: Table 4 · p. 15

B.2 More Detailed Experimental Results

• Human annotators and the automatic analyzer showed similar shifts in perception, grounding, and reasoning error distributions under different interface heuristics.Their absolute error proportions did not always match.Source: §B.2 · p. 15

C Discussions

• The study focuses on general UI-centered agents using existing software through the interface surface shared with human users, not privileged back-end access.The framing emphasizes human supervision, interpretability, and intervention.Source: C Discussions · p. 15
• Automation-native systems using APIs, protocols, or structured tools are an alternative direction outside the paper’s primary target; human-understandable interfaces remain relevant where oversight and intervention matter.Source: C Discussions · pp. 15–16