Source-linked AI summary
ShowUI-Aloha: Human-Taught GUI Agent
Yichun Zhang, Xiangwu Guo, Yauhong Goh, Jessica Hu, Zhiheng Chen, Xin Wang, Difei Gao, Mike Zheng Shou
TL;DR
Complex GUI automation is hindered by scarce, difficult-to-annotate desktop interaction data, despite the richness of human demonstrations. ShowUI-Aloha converts in-the-wild demonstrations into structured teaching trajectories and uses planning and execution to operate desktop interfaces. It completes 217 of 361 evaluated tasks, while remaining limited by fine-grained element localization and precise drag-based text editing.
Problem
Desktop GUI training data is difficult to scale because human demonstrations are unannotated, untrimmed, and lack explicit task boundaries.
Method
ShowUI-Aloha records and parses in-the-wild human desktop interactions into structured teaching trajectories, then uses a planner and actor for grounded execution.
Results
60.1% overall success rate: Aloha successfully completes 217 of 361 evaluated OSWorld-style tasks.
Takeaways & Limitations
A single human demonstration often generalizes to an entire task group sharing the same workflow logic, with human-derived traces and temporally conditioned planning both important in ablations.
Takeaways & Limitations
Aloha remains weak in fine-grained element localization and precise drag-based text editing, especially when icons are visually similar or text boundaries are difficult to select.
Abstract
from arXiv · showhide
Graphical User Interfaces (GUIs) are central to human-computer interaction, yet automating complex GUI tasks remains a major challenge for autonomous agents, largely due to a lack of scalable, high-quality training data. While recordings of human demonstrations offer a rich data source, they are typically long, unstructured, and lack annotations, making them difficult for agents to learn from.To address this, we introduce ShowUI-Aloha, a comprehensive pipeline that transforms unstructured, in-the-wild human screen recordings from desktop environments into structured, actionable tasks. Our framework includes four key components: A recorder that captures screen video along with precise user interactions like mouse clicks, keystrokes, and scrolls. A learner that semantically interprets these raw interactions and the surrounding visual context, translating them into descriptive natural language captions. A planner that reads the parsed demonstrations, maintains task states, and dynamically formulates the next high-level action plan based on contextual reasoning. An executor that faithfully carries out these action plans at the OS level, performing precise clicks, drags, text inputs, and window operations with safety checks and real-time feedback. Together, these components provide a scalable solution for collecting and parsing real-world human data, demonstrating a viable path toward building general-purpose GUI agents that can learn effectively from simply observing humans.
1 Introduction
ShowUI-Aloha addresses the scarcity and poor structure of desktop GUI training data by converting in-the-wild human interactions into semantically grounded teaching trajectories. Its planner–actor system then generalizes those demonstrations across changing interfaces, supported by an open-source framework and evaluation on OSWorld-style tasks.
- Motivation: Existing GUI datasets require extensive manual annotation and focus largely on websites and mobile applications, limiting performance in complex desktop environments.Human workflows provide abundant interaction data, but their recordings are typically unannotated, untrimmed, and without explicit task boundaries.
- Approach: ShowUI-Aloha records natural desktop interactions and parses them into compact, semantically meaningful teaching trajectories that retain user intent while abstracting away low-level pixels.The record–parse–learn framework captures keyboard–mouse events and screen activity during normal workflows.
- Approach: Its planner–actor mechanism executes learned trajectories on live desktops while remaining robust to UI drift, layout changes, and unexpected system states.The system uses natural-language abstraction and grounding against the live desktop environment to generalize demonstrations to new tasks and unseen UI states.
- Evaluation: ShowUI-Aloha evaluates its generalization on a large-scale suite of OSWorld-style tasks spanning diverse desktop applications.The evaluation uses a user-oriented protocol focused on real-world applicability.
- Contributions: The framework is released fully open source, including its recorder, learner, planner, actor, and evaluation tools, to support reproducible and extensible GUI-agent research.The work presents this release as a practical and scalable foundation for demonstration-driven computer-use intelligence.
- Contributions: Aloha distills reusable task guidance from a single human demonstration and applies it to new task variants and interface layouts.The paradigm is designed to avoid trial-and-error interaction while supporting stable execution across changing interfaces.
2 Related Work
Related work extends language-model agents and vision-language-action models toward GUI automation, while human demonstrations offer a complementary source of action-rich experience. These lines of research motivate scalable methods for learning from real user behavior.
- GUI Agents: Early GUI agents used large language models to decompose user tasks and generate action plans, often in zero-shot settings based on web-derived knowledge.Later systems moved toward unified GUI vision-language-action models trained on screenshots and human-labeled interaction trajectories.
- Learning from Human Demonstration: Human demonstrations provide data-efficient records of experience and action for learning in both physical and digital environments.Prior work used demonstrations for video-based action understanding and complex robot manipulation.
3 Method
ShowUI-Aloha converts in-the-wild human desktop demonstrations into structured, semantically grounded traces, then plans and executes tasks through an actor that uses screenshots, demonstrations, and live UI context.
- Framework Overview: The framework combines recording, learning, planning, and execution to turn human demonstrations into actionable desktop workflows.The workflow records demonstrations, converts them into structured traces, generates execution plans, and performs actions on the computer.
- Recorder: The portable Recorder captures full-resolution screen video and dense, timestamped user actions, including clicks, movements, drags, scrolls, and keystrokes.Video capture runs at 30 FPS, while interaction logging records detailed action histories alongside timestamps.
- Aloha Learner: The Learner cleans noisy event streams into grouped interaction primitives, generates marked screenshots, and produces semantic natural-language traces with a vision–language model.It combines full-screen context, zoomed interaction crops, visual overlays, and recent action history to ground each trace step.
- Aloha Learner: The Trace Generator represents each step with an Observation, Think field, normalized Action, and Expectation grounded in marked screenshots and UI context.Structured prompts include action-specific priors, up to three previous steps, and high-resolution marked screenshots.
- Aloha Actor: The Aloha Actor couples task reasoning with GUI execution, integrating planning and execution backends to carry out reliable multi-step automation.It coordinates the framework’s execution process and connects high-level plans with reliable GUI actions.
- Aloha Actor: The Aloha Planner interprets goals, screenshots, action history, and demonstration cues to produce context-aware next-step plans under UI distribution shifts.Plans include Observation, Reasoning, Current Step, Action, and Expectation, using demonstrations as soft references rather than rigid scripts.
4 Experiments
ShowUI-Aloha was evaluated on a manually reinstantiated OSWorld-style suite across macOS and Windows using human demonstrations and binary end-to-end success. Comparisons are contextual because existing baselines use different supervision and scoring settings.
- Evaluation setup: Evaluations covered macOS and Windows on three representative desktop platforms to assess robustness across heterogeneous operating systems.The test devices included one macOS system and two Windows systems.
- Baseline considerations: Unguided agents were reported as contextual anchors because no strict apples-to-apples baseline matches Aloha’s supervision, execution protocol, and evaluation metric.Existing OSWorld baselines are unguided and zero-shot, while Aloha receives a structured demonstration before execution.
- Evaluation setup: Aloha’s binary success metric grants no partial credit, unlike the official OSWorld score based on task-specific rewards in a closed evaluator.This makes exact final-state completion the criterion used for all evaluated systems.
- Results: Aloha completed 217 of 361 tasks, achieving a 60.1% overall binary success rate across diverse GUI environments.The result is reported under the study’s exact-final-state evaluation protocol.
Aloha Performance Across OSWorld Categories
Aloha shows broad coverage across OSWorld application categories, with strongest performance on browser and operating-system tasks and greatest difficulty on multi-application workflows.
- Category performance: Aloha achieves near-perfect success on browser tasks and broad coverage of everyday productivity workflows.The category-level results identify Chrome as the strongest application area.
- Category performance: ShowUI-Aloha demonstrates consistently stronger end-to-end task success than prior unguided and agentic GUI agents across OSWorld-style tasks.The comparison is presented as a broad task-level result rather than a single application category.
Teaching Mode Improves Performance over Strong Baseline Agents
Human-taught demonstrations are associated with higher end-to-end task success than strong unguided and agentic baselines, although the comparison uses different scoring conventions.
- Teaching-mode comparison: Human-taught demonstrations enable higher end-to-end task success than unguided and agentic models on OSWorld-style tasks.The result is framed as a comparison under a user-oriented evaluation protocol.
- Teaching-mode comparison: Aloha’s reported 60.1% binary success rate exceeds the listed unguided baseline scores, which range from 29.4 to 56.4.The baseline values are reported for UI-TARS-1.5-7B, OpenAI CUA 4o, Claude 4 Sonnet, GTA-1-7B w/ o3, Jedi-7B w/ o3, Agent S2.5 w/ o3, and CoAct-1.
- Evaluation caveat: The comparison is not an apples-to-apples benchmark because Aloha uses binary end-to-end success while OSWorld baselines report graded scores.Aloha’s metric is described as strictly harder to achieve because it grants no partial credit.
- Error analysis: Aloha’s remaining failures include incorrect element selection and imprecise text-editing drags, especially in visually dense interfaces.The reported weaknesses center on fine-grained localization and precise drag-based text editing.
Breakdown of Failure Modes in Unsuccessful Trials
Among unsuccessful trials, element localization is the dominant failure mode, followed by text or field editing and misaligned action execution.
- Failure distribution: Element localization errors account for 53.5% of unsuccessful trials, making them the dominant error category.The analysis identifies localization as the clearest opportunity for targeted improvement.
- Failure distribution: Text or field editing errors represent 16.0% of failures, while misaligned action execution represents 14.6%.Both categories occur substantially less often than element localization errors.
- Failure distribution: Stalled or looping trajectories account for 8.3% of failures, and other failures account for 7.6%.The percentages are computed across 144 failed trials.
- Analysis design: The ablation study evaluates human demonstration traces and planner temporal memory on 30 tasks covering all ten application categories.Each variant uses the same evaluation protocol and environment as the main experiments.
Impact of Human Teaching and Planner Memory
The ablation shows that human teaching and planner memory are both important for stable, long-horizon GUI execution. Human demonstrations contribute the largest improvement, while memory helps preserve task-state consistency and avoid drift.
- Human teaching and planner memory are both critical for stable long-horizon execution.
- Removing TeachTrace drops Success Rate from 63.3% to 36.7% and Step-Norm from 0.89 to 0.56.Success Rate measures exact task completion, while Step-Norm measures normalized progress across 30 OSWorld tasks.
- Disabling PlannerMemory reduces performance to 50.0% Success Rate and 0.68 Step-Norm.The result indicates that temporally aware planning helps maintain task-state consistency and avoid drift in longer sequences.
- Together, human-taught trajectories and memory-equipped planning support robust, generalizable GUI automation; neither component alone is sufficient.
- Aloha also generalizes to complex workflows including air-ticket booking, Excel manipulation, and batch PowerPoint background editing.
5 Conclusion
ShowUI-Aloha converts raw human desktop demonstrations into structured, executable trajectories and uses them with temporally aware planning for multi-step GUI tasks. The framework generalizes across diverse applications, while its current scope still depends on demonstrations and faces several interaction-level challenges.
- ShowUI-Aloha combines a cross-platform recorder, intent-aligned trace learner, and temporally aware planner–actor system for desktop GUI automation.
- Aloha demonstrates robust multi-step execution across diverse real-world desktop applications and can generalize one demonstration across a workflow-sharing task group.
- Aloha remains limited by fine-grained icon disambiguation, noise-sensitive drag-based text selection, and reliance on at least one demonstration per workflow family.
- Future work targets broader task-group coverage, better icon and text-structure understanding, and few-shot or demonstration-free generalization.
- Because Aloha is open-sourced, researchers can replace its VLMs, actuators, or planners and reuse components such as the recorder.