Source-linked AI summary
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou
TL;DR
GUI agents often depend on text-rich structural metadata, limiting screenshot-based interaction, while high-resolution visuals and device-specific actions make visual-agent training costly and complex. ShowUI addresses these challenges with UI-guided token selection, interleaved vision-language-action streaming, and curated balanced data. It achieves strong grounding and navigation performance with a lightweight model, while remaining primarily trained on offline data.
Problem
Existing GUI agents rely on text-rich HTML or accessibility-tree metadata, while screenshot-based perception must also handle costly high-resolution visuals and device-specific actions.
Method
ShowUI combines RGB-based UI-connected-graph token selection, interleaved vision-language-action streaming, and curated, rebalanced GUI instruction data.
Results
75.1% zero-shot screenshot-grounding accuracy is achieved by a lightweight 2B model trained on 256K data, with competitive navigation across web, mobile, and online environments.
Takeaways & Limitations
ShowUI demonstrates potential for more human-like GUI interaction and perception while reducing visual-token redundancy and computational cost.
Takeaways & Limitations
ShowUI is primarily trained on offline data; online reinforcement learning is proposed as future work.
Abstract
from arXiv · showhide
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (e.g., HTML or accessibility tree), they show limitations in perceiving UI visuals as humans do, highlighting the need for GUI visual agents. In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances. With above components, ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection further reduces 33% of redundant visual tokens during training and speeds up the performance by 1.4x. Navigation experiments across web Mind2Web, mobile AITW, and online MiniWob environments further underscore the effectiveness and potential of our model in advancing GUI visual agents. The models are available at https://github.com/showlab/ShowUI.
1. Introduction
ShowUI targets GUI agents that can perceive screenshots rather than relying on text-rich structural metadata. It combines UI-aware visual modeling, interleaved vision-language-action processing, curated data, and a lightweight model to support grounding and navigation.
- Text-only GUI agents rely on HTML or accessibility trees, whereas users typically interact with interfaces visually through screenshots.
- High-resolution screenshots create expensive visual modeling, while device-specific action types and parameters complicate interleaved vision-language-action training.
- UI-Guided Visual Token Selection represents screenshot patches as RGB-based connected components and uses their redundancy relationships to guide token selection.
- Interleaved Vision-Language-Action Streaming structures diverse actions and combines action with visual navigation history or multi-turn textual queries.
- The curated dataset and rebalanced sampling strategy target GUI data imbalance while emphasizing visually rich elements for strong grounding performance.
- ShowUI is a lightweight 2B model trained on 256K data, achieving 75.1% accuracy in zero-shot screenshot grounding and competitive navigation across web, mobile, and online environments.
2. ShowUI
ShowUI combines UI-connected-graph token selection with interleaved action, visual, and query streams to process GUI screenshots and trajectories efficiently. Its design adapts token compression to screenshot structure while preserving positional relationships and supports device-diverse actions.
- UI-Guided Visual Tokens Selection: High-resolution patching can produce 1296 visual tokens after merging, creating a computational challenge for self-attention.
- UI-Guided Visual Tokens Selection: The UI Connected Graph groups neighboring patches with similar RGB values into connected components, modeling redundancy adaptively.
- UI-Guided Visual Tokens Selection: Component counts adapt from 1296 to 291 on sparse Google search pages and from 1296 to 986 on text-rich Overleaf screenshots.
- UI-Guided Visual Tokens Selection: Token Merge loses positional information, whereas Token Select.-UI-Graph provides competitive accuracy with a 1.5× speedup and remains more reliable than random selection.
- UI-Guided Visual Tokens Selection: Token selection randomly skips part of each component during training, preserving single-patch components and positional relationships without additional learnable parameters.
- Interleaved VLA Streaming: ShowUI standardizes device-diverse actions in JSON and documents their usage, while interleaved streams manage visual history for navigation and query-action pairing for grounding.
3. Experiments
ShowUI is evaluated on zero-shot grounding and navigation across desktop, web, mobile, and online environments, alongside component and data ablations. Results show strong lightweight grounding, benefits from visual history and balanced high-quality data, and trade-offs in online generalization and visual-token selection.
- Evaluation Setup: ShowUI is evaluated on Screenspot grounding and navigation across Mind2Web, AITW, and MiniWob, covering multiple devices and offline or online settings.Screenspot measures zero-shot grounding, while navigation benchmarks include web, mobile, and interactive online environments.
- Grounding Tasks: 75.1% accuracy in zero-shot screenshot grounding is achieved by the lightweight 2B ShowUI model using minimal training data.The model is evaluated on Screenspot and is described as the most lightweight method with minimal training data.
- Grounding Tasks: ShowUI achieves state-of-the-art zero-shot grounding, with text grounding generally stronger than icon grounding and mobile scores higher than desktop and web.The results attribute the icon-track difficulty to visual grounding and identify limited non-mobile visual UI data as a gap.
- Navigation Tasks: 1.7% additional accuracy gain from visual history on AITW indicates that retaining prior screenshots helps mobile navigation across frequent software changes.Without interleaved streaming, ShowUI† improves only 1.1% over the VLM baseline; visual history adds a further 1.7% gain.
- Navigation Tasks: 4.6% Avg. Step SR improvement over Qwen2-VL-2B is reported on Mind2Web, while visual context is less significant on its visually stable, three-action webpages.ShowUI-2B’s zero-shot performance is comparable with the pretrained and fine-tuned SeeClick-9.6B, with relatively high Op. F1 above 80%.
- Navigation Tasks: Online MiniWob exposes a substantial gap between ShowUI zero-shot performance at 27.1% and fine-tuned Qwen-VL at 48.4%, motivating online learning strategies.The reported interpretation is that offline instruction-tuning does not adequately address novel out-of-distribution error cases.
- Ablation Studies: UI-Graph token selection offers a 1.5× speedup with competitive accuracy, while cross-layer insertion and a 0.5 selection ratio provide favorable ablation settings.Token merging performs worse than random selection, and test-time selection slightly reduces accuracy because of resolution loss.
4. Related Works
Related work spans training-free GUI automation using structural or visual UI representations, training-based multimodal models, and efficient visual-token methods. ShowUI builds on these directions with a UI-specific graph that preserves spatial information while reducing redundant computation.
- GUI Agents: Training-free GUI agents commonly convert interfaces into HTML, accessibility trees, OCR, or visual annotations before using language models to generate actions.These systems often rely on closed-source APIs and structural information that ordinary users do not directly perceive.
- GUI Agents: Training-based models pretrain on large-scale UI vision-text corpora to improve visual perception and reduce reliance on structural oracle information.This line of work targets GUI understanding directly from screenshots and related visual-text data.
- Efficient Visual Representations: Generic token pruning and merging are poorly suited to GUI grounding because discarding spatial details can impair fine-grained element localization.Mixture-of-Depth is presented as an alternative that preserves individual token positions while allocating computation selectively.
- Efficient Visual Representations: ShowUI constructs a UI connected graph from screenshot patches and uses it to skip redundant tokens during self-attention without extra learnable parameters.The method exploits structured redundancy such as excessive whitespace while retaining positional relationships for grounding.
5. Conclusions
ShowUI addresses UI visual and action-modeling challenges through efficient visual processing, interleaved multimodal interaction, and curated instruction-tuning data. The model demonstrates strong performance while remaining lightweight, but is primarily trained on offline data.
- ShowUI combines UI-guided visual token selection, interleaved vision-language-action streaming, and curated instruction-following data for GUI visual agents.These components target UI visual modeling, multimodal interaction, and training-data curation.
- ShowUI demonstrates competitive navigation ability across web, mobile, and online environments.
- The model is primarily trained on offline data, motivating future enhancement in online environments through reinforcement learning.
A.1. Instruction-Tuning data
The instruction-tuning data combine web, mobile, and desktop GUI sources with substantial screenshot, element, functionality, and annotation coverage. Each source contributes a different platform or application scope.
- Website: 22K web screenshots from 22 representative scenarios yield 926K element annotations, with 576K elements retained after removing static text.The retained set averages 26 elements per screenshot.
- Mobile: Mobile data from AMEX annotations cover 97K screenshots, 885K elements, and 178K functionalities.
- Desktop: Desktop data include 100 screenshots and 2,000 raw annotations from 15 applications across iOS, Windows, and Linux.The annotations are additionally augmented with GPT-4o-assisted prompting.
A.2. Downstream tasks
The downstream evaluation covers web, mobile, and online GUI environments with distinct task sources and action spaces. These benchmarks test generalist web navigation, Android interaction, and open-ended web tasks.
- Mind2Web: Mind2Web evaluates generalist web agents across task, website, and domain test splits using CLICK, TYPE, and SELECT actions.Its webpages pair HTML documents with corresponding screenshots and verified element visibility.
- AITW: AITW contains 30K instructions and 715K trajectories across five domains with 12 Android action types.The action space includes navigation, scrolling, input, and task-status actions.
- MiniWob: MiniWob comprises 2,000 open-ended tasks from 137 real web environments, using CLICK and TYPE actions.
B.1. Training details
Training uses multi-GPU instruction tuning with mixed-precision LoRA adaptation and memory- and attention-efficiency techniques. The configuration limits trainable parameters while supporting both the language model and visual encoder.
- 32 V100 GPUs are used for instruction-tuning, while downstream adaptation uses 8 V100 GPUs.
- LoRA tuning with rank 64 and alpha 128 is applied to both the language model and visual encoder, leaving 4% of parameters learnable.
- Training uses float16 precision, DeepSpeed ZeRO-2, and SDPA attention with batch size 1 per GPU and two gradient-accumulation steps.
B.2. Prompt templates
The prompt templates structure GUI navigation inputs, outputs, and action histories for consistent model interaction across tasks. They also guide screenshot descriptions through appearance, spatial, and situational dimensions.
- GPT-4o Assisted Prompts: GPT-4o prompts augment annotations with appearance, spatial-relationship, and situational descriptions of target elements.The target is identified by a red bounding box and element name.
- GPT-4o Assisted Prompts: Descriptions must be concise, specific, distinguish target elements, ignore the red box as the element, and use JSON formatting.Responses should contain appearance, spatial, and situational fields, using None when a dimension does not apply.
- Action README Template: Action-navigation prompts provide a device, task instruction, screen observation, and action history to predict the next action.The model outputs an action and waits for the next observation.
- Action README Template: The action space templates action types such as CLICK and TYPE, with coordinates required for CLICK but not TYPE.Actions are formatted with action type, value, and position fields, and relative coordinates are scaled to 0-1.
- Action README Template: A task prompt pairs the task and visual-action history before requesting the next action for a new screenshot.The template interleaves past images with their corresponding actions and the next image with its action.