Source-linked AI summary
ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands
Siyuan Hu, Kevin Qinghong Lin, Mike Zheng Shou
TL;DR
Existing GUI agents’ discrete coordinate actions inadequately support continuous, closed-loop dragging that requires on-the-fly visual adjustment. ShowUI-π combines discrete clicks and continuous drags in a flow-based model, supported by drag data and the ScreenDrag benchmark, and achieves 26.98 with 450M parameters while proprietary agents score lower.
Problem
Existing GUI agents represent coordinates discretely, limiting free-form continuous trajectories that require on-the-fly perception and adjustment.
Method
ShowUI-π uses a shared flow-based formulation for clicks and drags, trained with 20K dense trajectories and evaluated through ScreenDrag’s offline and online protocols.
Results
26.98% is ShowUI-π’s overall online closed-loop success rate, surpassing Gemini-2.5-CUA by 4.8% and OpenCUA-7B by 6.19%.
Takeaways & Limitations
ShowUI-π demonstrates continuous GUI trajectory control with a 450M-parameter model on a benchmark spanning five domains.
Takeaways & Limitations
The model was trained at a small model size and with limited training-data scale.
Abstract
from arXiv · showhide
Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions (x,y), which prohibits free-form, closed-loop trajectories (e.g. dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI-$π$, the first flow-based generative model as GUI dexterous hand, featuring the following designs: (i) Unified Discrete-Continuous Actions, integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes; (ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories; (iii) Drag Training data and Benchmark, where we manually collect and synthesize 20K drag trajectories across five domains (e.g. PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents' drag capabilities. Our experiments show that proprietary GUI agents still struggle on ScreenDrag (e.g. Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI-$π$ achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world. The code is available at https://github.com/showlab/showui-pi.
1. Introduction
ShowUI-π addresses the limits of discrete GUI actions by modeling continuous trajectories while unifying clicks and drags. The paper also introduces ScreenDrag and 20K dense drag trajectories, and reports stronger performance than proprietary agents with a 450M-parameter model.
- GUI agents commonly tokenize action coordinates, restricting complex, high-degree-of-freedom dragging.
- ShowUI-π unifies clicks and continuous drags as sequences of cursor coordinates and mouse-button states within one shared model.Clicks are represented as drags with negligible movement, avoiding task-specific head selection.
- Its flow-based action expert incrementally predicts cursor adjustments from continuous visual observations for stable trajectories.The action expert is trained with flow matching on a transformer backbone.
- ScreenDrag contains 505 real-world drag tasks across five domains and evaluates both offline trajectory accuracy and online task success.Each domain contains 101 tasks, spanning professional control and daily usage.
- 20K manually collected and synthesized drag trajectories cover five domains and 11 task categories with recorded UI states and dense coordinates.
- 26.98 is ShowUI-π’s ScreenDrag score with 450M parameters, compared with 13.27 for Operator and 22.18 for Gemini-2.5-CUA.
2. Related Work
Related work spans reasoning-and-tool-use paradigms, training-free VLM planning with external tools, and models that jointly learn perception and action. The paper situates ShowUI-π against discrete GUI action decoding and flow-matching action generation.
- Digital GUI agents have evolved from reasoning and tool-use paradigms such as Chain-of-Thought and ReAct.
- Training-free GUI pipelines plan with a vision-language model and execute actions through external tools.
- Training-based GUI models jointly learn perception and action from screenshots and instructions.
- Generalist vision-language-action models map language instructions and visual observations to action policies using multimodal backbones and robotics data.Representative lines include RT-1, OpenVLA, PaliGemma, and TinyVLA.
- Flow matching trains the action expert to predict a velocity field conditioned on the current observation and task instruction.The target is the velocity along the probability path over training timesteps.
3. ScreenDrag Dataset
ScreenDrag evaluates GUI agents on continuous drag tasks that require on-the-fly observation and control, rather than only start–end predictions. It combines offline trajectory-precision measures with online closed-loop task evaluation.
- Dataset Scope: ScreenDrag targets continuous GUI actions with intermediate state changes and potentially multiple valid trajectories, unlike existing start–end drag benchmarks.The benchmark is designed to assess agents’ continuous-action capabilities during execution.
- Dataset Scope: The dataset spans daily and professional drag tasks including PowerPoint manipulation, file sorting, canvas handwriting, Premiere Pro editing, and Captcha solving.These tasks require precise spatial control and observation during execution.
- Evaluation Protocols: Offline evaluation measures open-loop stepwise behavior from static screenshots using Average Trajectory Error and endpoint accuracy.Average Trajectory Error compares predicted and ground-truth coordinates across timesteps, while endpoint accuracy uses a spatial tolerance radius ϵ.
- Evaluation Protocols: Offline and online evaluation differ because offline trunks are independent distance comparisons, whereas online trunks are sequential and judged by final outcome.This distinction captures compounding effects and trajectory variation that static stepwise evaluation can miss.
- Evaluation Protocols: Online evaluation uses closed-loop rollouts that expose dynamic UI changes and scores whether trajectories achieve the task goal.A data-driven environment stores task videos, specifications, and dense trajectories to support reproducible rollouts.
4. ShowUI-πModel
ShowUI-π combines a pretrained VLM with a lightweight flow-matching action expert to generate unified click and drag action chunks. Its streaming design supports incremental, closed-loop cursor control while emphasizing trajectory endpoints and directional consistency.
- Architecture: ShowUI-π integrates a pretrained VLM with a flow-matching action expert for streaming GUI control.The VLM encodes screenshots, action states, and task instructions, while the action expert generates actions.
- Unified actions: Clicks and drags are unified as sequences of cursor coordinates and mouse-button states, with clicks represented as negligible-motion drags.This shared representation supports both interaction types without rigid, predefined action formats.
- Flow-based control: The model generates incremental trajectory chunks conditioned on the current visual observation and task instruction, with the flow parameter moving each segment from start to end.After predicted actions update the environment, the next observation is encoded and a new action chunk is produced.
- Trajectory objective: The training objective reweights initial and terminal trajectory steps to emphasize start-point conditioning and endpoint precision.The stated weighting scheme assigns weight 10 to start and end points and weight 1 otherwise.
- Trajectory objective: Directional regularization complements reweighted flow matching by targeting directional consistency in cursor trajectories.The final objective combines the reweighted flow-matching loss with a directional regularization term.
- Evaluation: ScreenDrag reports offline evaluation through endpoint accuracy and trajectory error, alongside online success-rate evaluation.The supplied table captions identify the two offline metrics and the online success-rate protocol.
5. Experiments
Experiments evaluate ShowUI-π against GUI-agent baselines and isolate the effects of flow matching, temporal weighting, action-head design, chunking, execution frequency, and directional regularization. ShowUI-π performs strongly on continuous and free-form drag tasks, while frequent re-observation and targeted design choices improve trajectory quality and success.
- Main Results: 26.98% overall online closed-loop success surpasses Gemini-2.5-CUA by 4.8% and OpenCUA-7B by 6.19%.ShowUI-π performs well on circular PowerPoint drags, nonlinear handwriting, and on-the-fly Captcha tasks.
- Main Results: ShowUI-π produces the least trajectory error in offline evaluation, indicating close alignment with learned drag trajectories.For ShowUI-π, all predicted waypoints are evaluated, unlike language-output baselines whose trajectory error uses only endpoints.
- Temporal Control: Chunk size 20 with execution step 1 achieves the tradeoff for reliable and precise actions.At five execution steps, increasing chunk size from 10 to 20 improves accuracy by 12.9% and reduces trajectory error by 36.4 px; fewer execution steps generally perform best.
- Temporal Weighting: Temporal reweighting at scale 10 raises overall success from 10.49% to 26.98%, whereas scale 15 reduces overall performance.The weighting prioritizes steps after drag starts and near drag ends, while excessive weighting can harm intermediate-action learning.
- Action Heads: The unified action head matches separate heads offline at 78.55% versus 79.22%, improves online drag success by 3.7%, and reduces model size by 100M parameters.It also avoids task-dependent head selection while supporting both clicks and drags.
- Trajectory Regularization: Directional regularization consistently improves performance across domains, especially when drag direction determines task success.Captcha solving is particularly sensitive to inaccurate drag direction.
- Qualitative Analysis: Qualitative examples show precise 45-degree PowerPoint rotation, smooth S-shaped handwriting, and fine-grained Captcha control.These examples span geometric transformation, nonlinear drawing, and exact stopping behavior.
6. Conclusion
ShowUI-π is presented as a flow-based GUI agent that unifies clicks and continuous drags, trained with 20K dense trajectories and evaluated through ScreenDrag. It achieves leading results with 450M parameters, while its small model and limited training-data scale constrain grounding performance and motivate future scaling.
- Conclusion: ShowUI-π unifies discrete clicks and continuous drags within a shared flow-based formulation for continuous trajectory control.The design is intended to support precise, real-time cursor adjustment.
- Conclusion: 20K dense trajectories support ScreenDrag, a five-domain benchmark with offline and online rollout protocols.The benchmark evaluates drag behavior across both professional and everyday GUI tasks.
- Conclusion: ShowUI-π sets leading results with only 450M parameters, while proprietary GUI agents still struggle with continuous manipulation.The conclusion frames the result as progress toward GUI agents with human-like dexterity.
- Limitations: ShowUI-π achieves grounding performance comparable to much larger models, but its 450M-parameter size weakens vision capacity and restricts grounding performance.The authors identify larger models and more training data as future directions.
A.2. Drag Performance of ShowUI-π on public benchmark
On the image-based Drag subset of VideoGUI-Action, ShowUI-π generates the complete drag action chunk from a single screenshot. It remains comparable to much larger baseline models under this non-interactive evaluation setting.
- Public Benchmark: ShowUI-π remains comparable to much larger baseline models on the public VideoGUI-Action drag benchmark.Because the benchmark is image-based, the model generates the entire action chunk at once without on-the-fly observations or continuous actions.
A.3. Effects of Co-training with Drag Data and Grounding Data
Co-training on drag and grounding data produces the strongest performance on both ScreenDrag online evaluation and ScreenSpot-Pro. ScreenDrag data also boosts Creative and Office grounding categories that overlap with its domains.
- Co-training: Training on both Drag data and Grounding data yields the highest performance on ScreenDrag online evaluation and ScreenSpot-Pro.ScreenDrag training also boosts ScreenSpot-Pro’s Creative and Office categories because their domains overlap.
B.1. Training Details
ShowUI-π is trained end-to-end as a compact model using bfloat16 precision, four H200 GPUs, and a limited corpus centered on click data plus ScreenDrag. The supplied tables provide performance breakdowns for ScreenSpot-Pro and ScreenDrag but no numerical entries.
- Training Setup: ShowUI-π is trained end-to-end with bfloat16 precision on four H200 GPUs, using a batch size of 64 per GPU without gradient accumulation.The vision encoder and language model initialize from SmolVLM, while action-related components are randomly initialized.
- Evaluation Tables: Table 11 reports performance breakdowns across application categories on ScreenSpot-Pro.
- Evaluation Tables: Table 12 reports performance breakdowns across domains on ScreenDrag online evaluation.
- Training Data: The training corpus combines ScreenDrag with desktop Click subsets from GUIAct, WaveUI, UGround, and ShowUI-Desktop.The authors exclude non-Click public-dataset data and mobile data from training.
C. Dataset Construction
The dataset spans five GUI domains and pairs recorded trajectories with task metadata when accessible. ScreenDrag evaluation uses recorded videos, task specifications, and dense trajectories to support online closed-loop rollouts.
- Domains: The dataset covers PowerPoint, OS Desktop and File Manager, Captcha, Handwriting, and Adobe Premiere Pro.Data are collected on Windows machines using high-frequency mouse events and screen recordings; DOM or UIA metadata provide element locations where available.
- Domain-Specific Collection: Captcha trajectories are filtered using real-time metadata for puzzle positions, target positions, and task success or failure.The Captcha is refreshed after the previous one is solved, enabling collection of successfully completed tasks.
- Domain-Specific Collection: Handwriting trajectories are synthesized from sampled names and phrases, varied stroke styles, and canvas execution through the Win32 mouse interface.
- Domain-Specific Collection: Premiere Pro tasks use human demonstrations from video-editing workflows because complete UI metadata are unavailable.Two experienced student annotators design operations including trimming clips, adjusting timeline layers, arranging clips, and applying effects.
- Data Recording: The collection pipeline records screen video and trajectories across domains, while OBS is used for low-latency screen capture and the data-generation codebase is planned for release.
- Evaluation Pipeline: ScreenDrag online rollouts match predicted actions to nearby recorded trajectory states and return the corresponding next observation.The matching uses a tolerance such as 20 pixels from a ground-truth waypoint.
D. Failure Cases of Baseline Models
Baseline GUI agents fail on continuous drag tasks through tool limitations, discrete action habits, geometric errors, safety-oriented refusals, dialogue interruptions, and premature termination. These failures contrast with the need for on-the-fly observation and continuous cursor adjustment.
- Tool and Primitive Mismatches: Baselines can identify a rotation plan but fail when their tools support only linear drags rather than arc trajectories.This exposes a mismatch between task-required manipulation and the available action primitive.
- Tool and Primitive Mismatches: Discrete drag calls often begin correctly but stop midway because they cannot adjust cursor motion continuously from new observations.This failure appears in desktop file-to-folder tasks where the icon remains far from the target.
- Evaluation Setting: The benchmark’s online evaluation uses continuous actions with observations on-the-fly, while baseline failures reveal the gap between discrete tool use and closed-loop dragging.
- Execution Interruptions: Some baselines prioritize safety, clarification, or refusal over autonomous task completion during Captcha and handwriting interactions.Examples include halting after starting a Captcha drag, asking the user to finish, or ending with an “Instruction Unclear” error.
- Tool and Primitive Mismatches: Click-biased baselines may issue discrete clicks instead of the continuous drag required by a slider.The failure reflects an inductive bias inherited from GUI pre-training data dominated by discrete clicks.
- Geometric Control: Correct directional motion can still overshoot the target because baselines misestimate drag magnitude.The predicted trajectory follows the correct direction but lacks fine-grained geometric control.
E. Limitations and Future Work
The authors identify limited model size and training-data scale as current constraints and plan to address them by scaling the model and data while exploring text-centric planning integration.
- Current Constraints: ShowUI-π is currently trained with a small model and limited data scale, constraining the present training setup.Future work will scale parameters and expand data from the collection pipeline and external sources.
- Future Work: The authors also plan to explore integrating text-centric planning with ShowUI-π.