Source-linked AI summary
GTA1: GUI Test-time Scaling Agent
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Silvio Savarese, Caiming Xiong, Junnan Li
TL;DR
GUI agents face difficult choices among many valid action plans and must precisely ground actions in complex, high-resolution interfaces. GTA1 addresses these challenges with per-step test-time proposal scaling and RL-based direct coordinate prediction; it achieves state-of-the-art grounding and robust task-execution performance, with reasoning helping specifically in dynamic settings.
Problem
GUI agents need to select effective action sequences from an expansive space and accurately ground interactions in complex, high-resolution interfaces.
Method
GTA1 samples multiple action proposals per step for multimodal judging and trains a grounding model to directly predict coordinates rewarded when clicks land within target elements.
Results
GTA1 achieves state-of-the-art performance across diverse GUI grounding benchmarks and demonstrates robust task execution when integrated with a planner and test-time scaling.
Takeaways & Limitations
Test-time scaling improves planning robustness, while direct coordinate rewards align grounding training with successful GUI interactions.
Abstract
from arXiv · showhide
Graphical user interface (GUI) agents autonomously complete tasks across platforms (\eg, Linux) by sequentially decomposing user instructions into action proposals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning (\ie, the action proposal sequence) under expansive action space, where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, \ie, precisely interacting with visual targets. This paper investigates the aforementioned challenges with our \textbf{G}UI \textbf{T}est-time Scaling \textbf{A}gent, namely GTA1. First, we conduct test-time scaling to select the most appropriate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by a judge model. It trades off computation for better decision quality by concurrent sampling. Second, we propose a model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, GTA1 achieves state-of-the-art performance on both grounding and agent task execution benchmarks. The code and models are released here.
1 Introduction
GTA1 targets two GUI-agent challenges: selecting effective plans amid multiple feasible action sequences and grounding interactions in complex, high-resolution interfaces. It combines test-time planning with RL-based coordinate grounding and reports state-of-the-art performance across grounding and agent benchmarks.
- Motivation: GUI agents must plan among multiple valid action-proposal sequences while accurately locating targets in complex, high-resolution interfaces.Errors in early planning or grounding can derail the entire task, and center-point supervision does not match the validity of any click inside a target element.
- Planning: At each step, GTA1 samples multiple planner proposals and uses a multimodal judge to select the most contextually appropriate option.This test-time scaling strategy explores short-term alternatives without rolling out the full action sequence.
- Grounding: GTA1 directly predicts interaction coordinates and rewards predictions that fall within the target UI element, aligning optimization with successful clicks.The approach avoids requiring explicit reasoning during grounding optimization.
- Discussion: The study identifies a conditional role for reasoning: “thinking” can help in dynamic environments but often fails to generalize broadly.The paper presents this observation alongside its direct-coordinate RL grounding approach.
- Results: GTA1 reports state-of-the-art performance across diverse GUI grounding benchmarks and demonstrates precise interaction in real environments when paired with a planner and judge.The performance overview compares grounding accuracy on ScreenSpot-Pro and task success rate on OSWorld-Verified.
2 Related Work
Prior GUI-agent research includes visual grounding methods and two broad agent designs: native end-to-end agents and two-stage systems that separate planning from action. GTA1 is positioned within this landscape by combining modular planning and grounding while targeting realistic, dynamic interfaces.
- GUI Grounding: GUI grounding maps user instructions to coordinates of target UI elements, but SFT-based center prediction can generalize poorly to high-resolution, visually complex interfaces.RL-based approaches instead reward coordinates that fall within the target element region, although prior methods often add explicit reasoning.
- GUI Grounding: Recent RL grounding methods commonly prompt textual “thinking” before coordinate prediction and reward successful placement within the target element.The related-work discussion notes that this strategy may overlook the possibility that thinking degrades GUI-grounding performance.
- Two-stage GUI Agent: Two-stage GUI agents separate planning and action, using a planner to propose each step from task instructions, screenshots, and past trajectories.This modular design addresses the challenge of locating coordinates for intended interactions through separate models.
- Native GUI Agent: Native GUI agents complete tasks end to end across perception, memory, planning, and action, and have shown strong performance on dynamic benchmarks such as OSWorld.The paper states that GTA1 is the first to show a two-stage GUI agent can achieve competitive performance in such environments.
3 Method
GTA1 combines test-time proposal selection with grounding and data-quality improvements in a two-stage GUI agent. At each step, it samples and judges candidate actions, grounds coordinate-based actions, and trains grounding with click-based rewards and cleaned annotations.
- GTA1 uses a planner and grounding model to improve planning robustness and grounding accuracy in GUI agents.The framework combines test-time scaling for planning with grounding-model training and annotation cleaning.
- Test-time Scaling for Planning: At each step, the planner samples K candidate action proposals from the user instruction, trajectory, and current screenshot.Each proposal may be a coordinate-based action such as clicking a named button or a non-coordinate action such as a keystroke.
- Test-time Scaling for Planning: A multimodal judge evaluates candidate proposals against user intent and GUI state, selecting the most contextually appropriate action before execution.The judge may be the planner model itself, and sampling helps avoid overcommitting to suboptimal proposals.
- Test-time Scaling for Planning: For selected coordinate-based actions, the grounding model predicts an interaction point from the action proposal and screenshot; non-coordinate actions execute directly.The process repeats step by step until task completion or a termination condition.
- Grounding: Training data are cleaned by detecting UI-element boxes with OmniParser and discarding annotations whose maximum IoU falls below threshold τ.The cleaning strategy aims to keep annotated boxes consistent with actual visual targets and reduce noise from misalignment.
- Grounding: The grounding model directly outputs pixel coordinates and receives binary rewards for predictions inside the annotated target box, optimized with GRPO.The reward is one when both predicted coordinates lie within the bounding-box limits and zero otherwise; the advantage weights higher-reward predictions more strongly.
4 Experiment
GTA1 is evaluated on GUI grounding and agent task-execution benchmarks, with experiments spanning model comparisons, reward ablations, dynamic environments, and test-time scaling. The method achieves strong grounding and task-success results while showing that proposal scaling and contextual reasoning affect performance under specific conditions.
- Grounding Performance: 72.2% grounding accuracy on OSWorld-G establishes a new benchmark, while GTA1-32B matches Seed-1.5-VL on ScreenSpot-V2.ScreenSpot-V2 covers mobile, desktop, and web domains, whereas OSWorld-G focuses on Linux.
- Agent Performance: 45.2% task success makes GTA1-7B the strongest reported OSWorld result, exceeding CUA o3’s 42.9% despite using a shorter 100-step horizon versus 200 steps.The evaluation uses o3 as planner and judge for test-time scaling across 369 OSWorld tasks.
- Discussion and Ablation: “Thinking” improves AndroidWorld task success from 39% to 44% in dynamic conditions, although grounding on AndroidControl remains similar and differing samples suggest training instability.The dynamic setting provides the task object, past trajectories, and user instruction as context.
- Test-time Scaling: With K=8, OSWorld success rises from 43.4% to 45.2% when the execution horizon increases from 50 to 100 steps, versus K=1 baselines of 41.3% and 43.4%.The K=1 setting corresponds to no test-time scaling.
- Test-time Scaling: Test-time scaling generalizes to UI-TARS-1.5-7B: K=32 with 15 steps outperforms its unscaled 100-step baseline, with the largest overall gain at 50 steps.Concurrent sampling of candidate proposals also substantially reduces wall-clock time; 15 steps can be insufficient, while 100 steps may dilute scaling benefits.
5 Conclusion and Limitation
GTA1 addresses GUI planning and grounding with test-time proposal selection and RL-based direct coordinate prediction, achieving state-of-the-art benchmark performance while retaining difficulties in some image-editing scenarios.
- GTA1 samples multiple action proposals at each step and uses a multimodal judge to select the most suitable one.
- GTA1 directly rewards clicks that fall within target UI elements, bypassing the explicit reasoning required by prior grounding methods.
- GTA1 achieves state-of-the-art performance on standard grounding benchmarks and robust behavior for user-task execution.
- Despite achieving the highest accuracy on ScreenSpot-Pro, GTA1 struggles to select custom foregrounds and backgrounds in GMIP image-editing tasks.
A Example
Without test-time scaling, early grounding or planning errors can propagate through execution and derail the entire task.
- Without test-time scaling, errors in early grounding or planning stages can propagate and derail the entire task execution.
- The resulting agent is highly susceptible to cascading failures during task execution.
- The comparison evaluates UI-TARS-1.5-7B with and without test-time scaling strategies in Fig. 5 and Fig. 6.
B Model Training Details
GTA1 training uses specified model initializations, optimization settings, data sources, and hardware configurations, while evaluation defaults to eight sampled action proposals per step.
- GTA1 models are initialized from UI-TARS-1.5-7B and OpenCUA-32B, using learning rates of 10^-6 and 10^-5, respectively.
- Training rolls out N=8 responses per input, uses batch size 256, and typically converges after approximately 250 iterations.
- Images are resized to dimensions divisible by 28, with ground-truth boxes scaled by the same ratio for reward calculation.
- Real-world dynamic-environment evaluation samples K=8 action proposals per step and uses o3 and GPT-5 as planners.
- Training data come from Aria-UI-Web, OmniACT, UI Vision, Widget Caption, and OS-Altas-Desktop, with 70K datasets sampled.
C Evaluation Details
The evaluation covers GUI grounding benchmarks and agent task-execution benchmarks spanning Linux, mobile, desktop, web, and dynamic real-world environments.
- GUI grounding is evaluated on ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G, while task execution is evaluated on OSWorld and WindowsAgentArena.
C.1 ScreenSpot-V2
ScreenSpot-V2 extends ScreenSpot by correcting and re-annotating incorrect samples, and evaluates GUI grounding across mobile, desktop, and web domains.
- ScreenSpot-V2 corrects and re-annotates 11.32% of incorrect ScreenSpot samples.Examples include spelling errors and incorrect bounding boxes.
- The benchmark assesses grounding ability across mobile, desktop, and web domains.
C.2 ScreenSpot-Pro
ScreenSpotPro evaluates GUI grounding for professional, high-resolution computer use across six application domains and 23 applications; the examples illustrate how test-time scaling improves trajectories.
- C.2 ScreenSpot-Pro: ScreenSpotPro covers professional computer use at resolutions up to 3840×2160 across 23 applications.
- C.2 ScreenSpot-Pro: The benchmark categorizes tasks into Development, Creative, CAD, Scientific, Office, and OS domains.
- Example trajectories: Without test-time scaling, an early error in the “From” field shifts the trajectory from modifying the search field to scrolling for the ticket.
- Example trajectories: With test-time scaling, the agent consistently modifies the search information to complete the flight-search task.
- Example trajectories: Without the strategy, an early shortcut-panel error leads the agent to close Chrome, encounter authentication, and get stuck.
- Example trajectories: With the strategy, the agent opens Chrome settings and successfully completes the desktop-shortcut task.
- Implementation: Both 7B and 32B models perform inference efficiently on a single 80GB GPU with bfloat16 precision.
C.3 OSWorld-G
OSWorld-G provides fine-grained annotations for GUI task understanding, while the evaluation setup uses multiple planners, grounding models, prompts, and agent actions across virtual environments.
- C.3 OSWorld-G: OSWorld-G covers text matching, element recognition, layout understanding, precise manipulation, and refusal.
- C.3 OSWorld-G: The benchmark reports performance for both instruction-level and fine-grained annotations that explicitly decompose the GUI knowledge required for each task.
- Evaluation setup: Evaluations use o3 and GPT-5 as planners, while GTA1-7B and GTA1-32B provide grounding through vllm-served models.
- Evaluation setup: The evaluation setup includes 48 Docker instances for OSWorld evaluations and 8 Docker instances with 8 served models for WindowAgentArena.
- Agent actions: The agent prompt supports actions including scrolling, switching applications, opening files, typing, waiting, hotkeys, and spreadsheet cell updates.
- Agent interaction protocol: The agent must base each action on the current screenshot, avoid repeating unsuccessful actions, and return one executable action per step.
- Planning prompt: The planning prompts require detailed screenshot observations and step-by-step progress assessment before proposing the next action.