Source-linked AI summary
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, Guang Shi
TL;DR
GUI agents need to operate across dynamic interfaces without the brittleness of text-dependent or manually orchestrated frameworks. This paper introduces UI-TARS, an end-to-end native model integrating perception, action, reasoning, and memory, and reports state-of-the-art performance across 10+ GUI-agent benchmarks. The results support UI-TARS as a scalable, adaptive approach for complex GUI tasks, while the broader framework-based comparison remains bounded by documented brittleness and maintenance demands.
Problem
Textual and modular GUI-agent approaches face platform inconsistencies, scalability limits, manual workflow dependence, and coordination problems as tasks and interfaces evolve.
Method
UI-TARS integrates screenshot perception, unified action modeling, system-2 reasoning, memory, and iterative training with interaction traces in a native end-to-end model.
Results
UI-TARS achieves state-of-the-art performance across 10+ GUI-agent benchmarks covering perception, grounding, and task execution, outperforming existing systems such as Claude and GPT-4o.
Takeaways & Limitations
The paper positions scalable, data-driven native agents as a path toward adaptive GUI systems that handle complex tasks with minimal human oversight.
Takeaways & Limitations
Framework-based comparisons remain bounded by manually maintained workflows, disconnected experience learning, and module incompatibility in evolving scenarios.
Abstract
from arXiv · showhide
This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions (e.g., keyboard and mouse operations). Unlike prevailing agent frameworks that depend on heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts and workflows, UI-TARS is an end-to-end model that outperforms these sophisticated frameworks. Experiments demonstrate its superior performance: UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating perception, grounding, and GUI task execution. Notably, in the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld, UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of GUI screenshots for context-aware understanding of UI elements and precise captioning; (2) Unified Action Modeling, which standardizes actions into a unified space across platforms and achieves precise grounding and interaction through large-scale action traces; (3) System-2 Reasoning, which incorporates deliberate reasoning into multi-step decision making, involving multiple reasoning patterns such as task decomposition, reflection thinking, milestone recognition, etc. (4) Iterative Training with Reflective Online Traces, which addresses the data bottleneck by automatically collecting, filtering, and reflectively refining new interaction traces on hundreds of virtual machines. Through iterative training and reflection tuning, UI-TARS continuously learns from its mistakes and adapts to unforeseen situations with minimal human intervention. We also analyze the evolution path of GUI agents to guide the further development of this domain.
1 Introduction
GUI agents aim to automate interaction with graphical interfaces, but textual and current native approaches face platform, perception, reasoning, and scalability challenges. UI-TARS addresses these challenges as an end-to-end native model with enhanced perception, unified actions, and strong benchmark performance.
- GUI agents automate interaction with buttons, text boxes, and images to streamline workflows and enhance accessibility.
- Textual GUI methods suffer from platform-specific inconsistencies, verbosity, limited scalability, and dependence on system-level access.
- Native GUI agents increasingly rely on screenshots and end-to-end modeling rather than textual representations and modular frameworks.
- Current native agents still struggle with high-density, evolving interfaces and the reasoning and planning required for robust interaction.
- UI-TARS integrates enhanced screenshot perception, unified action modeling, and system-2 reasoning into a native GUI agent model.
- UI-TARS demonstrates SOTA performance across 10+ GUI agent benchmarks spanning perception, grounding, and task execution.
2 Evolution Path of GUI Agents
GUI agents have evolved from rigid rule-based automation through modular foundation-model frameworks toward native, end-to-end models. The next stage emphasizes active and lifelong learning to reduce dependence on human-authored workflows and training data.
- Stage 1: Rule-based Agents: Rule-based agents reproduce predefined human actions effectively in structured, repetitive environments but remain constrained by their rules.
- Stage 1: Rule-based Agents: Visual GUI interaction expands flexibility and adaptability beyond low-level system access and rigid predefined rules.
- Stage 2: Agent Framework: Foundation-model agent frameworks improve execution flexibility through APIs, code, prompts, workflows, memory, and reflection-based reasoning.
- Stage 2: Agent Framework: Manually encoded workflows create maintenance overhead, disconnected learning, and module incompatibility as tasks and interfaces change.
- Stage 2: Agent Framework: Frameworks remain brittle and non-scalable in evolving real-world scenarios because they depend on developer foresight and continuous human involvement.
- Stage 3: Native Agent Model: Native agent models unify perception, reasoning, memory, and action end to end, enabling data-driven adaptation to changing tasks and interfaces.
- Stage 4: Action and Lifelong Agent: Native agents still depend heavily on human experts for data labeling and training guidance.
- Stage 4: Action and Lifelong Agent: Active and lifelong agents propose tasks, execute them, evaluate outcomes, and refine capabilities through continuous feedback.
3 Core Capabilities of Native Agent Model
Native GUI agents require real-time perception, versatile actions, system-1 and system-2 reasoning, and memory to operate across dynamic interfaces. UI-TARS emphasizes deliberate planning and reflection-driven learning for complex workflows.
- Native agent models internalize perception, reasoning, memory, and action into an end-to-end structure.
- GUI-agent analysis centers on perception, action, system-1 and system-2 reasoning, and memory.
- Perception: Effective perception interprets screenshots in real time and adapts as interface elements and states change.
- Perception: Pure text and hybrid text-visual approaches face privacy concerns and perceptual noise from DOM representations.
- Action: A unified action space abstracts platform-specific operations such as clicking, typing, scrolling, and dragging.
- Reasoning with System 1&2 Thinking: System 1 supports fast intuitive responses, whereas System 2 supports deliberate analytical reasoning for complex tasks.
- Reasoning with System 1&2 Thinking: Effective agents must select between rapid heuristic responses and detailed reasoning to improve efficiency, adaptability, and reliability.
- Reasoning with System 1&2 Thinking: UI-TARS uses task decomposition, long-form Chain-of-Thought reasoning, and reflection to handle intricate workflows and adjust after mistakes.
4 UI-TARS
UI-TARS is a native GUI agent that directly perceives screenshots, reasons over interaction history, and iteratively generates actions while learning from environment feedback.
- UI-TARS directly perceives screenshots, applies reasoning, and autonomously generates valid actions without cumbersome manual rules or cascaded modules.
- At each step, the model conditions on the task instruction, prior interactions, and the current device screenshot to output an action.The interaction history contains observations and actions, and the device returns a new observation after execution.
- UI-TARS inserts a thought before each action so the agent can reconsider previous observations and actions during deliberate decision-making.The thoughts provide structured, goal-oriented deliberation, especially for complex or ambiguous situations.
- The model represents trajectories as alternating observations, thoughts, and actions, extending the interaction sequence beyond observation-action pairs.
- To control memory use, UI-TARS retains the full history of prior actions and thoughts while limiting the input to the last N observations.This design targets constrained token budgets such as a 32k sequence length.
4.2 Enhancing GUI Perception
UI-TARS improves GUI perception and grounding through large-scale screenshot data, bottom-up interface understanding, diverse captioning tasks, and a unified cross-platform action space.
- GUI perception is difficult because screenshots are scarce, information-dense, spatially structured, and filled with small elements requiring precise interpretation.
- UI-TARS builds a large-scale screenshot dataset from websites, apps, and operating systems with metadata including element types, depth, bounding boxes, and text.Collection combines automated crawling with human-assisted exploration.
- A bottom-up construction strategy progresses from individual elements to holistic interface understanding, balancing component precision with layout interpretation.The resulting data supports five core tasks, including element description, dense captioning, state-transition captioning, question answering, and Set-of-Mark annotations.
- UI-TARS standardizes semantically equivalent actions across devices while adding platform-specific optional actions and terminal actions for completion or user intervention.
- The training data combines annotated action traces with open-source datasets standardized into the unified action-space format across web, mobile, and desktop platforms.Table 2 reports element and action-trace counts for the combined data sources.
- Directly mapping element descriptions to normalized coordinates improves the model’s ability to locate and ground visual GUI elements accurately.
4.4 Infusing System-2 Reasoning
UI-TARS infuses System-2 reasoning by enriching GUI knowledge, augmenting action traces with explicit thoughts, and selecting thought-action pairs that preserve causal alignment.
- System-1 intuition alone is insufficient for complex, changing environments, motivating flexible System-2 planning over global task structure.
- Approximately 6M filtered GUI tutorials provide text-image interaction examples that establish GUI knowledge and reasoning patterns.Each tutorial averages 510 text tokens and 3.3 images.
- Thought augmentation transforms action-focused traces into sequences containing observations, explicit thoughts, and actions to connect perception with decision-making.
- ActRe generates thoughts iteratively from preceding context and target actions while prompting deliberate, step-by-step reasoning and reflection.
- UI-TARS uses task decomposition, long-term consistency, milestone recognition, and trial-and-error patterns to structure multi-step reasoning.
- Reverse annotation can produce thoughts that superficially match actions without a true causal relationship because the known action biases the generated reasoning.
- Bootstrapping samples multiple thought-action pairs without prior ground-truth action knowledge and selects the pair leading to the correct action.This is intended to produce reasoning that simulates decision-making rather than merely justifying a predetermined action.
4.5 Learning from Prior Experience in Long-term Memory
UI-TARS uses online trace bootstrapping to collect, filter, refine, and reuse GUI interaction experience. Repeated training rounds expand the trace data while reducing reliance on manual intervention.
- Learning from Prior Experience: UI-TARS dynamically learns from real-world device interactions through semi-automated collection, filtering, and refinement, minimizing manual intervention.The paper states that accumulated knowledge supports more efficient adaptation to new tasks over successive iterations.
- Online Trace Bootstrapping: Online bootstrapping begins with human-annotated and model-generated task goals executed by the current agent in target GUI environments.The process produces raw interaction traces containing observations, thoughts, and actions.
- Online Trace Bootstrapping: A multi-level filtering process removes anomalous traces, low-scoring traces, and invalid suffixes after human-identified errors.Filtering combines rule-based rewards, VLM quality scoring, and human review that retains only valid prefixes.
- Online Trace Bootstrapping: Filtered traces are used to fine-tune the next model, while annotators refine or expand the instruction set for subsequent rounds.The process jointly updates the model and its task instructions.
- Online Trace Bootstrapping: The procedure runs for multiple rounds on hundreds of virtual PCs, with each latest model generating new traces that expand and refine the dataset.This creates an iterative data-generation and training loop.
4.5 Learning from Prior Experience in Long-term Memory
Reflection tuning trains UI-TARS on its own mistakes together with corrected continuations, enabling recovery from suboptimal decisions. DPO further uses erroneous and corrected actions to encode an explicit preference for better actions.
- Reflection Tuning: Reflection tuning exposes UI-TARS to errors it makes during realistic interactions and their corrections, addressing the limitations of idealized offline trajectories.The protocol is designed to teach recovery from suboptimal decisions rather than only error-free behavior.
- Reflection Tuning: Annotators identify an erroneous step, label a corrected thought and action, and continue the trace by specifying how to recover from the mistake.The recovery step acknowledges the error’s impact and realigns task progress.
- Reflection Tuning: SFT trains only on corrected steps, excluding the original error steps from the loss.This training choice uses positive correction samples while omitting the erroneous actions.
- Reflection Tuning: The reflective training process improves the agent’s ability to recognize and recover from errors in imperfect or uncertain conditions.The paper connects this reflective ability with greater adaptability to dynamic environments and tasks.
- Agent DPO: DPO incorporates both corrected and erroneous actions so the agent learns an explicit preference for corrected actions over suboptimal ones.The method addresses the limitation of SFT, which ignores negative examples.
4.6 Training
DPO trains UI-TARS from error-correction preference pairs, optimizing the policy to favor corrected actions while remaining close to the SFT policy.
- DPO Training: DPO optimizes UI-TARS using a preference dataset of corrected and erroneous actions rather than relying only on corrected examples.This supplies an explicit contrast between preferred and suboptimal behavior.
- DPO Training: The DPO loss increases the likelihood of corrected actions and decreases the likelihood of erroneous actions through an implicit reward function.The objective directly encodes the desired action preference.
- DPO Training: πθ denotes the optimized agent, πSFT the SFT agent, and β controls divergence between them.The objective uses the SFT policy as a reference while constraining the optimized policy.
4.6 Training
UI-TARS is trained in three phases that progressively emphasize broad GUI knowledge, higher-quality focused data, and reflective preference optimization.
- Training Process: The three-phase training process uses the Qwen-2-VL backbone and approximately 50B tokens for comparison and capability refinement across GUI tasks.Each phase incorporates progressively higher-quality data for complex reasoning tasks.
- Continual Pre-training Phase: Continual pre-training uses the full GUI dataset except reflection-tuning data to learn perception, grounding, and action-trace knowledge.This phase provides broad coverage of GUI elements and interactions.
- Annealing Phase: Annealing trains on high-quality perception, grounding, action-trace, and reflection-tuning subsets, producing UI-TARS-SFT.The phase focuses learning on decision-making strategies for real-world GUI interactions.
- DPO Phase: The DPO phase trains on annotated reflective pairs from online bootstrapping, producing UI-TARS-DPO.It reinforces optimal actions and penalizes suboptimal ones for more context-aware decisions.
5 Experiment
UI-TARS is evaluated across perception, grounding, online GUI execution, and reasoning, with results showing strong performance across diverse benchmarks and settings. Its advantages include cross-domain adaptability, efficient execution, and improved out-of-domain reasoning, while single-sample System-2 reasoning remains challenging.
- Evaluation Setup: UI-TARS is evaluated in three dimensions—perception, grounding, and agent task execution—using approximately 50B training tokens and 2B, 7B, and 72B model variants.The evaluation compares UI-TARS with commercial and academic baselines.
- Perception Capability: 82.8 on VisualWebBench is achieved by UI-TARS-72B, exceeding GPT-4o’s 78.5.UI-TARS-7B also leads WebSRC with 93.6, while UI-TARS-72B excels on ScreenQA-short.
- Grounding Capability: 38.1 on ScreenSpot Pro, 89.5 on ScreenSpot, and 91.6 on ScreenSpot v2 demonstrate strong grounding across professional, general, and mobile GUI settings.UI-TARS-72B scores 38.1 on ScreenSpot Pro, while UI-TARS-7B scores 89.5 and 91.6 on ScreenSpot and ScreenSpot v2, respectively.
- Online Agent Capability: UI-TARS-7B and UI-TARS-72B surpass OS-Atlas-7B by an absolute performance increase of 25 on AndroidControl and GUI Odyssey.UI-TARS-72B achieves SOTA performance across key Multimodal Mind2Web metrics, and UI-TARS performs strongly in both website and mobile domains.
- Online Agent Capability: 24.6 on OSWorld with 50 steps and 22.7 with 15 steps exceed Claude’s 22.0 and 14.9, while AndroidWorld reaches 46.6 versus the previous framework’s 44.8.DPO improves OSWorld performance, and negative samples help distinguish optimal from suboptimal actions.
- System-1 and System-2 Reasoning: System-2 reasoning is slightly worse than System-1 at N=1 but gains a clear advantage at N=16 and N=64 through greater candidate diversity.On AndroidWorld, an out-of-domain benchmark without corresponding training data, System-2 significantly outperforms System-1 at Bo1.
- System-1 and System-2 Reasoning: Achieving optimal System-2 performance with a single decisive output remains a significant challenge.The paper identifies reinforced fine-tuning as a possible direction for improving single-pass confidence.
6 Conclusion
The conclusion presents UI-TARS as a scalable native GUI agent integrating core capabilities and achieving state-of-the-art benchmark performance. It also points toward active and lifelong learning through continuous real-world interaction.
- Conclusion: UI-TARS integrates perception, action, reasoning, and memory into a scalable and adaptive native GUI agent framework.The paper positions these capabilities as foundations for future GUI-agent development.
- Conclusion: UI-TARS achieves state-of-the-art performance on challenging benchmarks such as OSWorld, outperforming systems including Claude and GPT-4o.The conclusion attributes the system’s capabilities to enhanced perception, unified action modeling, System-2 reasoning, and iterative refinement using online traces.
- Future Direction: Future GUI agents are described as moving toward active and lifelong learning driven by continuous real-world interactions.This direction is framed as autonomous learning with reduced human intervention and broader generalization.
A Case Study
The case study presents UI-TARS executing GUI tasks across Android, Ubuntu applications, and Windows Chrome. The examples cover media playback, extension installation, presentation editing, and browser configuration.
- Case Studies: UI-TARS changes slide 2’s background color to match the title color from slide 1 in Ubuntu Impress.
- Case Studies: UI-TARS plays the song under Mount Fuji in an Android task.
- Case Studies: UI-TARS installs the autoDocstring extension in VS Code on Ubuntu.
- Case Studies: UI-TARS configures Chrome on Windows to show the bookmarks bar by default.
B Data Example
The data examples illustrate perception-training tasks for GUI understanding, including dense captioning, state transitions, question answering, set marking, and element description.
- Perception Data: The perception dataset includes dense-caption examples for holistic understanding of GUI screenshots.
- Perception Data: Question answering and set-mark examples train models to answer interface questions and identify relevant GUI elements.
- Perception Data: Element-description examples provide descriptions of GUI components, with the target element highlighted by a pink bounding box.