PaperlayerView exact revision on arXiv

Source-linked AI summary

OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao

arXiv:2410.23218v1cs.CLcs.CVcs.HC

TL;DR

Open-source GUI action models have struggled with grounding and out-of-distribution generalization. OS-Atlas addresses this gap with cross-platform data synthesis, a large GUI grounding corpus, and training innovations, achieving state-of-the-art performance across six benchmarks and three platforms.

  • Problem

    Existing open-source GUI action models show poor GUI grounding and out-of-distribution generalization, limiting their applicability in real-world situations.

  • Method

    OS-Atlas combines a multi-platform data-synthesis toolkit, a corpus exceeding 2.3 million screenshots and 13 million GUI elements, and training that resolves action-naming conflicts.

  • Results

    OS-Atlas achieves state-of-the-art performance across six benchmarks spanning desktop, mobile, and web, including unseen tasks across all six OOD evaluation datasets.

  • Takeaways & Limitations

    OS-Atlas has potential as an open-source alternative to leading commercial VLMs for developing future GUI agents.

Abstract

from arXiv · show

Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas - a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces. Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.

1 INTRODUCTION

Existing open-source VLM-based GUI action models struggle with GUI grounding and Out-Of-Distribution generalization because of limited multi-platform screenshot pretraining and inconsistent action naming. OS-Atlas addresses these problems with a cross-platform data toolkit, a large grounding corpus, action-conflict resolution, and broad evaluation across GUI platforms.

  • Motivation: Existing open-source VLM-based GUI action models perform poorly on GUI grounding and Out-Of-Distribution scenarios, limiting their real-world applicability.The paper attributes this ineffectiveness primarily to insufficient GUI screenshot pretraining and inconsistent action naming across datasets.
  • Motivation: A lack of large-scale, open-source screenshot corpora spanning platforms, applications, and resolutions limits GUI grounding and OOD generalization.The paper argues that comprehensive pretraining across Windows, MacOS, Linux, iOS, and Android could improve generalization because GUIs share design principles.
  • Motivation: Different labels for logically equivalent actions, such as “tap” and “click,” create training confusion and decrease cross-platform performance.The inconsistency reflects heterogeneity in existing datasets and action naming conflicts across platforms.
  • Contributions: The released toolkit automatically synthesizes GUI grounding data across Windows, macOS, Linux, Android, and the Web, reducing future data-curation engineering effort.This is presented as the first multi-platform GUI grounding data synthesis toolkit.
  • Contributions: Over 2.3 million distinct screenshots and more than 13 million GUI elements form the largest multi-platform GUI grounding corpus to date.The corpus adds desktop grounding data, while ScreenSpot-V2 re-annotates 11.32% incorrect samples in ScreenSpot.
  • Contributions: OS-Atlas resolves action naming conflicts, operates universally across GUIs, and is evaluated on six benchmarks spanning desktop, mobile, and web platforms.The paper presents OS-Atlas as an open-source alternative to commercial VLMs such as GPT-4o for future GUI agents.

2 RELATED WORK

Related work frames GUI agents as a response to the limited access imposed by closed-source software, while distinguishing referring-expression grounding from broader instruction grounding. OS-Atlas addresses these gaps as the first language action model specifically designed for GUI agents, using multi-platform data infrastructure.

  • GUI Agents and Large Action Models: Closed-source software limits agents’ access to internal APIs and code, motivating research on agents that interact through graphical user interfaces.
  • GUI Agents and Large Action Models: OS-Atlas is the first language action model specifically designed for GUI agents.
  • GUI Executable Language Grounding: GUI grounding converts natural-language instructions into actions and parameters, including element coordinates; its training data comprises referring expression grounding and instruction grounding.
  • GUI Executable Language Grounding: Instruction grounding is more important for real-world applications because it includes coordinate-free actions and nuanced instructions requiring reasoning about the target action and element.
  • GUI Executable Language Grounding: OS-Atlas develops multi-platform infrastructure for collecting GUI grounding data, whereas concurrent work focuses on scaling web data בלבד.

3 OS-ATLAS

OS-Atlas is a foundation action model for GUI agents built through data and methodological innovations. Its pipeline combines large-scale GUI grounding pre-training with multitask action fine-tuning and a unified action space for cross-platform generalization.

  • OS-Atlas is proposed as the first foundation action model specifically designed for GUI agents, with innovations in both data and methodology.
  • The two-stage pipeline first pre-trains VLMs to identify GUI elements from screenshots, then fine-tunes them to transform instructions into executable actions.Grounding uses screenshot, referring-expression or instruction, and coordinate triplets; action fine-tuning predicts actions from screenshots, instructions, and action history.
  • 13 million GUI grounding examples are used to build OS-Atlas-Base before multitask fine-tuning on agent data produces OS-Atlas.
  • The grounding corpus addresses desktop-data scarcity by collecting GUI data across web, Android, Linux, Windows, and MacOS environments.Android and Linux use simulated environments, while Windows and MacOS data are collected on physical machines using accessibility trees.
  • A unified action space resolves cross-platform conflicts by standardizing three basic actions—click, type, and scroll—while custom actions support unseen platform-specific tasks.Custom actions include operations such as opening an app and dragging an object.

4 EXPERIMENTS: GROUNDING TASKS

OS-Atlas-Base is evaluated on cross-platform GUI grounding with corrected ScreenSpot annotations, multiple model settings and backbones, and accuracy plus IoU metrics. It achieves state-of-the-art grounding results, benefits from scaled data and targeted pretraining, and improves GPT-4o-based OSWorld agents when used as the grounding module.

  • ScreenSpot results: Under both settings, OS-Atlas-Base significantly outperforms previous grounding models on ScreenSpot across mobile, desktop, and web platforms.The same trend appears on ScreenSpot-V2, and grounding pretraining further improves GUI-pretrained models such as Qwen2-VL.
  • Data scaling: Grounding accuracy and IoU positively correlate with grounding-data scaling, especially IoU and web grounding in the domain containing nearly 10 million elements.The weaker accuracy correlation reflects that accuracy does not capture finer-grained localization errors.
  • Ablation: Ablations show that referring-expression data is nearly sufficient for a strong, scalable grounding model, while removing platform data tests cross-platform generalization from web-only pretraining.The ablation removes instruction-grounding data and then mobile and desktop data to isolate these effects.
  • OSWorld evaluation: On OSWorld, GPT-4o paired with OS-Atlas-Base significantly outperforms SeeClick and Set-of-Mark while still lagging behind human performance.This setup replaces the agent’s generated coordinates with coordinates from an external grounding module, demonstrating OS-Atlas-Base’s standalone grounding potential.

5 EXPERIMENTS: AGENT TASKS

OS-Atlas is evaluated on five agent benchmarks spanning mobile, web, and desktop under zero-shot OOD and supervised fine-tuning settings. It achieves state-of-the-art performance across three platforms, six datasets, and both settings, while ablations show the importance of grounding pre-training and a unified action space.

  • Evaluation setup: Five benchmarks cover Android, web, and Windows desktop agents, using only their test splits for evaluation.The benchmarks are AndroidControl, GUI-Odyssey, GUI-Act-Web, OmniAct-Web, and OmniAct-Desktop.
  • Evaluation setup: The experiments assess zero-shot OOD generalization and supervised fine-tuning, with GPT-4o as the OOD baseline and InternVL-2, Qwen2-VL, and SeeClick as fine-tuning backbones.The two settings represent direct evaluation on unseen tasks or domains and downstream task-specific agent training.
  • Metrics: Type, Grounding, and SR measure action-type accuracy, GUI grounding accuracy, and step-wise success when both the action and its arguments are correct.Type is the exact-match score between predicted and ground-truth action types; SR is the step-wise success rate.
  • Main results: OS-Atlas achieved SOTA performance across three platforms, six datasets, and two evaluation settings, outperforming GPT-4o on all six OOD datasets, including unseen desktop-domain tasks.The results support OS-Atlas as a potential open-source alternative to leading VLMs for GUI agents.
  • Ablations: Removing grounding pre-training significantly degrades performance, especially on desktop and web, where fine-tuning data are limited to none and 7k samples, respectively.The ablation highlights the importance of the grounding-data synthesis infrastructure and scaling the pre-training corpus for OOD performance.
  • Ablations: Removing the unified action space also causes a noticeable performance drop, indicating that conflicting action spaces degrade model performance.The comparison uses the optimal action-space design proposed by state-of-the-art models for each fine-tuning dataset.

6 CONCLUSION

OS-Atlas is presented as a foundation action model for GUI agents that performs exceptionally on open-environment tasks across six complex benchmarks. Its results highlight its potential as an open-source alternative to powerful commercial VLMs such as GPT-4o.

  • OS-Atlas is presented as a foundation action model for GUI agents.
  • OS-Atlas demonstrates exceptional performance on open-environment GUI tasks across six complex benchmarks.
  • OS-Atlas has potential as an open-source alternative to powerful commercial VLMs such as GPT-4o for developing future GUI agents.

ETHICS AND REPRODUCIBILITY STATEMENT · A AUTHOR CONTRIBUTIONS · B DATA STATISTICS

The paper reports its ethical basis, reproducibility provisions, contributor roles, and pre-training corpus statistics. The data and benchmarks are cited, experimental details are provided, and responsibilities span infrastructure, modeling, experiments, writing, discussion, demonstrations, and strategy.

  • ETHICS AND REPRODUCIBILITY STATEMENT: The research uses synthesized or reprocessed previously released datasets and properly cites all datasets and benchmarks.The authors report no discrimination, bias, or fairness issues and do not expect the models to generate potentially harmful content.
  • ETHICS AND REPRODUCIBILITY STATEMENT: Experimental details are provided in Section 5 to support reproducibility.
  • A AUTHOR CONTRIBUTIONS: Project concept and leadership were led by Zhiyong Wu, while data infrastructure involved Yian Wang, Liheng Chen, and Fangzhi Xu.
  • A AUTHOR CONTRIBUTIONS: Data curation involved Yian Wang, Zhenyu Wu, Fangzhi Xu, Liheng Chen, Qiushi Sun, Kanzhi Cheng, and Zichen Ding.
  • A AUTHOR CONTRIBUTIONS: The grounding model was developed by Zhenyu Wu, Yian Wang, and Zhiyong Wu, and the action model by Zhiyong Wu, Fangzhi Xu, and Zhenyu Wu.
  • A AUTHOR CONTRIBUTIONS: Experiments, analysis, paper writing, research discussions, technical advice, demos, websites, and strategic advice were distributed among the listed authors.Experiments and analysis included Chengyou Jia; discussions involved all authors; demos and websites involved Zhenyu Wu and Yian Wang.
  • B DATA STATISTICS: The statistics of the collected pre-training corpus are detailed in Table 7.

C SCREENSPOT-V2

Screenspot-v2 revises Screenspot to address annotation and question-quality problems that could bias evaluation, while preserving the original question count. The revision modifies questions across web, desktop, and mobile domains and corrects instructions and bounding boxes.

  • Annotation and question issues: Screenspot error analysis identified spelling mistakes, absent screenshot elements, ambiguous questions, similar questions, and incorrectly labeled ground-truth bounding boxes.These issues could produce incorrect or non-unique answers and undermine benchmark validity.
  • Revision approach: Screenspot-v2 removes problematic questions, rewrites REG-form instructions as natural language, and corrects mislabeled ground-truth bounding boxes.The revised release keeps the total number of questions unchanged.
  • Revision scope: 63 out of 436 (≈14.4%) web questions, 28 out of 334 (≈8.4%) desktop questions, and 53 out of 502 (≈10.6%) mobile questions were modified.The benchmark evaluation results are reported in Table 8.

D DETAILS OF EVALUATION BENCHMARKS · 1. Basic Actions

The evaluation benchmarks cover grounding and executable language grounding across desktop, mobile, and web environments. The basic action space standardizes cross-platform operations, while custom actions support platform- and dataset-specific behavior; evaluation also includes diverse OOD settings.

  • D DETAILS OF EVALUATION BENCHMARKS: AndroidControl-Low provides low- and high-level instructions, whereas AndroidControl-High provides only the high-level instruction.AndroidControl is still classified as OOD because its diverse OOD splits differ from the training set.
  • D DETAILS OF EVALUATION BENCHMARKS: The model is evaluated across desktop systems, mobile platforms, and web browser environments by interpreting screenshots and taking actions.The described environments include Windows, macOS, Linux, Android, and iOS.
  • D DETAILS OF EVALUATION BENCHMARKS: Grounding requires locating or inferring screenshot elements from a description.The task may require identifying elements that are not explicitly stated.
  • D DETAILS OF EVALUATION BENCHMARKS: Executable language grounding maps a screenshot and task instruction to executable actions, returned as Python code in the specified format.The action set includes both basic and custom actions.
  • 1. Basic Actions: Basic actions are standardized across platforms and defined with fixed formats for consistency and reliability.The basic action set includes CLICK, TYPE, and SCROLL.
  • 1. Basic Actions: CLICK uses a specified coordinate, with the format CLICK <point>[[x-axis, y-axis]]</point>.An example is CLICK <point>[[101, 872]]</point>.
  • 1. Basic Actions: TYPE enters specified text, while SCROLL moves in a specified direction using UP, DOWN, LEFT, or RIGHT.The formats are TYPE [input text] and SCROLL [direction (UP/DOWN/LEFT/RIGHT)].
  • 1. Basic Actions: Custom actions are platform- and environment-specific extensions whose implementations vary across datasets.They extend the basic action set to support new and unseen actions.

E DETAILS OF EVALUATION METRICS

The evaluation standardizes metrics across baseline methods by using action-specific correctness criteria. Click actions require coordinate proximity, typed actions require correct content and action type, while scroll and other actions require exact matches.

  • Click-based actions: Click actions are correct when predicted coordinates fall within 14% of screen width from the ground-truth coordinates.Models must generate both the action type and position coordinates (x,y).
  • Type-based actions: Type-based actions are correct only when both action type and content match, with text accepted when F1 >0.5.The F1 score is calculated between predicted text and ground truth.
  • Scroll actions: Scroll actions are correct only when the direction—UP, DOWN, LEFT, or RIGHT—exactly matches the ground truth.The direction argument must precisely match the reference.
  • Other actions: Other actions, such as PRESS BACK, are correct only when they exactly match the ground truth.The evaluation metrics are standardized across baseline methods for each action.

F TRAINING DETAILS · G CASE STUDY: OS-WORLD

The training details specify platform-specific visual-input handling and diverse grounding-data formatting, while the OS-World case study illustrates grounding-mode interaction with GPT-4o task planning. Together, these passages describe how OS-Atlas processes GUI inputs, structures training samples, and executes click steps.

  • F TRAINING DETAILS: OS-Atlas-Base and OS-Atlas (4B) InternVL-2 use Dynamic Aspect Ratio Matching with up to 6 tiles of 448×448 pixels plus a global thumbnail.The max dynamic patch parameter is set to 6 to capture sufficient pixel information after resizing.
  • F TRAINING DETAILS: The agentic benchmark details define whether previous-action history is provided and count unified actions as basic actions plus custom actions.These benchmark descriptions distinguish input history conditions and task action-space size.
  • F TRAINING DETAILS: OS-Atlas-Base and OS-Atlas (7B) Qwen2-VL use dynamic visual tokens for arbitrary resolutions, with a 1024x1024 max-pixel setting during training and inference.The setting is reported to yield excellent GUI grounding results while optimizing training and inference cost.
  • F TRAINING DETAILS: Grounding pre-training data follows SeeClick by formatting REG samples as point grounding, box grounding, and OCR.Each type uses 30 GPT-generated prompts; 15 samples are grouped per conversation with 100 predefined prefix prompts.
  • G CASE STUDY: OS-WORLD: The OS-World case study asks the agent to modify VS Code settings to hide all "__pycache__" folders in the explorer view.This task exemplifies the type of desktop interaction used in the case study.
  • G CASE STUDY: OS-WORLD: In OS-World grounding mode, OS-Atlas-Base integrates GPT-4o as a task planner and outputs coordinates for each Click step from step-level instructions.The figure presents this setup as an agent built from OS-Atlas-Base’s grounding capabilities and GPT-4o planning.
Loading 2410.23218v1…