Source-linked AI summary
You Only Look at Screens: Multimodal Chain-of-Action Agents
Zhuosheng Zhang, Aston Zhang
TL;DR
Existing GUI agents commonly depend on environment parsing and application-specific APIs, which can make inference inefficient and error-prone. Auto-GUI directly interacts with screens and uses previous action histories plus future action plans to predict actions, achieving state-of-the-art results on AITW.
Problem
Existing GUI agents rely on external parsing tools and application-specific APIs, causing inference inefficiency and risks of error propagation or information loss.
Method
Auto-GUI is a multimodal screen-interacting agent that uses previous action histories and future action plans to predict each action.
Results
On AITW's 30K instructions spanning application operation, web searching, and web shopping, Auto-GUI achieves state-of-the-art performance with 90% action type prediction accuracy and 74% action success rate.
Takeaways & Limitations
The results show that an end-to-end multimodal model can achieve state-of-the-art GUI-agent performance without external parsing tools or application-specific APIs.
Takeaways & Limitations
Experiments were conducted exclusively on AITW, and the approach was not extended to extremely large models.
Abstract
from arXiv · showhide
Autonomous graphical user interface (GUI) agents aim to facilitate task automation by interacting with the user interface without manual intervention. Recent studies have investigated eliciting the capabilities of large language models (LLMs) for effective engagement in diverse environments. To align with the input-output requirement of LLMs, most existing approaches are developed under a sandbox setting where they rely on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. Consequently, those approaches often grapple with inference inefficiency and error propagation risks. To mitigate the challenges, we introduce Auto-GUI, a multimodal solution that directly interacts with the interface, bypassing the need for environment parsing or reliance on application-dependent APIs. Moreover, we propose a chain-of-action technique -- leveraging a series of intermediate previous action histories and future action plans -- to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions, spanning multi-step tasks such as application operation, web searching, and web shopping. Experimental results show that Auto-GUI achieves state-of-the-art performance with an action type prediction accuracy of 90\% and an overall action success rate of 74\%. Code is publicly available at https://github.com/cooelf/Auto-GUI.
1 Introduction
Auto-GUI addresses limitations of sandboxed GUI agents by directly interacting with screens and using prior action histories plus future plans to choose actions. On AITW, it reports state-of-the-art performance across multi-step device-control tasks.
- Existing GUI agents often parse screens with external tools, creating lengthy inputs that increase inference cost and risking error propagation or information loss.
- Sandboxed approaches also require internal application APIs, which are often inaccessible in third-party applications.
- 30K unique AITW instructions span application operation, web searching, and web shopping, providing the evaluation setting.
- Auto-GUI directly interacts with screens without environment parsing or application-specific API access.
- Its chain-of-action technique uses previously executed actions and future action plans to guide each action decision.
- 90% action type prediction accuracy and 74% action success rate are reported for Auto-GUI.
2 Related Work
Language agents follow user instructions while interacting with environments, and this work focuses on autonomous GUI control for multi-step tasks. Auto-GUI avoids fine-grained parsing by reading screens directly and uses action histories plus future plans to guide decisions.
- 2.1 Language Agents: Language agents interact with environments to complete user-directed tasks, including application operation, web searching, and web shopping.
- 2.1 Language Agents: Autonomous agents aim to assist humans in achieving specific real-world goals, unlike communicative agents.
- 2.1 Language Agents: Mobile GUI agents must interpret natural-language instructions and directly control interfaces to complete multi-step tasks without manual intervention.
- 2.2 GUI Control with Natural Language: Existing GUI-control methods convert screens and actions into text through icon recognition, OCR, external tools, and application-specific APIs.
- 2.2 GUI Control with Natural Language: Auto-GUI instead reads GUI screens directly and outputs executable action types, gesture coordinates, and typed text without extra APIs.
- 2.2 GUI Control with Natural Language: The chain-of-action method uses previously executed actions and future action plans to inform decisions at each step.
3 Methodology
Auto-GUI maps screenshots, user goals, and action histories into executable GUI actions, while planning future action types to improve stepwise decisions. Its multimodal pipeline encodes vision and language inputs, interacts them through attention and fusion, and decodes both plans and current actions.
- Framework Overview: Auto-GUI receives a screenshot and user instruction, then predicts actions until the multi-step task is complete.An episode consists of successive screens, with one screenshot supplied at each interaction step.
- Encoding: The language input combines the user goal with previously executed action histories, which contain action type, touch point, lift point, and typed text.At the first step, the history is empty; at later steps, it records earlier actions.
- Encoding: A frozen image encoder extracts screenshot features, while a language encoder represents the goal and action history before multimodal interaction.A trainable projection converts screenshot features to the language-feature dimensionality.
- Interaction: Attention and gated fusion combine the language and screen representations into a fused representation for decoding.The design uses a single-head interaction layer and remains lightweight, with fewer than 1 billion parameters.
- Decoding: The decoder outputs future action plans and a current action containing action type, touch point, lift point, and typed text.Future plans are sequences of action types; supported actions include gestures, typing, system actions, and completion.
- Action Representation: Normalized coordinates represent click and scroll gestures, using fixed directional coordinates for scrolling.Click coordinates retain four decimal places, while gesture coordinates follow the benchmark’s [y, x] convention.
4 Experiments
The experiments evaluate Auto-GUI on the AITW benchmark using multiple LLM and specialized-agent baselines, unified and separate training variants, and ablations of its design choices. Auto-GUIunified achieves the best overall performance, while chain-of-action and coordinate normalization improve results.
- 4.1 Dataset: AITW contains 715K episodes covering 30K unique instructions across diverse multi-step GUI tasks, Apps, websites, devices, and operating systems.The benchmark includes General, Install, GoogleApps, Single, and WebShopping subsets.
- 4.2 Baselines: The baselines include in-context and fine-tuned language models plus specialized Transformer-based behavioral-cloning agents.Language-model baselines use textual HTML-like screen descriptions, while behavioral cloning can use stacked screen-action histories.
- 4.5 Main Results: Auto-GUIunified achieves the best overall performance compared with all baselines and generalizes across the benchmark’s task subsets.The unified model is trained across subsets, whereas separate models are trained and evaluated independently on each subset.
- 4.5 Main Results: +5.74% and 4.04% are the respective contributions of chain of actions and coordinate normalization in the ablation study.The selected setup uses up to 8 previous actions and 4 future actions without exceeding the model’s sequence limits.
- 4.5 Main Results: Prompting and fine-tuning language models are not competitive with multimodal approaches on this evaluation.The authors attribute this to information loss from parsed HTML elements relative to screen vision features.
5 Analysis
The analysis examines category-level accuracy, transfer across task subsets, feature and model-scale choices, and computational cost. Auto-GUI’s main weaknesses are low-level grounding errors, while vision features and efficient medium-scale modeling support practical deployment.
- 5.1 Analysis of Auto-GUI by Category: Auto-GUI exceeds 90% average action type accuracy, but click-region and scroll-direction prediction remain the principal challenges.The model often selects the correct action type while grounding it at the wrong location or with the wrong direction.
- 5.1 Analysis of Auto-GUI by Category: ChatGPT reaches 41.72% action type accuracy but only 8.5% clicking-position and 4.0% scrolling-direction accuracy on the General test set.The comparison indicates that HTML-based layout information alone does not accurately execute lower-level actions.
- 5.2 Transferability: Auto-GUI achieves decent transfer performance across varying domains, indicating captured GUI-control knowledge applicable to different subsets.The unified model offers broader training-data coverage for potential real-world use.
- 5.3 Pre-trained Features and Model Scale: BLIP-2 provides relatively better performance than CLIP, while increasing model scale yields comparatively marginal gains.The authors therefore use BLIP-2 by default and focus on base and large models under memory and computation constraints.
- 5.5 Discussion: Vision features are critical, whereas larger model sizes do not necessarily produce dramatic performance improvements.These findings identify perception and model scaling as distinct determinants of efficacy.
- 5.4 Computation Cost: Less than 10GB GPU memory and under one second per action prediction give Auto-GUI nearly real-time inference, over 10 times faster than Llama 2.The reported comparison links competitive performance with fast inference and modest resource cost.
6 Conclusion
Auto-GUI is a multimodal autonomous GUI agent that directly interacts with screens and uses previous actions plus future plans to choose actions. It achieves state-of-the-art performance without external parsing tools or application-specific APIs.
- Auto-GUI is a multimodal agent for autonomous GUI control that directly interacts with screens.
- The chain-of-action approach uses intermediate previous action histories and future action plans to predict actions.
- Auto-GUI achieves state-of-the-art performance without relying on external tools or application-specific APIs to parse environments and interpret actions.
- Auto-GUI infers an action in less than one second and demonstrates generality across domains.
Limitations
The study reports two limitations: it does not evaluate extremely large models, and its experiments are conducted exclusively on AITW. The authors suggest future evaluation on other benchmark datasets as they become available.
- The study does not extend Auto-GUI to extremely large models because it targets a simple yet effective GUI-agent solution.
- The experiments and analysis are conducted exclusively on AITW, despite the field’s rapid development.
- Future studies are anticipated to apply the approach to other benchmark datasets as they become available.
A.2 Data Statistics
AITW is a large-scale GUI-control benchmark with 715K episodes and 30K unique instructions across diverse multi-step tasks, Apps, websites, devices, and operating systems. Its five subsets cover general interaction, installation, Google applications, atomic actions, and web shopping.
- AITW contains 715K episodes spanning 30K unique instructions across application operation, web searching, and web shopping.
- The benchmark covers over 350 Apps and websites, multiple device types and operating systems, and varying screen resolutions.
- Subsets: The General subset contains miscellaneous third-party-App, website-interaction, and question-answering tasks.
- Subsets: The Install subset covers installing, uninstalling, logging Apps, and App-login support.
- Subsets: GoogleApps contains tasks involving Gmail, Calendar, Photos, and Settings, while Single contains atomic tasks whose preceding actions are completed.
- Subsets: WebShopping covers online-shopping tasks such as searching for items, adding items to carts, and viewing shopping carts.
- Splits: Each subset is split episode-wise into training, validation, and test sets using an 80/10/10% split.
B Implementation Details
The implementation represents GUI actions with normalized coordinates and typed text, compares Auto-GUI with in-context, fine-tuned, and specialized GUI-agent baselines, and reports results through tables and ablations.
- Action Representation: Target actions contain an action type, touch point, lift point, and typed text, with six supported action types.
- Action Representation: Dual-point gestures represent clicks and scrolling through touch and lift coordinates, while type actions store text in the typed_text field.
- Coordinate Normalization: Click and scroll coordinates are normalized to accelerate convergence and mitigate coordinate ambiguity.
- Baselines: The study compares in-context-learning LLMs, fine-tuned LLMs, and a specialized Behavioural Cloning GUI agent.
- Baselines: In-context-learning baselines receive HTML screen descriptions derived from OCR and icon detection and predict predefined actions.
- Baselines: The fine-tuned Llama 2 baseline uses LoRA and randomly samples 1% of the training data because fine-tuning is expensive.
- Baselines: The Behavioural Cloning agent combines instruction, screen, OCR text, icon representations, and optionally stacked screen-action history to predict actions.
C.1 Subset Analysis
Auto-GUIunified underperforms BC-history on the Install and GoogleApps subsets, while Auto-GUIseparate performs better than BC-history. Using all GoogleApps training data does not improve performance, possibly because of data imbalance.
- Auto-GUIunified performs relatively worse than BC-history on the Install and GoogleApps subsets.
- Using only 10% of GoogleApps training data limits Auto-GUIunified because of data-balance and computation-overhead considerations.
- Auto-GUIunified does not improve with all GoogleApps training data, possibly because of data imbalance, whereas Auto-GUIseparate outperforms BC-history.
C.2 Ablation Study
The ablation study finds that chain-of-action and coordinate normalization improve overall performance. Screen annotations provide further gains, but are excluded by default because they are not always available in real-world applications.
- Ablation Study: Chain-of-action and coordinate normalization contribute +5.74% and 4.04%, respectively, to overall performance.
- Screen Annotations: The annotation variant adds icon and text screen descriptions, organized in HTML syntax, to the language input.
- Screen Annotations: Auto-GUI performs better with annotated screen descriptions, indicating room for further performance gains.
- Screen Annotations: Annotations are excluded by default because they are not always available in real-world applications.