Source-linked AI summary
Android in the Zoo: Chain-of-Action-Thought for GUI Agents
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang
TL;DR
GUI agents must connect action history, current screens, and possible actions, but these relations and their underlying semantics are difficult to model. The paper introduces CoAT and the AITZ dataset, finding improved GUI-agent performance and comparable results between a ~1B fine-tuned agent and an LLM-based agent.
Problem
Relations between navigation history, current observations, and possible actions are highly implicit, while data capturing these semantics is lacking for small models.
Method
CoAT combines screen descriptions, action thinking, next-action descriptions, and possible outcomes, while AITZ provides data connecting screen perception with action decision-making.
Results
CoAT improves action prediction across three off-the-shelf LMMs and improves GUI-agent goal progress and learning efficiency.
Takeaways & Limitations
Fine-tuning a ~1B AUTO-UI-base agent with CoAT achieves comparable performance with an LLM-based GUI agent, indicating CoAT's potential for GUI navigation.
Takeaways & Limitations
The effects of image resolution and GUI-related pretraining abilities on navigation performance remain under-explored.
Abstract
from arXiv · showhide
Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typically consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon three off-the-shelf LMMs, CoAT significantly improves the action prediction compared to previous proposed context modeling. To further facilitate the research in this line, we construct a dataset Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 1B model (i.e. AUTO-UI-base) on our AitZ dataset achieves on-par performance with CogAgent-Chat-18B.
1 Introduction
GUI agents must reason over screenshots, action semantics, and outcomes rather than relying mainly on operation coordinates. The paper introduces CoAT and AITZ to provide this capability and evaluates them through zero-shot prompting and fine-tuning.
- GUI navigation matters for everyday smartphone use, including accessibility for people with physical disabilities.
- Prior GUI agents often focus on operation coordinates while overlooking the semantics connecting screens, decisions, and action outcomes.
- CoAT interleaves screen description, action thinking, next-action description, and possible action outcomes to represent navigation semantics.
- CoAT improves action prediction across GPT-4V, Gemini-Pro-Vision, and Qwen-VL-Max, while fine-tuning a small multimodal agent improves goal progress and learning efficiency.
- AITZ contains 2,504 unique instructions and 18,643 screen-action pairs with four semantic annotation types across over 70 Android apps.
2 Chain-of-Action-Thought (CoAT)
CoAT is a structured representation of GUI navigation that makes implicit relations among history, current screens, decisions, and outcomes explicit. Its components can be combined flexibly to support action prediction and coherent sequential decision-making.
- 2.1 Definition: Direct policy learning is difficult because relations among history, current observations, and possible actions are highly implicit.
- 2.1 Definition: CoAT provides a shortcut for comprehending interaction dynamics when prior state, such as an active search bar, determines the next action.
- 2.1 Definition: Screen Description supplies textual context, while Action Think analyzes the query, current screen, and history to infer actions that fulfill the target.
- 2.1 Definition: Action Result links an action to future observations by synthesizing outcome changes between screenshots and maintaining a consistent history.
- 2.2 Comparison: CoAT carries more explicit screen and action semantics than Standard, Chain-of-Action, and Chain-of-Thought prompting methods.
3 Android in the Zoo (AITZ)
AITZ addresses the lack of data capturing CoAT’s underlying semantics by constructing a fine-grained Android GUI navigation dataset from sampled and filtered AITW episodes. It adds semantic annotations to screen-action data and offers richer instruction coverage than related datasets.
- AITZ targets the scarcity of data capturing the underlying semantics of CoAT, which limits small models’ ability to acquire this capability through fine-tuning.
- Data Collection: The dataset is built from AITW by sampling instructions and episodes to reduce redundancy and filter screenshot-instruction mismatches.
- Semantic Annotation: AITZ uses GPT-4V to generate screen descriptions, action thinking, next action descriptions, and action-result summaries for navigation episodes.
- Dataset Analysis: Compared with related Android GUI navigation datasets, AITZ provides richer instruction coverage while retaining a similar magnitude of human demonstrations.
- Dataset Analysis: Most screen descriptions contain 80–120 words, action thinking 30–70 words, and action results 20–80 words.
4 Experimental Setup
The experiments evaluate CoAT with CogAgent and AUTO-UI on AITZ using action-level and sequence-level metrics. The setup uses separate training and testing episodes and reports both zero-shot and fine-tuning conditions.
- Baseline Models: CogAgent is a multimodal GUI agent with high-resolution visual encoding, while AUTO-UI is a specialized AITW-trained navigation model retrained on AITZ.
- Evaluation Setup: The study reports zero-shot evaluation for CogAgent and fine-tuning evaluation for AUTO-UI on AITZ.
- Evaluation Metrics: An action matches the gold action only when both its action type and action details, such as direction, text, position, or button, are correct.
- Evaluation Metrics: Goal progress measures the relative position of the first error in the agent’s action sequence toward the user query.
- Evaluation Setup: The data split contains 70% training episodes and 30% testing episodes, with 1998 training and 506 testing episodes.
5 Experiments
Experiments show that CoAT improves GUI action prediction and that semantic annotations strengthen AUTO-UI’s fine-tuning performance. The analyses connect these gains to more coherent action histories and better training efficiency.
- Zero-Shot Evaluation: CoAT contributes significant overall performance improvements in zero-shot CogAgent evaluation, while a fine-tuned approximately 1B AUTO-UI model achieves comparable performance with a language-model-based agent.
- Fine-tuning Evaluation: Previous action results, especially combined with action thinking and action descriptions, significantly improve AUTO-UI’s overall action prediction accuracy.
- Fine-tuning Evaluation: The STOP action-matching score rises from 60.12 to 79.17 when previous action results strengthen the coherence of decision-making.
- Fine-tuning Evaluation: Adding screen descriptions and/or previous action results improves performance immediately, especially for predicting CLICK actions.
- Fine-tuning Evaluation: Screen descriptions can slightly decrease action-matching score and goal progress for AUTO-UI, which the authors attribute to limitations of its low-resolution visual encoder.
- Qualitative Analysis: Qualitative analysis finds that previous action results help AUTO-UI judge task progress by describing prior outcomes in words and connecting adjacent time steps.
6 Related Works
GUI navigation requires agents to perceive interfaces and infer actions that fulfill user instructions. Prior work separates element recognition from action inference or uses limited visual and semantic representations, whereas CoAT connects screen perception, action history, and future decisions.
- GUI navigation requires agents to both perceive smartphone or website interfaces and deduce actions that fulfill user instructions.
- Earlier GUI navigation studies often separate UI-element recognition from action inference, creating a discrepancy between user intent and performed actions.
- Large Multimodal Models: Large multimodal models commonly encode visual signals with vision transformers before incorporating them into language models.
- Large Multimodal Models: Recent multimodal GUI agents increasingly use high-resolution image encoders, including CogAgent’s 1120×1120 visual features and other high-resolution architectures.
- LMM as GUI Agents: LMM-based GUI-agent research includes direct prompting of GPT-4V and fine-tuning smaller models on GUI-related datasets.
7 Conclusion
The work introduces CoAT to make GUI agents reason over screen and action semantics, and builds AITZ to inject these capabilities into existing agents. Experiments demonstrate the paradigm’s efficiency and usefulness for GUI navigation.
- CoAT analyzes human orienteering processes to improve navigation ability in LMM-based GUI agents.
- AITZ provides high-quality CoAT-driven data generated through cooperation between human experts and GPT-4V.The dataset bridges perception and cognition for GUI navigation agents.
- Experiments demonstrate the efficiency and usefulness of the proposed CoAT paradigm.
8 Limitations
The paper shows that CoAT and AITZ can stimulate GUI reasoning, but comparisons across GUI models remain difficult because their architectures and training data differ. The effects of image resolution and GUI-related pretraining abilities remain under-explored.
- Comparisons among current GUI models are difficult because they use different model structures and training data.
- The effects of image resolution and GUI-related pretraining tasks on navigation performance remain under-explored.The paper identifies text recognition, GUI imagery, screen question-answering, and GUI grounding as relevant abilities.
- Future work will precisely measure how image resolution, text recognition, and GUI grounding abilities affect GUI navigation.
9 Ethics
AITZ derives from an open-source dataset permitted for academic use and was checked for privacy during data collection. Its semantic annotations concern smartphone operations and are described as posing neither ethical risks nor harmful guidance.
- AITZ is sourced from AITW, an open-source dataset permitted for academic use.
- Instruction-episode checks addressed privacy concerns, and sampled data contains no real personal information.Fake or meaningless data were allowed.
- Because AITZ contains only semantic annotations on smartphone operations, the authors report neither ethical risks nor harmful guidance.
Screen Description
The dataset pipeline combines GPT-4V generation with human verification, while defining screen descriptions and a simplified action space for GUI navigation. Instruction sampling addresses distribution imbalance and potential leakage in the source data.
- Screen Description: The screen-description prompt asks for a mobile screen’s type, main content, and primary icons or apps in fewer than five sentences.
- Data Construction: Human verification complements GPT-4V generation to detect factual errors and support high-quality data construction.
- Instruction Sampling: Instruction sampling was introduced because source-set distributions were nearly identical across splits, indicating serious data leakage.
- Instruction Sampling: The SINGLE subset is clustered by instruction verbs and then balanced-sampled, with manual classification, tf-idf clustering, and verification used for larger groups.
- Data Construction: GPT-4V generates screen descriptions and action descriptions for five action categories: SCROLL, TYPE, PRESS, CLICK, and STOP.The generation process uses correct next-action coordinates for action grounding.
- Action Space: CLICK predicts relative screen coordinates, whereas SCROLL uses textual direction because scroll points convey little information in most cases.The action-space redesign splits the source DUAL_POINT type into CLICK and SCROLL.
B.1 Comparison between Prompting Methods
CoAT leads the prompting comparison by incorporating richer semantic context into GUI-agent inputs, while set-of-mark tagging generally outperforms plain textual UI representations. The baseline comparison further reports that Auto-UI + CoAT reaches performance on par with CogAgent-Chat-18B after fine-tuning on AITZ.
- B.1 Comparison between Prompting Methods: Set-of-mark tagging adds an annotated screenshot, whereas textual representation appends a textual description of UI elements to the input.
- B.1 Comparison between Prompting Methods: GPT-4V prompted by CoAT achieves the leading overall performance and leads in predicting each action type.
- B.1 Comparison between Prompting Methods: Set-of-mark tagging generally performs better than plain textual representations of UI elements.
- B.2 Comparison between Baselines: Auto-UI + CoAT achieves performance on par with CogAgent-Chat-18B after fine-tuning on AITZ, with Auto-UI slightly higher on goal progress.
- B.2 Comparison between Baselines: Auto-UI uses less training data and a 224 x 224 visual encoder, whereas CogAgent uses broader GUI training and a 1120 x 1120 visual encoder.
- B.1 Comparison between Prompting Methods: The prompting comparison evaluates Qwen-VL-Max, Gemini-1.0-Pro-Vision, and GPT-4V using action prediction accuracy and hit rate.
- B.1 Comparison between Prompting Methods: Figure 10 visualizes set-of-mark tagging and its corresponding textual representations for screen elements.