PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection

Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, Fei Wu

arXiv:2501.04575v1cs.AIcs.CLcs.HC

TL;DR

GUI agents can struggle to leverage information from previous steps. InfiGUIAgent uses two-stage supervised fine-tuning to develop fundamental GUI skills and native reasoning, and performs well across GUI tasks, surpassing several open-source baselines.

  • Problem

    Existing GUI agents may perform single-step reasoning but struggle to leverage information from previous steps and reflect on past experience.

    Source: §1 · p. 1

  • Method

    InfiGUIAgent uses two-stage supervised fine-tuning: Stage 1 develops GUI understanding and grounding, while Stage 2 integrates hierarchical and expectation-reflection reasoning into synthesized training data.

    Source: §1 · p. 1

  • Results

    Across GUI tasks, InfiGUIAgent performs well and surpasses several open-source baselines; on ScreenSpot, InfiGUIAgent-2B achieves 76.3% accuracy, exceeding ShowUI (75.1%).

    Source: §4.2 · p. 6, §5 · p. 7

  • Takeaways & Limitations

    Evaluation on raw screenshots without additional GUI metadata demonstrates the model’s applicability to real-world scenarios.

    Source: §5 · p. 7

    Expectation generation excludes the next state, which may impair the agent’s ability to handle expectation mismatches during deployment.

    Source: §3.2.5 · p. 6

Abstract

from arXiv · show

Graphical User Interface (GUI) Agents, powered by multimodal large language models (MLLMs), have shown great potential for task automation on computing devices such as computers and mobile phones. However, existing agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. We introduce \textit{InfiGUIAgent}, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills using synthesized data to enable native reasoning abilities of the agents. \textit{InfiGUIAgent} achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks. Resources are available at \url{https://github.com/Reallm-Labs/InfiGUIAgent}.

1 Introduction

InfiGUIAgent uses two-stage supervised fine-tuning to build GUI fundamentals and native reasoning skills. The authors report competitive performance on several GUI benchmarks.

1 Introduction

  • Existing MLLM-based GUI agents struggle to use information from prior steps, and text-based GUI representations can lose or duplicate visual information and add overhead.Textual representations also vary in availability and consistency across platforms, hindering practical deployment.Source: §1 · p. 1
  • InfiGUIAgent’s two-stage supervised fine-tuning pipeline targets both fundamental GUI abilities and advanced reasoning.Stage 1 improves GUI understanding and instruction grounding; Stage 2 integrates hierarchical and expectation-reflection reasoning.Source: §1 · pp. 1–2
  • Stage 1 uses vision-language understanding, GUI-specific QA, and tool-use data to improve GUI understanding and instruction grounding.Source: §1 · p. 1
  • Synthesized SFT data adds hierarchical and expectation-reflection reasoning, enabling agents to natively perform complex reasoning.The reasoning skills are integrated into data synthesized by MLLMs from existing trajectories.Source: §1 · pp. 1–2
  • Experiments on several GUI benchmarks show that InfiGUIAgent achieves competitive performance.Source: §1 · p. 2

2 Related Works

Prior work established multimodal language-model architectures and applied them to GUI agents, while this study focuses on a lightweight, simplified agent for GUI tasks.

2.1 Multimodal LLMs

  • MLLMs combine a pretrained language model, modality encoder, and interface, with vision features integrated through adapters, cross-attention, or visual expert modules.Examples include Qwen-VL, GPT-4 Vision (OpenAI, 2023), BLIP-2, and InfiMM (Liu et al., 2024).Source: §2.1 · p. 2

2.2 MLLM-based GUI Agents

3 Method

InfiGUIAgent uses a two-stage supervised fine-tuning strategy: Stage 1 improves GUI understanding and grounding, while Stage 2 develops native reasoning for complicated GUI tasks.

3 Method

  • The method builds InfiGUIAgent through a two-stage supervised fine-tuning strategy.The stages target fundamental GUI abilities first, then native reasoning.Source: §3 · p. 2
  • Stage 1 improves fundamental GUI abilities, particularly understanding and grounding.The approach accounts for the complexity of GUIs.Source: §3 · p. 2
  • Stage 2 improves agents’ native reasoning abilities for complicated GUI tasks.Source: §3 · p. 2

3.1 Stage 1: Training for Fundamental Abilities

Stage 1 addresses MLLMs’ limited GUI understanding and action grounding by training on diverse supervised datasets. Data preprocessing and reference-augmented annotations standardize spatial information and connect GUI elements with text.

3.1 Stage 1: Training for Fundamental Abilities

  • Stage 1 combines datasets for GUI understanding, grounding, GUI question answering, general knowledge, and tool use.The collected sources cover element recognition and layout comprehension, interaction sequences, GUI-specific QA, general multimodal capabilities, and function-calling.Source: §3.1 · pp. 2–3

3.1.1 Data Preprocessing and Standardization

  • Coordinates are normalized to a [0, 1000] screen scale; ambiguous instructions receive templates, and complex or inconsistent responses are reformulated and validated.Points and boxes use JSON coordinates with the origin at the top-left; Qwen2-VL-72B reformulates responses while preserving their meaning.Source: §3.1.1 · p. 3

3.1.2 Reference-Augmented Annotation

  • Reference-augmented annotations link GUI elements and text using point or box coordinates, optional notes, and corresponding natural-language content.Qwen2-VL-72B integrates spatial information with original responses while preserving natural language flow and precise spatial references.Source: §3.1.2 · pp. 3–4

3.2 Stage 2: Training for Native Reasoning

Stage 2 continues fine-tuning from Stage 1 with synthesized trajectory data that teaches agents hierarchical and expectation-reflection reasoning natively, without extra prompting.

3.2 Stage 2: Training for Native Reasoning

  • Stage 2 adds hierarchical planning and expectation-reflection skills to Stage 1 capabilities by fine-tuning on synthesized reasoning data, enabling native reasoning without extra prompting.The two targeted skills support task decomposition and adaptive self-correction from past actions.Source: §3.2 · p. 4

3.2.1 Hierarchical Reasoning

  • Hierarchical reasoning divides work between strategic subtask planning and tactical selection and grounding of concrete GUI actions.The strategic layer orders subtasks; the tactical layer selects GUI operations and adjusts their parameters to match targets.Source: §3.2.1 · p. 4

3.2.2 Expectation-Reflection Reasoning

  • Expectation-reflection reasoning iterates through reasoning, action, expected outcome, and reflection to support action consistency and self-correction.After each action, the agent records an expected outcome; at the next step, it evaluates the result and summarizes its reflection before further reasoning, except at the first step.Source: §3.2.2 · pp. 4–5

3.2.3 Agent-Environment Interface

  • At each step, the agent uses a screenshot observation, task goal, and history window to reason and act in a mobile environment that transitions in response to its actions.The history contains prior observations, reasoning, and actions; the environment’s next state is sampled according to its transition function.Source: §3.2.3 · p. 5

3.2.4 Modular Action Space

  • The action space standardizes names and parameters across datasets, merging similar operations into composable actions for flexible configuration across platforms and scenarios.This preserves a consistent interface while allowing operations to be combined according to task requirements.Source: §3.2.4 · p. 5

3.2.5 Reasoning Process Construction

  • More capable MLLMs generate structured reasoning from existing trajectories, using screenshot descriptions to align observations with steps and construct reflection and strategic planning.Reflection evaluates the previous outcome against its expectation; strategic reasoning summarizes history and the current observation, then plans in alignment with the trajectory’s actual action.Source: §3.2.5 · pp. 5–6
  • Expectations use the current observation, reasoning, and action—not the next state—to preserve robustness to expectation mismatches at deployment.Tactical reasoning combines reflection and strategic output with the trajectory’s action; a parallel task predicts next-state descriptions from the current observation and action.Source: §3.2.5 · p. 6

4 Experiments

InfiGUIAgent-2B outperforms the cited baselines on ScreenSpot grounding and AndroidWorld success rate. Experiments use two-stage training and evaluate AndroidWorld agents on raw screenshots rather than annotated images.

4.1.1 Implementation Details

  • Stage 1 samples 1M examples, while Stage 2 synthesizes 45K trajectory-based samples; training continual-finetunes Qwen2-VL-2B with ZeRO0 across eight A800 80GB GPUs.Source: §4.1.1 · p. 6

4.1.2 Evaluation Benchmarks

  • ScreenSpot evaluates GUI grounding with over 1,200 instructions across iOS, Android, macOS, Windows, and Web, with annotated element types.Source: §4.1.2 · p. 6
  • AndroidWorld provides reward signals for 116 tasks across 20 Android apps; agents respond to raw images rather than Set-of-Marks-annotated images to test pixel-level grounding.The authors argue that reliance on Set-of-Marks annotations may reduce focus on pixel-level grounding.Source: §4.1.2 · p. 6

4.2 Main Results

  • 76.3% accuracy makes InfiGUIAgent-2B the highest-performing model on ScreenSpot, ahead of ShowUI (75.1%) and UGround-7B (73.3%).The comparison spans Mobile, Desktop, and Web platforms and Text and Icon element types.Source: §4.2 · p. 6
  • 0.09 overall success rate puts InfiGUIAgent-2B ahead of ShowUI-2B (0.07), LLaVa-OV-7B (0.00), and Qwen2-VL-72B (0.05) on AndroidWorld.Source: §4.2 · p. 6

5 Conclusion

InfiGUIAgent uses two-stage supervised fine-tuning to strengthen GUI understanding, reasoning, and interaction. Evaluations using raw screenshots indicate applicability to real-world scenarios and strong performance on GUI tasks, surpassing several open-source baselines.

5 Conclusion

  • Comprehensive training datasets and two-stage supervised fine-tuning enhance InfiGUIAgent’s ability to understand, reason, and interact with GUIs.Source: §5 · p. 6
  • InfiGUIAgent performs well on GUI tasks and surpasses several open-source baselines.The supplied conclusion gives no numeric results or specific baseline comparisons.Source: §5 · p. 7, Table 4 · p. 7
  • Evaluation on raw screenshots without additional GUI metadata demonstrates the model’s applicability to real-world scenarios.Source: §5 · p. 7, Table 4 · p. 7

A Cases · A.1 Stage 1: Fundamental Abilities

Stage 1’s fundamental abilities are demonstrated through three cases: GUI understanding, grounding, and question answering.

A.1 Stage 1: Fundamental Abilities

  • The cases demonstrate GUI understanding as a fundamental ability trained in Stage 1.The paper associates GUI understanding with Figure 2.Source: §A.1 · p. 11
  • The cases demonstrate grounding as a fundamental ability trained in Stage 1.The paper associates grounding with Figure 3.Source: §A.1 · p. 11
  • The cases demonstrate question answering as a fundamental ability trained in Stage 1.The paper associates question answering with Figure 4.Source: §A.1 · p. 11

A.2 Stage 2: Native Reasoning

Two representative cases illustrate InfiGUIAgent’s reasoning and interaction process through replying to a message and creating a new contact.

A.2 Stage 2: Native Reasoning

  • Two representative cases demonstrate InfiGUIAgent’s reasoning and interaction process.Source: §A.2 · p. 11
  • To reply to a specific message, the agent identifies and grounds an action on the “Start chat” button to initiate the reply process.Source: §A.2 · p. 11, Figure 5 · p. 13
  • To create a contact, the agent first navigates to Contacts, then identifies and taps “Create new contact,” demonstrating sequential hierarchical reasoning and grounding.Source: §A.2 · p. 11, Figure 6 · p. 14, Figure 7 · p. 14

Related papers

Loading 2501.04575v1…