Source-linked AI summary
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, Fei Wu
TL;DR
Existing MLLM-based GUI agents struggle with multi-step reasoning and reliance on textual GUI annotations. InfiGUIAgent addresses these limitations through two-stage supervised fine-tuning with synthesized hierarchical and expectation-reflection reasoning, and achieves competitive performance on several GUI benchmarks using raw screenshots without additional GUI metadata.
Problem
MLLM-based GUI agents have limited multi-step reasoning and often rely on textual GUI representations that can lose information, add overhead, and vary across platforms.
Method
InfiGUIAgent uses two-stage supervised fine-tuning: Stage 1 develops fundamental GUI understanding and grounding, while Stage 2 synthesizes hierarchical and expectation-reflection reasoning into training data.
Results
InfiGUIAgent achieves competitive performance across several GUI benchmarks, including 76.3% accuracy on ScreenSpot and a 0.09 overall success rate on AndroidWorld.
Takeaways & Limitations
Evaluation on raw screenshots without additional GUI metadata demonstrates the model’s applicability to real-world GUI scenarios and performance beyond several open-source baselines.
Takeaways & Limitations
Expectation generation deliberately excludes the next state, accepting less accurate transition modeling to preserve the agent’s ability to handle expectation mismatches during deployment.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) Agents, powered by multimodal large language models (MLLMs), have shown great potential for task automation on computing devices such as computers and mobile phones. However, existing agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. We introduce \textit{InfiGUIAgent}, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills using synthesized data to enable native reasoning abilities of the agents. \textit{InfiGUIAgent} achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks. Resources are available at \url{https://github.com/Reallm-Labs/InfiGUIAgent}.
1 Introduction
MLLM-based GUI agents offer strong visual interaction capabilities, but existing systems struggle with multi-step reasoning and dependence on textual GUI representations. InfiGUIAgent addresses these issues through two-stage supervised fine-tuning and synthesized native reasoning data.
- Challenges: Existing GUI agents can support complex automation, but many MLLM-based systems struggle to leverage information from previous steps.This limitation can lead to repetitive errors during task execution.
- Challenges: Reliance on accessibility trees or Set-of-Marks can cause information loss, redundancy, computational overhead, and platform-dependent deployment constraints.These textual representations vary in availability and consistency across platforms.
- Approach: InfiGUIAgent uses a two-stage supervised fine-tuning pipeline to improve fundamental GUI abilities and native advanced reasoning.Stage 1 targets GUI understanding and instruction grounding, while Stage 2 incorporates synthesized reasoning data.
- Approach: The synthesized Stage 2 data integrates hierarchical reasoning and expectation-reflection reasoning for complex GUI interaction.These skills are designed to support native reasoning rather than requiring extra prompting.
- Results: InfiGUIAgent achieves competitive performance on several GUI benchmarks.The authors evaluate the supervised-fine-tuned model across multiple GUI benchmarks.
2 Related Works
Research on GUI agents has progressed from language- and vision-based systems toward MLLM-based agents that directly process visual interfaces. Prior work explores general GUI understanding, autonomous app interaction, and high-resolution visual encoding.
- MLLM foundations: MLLMs provide visual understanding and multimodal reasoning capabilities that support sophisticated GUI agents.Their ability to process images and semantic context motivates their use as GUI-agent foundations.
- GUI agent development: LLMs have been used to perceive GUIs through HTML code, while visual interaction can provide superior performance for GUI tasks.This motivates the development of MLLM-based GUI agents.
- Representative approaches: Prior systems improve GUI capabilities through general understanding fine-tuning, autonomous app interaction, and high-resolution vision encoders.Examples include ILuvUI, AppAgent, and CogAgent.
3 Method
InfiGUIAgent is built with a two-stage supervised fine-tuning strategy: Stage 1 develops fundamental GUI understanding and grounding, while Stage 2 develops native reasoning for complicated GUI tasks.
- Stage 1: Stage 1 improves fundamental abilities such as GUI understanding and action grounding using diverse GUI-related data.The strategy is designed to address the complexity of GUI formats and interactions.
- Stage 2: Stage 2 improves agents’ native reasoning abilities for handling complicated GUI tasks.The method moves beyond foundational understanding and grounding to advanced reasoning.
3.1 Stage 1: Training for Fundamental Abilities
Stage 1 assembles and standardizes diverse datasets to strengthen GUI understanding, grounding, question answering, general knowledge, and tool usage. It also uses reference-augmented annotations to connect textual responses with precise GUI locations.
- Data composition: Stage 1 collects diverse visual-language and GUI datasets to improve fundamental understanding and action grounding.The data addresses the complexity of HTML, high-resolution interfaces, small icons, and dense text.
- Data composition: The Stage 1 datasets cover GUI understanding, grounding, question answering, general knowledge, and tool usage.GUI understanding includes element recognition and layout comprehension, while grounding covers interaction sequences and operation patterns.
- Training design: Stage 1 uses a two-stage training design in which fundamental abilities precede native advanced reasoning.The training overview distinguishes Stage 1 fundamental abilities from Stage 2 native advanced reasoning.
- Training design: The training example pairs a task such as setting an alarm with an expected action outcome and success state.The figure materials illustrate this interaction-oriented training structure.
- Annotation and preprocessing: Dataset formats are standardized, and reference-augmented annotations link GUI elements bidirectionally with textual responses.The format supports precise spatial references while maintaining natural-language responses.
- Annotation and preprocessing: Coordinates are normalized to the [0, 1000] scale, with points and boxes represented in JSON using a top-left origin.The x-axis extends rightward, the y-axis downward, and the bottom-right corner is [1000, 1000].
- Annotation and preprocessing: Instruction templates clarify ambiguous commands, while response refinement reformulates complex or inconsistent answers and validates them for accuracy.Response refinement uses Qwen2-VL-72B while preserving semantic content.
3.2 Stage 2: Training for Native Reasoning
Stage 2 trains InfiGUIAgent with native hierarchical and expectation-reflection reasoning for complex GUI interaction. Synthesized trajectory data links high-level planning, concrete action selection, expected outcomes, and reflection-based correction.
- Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning into training data so agents can use these skills natively without extra prompting.
- Hierarchical reasoning: Hierarchical reasoning separates strategic task decomposition and sub-goal planning from tactical GUI action selection and grounding.
- Expectation-reflection reasoning: Expectation-reflection reasoning cycles through reasoning, action, expected outcome generation, and reflection on whether the outcome was achieved.
- Agent-environment interface: The agent interacts with a mobile environment by observing screenshots, using a history window, generating reasoning, and predicting actions through tool calls and responses.
- Reasoning-data construction: Training data construction uses screenshot descriptions, reflections, strategic and tactical reasoning, and expectations generated from current observations, reasoning, and actions.
- Expectation design: Expectations deliberately exclude the next state, while an auxiliary next-state prediction task separately models transition dynamics.
4 Experiments
Experiments evaluate InfiGUIAgent on GUI understanding and interaction benchmarks using large-scale supervised fine-tuning data. The model achieves strong results on ScreenSpot and AndroidWorld relative to open-source baselines.
- Training setup: Stage 1 uses 1M samples, while Stage 2 synthesizes 45K trajectory-based samples for continued supervised fine-tuning.
- Benchmarks: ScreenSpot contains over 1,200 instructions spanning iOS, Android, macOS, Windows, and Web environments with annotated element types.
- AndroidWorld: AndroidWorld provides reward signals for 116 programmatic tasks across 20 real-world Android apps, and experiments use raw images rather than annotated images.
- ScreenSpot: 76.3% accuracy on ScreenSpot makes InfiGUIAgent-2B the highest-performing model, ahead of ShowUI at 75.1% and UGround-7B at 73.3%.
- AndroidWorld: 0.09 overall success rate on AndroidWorld exceeds ShowUI-2B at 0.07, LLaVa-OV-7B at 0.00, and Qwen2-VL-72B at 0.05.
5 Conclusion
InfiGUIAgent combines two-stage supervised fine-tuning to improve GUI understanding, reasoning, and interaction. Evaluations using raw screenshots show strong GUI-task performance and results surpassing several open-source baselines.
- InfiGUIAgent is an MLLM-based GUI agent trained with comprehensive two-stage supervised fine-tuning datasets.
- Table 4 reports performance across Mobile, Desktop, and Web platforms using raw screenshot information.
- Table 5 reports performance comparisons on AndroidWorld.
- The evaluation uses raw screenshots without additional GUI metadata to assess applicability to real-world scenarios.
A.1 Stage 1: Fundamental Abilities
Stage 1 trains the agent’s fundamental GUI abilities before advanced reasoning. The paper demonstrates these abilities through GUI understanding, grounding, and question-answering cases.
- Stage 1 fundamental abilities are demonstrated through three cases: GUI Understanding, Grounding, and Question Answering.
A.2 Stage 2: Native Reasoning
Stage 2 demonstrates InfiGUIAgent’s native reasoning and interaction process through representative GUI tasks, including replying to messages and creating contacts. These cases combine reasoning, grounding, sequential interaction, and reflection on whether previous actions achieved their expectations.
- Stage 2 presents representative cases demonstrating InfiGUIAgent’s reasoning and interaction process.
- Reply to a Message: The reply-to-message case requires identifying the “Start chat” button and grounding the action that initiates the reply process.
- Creating a New Contact: Creating a new contact involves navigating to “Contacts” and then selecting “Create new contact” in sequential steps.The first action corresponds to Step K, while the following action corresponds to Step K+1.
- Creating a New Contact: The sequential contact-creation steps highlight hierarchical reasoning and grounding abilities.
- Reflection: Reflection compares expected and observed screens, identifying both mismatches and successful completion of prior actions.Examples include expecting the Simple SMS Messenger or Contacts app but observing another screen, and confirming successful navigation to Contacts.