Source-linked AI summary
InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, Fei Wu
TL;DR
GUI agents often lack robust, adaptive reasoning for complex tasks requiring planning and error recovery. InfiGUI-R1 uses Actor2Reasoner, a two-stage framework that injects spatial reasoning and enhances deliberation with reinforcement learning; across grounding and trajectory benchmarks, it achieves strong performance, including state-of-the-art results among similar-sized models.
Problem
Many GUI agents rely on implicit reasoning or manually designed templates, leaving a need for more robust and adaptive reasoning for complex GUI tasks.
Method
Actor2Reasoner first uses Spatial Reasoning Distillation to establish cross-modal reasoning, then uses reinforcement learning for sub-goal planning and error recovery.
Results
Across diverse grounding and trajectory benchmarks, InfiGUI-R1-3B matches or surpasses larger models and achieves state-of-the-art performance among similar-parameter models.
Takeaways & Limitations
The results support progressively transforming a base MLLM into a more effective deliberative GUI reasoner through spatial reasoning and reinforcement-learning-based deliberation.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have powered Graphical User Interface (GUI) Agents, showing promise in automating tasks on computing devices. Recent works have begun exploring reasoning in GUI tasks with encouraging results. However, many current approaches rely on manually designed reasoning templates, which may result in reasoning that is not sufficiently robust and adaptive for complex GUI environments. Meanwhile, some existing agents continue to operate as Reactive Actors, relying primarily on implicit reasoning that may lack sufficient depth for GUI tasks demanding planning and error recovery. We argue that advancing these agents requires a shift from reactive acting towards acting based on deliberate reasoning. To facilitate this transformation, we introduce InfiGUI-R1, an MLLM-based GUI agent developed through our Actor2Reasoner framework, a reasoning-centric, two-stage training approach designed to progressively evolve agents from Reactive Actors to Deliberative Reasoners. The first stage, Reasoning Injection, focuses on establishing a basic reasoner. We employ Spatial Reasoning Distillation to transfer cross-modal spatial reasoning capabilities from teacher models to MLLMs through trajectories with explicit reasoning steps, enabling models to integrate GUI visual-spatial information with logical reasoning before action generation. The second stage, Deliberation Enhancement, refines the basic reasoner into a deliberative one using Reinforcement Learning. This stage introduces two approaches: Sub-goal Guidance, which rewards models for generating accurate intermediate sub-goals, and Error Recovery Scenario Construction, which creates failure-and-recovery training scenarios from identified prone-to-error steps. Experimental results show InfiGUI-R1 achieves strong performance in GUI grounding and trajectory tasks. Resources at https://github.com/Reallm-Labs/InfiGUI-R1.
1 Introduction
GUI agents promise broad automation, but many remain reactive or depend on manually designed reasoning, limiting robustness for complex tasks. Actor2Reasoner addresses this by injecting spatial reasoning and refining planning and recovery capabilities in two stages.
- Many GUI agents rely on implicit reasoning or manually designed templates that may be insufficiently robust and adaptive for complex environments.
- Deliberative GUI agents explicitly reason between perception and action to plan, decompose goals, understand spatial relationships, and correct mistakes.
- Actor2Reasoner progressively transforms reactive MLLM-based GUI agents into deliberative reasoners through reasoning injection and refinement.
- Stage 1 uses Spatial Reasoning Distillation and supervised fine-tuning on explicit teacher trajectories to establish foundational cross-modal reasoning.
- Stage 2 constructs error-and-recovery scenarios and applies targeted reinforcement learning to develop reflective self-correction.
2 Related Works
Related work advances multimodal GUI agents through visual understanding, specialized architectures, planning, reflection, and increasingly complex evaluation environments. These efforts span general GUI comprehension, platform-specific interaction, grounding, and autonomous adaptation.
- MLLMs combine language models, visual encoders, and modality interfaces to process multimodal inputs for GUI tasks.
- Visual GUI interaction has increasingly replaced structured representations such as HTML, leveraging visual perception alongside language understanding.
- Representative systems improve GUI comprehension, mobile interaction, UI detail capture, or resolution flexibility through specialized designs.
3 Actor2Reasoner
Actor2Reasoner is a two-stage training methodology that progressively changes GUI agents from reactive execution to deliberative reasoning. It first establishes reasoning, then enhances planning and reflection with reinforcement learning.
- The framework targets a transition from reactive behavior to deliberative reasoning in MLLM-based GUI agents.
- Stage 1: Reasoning Injection: Stage 1 uses Spatial Reasoning Distillation and supervised fine-tuning to establish foundational reasoning.
- Stage 2: Deliberation Enhancement: Stage 2 applies reinforcement learning with Sub-goal Guidance and Error Recovery Scenario Construction to enhance planning and reflection.
3.1 Stage 1: Reasoning Injection
Stage 1 identifies steps where reasoning, rather than perception or execution, is the likely bottleneck, then distills explicit spatial reasoning trajectories into the base MLLM. Supervised fine-tuning thereby establishes a reasoner that connects perception, reasoning, and action.
- Stage 1 aims to replace the direct Perception →Action pathway with Perception →Reasoning →Action for GUI tasks requiring precise spatial understanding.
- Spatial Reasoning Distillation trains the student to learn intermediate spatial-logic steps, not only the correct action, from a capable teacher model.
- 3.1.1 Pinpointing Reasoning Bottleneck Samples: The bottleneck criterion compares action prediction from the screenshot and goal alone with prediction after adding the step-specific sub-goal.
- 3.1.1 Pinpointing Reasoning Bottleneck Samples: Reasoning Bottleneck Samples are steps where the base MLLM fails without a sub-goal but succeeds when given the ground-truth sub-goal.
- Teacher trajectories combine compressed accessibility-tree spatial descriptions, action-space information, and the overall goal to produce explicit reasoning and actions.
- Quality-filtered teacher reasoning-action pairs supervise the base MLLM to generate or simulate reasoning before acting.
- 3.1.3 Injecting Reasoning via SFT: After Stage 1, the model has enhanced spatial understanding and a foundational ability to connect perception to action through intermediate reasoning.
3.2 Stage 2: Deliberation Enhancement
Stage 2 uses reinforcement learning with rule-based rewards to refine a foundational reasoner into a deliberative GUI agent. It targets forward-looking planning through Sub-goal Guidance and backward-looking correction through Error Recovery Scenario Construction.
- 3.2 Stage 2: Deliberation Enhancement: Stage 2 applies reinforcement learning with rule-based rewards to improve forward-looking planning and backward-looking reflection and correction.It builds on the foundational Reasoner established in Stage 1.
- Sub-goal Guidance: Sub-goal Guidance rewards accurate intermediate goals extracted from the agent’s reasoning and compared with annotated ground-truth sub-goals.A scoring LLM assigns a semantic-match score from 1 to 10, normalized to a reward in [0, 1].
- Sub-goal Guidance: The sub-goal reward provides planning feedback even when action execution fails, by entering the trajectory reward when action parameters are incorrect.This conditional design is intended to guide task decomposition and intermediate reasoning.
- Error Recovery Scenario Construction: Error Recovery Scenario Construction trains agents on failure states and post-recovery states requiring escape actions followed by the originally correct action.The modified histories preserve the preceding error and escape actions, testing whether the agent can re-evaluate and resume the task.
- Error Recovery Scenario Construction: The constructed recovery samples are added to Stage 2 reinforcement-learning data, rewarding successful escapes and correct subsequent actions.The same comprehensive reward function evaluates outputs in both scenario types.
4 Experiments
The experiments evaluate InfiGUI-R1-3B across grounding and multi-step AndroidControl tasks, comparing it with open-source and proprietary GUI agents. Results show strong grounding and trajectory performance, while reinforcement-learning rewards rise across task types during training.
- Experimental setup: The evaluation covers ScreenSpot, ScreenSpot-Pro, and AndroidControl benchmarks spanning GUI grounding, complex desktop interfaces, and long-horizon Android tasks.ScreenSpot evaluates Mobile, Desktop, and Web grounding; ScreenSpot-Pro targets complex high-resolution desktop applications; AndroidControl measures planning and state tracking in multi-step trajectories.
- Comparison: The experiments compare InfiGUI-R1-3B against a range of open-source and proprietary GUI agents.The reported comparisons are used to assess the effectiveness of the Actor2Reasoner framework.
- ScreenSpot: InfiGUI-R1-3B achieves state-of-the-art ScreenSpot grounding with 87.5% average accuracy across Mobile, Desktop, and Web platforms.It ranks first across text-based and icon-based grounding tasks on all three platforms.
- ScreenSpot-Pro: 35.7 overall average on ScreenSpot-Pro matches UI-TARS-7B and exceeds OS-Atlas-7B at 18.9 and UGround-7B at 16.5.The model records category averages of 28.4 for CAD, 57.0 for Office, and 29.6 for OS, although it does not lead every category.
- AndroidControl: 92.1% and 71.1% Success Rate on AndroidControl-Low and AndroidControl-High surpass UI-TARS-2B’s 89.3% and 68.9%.The model also exceeds Aguvis-72B, which records 84.4% and 66.4% on the corresponding splits.
- Training analysis: Overall and task-specific rewards increase consistently during reinforcement learning for Low-level, High-level, and Grounding tasks.The plotted curves indicate improvement across all three task categories as training progresses.
5 Conclusion
InfiGUI-R1-3B uses Actor2Reasoner to inject and refine reasoning through spatial distillation and reinforcement-learning-based deliberation. Across diverse benchmarks, it matches or surpasses larger models in grounding and performs strongly on long-horizon execution with planning and reflection.
- Conclusion: InfiGUI-R1-3B bridges reactive execution and deliberative reasoning through two-stage Actor2Reasoner training.The stages are Spatial Reasoning Distillation and reinforcement-learning-based Deliberation Enhancement.
- Conclusion: The agent supports sub-goal planning and error recovery while matching or surpassing larger models in grounding accuracy.The conclusion reports these capabilities across diverse benchmarks.
- Conclusion: InfiGUI-R1-3B excels in long-horizon task execution with robust planning and reflection.This outcome is reported alongside its grounding performance across diverse benchmarks.