PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization

Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xavier Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, Shengyu Zhang, Hongxia Yang, Fei Wu

arXiv:2508.05731v2cs.AIcs.CL

TL;DR

GUI grounding requires models to locate interface elements and match them correctly to instructions, but inefficient RLVR exploration bottlenecks semantic alignment. AEPO broadens candidate search with multi-answer generation and guides it using an adaptive reward. InfiGUI-G1 achieves state-of-the-art performance, with analyses attributing its effectiveness to adaptive exploration, efficiency, and learning signals for previously unlearnable samples.

  • Problem

    Inefficient exploration in standard RLVR can trap GUI agents on confident errors, limiting discovery of correct actions and hindering semantic alignment.

    Source: §1 · pp. 1–2

  • Method

    AEPO combines multi-answer generation with an adaptive exploration reward to guide broader search and encourage efficient successful predictions.

    Source: §1 · p. 2

  • Results

    InfiGUI-G1 achieves state-of-the-art performance, with analyses reporting adaptive exploration, efficiency over naive sampling, and learning signals for previously unlearnable samples.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The paper’s analyses attribute InfiGUI-G1’s effectiveness to adaptive exploration, efficiency compared with naive sampling, and learning signals for previously unlearnable samples.

    Source: §5 · p. 9

    Multi-answer generation adds computational overhead, and performance is limited by the backbone MLLM’s visual capabilities.

    Source: §5 · p. 9

Abstract

from arXiv · show

The emergence of Multimodal Large Language Models (MLLMs) has propelled the development of autonomous agents that operate on Graphical User Interfaces (GUIs) using pure visual input. A fundamental challenge is robustly grounding natural language instructions. This requires a precise spatial alignment, which accurately locates the coordinates of each element, and, more critically, a correct semantic alignment, which matches the instructions to the functionally appropriate UI element. Although Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be effective at improving spatial alignment for these MLLMs, we find that inefficient exploration bottlenecks semantic alignment, which prevent models from learning difficult semantic associations. To address this exploration problem, we present Adaptive Exploration Policy Optimization (AEPO), a new policy optimization framework. AEPO employs a multi-answer generation strategy to enforce broader exploration, which is then guided by a theoretically grounded Adaptive Exploration Reward (AER) function derived from first principles of efficiency eta=U/C. Our AEPO-trained models, InfiGUI-G1-3B and InfiGUI-G1-7B, establish new state-of-the-art results across multiple challenging GUI grounding benchmarks, achieving significant relative improvements of up to 9.0% against the naive RLVR baseline on benchmarks designed to test generalization and semantic understanding. Resources are available at https://github.com/InfiXAI/InfiGUI-G1.

1 Introduction

GUI grounding requires both locating an element precisely and identifying the element that matches an instruction. The paper targets inefficient RLVR exploration as a bottleneck to semantic alignment and introduces AEPO to address it.

1 Introduction

  • GUI grounding has two dimensions: spatial alignment locates an element precisely, while semantic alignment identifies the appropriate element for an instruction.Reliable performance in complex scenarios depends on both dimensions, with semantic alignment particularly critical.Source: §1 · p. 1
  • RLVR improves spatial alignment, but inefficient exploration can trap models on confident errors and prevent discovery of low-probability correct actions.The paper illustrates this with repeatedly selecting a generic Camera icon instead of the correct Google Lens icon.Source: §1 · p. 2
  • AEPO combines multi-answer generation with an adaptive exploration reward to broaden candidate search and guide exploration during failure toward convergence on success.A quality-of-exploration penalty discourages inefficient, near-collinear outputs to promote semantic diversity.Source: §1 · p. 2
  • InfiGUI-G1 offers 3B and 7B variants, and evaluation across diverse benchmarks establishes state-of-the-art GUI-grounding performance.The paper presents the models as a contribution built on the AEPO framework.Source: §1 · p. 2

2 Related Work

Prior work developed visually driven GUI agents, grounding techniques, and reinforcement-learning approaches for multimodal models. The paper distinguishes its approach by replacing single-answer generation with broader search guided by an adaptive reward.

2.1 MLLM-based GUI Agents and Grounding

2.2 Reinforcement Learning in MLLM

  • RL has been applied to MLLM reasoning and GUI grounding through rule-based rewards, unified action spaces, self-supervision, self-evolution, and zoom-in grounding.The section presents RL as an effective and efficient approach for MLLM training and GUI-grounding performance.Source: §2.2 · p. 2
  • The paper identifies single-answer generation as a limitation of prior methods and contrasts it with multi-answer search guided by an adaptive reward.The broader search is intended to provide richer learning signals.Source: §2.2 · pp. 2–3

3 Methodology

AEPO formulates GUI grounding as policy optimization and extends it with multi-answer generation, an adaptive reward, and a penalty for low-quality exploration. Its total training reward combines output-format validity with accuracy feedback.

3.1 Problem Formulation

  • GUI grounding is formulated as training an MLLM policy to generate an action that corresponds to a screenshot-and-instruction context.The context contains a GUI screenshot and natural-language instruction; the action is a coordinate point, evaluated against the target bounding box.Source: §3.1 · p. 3
  • The objective maximizes expected reward over the data distribution, and autoregressive coordinate generation supports policy-gradient optimization.The paper lists PPO, GRPO, and RLOO as suitable algorithms.Source: §3.1 · p. 3

3.2 Adaptive Exploration Policy Optimization

  • AEPO generates a set of candidate points in one forward pass, broadening search beyond the policy’s most confident prediction.This is intended to increase the chance of sampling a correct action from the policy’s tail, particularly for semantically challenging samples.Source: §3.2 · p. 3
  • AER derives its reward from efficiency η = U/C, with utility determined by whether any candidate falls within the target box.Success has utility +1 and failure −1; the reward structure encourages wider exploration after failure and efficient, confident predictions after success.Source: §3.2 · pp. 3–4
  • The cost model combines candidate-generation effort with verification effort, using candidate-set size N and the rank k of the first correct point.On failure, all N candidates must be checked; on success, verification cost is the first correct point’s rank.Source: §3.2 · p. 3
  • A collinear-set penalty overrides the accuracy reward with −1 to discourage linear scanning and promote spatially diverse exploration.Collinearity is checked using the area of triangles formed by triples of generated points.Source: §3.2 · p. 4

3.3 Overall Training Objective

  • The total reward combines format and accuracy rewards; format validity is a prerequisite for evaluating accuracy, and the resulting reward guides policy updates.The training loop checks format, extracts candidate points, evaluates collinearity and first-correct rank, then computes an advantage estimate.Source: §3.3 · p. 4

4 Experiments

InfiGUI-G1 performs strongly across five GUI-grounding benchmarks, while ablations and targeted analyses examine how its exploration components contribute and where gains are most pronounced.

4.1 Experimental Setup

  • The evaluation spans five benchmarks covering varied task complexity, professional high-resolution screens, unseen applications, platforms, and implicit instructions.Accuracy counts a point inside the ground-truth box as correct; the study also reports exploration success rate.Source: §4.1 · p. 7
  • The models use Qwen2.5-VL 3B/7B backbones and RLOO; training uses about 44k GUI samples, excluding examples answered correctly in all eight initial responses.Training ran for three epochs on 16 H800 GPUs; prompts requested reasoning before the final answer.Source: §4.1 · p. 7

4.2 Main Results

  • InfiGUI-G1 establishes new open-source state-of-the-art performance in both model-size categories and is competitive with or superior to several larger proprietary models.Results are reported across five benchmarks.Source: Table 1 · p. 5, Table 2 · p. 5, Table 3 · p. 6
  • InfiGUI-G1 consistently and substantially outperforms the Naive RLVR baseline across all benchmarks.The controlled baseline uses the same training data and optimized hyperparameters.Source: §4.1 · p. 7, §4.2 · p. 8

4.3 Ablation Studies

  • Removing multi-answer generation reduced performance; removing AER caused a further decline, while removing k impaired ranking and removing the collinear penalty encouraged low-quality answer proliferation.Without the collinear penalty, accuracy plummeted despite a high number of generated answers.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 Analysis of AEPO’s Effectiveness

  • UI-Vision's 26.1% Acc corresponded to 2.1 answers, versus ScreenSpot-V2's 93.5% Acc and 1.4 answers, suggesting exploration adapts to task difficulty.These are reported results for the 7B model.Source: §4.4 · p. 8
  • On ScreenSpot-Pro, InfiGUI-G1's single-pass exploration success rate exceeded Naive RLVR's pass@4 despite generating about two answers per instance.The comparison concerns finding at least one correct answer, not only the first answer's accuracy.Source: §4.4 · p. 8, Table 7 · p. 9
  • The 7B model's relative improvement in Accuracy on ScreenSpot-Pro 'hard' samples exceeded 60%.Samples were labeled hard when the base model failed all 16 stochastic responses.Source: §4.4 · p. 9, Table 8 · p. 9

5 Conclusion

The conclusion identifies inefficient exploration in standard RLVR as a bottleneck to semantic alignment and presents AEPO, which combines multi-answer generation with AER. InfiGUI-G1 achieves state-of-the-art performance, while the approach has computational overhead and a performance ceiling imposed by backbone visual capabilities.

5 Conclusion

  • InfiGUI-G1 achieves state-of-the-art performance, with analyses linking its effectiveness to adaptive exploration, efficiency over naive sampling, and learning signals for previously “unlearnable” samples.The conclusion also identifies inefficient exploration in standard RLVR as a key bottleneck to semantic alignment.Source: §5 · p. 9
  • AEPO combines multi-answer generation with a theoretically grounded AER function to enable effective exploration.The framework is proposed to address inefficient exploration in standard RLVR.Source: §5 · p. 9
  • The work identifies inefficient exploration in standard RLVR as a key bottleneck to semantic alignment in MLLM-based GUI agents.Source: §5 · p. 9
  • Multi-answer generation adds computational overhead, and the backbone MLLM’s visual capabilities impose a performance ceiling.The conclusion suggests more efficient sampling and more advanced visual encoders as future directions.Source: §5 · p. 9

Related papers

Loading 2508.05731v2…