PaperlayerView exact revision on arXiv

Source-linked AI summary

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang

arXiv:2604.06995v2cs.AI

TL;DR

Current GUI reasoning methods struggle to interpret complex screens and understand UI elements comprehensively. UILoop uses a cyclic screen–UI elements–action process, achieving state-of-the-art UI comprehension and superior GUI reasoning results.

  • Problem

    Current GUI agents struggle with complex screen layouts and diverse UI elements, while direct screen-to-action decisions lack interpretability and comprehensive UI-element understanding.

    Source: §1 · p. 1

  • Method

    UILoop reframes GUI reasoning as a cyclic screen–UI elements–action process and uses UI Element-Driven Reinforcement Fine-Tuning to teach element localization, semantic functions, and practical usage.

    Source: §1 · p. 2, §4 · p. 3, §6 · p. 8

  • Results

    UILoop achieves state-of-the-art performance in UI comprehension and superior results in GUI reasoning tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The paper concludes that comprehensive UI understanding significantly enhances GUI agent reasoning.

    Source: §6 · p. 8

    UILoop focuses on fine-grained UI elements and does not consider screen layouts at coarser granularities.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding. Current methods typically rely on direct screen-based decision-making, which lacks interpretability and overlooks a comprehensive understanding of UI elements, ultimately leading to task failure. To enhance the understanding and interaction with UIs, we propose an innovative GUI reasoning paradigm called UI-in-the-Loop (UILoop). Our approach treats the GUI reasoning task as a cyclic Screen-UI elements-Action process. By enabling Multimodal Large Language Models (MLLMs) to explicitly learn the localization, semantic functions, and practical usage of key UI elements, UILoop achieves precise element discovery and performs interpretable reasoning. Furthermore, we introduce a more challenging UI Comprehension task centered on UI elements with three evaluation metrics. Correspondingly, we contribute a benchmark of 26K samples (UI Comprehension-Bench) to comprehensively evaluate existing methods' mastery of UI elements. Extensive experiments demonstrate that UILoop achieves state-of-the-art UI understanding performance while yielding superior results in GUI reasoning tasks.

1 Introduction

The introduction argues that direct screen-to-action GUI reasoning obscures UI-element understanding, contributing to task failures. It presents UILoop’s cyclic screen–UI elements–action paradigm and a benchmark for evaluating UI comprehension.

1 Introduction

  • Direct screen-to-action methods lack interpretability and comprehensive UI-element understanding, making accurate localization and understanding of element semantics and functions difficult and contributing to task failure.The introduction attributes this limitation to generating actions directly from screen inputs.Source: §1 · p. 1
  • Existing models average below 0.1 on UI localization, semantic-function description, and practical-usage; correct UI information improves reasoning, while misleading information increases failures.The reported effect spans zero-shot MLLMs, GUI experts, and models of varying scales.Source: §1 · p. 1, Figure 1 · p. 2
  • UILoop replaces direct screen-to-action decisions with a cyclic screen–UI elements–action process, using UI elements as a bridge to action.UI-Element-Driven Reinforcement Fine-Tuning teaches models to locate key elements, infer their semantic functions, and use them practically, supporting precise parsing and interpretable reasoning.Source: §1 · p. 2
  • The paper introduces a more challenging UI Comprehension task and UI Comprehension-Bench, a 26K-sample benchmark for assessing UI-element capabilities.Its three metrics are UI Locate, Lingualize, and Leverage.Source: §1 · p. 2
  • Experiments report state-of-the-art UI comprehension and superior GUI reasoning results for UILoop.The introduction states these findings without giving numerical results here.Source: §1 · p. 2

2 Related Work

Prior GUI agents embed UI comprehension in action prediction, while UI-element methods emphasize localization but omit semantics and practical use. UILoop explicitly teaches UI-element mastery within a Screen-UI Element-Action process.

Screen-to-Action GUI Agent

UI Elements-Enhanced GUI Agent

  • UI-element methods improve localization but neglect semantic functions and practical usage, which can cause errors such as clicking a scrollbar instead of dragging it.Examples include SeeClick (Cheng et al., 2024), GUI-explorer (Xie et al., 2025), ScreenSpot-Pro (Li et al., 2025a), and UGround (Gou et al., 2025).Source: §2 · p. 3
  • UILoop addresses these gaps with a Screen-UI Element-Action paradigm that explicitly teaches models to master UI elements and achieves superior GUI reasoning performance.Source: §2 · p. 3

3 Preliminary

The section formalizes GUI reasoning as iterative screen-to-action decisions mediated by useful UI elements, and identifies GRPO as the method used to optimize the policy.

GUI Reasoning

  • GUI reasoning is framed as a multi-turn process in which the agent observes each screen and outputs an action to complete the user instruction.The policy πθ is trained to produce the correct action at each iteration.Source: §3 · p. 3
  • The agent analyzes useful UI elements by their location, semantic and functional description, and usage, forming a Screen–UI Elements–Action loop.Usage includes an action and its associated box.Source: §3 · p. 3

GRPO

  • The model is optimized with GRPO (Guo et al., 2025), which estimates each response’s relative advantage within a group for the same prompt without a value function.The objective also includes a clipping range and a KL-divergence penalty between the current and reference policies.Source: §3 · p. 3

4 UI-in-the-Loop Framework

UILoop frames GUI reasoning as a Screen-UI Element-Action process, pairing a UI Comprehension benchmark with reinforcement fine-tuning to improve explicit understanding and use of UI elements.

4 UI-in-the-Loop Framework

  • UILoop combines UI Comprehension data synthesis with UI Element-Driven Reinforcement Fine-Tuning to address the Screen-to-Action limitation.The framework explicitly trains models to localize UI elements, understand their semantic functions, and use them in decisions.Source: §4 · p. 3, §4.2 · p. 5

4.1 Scaling Data for UI Comprehension

  • The benchmark augments existing GUI reasoning data with key UI-element locations, semantic functions, and practical usage to support instruction completion.The process marks identifiable elements, selects those beneficial for the instruction, and adds fine-grained UI information to produce data in (I, S, U*, a) format.Source: §4.1 · p. 4
  • UI Comprehension-Bench adds 26K samples with ground-truth UI elements and an interpretable Screen-UI Element-Action chain, evaluated by Locate, Lingualize, and Leverage.Its Overall score is defined as Locate * Lingualize * Leverage.Source: §4.1 · pp. 4–5
  • The benchmark contains 1,576,068 UI elements but only 57,332 ground-truth elements (<4%); cases with one ground-truth element make up 26.5% of samples.Text coverage exceeds 90% for most action types, while only a few actions fall below 80%.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 UI Element-Driven Reinforcement Fine-Tuning

  • UI Element-Driven Reinforcement Fine-Tuning uses format, location, lingualization, and leverage rewards to train models to find, understand, and use UI elements.The location, lingualization, and leverage rewards align with the benchmark’s corresponding evaluation metrics.Source: §4.2 · pp. 5–6

5 Experiments

Experiments evaluate UILoop on GUI reasoning and UI comprehension, finding advantages over Screen-to-Action methods and gains from UI-element rewards and training. A case study illustrates how semantic analysis supports more interpretable action selection.

5.1 Experiment Setting

  • Experiments use Android Control-High and ScreenSpot-Pro for GUI reasoning, UI Comprehension-Bench for element comprehension, and zero-shot MLLMs and Screen-to-Action models as baselines.Metrics include Type, GR, and SR for GUI reasoning, and Locate, Lingualization, and Leverage for UI comprehension; the base models are Qwen2.5-VL-3B and 7B, trained on the benchmark’s training set.Source: §5.1 · p. 6

5.2 Main Result

  • On Android Control-High, UILoop-7B exceeds OS-Atlas-7B, OS-Atlas-Pro-7B, and GUI-OWL-7B by 46.5%, 58.0%, and 38.8% in SR, respectively, and surpasses Screen-to-Action models on both datasets.On ScreenSpot-Pro, the passage also reports that UILoop’s 3B and 7B models outperform similarly sized Qwen2.5-VL and GUI-R1, but its figures are fragmented and do not clearly preserve the model-to-value pairings.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 Ablation Study

  • Locate and Lingualize rewards add 7.9% and 8.6% GR for 3B and 7B, respectively, while Lingualize further adds 11.1% and 13.7% SR.Leverage Reward improves all metrics; the reported progression compares Direct SFT, Direct RFT with Leverage only, Locate plus Leverage, and full UILoop.Source: §5.3 · p. 7, Figure 5 · p. 7

5.4 Impact of UI Elements

  • UILoop Training improves Qwen2.5-VL-3B and 7B by 31.6% and 22.8%, versus 16.0% and 18.7% for key-UI context alone.For GUI-Owl-7B and OS-Atlas-Pro-7B, the corresponding improvements are 17.8% and 29.0%, versus 12.8% and 20.7% for context alone; false UI information impairs GUI reasoning.Source: §5.4 · pp. 7–8, Table 3 · p. 8

5.5 Experiment of UI Comprehension-Bench

  • On UI Comprehension-Bench, Screen-to-Action models score below 10% across Locate, Lingualize, and Leverage, while UILoop-7B achieves a 26.1 score.UILoop also boosts GUI-Owl-7B and OS-Atlas-Pro-7B overall scores by 18.4 and 9.0, respectively.Source: §5.5 · p. 8

5.6 Case Study

  • UILoop correctly identifies “Medium 10” and the “ADD” button by analyzing UI semantics, while Screen-to-Action methods click the misleading “P. PAN 7” option.The case illustrates UILoop’s explicit reasoning from screen to key UI elements to action.Source: §5.6 · p. 8

6 Conclusion

UILoop reframes GUI reasoning as a cyclic Screen–UI Elements–Action process and uses UI Element-Driven Reinforcement Fine-Tuning to improve UI comprehension and interpretability. The paper introduces a three-metric UI Comprehension task and a 26K-sample benchmark, reporting state-of-the-art UI comprehension and superior GUI reasoning results.

6 Conclusion

  • UILoop reframes GUI reasoning as a cyclic Screen–UI Elements–Action process and uses UI Element-Driven Reinforcement Fine-Tuning to improve element comprehension and interpretability.The paradigm contrasts with conventional Screen-to-Action reasoning.Source: §6 · p. 8
  • The paper introduces a UI Comprehension task with UI Locate, Lingualize, and Leverage metrics, plus UI Comprehension-Bench, a 26K-sample benchmark for comprehensive UI assessment.Source: §6 · p. 8
  • UILoop achieves state-of-the-art UI comprehension performance and delivers superior results in GUI reasoning tasks.The supplied passages report these outcomes but provide no numeric values.Source: §6 · p. 9, Table 4 · p. 9

Limitations

The paper identifies two limitations: UILoop focuses on fine-grained UI elements rather than layouts at multiple granularities, and experiments focus predominantly on Qwen2.5-VL.

Limitations

  • UILoop does not account for screen layouts at different granularities, including coarse-grained layouts composed of multiple fine-grained UI elements.Future work will investigate how UI elements at varying granularities affect GUI reasoning.Source: Limitations · p. 9
  • Experiments predominantly evaluate UILoop with Qwen2.5-VL, leaving its performance across a broader range of MLLMs unexplored.The authors plan to study UILoop across more MLLMs.Source: Limitations · p. 9

Ethics Statement

The authors describe how they assembled and reviewed UI Comprehension-Bench data, screened screenshots for personal or sensitive information, and plan to release the code, models, and datasets upon publication.

Ethics Statement

  • UI Comprehension-Bench combines six existing GUI datasets with externally collected webpage, mobile-app, and OS data, which the authors manually verified and filtered for quality and compliance.The source datasets are Android Control, OmniAct, GUI-Act, ScreenSpot, ScreenSpot-Pro, and OS-Atlas.Source: Ethics Statement · p. 9
  • All UI screenshots were reviewed to exclude or anonymize personal or sensitive information.Source: Ethics Statement · p. 9
  • The authors commit to releasing all code, models, and datasets upon publication to support transparency, reproducibility, and further research.Source: Ethics Statement · p. 9

A Details of UI Comprehension-Bench Collection

UI Comprehension-Bench is assembled through source-data collection, key-element identification and parsing, and human verification. The resulting benchmark contains 26,207 samples, with isolated training and test sets.

Source Data Collection

  • Sources include browser-captured webpages, Android applications and operating systems, and training data from six existing GUI datasets.BrowserGym (Chezelles et al., 2024) and Playwright are used for webpage capture and simulated actions; Droid-Bot3 (Wen et al., 2023) supports capture and action execution on Android applications and operating systems. Samples are normalized as instruction, screen, and action.Source: A Details of UI Comprehension-Bench… · p. 13

Key UI Element Identification and Parsing

  • OmniParser V2 annotates screen elements, and GPT-4o selects instruction-relevant elements and generates their semantic functions, practical usage, and reasoning processes.The selection model receives the instruction, screen, element coordinates, and action; the data format is expanded to include key UI element information.Source: A Details of UI Comprehension-Bench… · p. 13

Human Verification

  • Manual screening excludes samples with incorrect instructions, erroneous answers, or misidentified key UI elements.The verification process produces the curated benchmark.Source: A Details of UI Comprehension-Bench… · p. 13
  • 26,207 samples make up the benchmark: 3,471 training samples and 22,736 test samples, with complete data isolation between the sets.The training set is selected from Android Control, OmniAct, GUI-Act, ScreenSpot, ScreenSpot-Pro, and OS-Atlas.Source: A Details of UI Comprehension-Bench… · p. 13

B Demonstrations of UI Comprehension-Bench · C Prompt Details

The benchmark examples show how UI Comprehension-Bench extends Screen-to-Action data with key UI elements and reasoning chains. The prompt details specify task-dependent action spaces and require explicit UI-element descriptions and reasoning alongside actions.

B Demonstrations of UI Comprehension-Bench

  • Existing GUI-reasoning datasets follow Screen-to-Action and lack fine-grained UI-element location, semantic functionality, and practical-usage information.The section compares UI Comprehension-Bench with existing datasets in Table 5.Source: B Demonstrations of UI… · p. 13, Table 5 · p. 15
  • UI Comprehension-Bench adds Key UI Elements and Reasoning_Chains to Screen-to-Action fields, representing UI-element localization, semantic function, and practical usage.The examples cover open_app, type, and click actions.Source: B Demonstrations of UI… · p. 13

C Prompt Details

  • The prompt specifies task-dependent action spaces, following reasoning-prompt principles from Li et al. (2025b,c) and Liu et al. (2026).For GUI reasoning, the action choices include wait, long_press, click, press_back, type, open_app, and scroll, with corresponding point and input-text fields.Source: C Prompt Details · pp. 13, 16
  • For GUI grounding, the prompt requires an action output with a point and input text, and a note describing each element’s location, semantics, and functionality.The example answer uses a click action and a coordinate pair.Source: C Prompt Details · p. 16
  • The GUI-reasoning prompt asks UILoop to identify useful UI elements, reason from them, and return the action in <ui>, <think>, and <answer> tags.It requires each UI element’s location, semantics, and functionality; certain actions require specific input text.Source: C Prompt Details · p. 16
  • The selection-model prompt supplies instruction, action history, ground-truth action details, and UI information, then requests concise element descriptions and a reasoning chain of at most five sentences.It specifies <ui> tags for element descriptions and <think> tags for reasoning.Source: C Prompt Details · p. 17

D Error Analysis

A manual comparison on 100 Android Control-High test instances found fewer UI-element errors with the Screen-UI Elements-Action method than with the three Screen-to-Action methods.

D Error Analysis

  • UILoop recorded 1 Locate Error, 8 Lingualize Errors, and 31 Leverage Errors, substantially fewer than UI-R1-3B, GUI-R1-7B, and GUI-OWL-7B.The comparison is reported in Fig. 10.Source: D Error Analysis · p. 17
  • The analysis classified UI-element failures as Locate Error, Lingualize Error, and Leverage Error.Source: D Error Analysis · p. 17
  • Researchers manually tallied errors in a random sample of 100 instances from the Android Control-High test set.Source: D Error Analysis · p. 17

Related papers

Loading 2604.06995v2…