PaperlayerView exact revision on arXiv

Source-linked AI summary

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang

arXiv:2604.06995v2cs.AI

TL;DR

GUI reasoning remains limited by screen-to-action methods that lack interpretability and comprehensive UI-element understanding. The paper proposes UILoop, a cyclic Screen–UI Elements–Action paradigm with reinforcement fine-tuning and a 26K UI Comprehension-Bench, and reports state-of-the-art UI comprehension with superior GUI reasoning results.

  • Problem

    Screen-to-action GUI agents struggle to interpret complex interfaces because direct action generation lacks interpretability and comprehensive understanding of UI elements.

  • Method

    UILoop cyclically connects screens, UI elements, and actions while reinforcement fine-tuning teaches localization, semantic functions, and practical usage.

  • Results

    UILoop achieves state-of-the-art UI comprehension performance and superior results in GUI reasoning tasks.

  • Takeaways & Limitations

    UI Comprehension-Bench provides a 26K-sample evaluation of UI localization, semantic understanding, and practical usage through three dedicated metrics.

  • Takeaways & Limitations

    UILoop primarily addresses fine-grained UI elements and does not consider UI layouts at different granularities, including coarse-grained layouts.

Abstract

from arXiv · show

Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding. Current methods typically rely on direct screen-based decision-making, which lacks interpretability and overlooks a comprehensive understanding of UI elements, ultimately leading to task failure. To enhance the understanding and interaction with UIs, we propose an innovative GUI reasoning paradigm called UI-in-the-Loop (UILoop). Our approach treats the GUI reasoning task as a cyclic Screen-UI elements-Action process. By enabling Multimodal Large Language Models (MLLMs) to explicitly learn the localization, semantic functions, and practical usage of key UI elements, UILoop achieves precise element discovery and performs interpretable reasoning. Furthermore, we introduce a more challenging UI Comprehension task centered on UI elements with three evaluation metrics. Correspondingly, we contribute a benchmark of 26K samples (UI Comprehension-Bench) to comprehensively evaluate existing methods' mastery of UI elements. Extensive experiments demonstrate that UILoop achieves state-of-the-art UI understanding performance while yielding superior results in GUI reasoning tasks.

1 Introduction

Existing GUI agents often make opaque screen-to-action decisions without comprehensively understanding UI elements, limiting localization, semantic interpretation, and practical use. UILoop addresses this gap by inserting explicit UI-element understanding into a cyclic screen–UI elements–action process and evaluates it with a dedicated benchmark.

  • Motivation: Existing GUI agents struggle with complex layouts and diverse UI elements while directly generating actions from screen inputs.This screen–action process is described as lacking interpretability and comprehensive UI-element understanding.
  • Motivation: Average scores below 0.1 across localization, semantic function description, and practical usage reveal substantial UI-element comprehension deficiencies.The evaluation covers three critical dimensions of UI understanding.
  • UILoop: UILoop reframes GUI reasoning as a cyclic Screen–UI Elements–Action process, making UI elements the bridge between screens and actions.The paradigm is designed to support more accurate reasoning based on correct UI elements.
  • UILoop: UI Element-Driven Reinforcement Fine-Tuning teaches models to locate key elements, infer semantic functions, and master their practical usage.The method targets precise UI parsing and interpretable reasoning.
  • Evaluation: UI Comprehension-Bench contains 26K samples and evaluates UI mastery with UI Locate, Lingualize, and Leverage metrics.The benchmark is introduced alongside a more challenging UI Comprehension task.
  • Results: Extensive experiments report state-of-the-art UI comprehension and superior results in GUI reasoning tasks for UILoop.The reported outcome spans both UI understanding and GUI reasoning.

2 Related Work

Prior GUI-agent work uses pretraining, supervised fine-tuning, and reinforcement learning for screen-to-action reasoning, while UI-element methods have largely emphasized localization. UILoop extends this line by explicitly teaching models UI-element mastery for improved GUI reasoning.

  • Screen-to-Action GUI Agent: Screen-to-Action agents improve GUI reasoning through large-scale pretraining, supervised fine-tuning, and reinforcement learning.Examples include GUI-OWL, Aguvis, CoCo-Agent, Show-UI, Aria-UI, and several reinforcement-learning systems.
  • UI Elements-Enhanced GUI Agent: Existing UI-element-enhanced methods focus mainly on localization while overlooking semantic functions and practical usage.GUI-explorer retrieves UI information externally but does not enhance intrinsic understanding.
  • UI Elements-Enhanced GUI Agent: UILoop explicitly teaches models to master UI elements through a Screen-UI Element-Action paradigm, achieving superior GUI reasoning performance.This positions UILoop as an extension beyond localization-focused approaches.

3 Preliminary

GUI reasoning is formulated as iterative action selection over the current screen and task-relevant UI elements. UILoop uses reinforcement learning to optimize this sequential process while modeling UI location, semantics, and usage.

  • GUI reasoning is modeled as a multi-turn decision process in which the agent observes the current screen and outputs an action at each iteration.
  • Task-relevant UI elements are represented through location, semantic or functional description, and practical usage information.Examples include an element's coordinates, its likely meaning, and an associated action and bounding box.
  • Using UI elements to obtain actions yields a Screen–UI Elements–Action iteration loop.
  • GRPO optimizes the policy for complex sequential decision-making by comparing the relative advantages of multiple responses to the same prompt without a value function.
  • The optimization includes a UI advantage, clipping range, and KL-divergence penalty relative to a reference policy.

4 UI-in-the-Loop Framework

UILoop builds a two-stage framework: a benchmark for explicit UI comprehension and reinforcement fine-tuning that integrates UI elements into GUI reasoning. The framework evaluates and trains localization, semantic understanding, and practical use of key UI elements before action prediction.

  • Framework Overview: UILoop consists of benchmark construction followed by UI Element-Driven Reinforcement Fine-Tuning to strengthen UI comprehension.The first stage synthesizes UI Comprehension-Bench; the second uses it to address the missing UI-understanding step in Screen-to-Action reasoning.
  • Scaling Data for UI Comprehension: Existing GUI datasets lack fine-grained information about key UI elements’ locations, semantic functions, and practical usage.The benchmark augments Screen-to-Action data with UI-element information needed for interpretable intermediate reasoning.
  • UI Comprehension Task: UI Comprehension introduces Locate, Lingualize, and Leverage metrics for localization, semantic-function understanding, and utilization accuracy.The overall score is defined as Overall = Locate ∗Lingualize ∗Leverage.
  • Benchmark Statistics: Ground-truth UI elements exceed 90% text coverage for most action types, while only minimal actions fall below 80%.The benchmark therefore provides UI elements with reported logical coherence and interpretability; long_press has 14 samples below the higher coverage range.
  • UI Element-Driven Reinforcement Fine-Tuning: The reinforcement objective combines format, location, lingualization, and leverage rewards to prioritize locating, understanding, and using UI elements.The training output explicitly contains UI descriptions, reasoning, and an action answer; the overall reward uses these components to guide sequential decisions.

5 Experiments

Experiments evaluate UILoop on GUI reasoning and UI comprehension, comparing it with zero-shot and Screen-to-Action models, intervention methods, and reward components. UILoop generally improves reasoning, UI comprehension, and interpretability.

  • Evaluation Setup: UILoop is evaluated on Android Control-High, ScreenSpot-Pro, and UI Comprehension-Bench using action, grounding, success-rate, Locate, Lingualize, and Leverage metrics.The baselines include zero-shot MLLMs and Screen-to-Action models.
  • Main Results: On ScreenSpot-Pro, UILoop’s 3B and 7B models outperform similarly sized Qwen2.5-VL and GUI-R1 models by 13.3% and 3.2% in overall scores, respectively.
  • Main Results: On Android Control-High, UILoop-7B exceeds OS-Atlas-7B, OS-Atlas-Pro-7B, and GUI-OWL-7B by 46.5%, 58.0%, and 38.8% in SR, respectively.
  • Ablation Study: Leverage Reward improves all metrics, Locate Reward increases GR by 7.9% and 8.6% for 3B and 7B models, and Lingualize Reward raises SR by 11.1% and 13.7%.The rewards target UI-element utilization, localization, and semantic understanding, respectively.
  • UI Intervention: False UI information impairs GUI reasoning, while key UI information improves accuracy; UILoop Training further improves results over context alone across four evaluated models.The reported UILoop Training improvements are 31.6% and 22.8% for Qwen2.5-VL-3B and 7B, and 17.8% and 29.0% for GUI-Owl-7B and OS-Atlas-Pro-7B.

6 Conclusion

The paper concludes that UILoop reframes GUI reasoning around UI elements and combines UI comprehension training with a 26K benchmark. Experiments report state-of-the-art UI comprehension and superior GUI reasoning results.

  • Conclusion: UILoop reframes GUI reasoning as a cyclic Screen–UI Elements–Action process rather than conventional Screen-to-Action reasoning.
  • Conclusion: UI Comprehension-Bench contains 26K samples and evaluates UI Locate, Lingualize, and Leverage capabilities.
  • Conclusion: UILoop achieves state-of-the-art UI comprehension performance and superior results in GUI reasoning tasks.

Limitations

The authors identify two limitations: UILoop focuses on fine-grained UI elements without modeling layouts at multiple granularities, and experiments predominantly use Qwen2.5-VL.

  • Limitations: UILoop focuses on fine-grained UI elements but does not consider coarse-grained layouts composed of multiple fine-grained elements.
  • Limitations: Current experiments predominantly focus on Qwen2.5-VL, leaving performance across a broader range of MLLMs for future investigation.

Ethics Statement

The paper states that UI Comprehension-Bench is constructed from existing GUI datasets and externally collected data, with manual verification and planned release of research artifacts.

  • Ethics Statement: UI Comprehension-Bench combines existing GUI reasoning datasets with externally collected webpages, mobile apps, and operating-system data.
  • Ethics Statement: The authors report manual verification and exclusion of low-quality or non-compliant data, and commit to releasing code, models, and datasets upon publication.

A Details of UI Comprehension-Bench Collection

UI Comprehension-Bench is constructed through source collection, key UI-element identification and parsing, and human verification. The resulting benchmark contains 26,207 isolated training and test samples.

  • Collection Pipeline: The synthesis pipeline comprises Source Data Collection, Key UI Element Identification and Parsing, and Human Verification.These steps organize the benchmark construction process.
  • Source Data Collection: Sources include webpages, mobile applications, operating systems, and existing GUI reasoning datasets, with samples normalized as (instruction, screen, action).Web screens are captured with BrowserGym and actions are simulated with Playwright; mobile and OS data use Droid-Bot.
  • UI Element Identification and Parsing: OmniParser V2 annotates candidate UI elements and GPT-4o selects instruction-relevant elements while generating semantic reasoning about their task contribution.The selection model receives the instruction, screen, coordinates, and action.
  • Human Verification: Human screening removes incorrect instructions, erroneous answers, and misidentified key UI elements before benchmark curation.Verification is the final quality-control stage.
  • Benchmark Composition: 26,207 samples comprise 3,471 training examples and 22,736 test examples, with complete data isolation between the sets.The training set is selected from existing GUI datasets, while the test set contains 22,736 samples.

B Demonstrations of UI Comprehension-Bench

UI Comprehension-Bench extends existing GUI reasoning data with key UI elements and reasoning chains that expose localization, semantics, and functional use across common actions. Its demonstrations show how these fields support structured UI analysis and action selection.

  • Action Examples: The benchmark is presented through examples for open_app, type, and click actions, illustrating their corresponding data fields and values.Figures 7–9 provide action-specific cases.
  • Benchmark Demonstrations: UI Comprehension-Bench adds Key UI Elements and Reasoning_Chains to existing dataset fields, representing Locate, Lingualize, and Leverage information.Examples cover open_app, type, and click actions.
  • Prompt Design: Prompts specify task-specific action spaces, including GUI grounding and GUI reasoning tasks.The prompts adapt the available actions to each task.
  • Dataset Comparison: A comparative table organizes existing datasets by episode counts, unique instructions, and availability of reasoning or UI-element information.The table includes datasets such as Android Control, AITZ, ScreenSpot, and UI-E2I-Bench.
  • Worked Example: In a calendar example, the date 10, month context, and OK button identify the target selection and confirmation action.The associated reasoning selects a click action near the date element.
  • Prompt Design: UILoop prompts require UI-element location, semantics, and functionality, followed by reasoning and a structured final action answer.The output uses <ui>, <think>, and <answer> sections and supports actions such as click, type, open_app, and scroll.
  • Element Parsing Prompt: The selection prompt asks for concise descriptions of instruction-relevant elements and a reasoning chain from instruction analysis to action determination.Reasoning is limited to no more than five sentences and uses structured XML tags.

D Error Analysis

The error analysis compares Screen-to-Action methods with UILoop using three UI-element-centered error categories: locating, verbalizing, and leveraging elements.

  • Experimental Setup: The analysis compares UI-R1-3B, GUI-R1-7B, GUI-OWL-7B, and UILoop on 100 randomly sampled Android Control-High test instances.Manual statistics are reported in Figure 10.
  • Error Categories: The three primary error types are Locate Error, Lingualize Error, and Leverage Error.These categories target element location, semantic description, and practical use in reasoning.
Loading 2604.06995v2…