PaperlayerView exact revision on arXiv

Source-linked AI summary

UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning

Liangyu Chen, Hanzhang Zhou, Chenglin Cai, Jianan Zhang, Panrong Tong, Quyu Kong, Xu Zhang, Chen Liu, Yuqi Liu, Wenxuan Wang, Yue Wang, Qin Jin, Steven Hoi

arXiv:2510.20286v1cs.CVcs.AI

TL;DR

GUI grounding depends on natural-language instructions, but prior work largely treated them as static inputs and underexplored their quality and diversity. UI-Ins treats instruction perspectives as reasoning pathways and trains models to use and select them. Its models establish state-of-the-art results across five grounding benchmarks.

  • Problem

    Prior work largely treated instructions as simple input strings, leaving their impact on GUI grounding underexplored.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    Instruction-as-Reasoning treats distinct instruction perspectives as reasoning pathways, using supervised fine-tuning to teach their use and reinforcement learning to select an effective pathway.

    Source: §1 · pp. 2–3

  • Results

    UI-Ins-7B and UI-Ins-32B establish state-of-the-art results across five grounding benchmarks, with UI-Ins-32B scoring 87.3% on UI-I2E-Bench.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    Instruction quality and diversity are consequential grounding variables: existing datasets have a 23.3% instruction flaw rate, while inference-time instruction diversity yields up to a 76% relative performance gain.

    Source: §1 · pp. 1, 3

    Error analysis identifies failures involving missing domain-specific knowledge, weak layout understanding, and visually similar distractors.

    Source: §4.6 · p. 13

Abstract

from arXiv · show

GUI grounding, which maps natural-language instructions to actionable UI elements, is a core capability of GUI agents. Prior works largely treats instructions as a static proxy for user intent, overlooking the impact of instruction diversity and quality on grounding performance. Through a careful investigation of existing grounding datasets, we find a 23.3% flaw rate in their instructions and show that inference-time exploitation of instruction diversity yields up to a substantial 76% relative performance improvement. In this paper, we introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives and enabling the model to select the most effective pathway during reasoning. To achieve this, we propose a two-stage training framework: supervised fine-tuning (SFT) on synthesized, diverse instructions to instill multi-perspective reasoning, followed by reinforcement learning (RL) to optimize pathway selection and composition. Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks and exhibit emergent reasoning, selectively composing and synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model demonstrates strong agentic potential, achieving a 74.1% success rate on AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals additional insights such as how reasoning can be formulated to enhance rather than hinder grounding performance, and how our method mitigates policy collapse in the SFT+RL framework. All code and model checkpoints will be publicly released in https://github.com/alibaba/UI-Ins.

1 INTRODUCTION

GUI grounding depends on instructions, yet prior work often treats them as static inputs rather than examining how their diversity and quality affect performance. UI-Ins reframes instruction perspectives as reasoning pathways and trains models to select among them.

1 INTRODUCTION

  • UI-Ins-7B and UI-Ins-32B establish new state-of-the-art performance across five well-known grounding benchmarks; UI-Ins-32B achieves 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2.Source: §1 · p. 3
  • Models trained in a narrow, fixed instruction style have limited capacity to reason across perspectives, whereas humans switch among appearance, function, location, and intent descriptions.The paper names the dynamic-pathway framing Instruction-as-Reasoning.Source: §1 · p. 2
  • A manual inspection of 1,909 entries sampled from OS-Atlas, Widget Captioning, and AMEX found substantive instruction flaws that could introduce noise into training.The reported flaw rate is 23.3%.Source: §1 · p. 2
  • The framework cleans and diversifies instruction data, then uses SFT to teach multi-perspective pathways and GRPO-based RL to select a pathway for each GUI scenario.The pipeline first establishes clean data and then augments it with diverse instruction styles.Source: §1 · pp. 2–3
  • A systematic analysis of GUI-grounding instructions found 23.3% of samples’ instructions in major datasets flawed and suggested leveraging instruction diversity could unlock up to a 76% relative performance gain even without training.Source: §1 · p. 3
  • Analysis reports that instruction-pathway reasoning enhances rather than hinders grounding, mitigates policy collapse in SFT+RL, and supports reasoning from novel perspectives.The model can combine perspectives and generate diverse RL rollouts after SFT.Source: §1 · pp. 2–3

2 HOW MUCH DO INSTRUCTIONS REALLY MATTER?

The section tests whether instruction perspective and correctness affect GUI grounding, finding that choosing among diverse perspectives can substantially improve performance and that existing datasets contain flawed instructions. It also reports performance gains from training on cleaned data.

2 HOW MUCH DO INSTRUCTIONS REALLY MATTER?

  • The analysis asks how instruction perspective and correctness affect grounding, an understudied question because instructions carry high-level intent.The section frames these as research questions about perspective diversity and dataset quality.Source: §2 · p. 3

2.1 DOES INSTRUCTION DIVERSITY UNLOCK HIGHER PERFORMANCE?

  • The controlled ScreenSpot-Pro experiment rewrote instructions into four perspectives and evaluated Qwen2.5-VL-7B zero-shot.The perspectives were Appearance, Functionality, Location, and Intent.Source: §2.1 · p. 4
  • 76% relative improvement is achieved by the “Combined” strategy that always selects the best-performing instruction perspective for each sample.The result exceeds any single instruction perspective.Source: §2.1 · p. 4
  • Appearance-, function-, and intent-based instructions substantially outperform the original instructions, even without retraining.These results motivate dynamically choosing the most effective analytical angle.Source: §2.1 · p. 4

2.2 CAN WE TRUST EXISTING DATASETS FOR INSTRUCTION QUALITY?

  • The quality analysis manually examined 1,909 samples from OS-Atlas, AMEX, and Widget Captioning.Source: §2.2 · p. 4
  • 23.3% of examined instructions had substantive flaws, including ambiguous matches or no matching UI element.Figure 4 distinguishes ambiguous matches from mismatches.Source: §2.2 · p. 4, Figure 4 · p. 4
  • Models trained on cleaned data achieved substantial and consistent gains across multiple benchmarks.The analysis concludes that flawed training instructions can degrade downstream performance.Source: §2.2 · p. 4
  • The high-quality data pipeline preprocesses ground-truth boxes, generates perspective-diverse instructions with GPT-4.1, then verifies alignment.Source: Figure 5 · p. 5

3 METHOD

The method builds a clean, multi-perspective instruction dataset and trains models to use instruction perspectives as reasoning pathways. SFT teaches diverse reasoning, then RL trains context-dependent pathway selection.

3 METHOD

  • The method combines a clean data foundation with a two-stage algorithm for using diverse instructions as pathways and selecting an analytical perspective.Source: §3 · p. 5, §3.2 · p. 5

3.1 TASK DEFINITION

  • GUI grounding predicts a coordinate point on a screenshot to identify the UI element corresponding to a natural-language instruction.Source: §3.1 · p. 5

3.2 DATA PIPELINE FOR MULTI-PERSPECTIVE REASONING

  • The data pipeline first refines or filters ground-truth boxes and removes flawed instructions, then augments the clean data with verified instructions from four perspectives.GPT-4.1 generates appearance, functionality, location, and intent phrasings and verifies that each refers only to the target element.Source: §3.2 · p. 5

3.3 INSTRUCTION-AS-REASONING

  • Instruction-as-Reasoning is a two-stage framework: SFT teaches diverse instructions as explicit reasoning pathways, and RL trains selection of the optimal analytical angle.The overview presents RL as exploring unconstrained perspectives after SFT teaches multiple reasoning paths.Source: §3.3 · pp. 5–6, Figure 6 · p. 6

3.3.1 SFT STAGE: LEARNING TO GENERATE DIVERSE REASONING

  • The SFT objective trains the model to generate an instruction-perspective reasoning text and predict the target coordinate conditioned on that reasoning.The reasoning text is sampled from a valid augmented perspective and concatenated with the ground-truth point.Source: §3.3.1 · p. 6

3.3.2 RL STAGE: LEARNING TO SELECT THE OPTIMAL PERSPECTIVE

  • RL iteratively trains the model to prioritize context-dependent reasoning pathways, and the model also learns to combine perspectives and formulate novel ones.The prompt invites open-ended reasoning, and reward feedback guides pathway selection.Source: §3.3.2 · pp. 6–7

4 EXPERIMENT AND RESULTS

UI-Ins performs strongly across grounding benchmarks and AndroidWorld, while ablations and qualitative analyses examine its data pipeline, training stages, reasoning pathways, and failure modes.

4.2 GROUNDING RESULTS

  • UI-Ins-32B achieves state-of-the-art performance on benchmarks emphasizing complex instruction understanding, and leads across ScreenSpot-Pro, ScreenSpot-V2, and Showdown evaluations.UI-Ins-7B also outperforms similarly scaled models; the reported advantage is larger on harder instruction subsets.Source: §4.2 · pp. 9–10
  • UI-Ins-7B’s advantage over Qwen2.5-VL-7B grows from 134.2% on MMBench-GUI L2 Basic to 159.4% on Advanced.For UI-Ins-32B, the advantage over Qwen2.5-VL-32B grows from 12.3% on Basic to 24.5% on Advanced; its gain over GTA1 on UI-I2E-Bench grows from 1.6% on explicit to 6.6% on implicit.Source: §4.2 · pp. 9–10

4.3 ONLINE AGENT RESULTS

  • 74.1% task success rate on AndroidWorld with UI-Ins-7B as the grounding executor under a GPT-5 planner.The evaluation uses a live, dynamically changing interface and the framework outperforms strong closed-source models.Source: §4.3 · p. 10, Table 5 · p. 11

4.4 ABLATION STUDY

  • The data pipeline reduced inspected-sample error from 23.3% in original data to below 8%, and cleaned-data SFT improved performance across multiple benchmarks.The ablation compares 210k original samples with 180k cleaned samples.Source: §4.4 · p. 11, Figure 7 · p. 12
  • Removing either SFT or RL degrades accuracy, supporting the two-stage sequence of teaching diverse-perspective reasoning before optimizing pathway selection.The training-stage ablation compares UI-Ins-7B with SFT-only and RL-only variants.Source: §4.4 · p. 11, Table 6 · p. 11

4.5 DEEPER INSIGHTS INTO INSTRUCTION-AS-REASONING

  • Removing intermediate reasoning lowers accuracy across all benchmarks, including by over 10% on UI-I2E-Bench; free-form reasoning degrades tested baselines, while Instruction-as-Reasoning improves accuracy.Instruction-as-Reasoning-based SFT also serves as an exploratory warm-up associated with performance gains during RL, unlike standard SFT initialization, which often degrades.Source: Table 7 · p. 11, §4.5 · p. 12, Table 8 · p. 12
  • After RL, UI-Ins selects different perspectives by scenario, combines multiple perspectives, and generates analytical angles beyond the four trained perspectives.Across 1,477 UI-I2E samples, the model produced 5,245 reasoning ways; synthesis was not explicitly taught and emerged during RL.Source: §4.5 · p. 13, Figure 8 · p. 13

4.6 ERROR ANALYSIS

  • Identified failures involve missing domain-specific knowledge, weak layout understanding, and confusion between visually similar targets and distractors.The examples include failure to associate a company description with the correct brand and difficulty identifying the clickable area.Source: §4.6 · p. 13, Figure 10 · p. 14

5 RELATED WORK

Related work examines free-form reasoning, instruction augmentation, and GUI-grounding training paradigms. UI-Ins contrasts with prior approaches by using diverse instructions as reasoning pathways within SFT+RL.

5.1 REASONING IN GUI GROUNDING

  • Free-form reasoning may degrade GUI-grounding performance; UI-Ins instead trains diverse pathways during SFT to support varied RL rollouts and mitigate policy collapse.The passage notes that several prior works using free-form reasoning did not provide ablations testing its effectiveness.Source: §5.1 · p. 14

5.2 INSTRUCTION IN GUI GROUNDING

  • Prior instruction augmentation mainly paraphrases instructions as static inputs, whereas Instruction-as-Reasoning treats diverse styles as learnable reasoning pathways.The paper also notes that prior methods lacked deep analysis of instruction impact and consistent performance gains.Source: §5.2 · p. 14, Figure 11 · p. 15

5.3 TRAINING PARADIGM IN GROUNDING

  • Prior SFT+RL approaches face policy collapse, while pure RL overlooks the benefit of SFT; UI-Ins uses SFT to teach broader reasoning formats and RL to select pathways.The discussion presents this as an example of addressing the practical difficulty of SFT+RL training.Source: §5.3 · p. 15

6 CONCLUSION

The conclusion presents Instruction-as-Reasoning as a two-stage SFT+RL approach to instruction diversity, reporting state-of-the-art grounding results and AndroidWorld performance.

6 CONCLUSION

  • UI-Ins establishes state-of-the-art results across five benchmarks, with UI-Ins-32B scoring 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2.UI-Ins-7B as executor achieves a 74.1% success rate on AndroidWorld.Source: §6 · p. 15

A DATA PIPELINE DETAILS

The data pipeline expands instructions into four perspectives and filters instruction quality by checking whether instructions uniquely identify GUI elements and whether bounding boxes fit their targets.

A.1 INSTRUCTION DIVERSITY AUGMENTATION

  • Instruction augmentation adds appearance-, function-, spatial-, and goal-based descriptions, using screenshots marked with the target point or box to ground GPT-4.1 generation.The augmentation prompt prioritizes clarity and uniqueness, requiring each generated instruction to identify only the highlighted element.Source: §A.1 · p. 20

A.2 INSTRUCTION QUALITY REFINEMENT

  • Quality refinement prompts GPT-4.1 to assess whether each instruction uniquely identifies one element and whether the ground-truth box appropriately encloses it.The verification task overlays the ground-truth annotation on the screenshot to reduce potential model hallucinations.Source: §A.2 · p. 21

B EXPERIMENT PROMPTS

The appendix examples show how SFT and RL frame GUI grounding as producing a coordinate-based action. The SFT example explicitly guides analysis through one of four perspectives, while the RL example asks for reasoning before output.

B.2 RL TRAINING EXAMPLE

B.1 SFT TRAINING EXAMPLE

  • The SFT example wraps a GUI-agent prompt, reasoning, and a grounding action with click coordinates in a structured tool-call format.The prompt describes the action space as a click at [x, y].Source: §B.1 · p. 22
  • SFT instructs the agent to analyze the screenshot through Appearance, Function, Spatial, or Goal perspectives before translating the instruction and returning an action.Source: §B.1 · p. 22
  • The SFT example rewrites the CSDN bookmark request as finding a bookmark with a red C icon and the label CSDN, then clicks it.Source: §B.1 · p. 23

B.2 RL TRAINING EXAMPLE

  • The RL example uses the same GUI-agent and coordinate-click tool-call framing shown in the SFT example.Its action-space entry likewise represents a click with [x, y] coordinates.Source: §B.2 · p. 23
  • The RL prompt asks the agent to reason inside think tags and then output the coordinate for the requested UI element.Source: §B.2 · p. 23

B.3 ONLINE BENCHMARK EVALUATION

The online AndroidWorld evaluation pairs a high-level planner with a grounding executor that converts planner descriptions into screen coordinates. The prompt defines action choices and a perception-action loop for carrying out tasks.

B.3 ONLINE BENCHMARK EVALUATION

  • The AndroidWorld framework pairs GPT-5 as planner with UI-Ins-7B as executor, which maps planner descriptions to screen coordinates for clicks or long presses.Screen updates and execution feedback return to the planner in an iterative perception-action loop.Source: §B.3 · pp. 23–24
  • The system prompt frames the agent as an Android phone operator that answers queries and performs precise UI actions.Source: §B.3 · p. 24
  • The action framework includes opening apps, clicking, and long-pressing visible elements, with examples specifying an app name or target description.Source: §B.3 · p. 24
  • Other listed actions cover text entry, answering, home and back navigation, scrolling, task status, and waiting for screen updates.Source: §B.3 · p. 24
  • Execution principles favor simple valid paths, retries with alternatives after two failures, and preferred navigation and text-entry procedures.The prompt also directs the agent to activate an input box before typing and to use selection controls for text manipulation.Source: §B.3 · p. 24
  • The decision process considers the goal, action history, and current screen, checks completion, then selects one action and returns valid JSON.The prompt supplies user goal, previous actions, and available apps as context; an example begins by opening Chrome.Source: §B.3 · p. 24

C.1 REASONING PERSPECTIVE ANALYSIS

The authors manually define ten reasoning perspectives and use GPT-4.1 to classify 1,477 UI-Ins-7B responses on the UI-I2E benchmark. The taxonomy distinguishes visual, functional, spatial, goal, structural, state, component, sequential, salience, and accessibility cues.

C.1 REASONING PERSPECTIVE ANALYSIS

  • The analysis manually defines ten perspectives and applies the taxonomy with GPT-4.1 to 1,477 UI-Ins-7B responses on the whole UI-I2E benchmark.Source: §C.1 · p. 25

1. Appearance

  • Appearance describes an element’s visual properties and literal text; functionality describes its purpose or interaction effect; location captures absolute or relative screen position; intent captures the high-level goal.Source: §1 · p. 25, §2 · p. 25, §3 · p. 25

5. Structural Relationship

  • Structural relationship identifies an element’s place in the UI hierarchy, state describes its dynamic condition, and component type identifies a reusable design pattern.Source: §5 · p. 25, §6 · p. 25, §7 · p. 25

8. Sequential Position

  • Sequential position tracks an element’s place in a multi-step workflow, salience its visual prominence, and accessibility its non-visual properties for assistive technologies.Source: §8 · p. 25, §9 · p. 25, §10 · p. 25

C.2 QUALITATIVE EXAMPLE

The authors present Figure 12 as success examples of UI-Ins-32B and describe its results as robust across diverse platforms. The examples include visually specified menu items, product-search refinements, and requests for help with shortcut editing.

C.2 QUALITATIVE EXAMPLE

  • One example specifies a menu option by its “Deselect All” label, gray background, and black text.Source: §C.2 · p. 26
  • Other examples ask for a Charizard product-search refinement and help with the current shortcut-editing process.Source: §C.2 · p. 26
  • UI-Ins-32B demonstrates robust performance across diverse platforms in the Figure 12 success examples.Source: §C.2 · p. 25, Figure 12 · p. 26
Loading 2510.20286v1…