PaperlayerView exact revision on arXiv

Source-linked AI summary

UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Aojun Zhou, Hongsheng Li

arXiv:2505.21496v1cs.CLcs.CVcs.LG

TL;DR

GUI agents face difficult trajectory verification and limited scalable, high-quality training data. UI-Genie pairs a GUI reward model with iterative reward-guided exploration and outcome verification to expand agent and reward-model training. The paper reports state-of-the-art performance across benchmarks after three self-improvement cycles, while noting that fully correct trajectories are not guaranteed across all tasks.

  • Problem

    GUI trajectory outcomes are hard to verify from final answers alone, and unreliable verification leaves high-quality training data costly and difficult to scale.

    Source: §1 · pp. 1–2

  • Method

    UI-Genie combines an image-text interleaved reward model for action- and task-level assessment with iterative reward-guided exploration, outcome verification, and model refinement.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    UI-Genie reports state-of-the-art performance across benchmarks after three self-improvement cycles; on AndroidControl, its 72B model reaches 77.0% high-level-task success, 2.3% above UI-TARS.

    Source: §1 · p. 2, §4.2.1 · p. 7

  • Takeaways & Limitations

    The work produces UI-Genie-RM-517k and UI-Genie-Agent-16k, including the first reward-specific GUI-agent dataset and synthetic trajectories generated without manual annotation.

    Source: §1 · p. 2, §5 · p. 11

    The reward model can occasionally provide suboptimal signals, and the framework cannot guarantee fully correct trajectories across all GUI tasks.

    Source: §5 · p. 11

Abstract

from arXiv · show

In this paper, we introduce UI-Genie, a self-improving framework addressing two key challenges in GUI agents: verification of trajectory outcome is challenging and high-quality training data are not scalable. These challenges are addressed by a reward model and a self-improving pipeline, respectively. The reward model, UI-Genie-RM, features an image-text interleaved architecture that efficiently pro- cesses historical context and unifies action-level and task-level rewards. To sup- port the training of UI-Genie-RM, we develop deliberately-designed data genera- tion strategies including rule-based verification, controlled trajectory corruption, and hard negative mining. To address the second challenge, a self-improvement pipeline progressively expands solvable complex GUI tasks by enhancing both the agent and reward models through reward-guided exploration and outcome verification in dynamic environments. For training the model, we generate UI- Genie-RM-517k and UI-Genie-Agent-16k, establishing the first reward-specific dataset for GUI agents while demonstrating high-quality synthetic trajectory gen- eration without manual annotation. Experimental results show that UI-Genie achieves state-of-the-art performance across multiple GUI agent benchmarks with three generations of data-model self-improvement. We open-source our complete framework implementation and generated datasets to facilitate further research in https://github.com/Euphoria16/UI-Genie.

1 Introduction

MLLM-based GUI agents have improved interface navigation, but verifying trajectory outcomes and scaling high-quality training data remain difficult. UI-Genie combines a specialized reward model with iterative self-improvement to generate synthetic trajectories and report state-of-the-art benchmark performance.

1 Introduction

  • MLLM-based GUI agents perceive screenshots, identify functional UI elements, and generate executable action sequences, outperforming text-only counterparts.The introduction situates this progress alongside broader LLM and MLLM advances.Source: §1 · p. 1
  • Trajectory verification is difficult because task completion depends on historical context, while existing assessment approaches do not reliably verify outcomes or intermediate steps.Human-annotated operation trajectories are time-consuming, expensive, difficult to scale, and limited in volume and diversity, especially for complex multi-step tasks.Source: §1 · pp. 1–2
  • UI-Genie-RM uses interleaved image-text context to assess actions and tasks, while UI-Genie iteratively expands synthetic training data without extensive human annotation.Its reward model supports both single-step and multi-step evaluation; the pipeline uses reward-guided exploration and outcome verification.Source: §1 · p. 2
  • UI-Genie reports state-of-the-art results with three self-improvement cycles and releases UI-Genie-RM-517k and UI-Genie-Agent-16k.The paper describes the former as the first reward-specific GUI-agent dataset and the latter as synthetic trajectories generated without manual annotation.Source: §1 · p. 2, Table 1 · p. 2

2 Related Work

Prior GUI-agent research spans commercial and open-source MLLMs, while reward-model research has explored process supervision and automated step assessment. UI-Genie distinguishes itself by targeting GUI-specific reward modeling and automatic trajectory synthesis and evaluation.

2 Related Work

  • Prior mobile and desktop GUI agents include commercial MLLM systems and open-source approaches such as OS-Atlas, UI-TARS, and methods using rule-based reinforcement learning or GRPO.Many approaches still depend on human-annotated trajectories, and synthetic-trajectory methods may rely on human effort or closed API models for verification.Source: §2 · p. 3
  • Process-supervision reward models can better assess intermediate reasoning steps than outcome-supervision models, and prior work has pursued automatic step supervision.GUI-related reward-model work includes GPT-4o trajectory-level assessment; UI-Genie instead builds a GUI-specific model on open-source foundation models for step-level rewards without costly proprietary models.Source: §2 · p. 3

3 Method

UI-Genie combines a GUI-specific reward model, deliberately constructed reward data, and iterative agent–reward-model improvement. Its method uses historical interaction context for unified action and task assessment, then expands training through reward-guided exploration and verification.

3.1 UI-Genie-RM: A Unified Reward Model

  • UI-Genie-RM adapts a standard MLLM backbone to sequential GUI interactions and unifies step-level and outcome-level assessment.It uses task goals, current screenshots, candidate actions, and recent screenshot-action history; earlier actions are summarized in language to balance context and computational efficiency.Source: §3.1 · pp. 3–4
  • A special task-completion action lets UI-Genie-RM judge terminal outcomes alongside individual actions through the same binary-reward objective.The model assigns correct or incorrect labels conditioned on the task goal, screenshot, and historical context, and predicts reward labels as next tokens.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Data Construction for Reward Model Training

  • The reward dataset contains approximately 517k samples: 458k synthetic samples from three complementary construction strategies, plus 59k process-reward samples from self-improvement rollouts.The strategies are rule-based verification, controlled trajectory corruption, and hard negative mining; together they cover varied difficulty levels and failure modes.Source: §3.2 · pp. 4–5

3.3 Iterative Self-Improvement of Agent and Reward Models

  • Reward-guided beam search evaluates ten candidate actions per step and retains the top five paths by cumulative reward, avoiding computationally expensive MCTS.The exploration produces successful paths and informative partial trajectories for data expansion.Source: §3.3 · pp. 5–6
  • Outcome verification adds successful trajectories to agent training, while continuation rollouts label potentially useful steps from failed trajectories for reward-model training.For unsuccessful trajectories, five continuations are sampled from each intermediate step; a successful completion marks that step correct. Three rounds increase task complexity from source tasks to generated and manually crafted scenarios.Source: Figure 3 · p. 5, §3.3 · p. 6

4 Experiments

Experiments evaluate UI-Genie agents and UI-Genie-RM on static and dynamic benchmarks, then test reward-guided scaling, architecture choices, and iterative self-improvement. Results report gains across agent benchmarks, reward assessment, and successive improvement rounds.

4.1 Implementation Details

  • The reward model uses Qwen2.5-VL-7B, and agent variants use 3B, 7B, and 72B backbones; training includes 458k initial reward samples, 59k iterative process-reward samples, and 16k synthetic agent trajectories.The agent trajectories are generated through three rounds of iterative improvement without manual annotation.Source: §4.1 · p. 6

4.2.1 Agent Model Evaluation

  • 93.8% low-level and 72.9% high-level success rates for the 3B agent exceed previous SOTA by 1.7% and 1.8%, respectively; the 72B model reaches 77.0% on high-level tasks, 2.3% above UI-TARS.The AndroidControl comparison covers commercial systems, open-source models, and UI-Genie across low- and high-level tasks.Source: §4.2.1 · p. 7, Table 2 · p. 7
  • UI-Genie outperforms commercial systems and open-source models and fine-tuned variants on AndroidLab task-completion metrics.AndroidLab evaluates agents in an emulator across 138 tasks in nine categories of offline applications.Source: §4.2.1 · p. 8, Table 3 · p. 8
  • UI-Genie exceeds UI-TARS by approximately 5% in function-based success rate, 3% in commercial-LLM-based success rate, and 5% in overall ESAR on A3.A3 evaluates 201 tasks across 20 online applications, including tasks requiring more than 15 interaction steps.Source: §4.2.1 · p. 8, Table 4 · p. 9

4.2.2 Reward Model Evaluation

  • On hard tasks, UI-Genie-RM achieves 68.7% step-level and 70.5% outcome-level F1, and consistently outperforms baselines across evaluation types and complexity levels.The reward-model benchmark uses step-level and outcome-level F1 to assess easy, medium, and hard tasks.Source: §4.2.2 · p. 9, Table 5 · p. 9

4.3.1 UI-Genie-RM for Test-time Scaling.

  • Best-of-10 sampling raises the Qwen2.5-VL-3B step-level success rate from 60.3% to 63.1% and task-level success rate from 7.5% to 10.2%.UI-Genie-RM ranks candidate actions; reported gains span both metrics and model sizes, including scaling from N=5 to N=10.Source: §4.3.1 · p. 10, Table 6 · p. 11

4.3.2 Effectiveness of UI-Genie-RM Architecture

  • With five historical images, step-level reward accuracy reaches 79.6% versus 61.3% without history, while outcome accuracy reaches 82.1% versus 60.5%.Accuracy increases with historical depth; unified action- and task-reward representation also outperforms separate models.Source: §4.3.2 · p. 10, Table 7 · p. 11

4.3.3 Effectiveness of Self-Improvement Approach

  • AndroidLab task success for UI-Genie-Agent-7B rises from 18.1% at round 0 to 38.7% at round 3, surpassing AutoGLM, while reward-model accuracy rises from 68.2% to 79.6%.The largest improvement occurs in round one, followed by continued gains as the framework incorporates more complex tasks and refines both models.Source: §4.3.3 · pp. 10–11, Figure 4 · p. 11

5 Discussions

UI-Genie combines synthetic trajectory generation with a specialized reward model and iterative agent-reward model co-evolution. The discussion notes both a trajectory-correctness boundary and a potential accessibility benefit alongside training-related carbon emissions.

5 Discussions

  • The framework produces two novel datasets and reports state-of-the-art performance, while presenting process-supervised synthetic data as a potential route to scalable GUI-agent development.The broader-impact discussion says the work can help people with disabilities use mobile devices, but notes significant carbon emissions from training computation.Source: §5 · p. 11
  • UI-Genie cannot guarantee fully correct trajectories across all GUI tasks, and occasional suboptimal reward signals can yield failed trajectories during training-data expansion.Source: §5 · p. 11

A UI-Genie Training

UI-Genie uses a unified action space across existing and synthetic datasets, with action-specific parameters for mobile interaction. Its dual-mode interaction supports both coordinate-based and Set-of-Mark targeting.

A.1 Definition of Action Space

  • UI-Genie defines eight action types—open, click, swipe, long_press, type, system_button, wait, and terminate—with parameters tailored to each action.The action space is used across AndroidControl, AMEX, AndroidLab, and newly generated synthetic data.Source: §A.1 · p. 15, Table 8 · p. 15
  • UI-Genie supports targeting interactive elements by either absolute coordinates or Set-of-Mark indices when SoM prompts are provided.Swipe also specifies direction and distance; wait specifies duration, and terminate indicates task-completion status.Source: §A.1 · p. 15
  • An action_desc parameter adds semantics to every action type in the historical operation record.Source: §A.1 · p. 15

A.2 Implementation Details

UI-Genie-RM and UI-Genie-Agent are based on the Qwen2.5-VL family, with variants developed at three model sizes.

A.2 Implementation Details

  • UI-Genie-RM and UI-Genie-Agent use Qwen2.5-VL backbones, with 3B, 7B, and 72B variants developed for experiments.Experiments used a distributed setup of 20 L40s machines with four GPUs each.Source: §A.2 · p. 15

A.2.1 Reward Model Training

UI-Genie-RM is trained as a reward predictor using supervised fine-tuning followed by iterative refinement on process-reward samples.

A.2.1 Reward Model Training

  • UI-Genie-RM uses a Qwen2.5-VL-7B backbone and represents positive and negative reward labels with special tokens for next-token prediction.It is initially fine-tuned on 458K samples, then refined using 59K process-reward samples from self-improvement cycles.Source: §A.2.1 · p. 15
  • Training fully fine-tunes the model while freezing the vision encoder, using AdamW with a 1e-5 learning rate and global batch size of 160.Input images are capped at 602,112 pixels for computational efficiency.Source: §A.2.1 · p. 15

A.2.2 Agent Model Training

The agent is trained progressively using existing datasets, then refined with trajectories collected under reward-model supervision. Separate prompts adapt baseline models for step-level and outcome-level reward evaluation.

A.2.2 Agent Model Training

  • The initial 7B baseline agent is trained on AndroidControl, AMEX, and AndroidLab before iterative self-improvement.The vision encoder is frozen during full fine-tuning.Source: §A.2.2 · p. 16
  • Three rounds of reward-model-supervised exploration in dynamic Android environments produce UI-Genie-Agent-16K, a dataset of 16K trajectories without manual annotation.After each round, the 7B agent is fine-tuned for one epoch on newly collected trajectories.Source: §A.2.2 · p. 16
  • The final agent is scaled to 3B and 72B using UI-Genie-Agent-16K combined with open-source datasets.The 72B model uses rank-stabilized LoRA because of GPU memory constraints.Source: §A.2.2 · p. 16

B Prompt for Evaluation

C.1 Reward Data Example

The reward-data examples pair positive and negative actions and trajectories. Synthetic examples show that reward guidance can accompany both task completion and failure.

C.1 Reward Data Example

  • Action-level samples include positive and negative actions, while trajectory-level samples pair complete successful and failed task completions.The figure examples omit history images or actions for visualization, although those histories are included in training.Source: §C.1 · p. 17, Figure 7 · p. 18, Figure 8 · p. 19

C.2 Synthetic Trajectory example

  • Successful trajectories can emerge under UI-Genie-RM guidance, but step-level reward guidance does not ensure a successful final outcome.The failure example ends with a negative outcome reward despite receiving process reward guidance at each step.Source: §C.2 · p. 20, Figure 9 · p. 20, Figure 10 · p. 21

D UI-Genie Trajectory Example on Online Evaluation

The examples show UI-Genie-Agent completing precise edits in Bluecoins and Calendar on AndroidLab, and a news search and settings change on Android Arena.

D.1 AndroidLab case

  • UI-Genie-Agent-72B adjusts a Bluecoins expenditure to 500 CNY for May 15, 2024.The example demonstrates locating and modifying transaction data from a precise instruction.Source: §D.1 · p. 22, Figure 11 · p. 22
  • UI-Genie-Agent-72B changes the end time of the calendar event titled “work” to 7:00 PM.The example concerns editing a time-based event detail.Source: §D.1 · p. 22, Figure 12 · p. 23

D.2 Android Arena (A3) case

  • UI-Genie-Agent-7B searches CNN for panda-related news and retrieves articles.The example requires navigating the CNN interface to use its search function.Source: §D.2 · p. 24, Figure 13 · p. 24
  • UI-Genie-Agent-7B navigates Coursera settings and switches the app to dark mode.The example demonstrates locating and toggling an application setting.Source: §D.2 · p. 24, Figure 14 · p. 25

Related papers

Loading 2505.21496v1…