PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Training

Linjia Kang, Zhimin Wang, Yongkang Zhang, Duo Wu, Jinghe Wang, Ming Ma, Haopeng Yan, Zhi Wang

arXiv:2601.22781v1cs.AI

TL;DR

Mobile GUI-agent training lacks scalable, high-quality trajectories whose difficulty is matched to agent capabilities. MobileGen profiles capability across structural and semantic difficulty dimensions, adaptively samples aligned tasks, and uses multi-agent generation to synthesize trajectories. Across multiple benchmarks, it improves average GUI-agent performance by 1.57× and consistently outperforms existing data-generation methods.

  • Problem

    GUI-agent training lacks scalable, high-quality trajectories with difficulty finely matched to agents’ current capabilities.

  • Method

    MobileGen profiles structural and semantic capability frontiers, adaptively generates difficulty distributions, and uses controllable multi-agent synthesis for training trajectories.

  • Results

    1.57×: MobileGen improves average GUI-agent performance across multiple challenging benchmarks and consistently outperforms existing data-generation methods.

  • Takeaways & Limitations

    Capability-aligned difficulty control supports effective mobile GUI-agent training and robust generalization across challenging tasks and environments.

Abstract

from arXiv · show

Large-scale, high-quality interaction trajectories are essential for advancing mobile Graphical User Interface (GUI) agents. While existing methods typically rely on labor-intensive human demonstrations or automated model exploration to generate GUI trajectories, they lack fine-grained control over task difficulty. This fundamentally restricts learning effectiveness due to the mismatch between the training difficulty and the agent's capabilities. Inspired by how humans acquire skills through progressively challenging tasks, we propose MobileGen, a novel data generation framework that adaptively aligns training difficulty with the GUI agent's capability frontier. Specifically, MobileGen explicitly decouples task difficulty into structural (e.g., trajectory length) and semantic (e.g., task goal) dimensions. It then iteratively evaluates the agent on a curated prior dataset to construct a systematic profile of its capability frontier across these two dimensions. With this profile, the probability distribution of task difficulty is adaptively computed, from which the target difficulty for the next round of training can be sampled. Guided by the sampled difficulty, a multi-agent controllable generator is finally used to synthesize high-quality interaction trajectories along with corresponding task instructions. Extensive experiments show that MobileGen consistently outperforms existing data generation methods by improving the average performance of GUI agents by 1.57 times across multiple challenging benchmarks. This highlights the importance of capability-aligned data generation for effective mobile GUI agent training.

1. Introduction

MobileGen addresses the shortage of scalable, high-quality GUI-agent training data by aligning generated task difficulty with each agent’s capability frontier. It decouples difficulty into structural and semantic dimensions, profiles capabilities, controls task distributions, and generates trajectories adaptively.

  • Large-scale, high-quality multi-step trajectories are needed because prompt-based methods cannot internalize failed interactions into improved GUI-agent capabilities.
  • Human demonstrations require costly annotation or restricted resources, while automated exploration methods lack fine-grained control over trajectory difficulty.
  • MobileGen aligns training difficulty with the agent’s capability frontier so tasks are neither too simple to be informative nor too complex to be tractable.
  • MobileGen decouples trajectory difficulty into structural and semantic dimensions, enabling controllable synthesis of high-quality interaction trajectories.
  • 1.57×: MobileGen improves GUI-agent performance over zero-shot baselines across multiple benchmarks while consistently outperforming existing SOTA methods.

2. Related Work

GUI agents need scalable, high-quality interaction data, but existing prompt-based, human-demonstration, and model-based approaches leave limited control over training difficulty. MobileGen addresses this mismatch by profiling agent capability, generating a difficulty distribution, and synthesizing difficulty-aware trajectories.

  • GUI agent methods generally comprise prompt-based and training-based approaches, with prompt-based methods being lightweight and easy to deploy but limited in scalability for complex tasks.
  • Human demonstrations provide training data but are labor-intensive and difficult to scale for GUI agents operating in vast, unstructured visual state spaces.
  • Model-based methods use VLMs to autonomously explore GUI environments and generate interactions, including reverse task synthesis from stochastic exploration trajectories in OS-Genesis [Sun et al., 2025].
  • MobileGen's pipeline profiles student-agent capabilities, generates a target difficulty distribution, and uses explorer and supervisor agents for difficulty-aware trajectory generation.
  • MobileGen aligns training difficulty with agent proficiency to address learning degradation caused by mismatched training difficulty and capability.

3. Methodology

MobileGen models GUI trajectory difficulty across structural and semantic dimensions, profiles the agent’s capability frontier, and uses that profile to generate capability-aligned training tasks. Its pipeline converts capability estimates into difficulty distributions and uses multi-agent collaboration to synthesize controlled, high-quality trajectories.

  • 3.1. Preliminaries: MobileGen decomposes trajectory difficulty into structural dimensions—Depth of Trajectory and Breadth of Trajectory—and semantic dimensions—Interaction Control Difficulty and Instruction Understanding Difficulty.Structural difficulty captures trajectory length and cross-application complexity, while semantic difficulty captures mapping goals to actions and interpreting instructions or implicit intent.
  • 3.2. Overall Pipeline: The methodology comprises agent capability profiling, difficulty distribution generation, and difficulty-aware trajectory generation.This pipeline customizes the training-task distribution before controllable synthesis of high-quality trajectories.
  • 3.2.1. Agent Capability Profiling: Agent capability profiling evaluates a student agent on a prior dataset to obtain structural and semantic capability statistics that guide subsequent difficulty regulation.The profile includes sequential execution, cross-application interaction, application vulnerability, interaction control, and instruction understanding measures; Pass@K evaluates action matching on the prior dataset.
  • 3.2.2. Difficulty Distribution Generation: A challenge point converts profiled capability into target difficulty, while independently shaped structural and semantic distributions align sampling with the agent’s learning frontier and retain moderate diversity.The parameter α controls training-task aggressiveness, and structural probabilities use a Gaussian kernel centered on the target capability; semantic affinities are normalized across easy, medium, and hard levels.
  • 3.2.3. Difficulty-Aware Trajectory Generation: Multi-agent collaboration generates trajectories from sampled difficulty parameters, with a supervisor allocating and regulating exploration budgets and a synthesizer reconstructing thoughts and task instructions.The workflow includes dynamic step management and rollbacks to correct interaction errors during exploration.

4. Experiments

MobileGen is evaluated across AndroidWorld, AndroidControl-Curated, and GUI-Odyssey against zero-shot and data-synthesis baselines, with ablations testing its difficulty-control components and challenge-point settings.

  • Main Results: 1.51× higher average Success Rates than zero-shot baselines were achieved on AndroidWorld, making MobileGen the strongest synthesis framework in that evaluation.Qwen3-VL-8B-Instruct fine-tuned with MobileGen also improved by up to 11.5% over UI-TARS-1.5-7B.
  • Main Results: A peak 1.44× Success Rate increase was achieved on AndroidControl-Curated’s Hard setting, where MobileGen consistently improved execution and planning across backbones.Only 20 applications were seen during training, making this setting an out-of-distribution generalization test.
  • Main Results: GUI-Odyssey evaluates transfer to unseen screen sizes and device types across high-level and low-level instructions, targeting robust cross-app interaction in diverse OOD environments.
  • Effectiveness of Critical Components: Removing the customized difficulty distribution, step budget, or rollback mechanism significantly degraded performance across both benchmarks.The no-rollback variant showed the largest decline, with a 4.4%-12.2% gap relative to full MobileGen.
  • Impact of Challenge Point Setting: The α = 0.5 challenge-point setting produced the best learning efficiency, outperforming other settings by 4.3% while achieving the minimum training loss.Both trivial and excessively difficult tasks yielded suboptimal learning.

5. Conclusion

MobileGen addresses the shortage of high-quality mobile GUI training data by aligning synthesized task difficulty with an agent’s capability frontier. Across three benchmarks, it improves success rates and generalization, including with only 500 synthetic trajectories.

  • MobileGen profiles agent capability, sets an adaptive challenge point, and synthesizes high-quality data through multi-agent collaboration across structural and semantic difficulty dimensions.
  • Across three benchmarks, MobileGen yields superior success rates and robust generalization, bridging the gap between general-purpose VLMs and specialized GUI agents in complex OOD scenarios.The framework uses 500 high-quality synthetic trajectories.

A.1.1. COMPARISON OF DATA QUALITY.

MobileGen produces higher-quality trajectories than comparison methods at both step and trajectory levels, with the advantage remaining across difficulty profiles from three VLMs.

  • MobileGen achieves superior trajectory-quality scores at both step-level and trajectory-level granularity compared with other methods.The comparison uses a fixed sample of 0.5k trajectories per method generated with the same Qwen3-VL-8B-Instruct backbone.
  • MobileGen’s quality advantage remains robust across varying difficulty profiles derived from three different VLMs.

A.1.2. VISUALIZATION ANALYSIS

Figure 5 illustrates semantic difficulty control in Simple Calendar Pro by varying ICD and IUD across trajectories with different operational and linguistic demands.

  • Figure 5 compares low, medium, and high semantic-complexity trajectories generated in the same application.The cases vary interaction requirements and instruction understanding while controlling the application context.
  • Trajectory (a) combines low operational complexity with an ambiguous instruction requiring intent-based interface exploration.
  • Trajectory (b) represents a standard event-creation task with medium difficulty in both linguistic logic and interaction.
  • Trajectory (c) requires intricate event-creation operations while keeping instruction understanding straightforward.

A.2. Details of Benchmarks

The paper evaluates GUI agents on AndroidWorld, AndroidControl-Curated, and GUIOdyssey, which cover dynamic mobile tasks, corrected evaluation data, and cross-application navigation.

  • AndroidWorld provides 116 dynamically instantiated tasks across 20 applications, with success verified through underlying system state and reported as Success Rate.
  • AndroidControl-Curated addresses evaluation bias and factual errors in the original AndroidControl dataset through a semi-automated purification pipeline.
  • Action-type Accuracy measures whether each predicted step uses the correct action category, such as Click, Type, or Scroll.
  • GUIOdyssey contains 8,334 cross-app episodes spanning 212 applications and evaluates step correctness using Action Matching Score.

A.3.1. ACTION SPACE

The appendix defines MobileGen’s action space, quality evaluation, difficulty-controlled dataset construction, action matching, and multi-agent exploration mechanisms.

  • Quality evaluation: Step-level quality evaluates grounding and goal alignment, whereas trajectory-level quality evaluates goal achievement and step efficiency.
  • Difficulty modeling: Each trajectory is characterized by structural metrics DoT and BoT plus semantic metrics ICD and IUD, enabling separate control of trajectory and task-goal difficulty.
  • A.5.1. DATASET CONSTRUCTION: MobileGen constructs balanced prior data by stratifying DoT, BoT, ICD, and IUD into easy, medium, and hard levels.Structural levels use DoT ranges of 10–15, 16–25, and 26–35 with BoT values 1, 2, and 3, respectively.
  • A.5.2. ACTION MATCHING PROTOCOL: The unified action matcher canonicalizes action types and strings before applying spatial, textual, or exact-match criteria.Coordinate actions use region or distance checks, text uses ANLS ≥0.5, and navigation actions follow strict categorical or canonical-name rules.
  • Multi-agent generation: The multi-agent generator allocates exploration budgets, maintains summarized interaction history, and rolls back anomalous interactions to improve trajectory quality.
Loading 2601.22781v1…