PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis

Kanzhi Cheng, Zehao Li, Zheng Ma, Nuo Chen, Jialin Cao, Qiushi Sun, Zichen Ding, Fangzhi Xu, Hang Yan, Jiajun Chen, Anh Tuan Luu, Jianbing Zhang, Lewei Lu, Dahua Lin

arXiv:2604.15093v1cs.AIcs.CLcs.CVcs.HC

TL;DR

OpenMobile targets the open-source mobile-agent data gap caused by closed training data and opaque synthesis recipes. It builds grounded instructions from global environment memory and uses policy switching to collect error-recovery trajectories, achieving competitive results across three benchmarks, including 51.7% and 64.7% AndroidWorld success for Qwen2.5-VL and Qwen3-VL. The authors report that gains reflect broad functionality coverage and enhanced error recovery rather than benchmark overfitting.

  • Problem

    Closed mobile-agent systems hide their training data and synthesis recipes, while open-data approaches remain far behind on AndroidWorld, limiting reproducibility and study of effective data properties.

  • Method

    OpenMobile decouples exploration from instruction generation using global environment memory, then applies policy-switching rollout to combine learner behavior with expert correction.

  • Results

    OpenMobile-trained agents achieve competitive results across AndroidWorld, AndroidLab, and MobileWorld, including 51.7% and 64.7% AndroidWorld success for Qwen2.5-VL-7B and Qwen3-VL-8B.

  • Takeaways & Limitations

    The reported gains stem from broad functionality coverage and enhanced error-recovery capability rather than benchmark overfitting, narrowing the gap with closed-data systems.

  • Takeaways & Limitations

    Random policy switching can be noisy because learner–expert disagreement does not necessarily indicate an error, and frequent switching can fragment trajectories.

Abstract

from arXiv · show

Mobile agents powered by vision-language models have demonstrated impressive capabilities in automating mobile tasks, with recent leading models achieving a marked performance leap, e.g., nearly 70% success on AndroidWorld. However, these systems keep their training data closed and remain opaque about their task and trajectory synthesis recipes. We present OpenMobile, an open-source framework that synthesizes high-quality task instructions and agent trajectories, with two key components: (1) The first is a scalable task synthesis pipeline that constructs a global environment memory from exploration, then leverages it to generate diverse and grounded instructions. and (2) a policy-switching strategy for trajectory rollout. By alternating between learner and expert models, it captures essential error-recovery data often missing in standard imitation learning. Agents trained on our data achieve competitive results across three dynamic mobile agent benchmarks: notably, our fine-tuned Qwen2.5-VL and Qwen3-VL reach 51.7% and 64.7% on AndroidWorld, far surpassing existing open-data approaches. Furthermore, we conduct transparent analyses on the overlap between our synthetic instructions and benchmark test sets, and verify that performance gains stem from broad functionality coverage rather than benchmark overfitting. We release data and code at https://njucckevin.github.io/openmobile/ to bridge the data gap and facilitate broader mobile agent research.

1 Introduction

OpenMobile addresses the open-source mobile-agent data gap with an open synthesis framework for grounded tasks and corrective trajectories. Its data supports competitive benchmark performance and analyses attributing gains to broad coverage and error recovery rather than overfitting.

  • Open-source mobile agents achieve only around 30% on AndroidWorld, while leading closed systems approach 70%.Closed systems also keep trajectory data and synthesis recipes opaque, limiting reproduction and analysis of successful data properties.
  • OpenMobile decouples environment exploration from instruction generation by building global memory and composing complex instructions from short- and long-term context.This design targets diversity and grounding beyond what a single local trajectory reveals.
  • OpenMobile performance gains stem from broad functionality coverage and enhanced error-recovery capabilities rather than benchmark overfitting.The authors support this conclusion through transparent contamination and overlap analyses.
  • OpenMobile combines task synthesis with policy-switching rollout to capture corrective signals missing from expert distillation.The framework alternates learner and expert behavior so trajectories include error-recovery experiences.

2 Related Work

Related work shows rapid progress in vision-centric mobile agents but a widening open-source performance gap. Existing synthesis methods face grounding, annotation-quality, and trajectory-diversity limitations, motivating scalable open recipes.

  • Vision-language models shifted mobile agents toward end-to-end GUI interaction from raw screenshots using clicks and typing.Earlier systems used accessibility trees or code-based computer interaction.
  • Open-source models trained on AndroidControl and AMEX plateau at roughly 30% on AndroidWorld.These human-annotated datasets contain significant annotation noise and lack rich thinking patterns.
  • Task-driven synthesis can produce generic, underspecified, or infeasible instructions because it lacks grounding in real-world context.This limitation motivates interaction-driven approaches that explore target environments before generation.
  • Expert distillation supplies high-quality demonstrations, while self-evolution bootstraps performance from the agent’s own successful trajectories.These are established alternatives for collecting training trajectories after task instructions are available.

3 OpenMobile

OpenMobile constructs grounded tasks from a structured global memory of explored app functionality and enriches training trajectories through policy switching. Its error-intervention strategy limits noisy switching by invoking the expert when learner progress deviates.

  • OpenMobile: OpenMobile produces diverse, grounded task instructions and trajectories enriched with error-recovery signals as complementary training assets.The framework is designed to cover broad mobile functionality while facilitating effective agent training.
  • Scalable Task Synthesis: The decoupled task-synthesis paradigm first explores the environment, then builds structured memory before composing instructions from relevant functionalities.This avoids restricting generation to a single local exploration trajectory.
  • Scalable Task Synthesis: Environment exploration collects screen-action transitions across application states using random walks, structured coverage methods, or human demonstrations.The framework is agnostic to the specific exploration strategy.
  • Scalable Task Synthesis: Global environment memory unifies fragmented trajectories through shared screens, neighboring-screen relations, and per-screen functionality descriptions.Perceptual hashing clusters visually similar screens, while functionality embeddings support per-app retrieval.
  • Scalable Task Synthesis: Task synthesis presents a VLM with each candidate screen plus short-term neighboring functionality and long-term semantically related functionality.Generated instructions are quality-filtered and deduplicated after VLM generation.
  • Policy-Switching Rollout: Policy-switching rollout alternates learner and expert policies so learner mistakes are followed by expert corrections, providing error-recovery experiences absent from pure distillation.Expert participation also avoids the capacity ceiling associated with self-evolution.
  • Policy-Switching Rollout: Random switching is noisy because policy disagreement may reflect multiple valid action sequences, and frequent switching can fragment complex-task trajectories.These limitations motivate detecting unproductive learner deviation rather than switching at every disagreement.
  • Policy-Switching Rollout: Error-intervention rollout starts with the learner and switches to the expert only when a monitor detects deviation from productive progress.The expert intervention both restores the trajectory and helps ensure task completion.

4 Experiments

OpenMobile is evaluated on three dynamic mobile-agent benchmarks using trajectories that preserve learner errors as context while training on expert steps. Models trained on this data outperform open-data baselines, generalize beyond AndroidWorld, and remain dependent on base-model capability.

  • Training setup: OpenMobile retains expert action steps while preserving learner errors in the interaction history to expose realistic error-recovery scenarios.The study fine-tunes Qwen2.5-VL-7B and Qwen3-VL-8B.
  • Additional experiments: Reinforcement learning does not yield significant improvements over supervised fine-tuning on dynamic benchmarks, although the synthesized trajectories remain effective.The experiments include step-level and trajectory-level agentic reinforcement learning.
  • Evaluation setup: OpenMobile evaluates models on AndroidWorld, AndroidLab, and MobileWorld, emphasizing dynamic settings that test error recovery.The evaluation includes unseen apps and long-horizon cross-app workflows.
  • Main results: OpenMobile models significantly outperform corresponding baselines across all three benchmarks and generalize to unseen apps and long-horizon cross-app tasks.On MobileWorld, performance improves from 9.4% to 17.4%, exceeding a 50% relative improvement.
  • Interpretation: Qwen3-VL consistently outperforms Qwen2.5-VL despite identical training data, showing that base-model capability remains critical.The authors identify GUI understanding and planning as examples of underlying capabilities that affect the performance ceiling.

5 Analysis

The analysis attributes OpenMobile’s effectiveness to decoupled task synthesis, error-intervention rollout, broad functionality coverage, and limited dependence on test-similar instructions. Removing highly similar data becomes harmful at larger ratios because it removes core functionality coverage.

  • Analysis overview: OpenMobile’s ablations examine task synthesis, rollout strategies, effectiveness drivers, and potential benchmark overfitting.The analysis compares design choices and investigates why the resulting data improves performance.
  • Task synthesis: OpenMobile instructions are judged more challenging than baseline instructions while maintaining comparable soundness in 50 pairwise human comparisons.The coupled baseline uses a single exploration trajectory, whereas OpenMobile uses global environment memory.
  • Rollout strategy: Error-intervention switching achieves the best downstream performance by introducing richer error-recovery signals during trajectory rollout.The trained model also shows strengthened error awareness, diagnosis, and correction during live execution.
  • Benchmark overfitting: Only 3.5% of synthetic instructions exceed 0.7 similarity with AndroidWorld instructions, indicating moderate functionality-level rather than task-level overlap.Removing 10% of the most test-similar instructions causes only a marginal performance drop.
  • Benchmark overfitting: At a 40% removal ratio, performance degrades notably because highly similar instructions also encode core app functionalities needed for training.The comparison is against random removal.
  • Functionality coverage: OpenMobile’s functionality coverage increases with instruction count and consistently exceeds the coupled pipeline’s coverage.Global environment memory and semantically related functionality retrieval support broader, more diverse instruction synthesis.
  • Functionality coverage: Tasks requiring more atomic functionalities are harder, while higher synthetic-data coverage is associated with higher success rates.The analysis treats task complexity and functionality coverage jointly.

6 Conclusion

OpenMobile is an open framework for synthesizing mobile-agent tasks and trajectories. Its data supports strong performance and generalization while analyses attribute gains to functionality coverage and error recovery rather than benchmark overfitting.

  • OpenMobile decouples exploration from instruction generation to produce diverse tasks and uses policy switching to add error-recovery signals.
  • Agents trained on OpenMobile data achieve strong performance and generalize to unseen dynamic environments, narrowing the gap with closed-data systems.
  • Transparent overlap analyses indicate that performance gains stem from broad functionality coverage and enhanced error-recovery capability rather than benchmark overfitting.
  • The authors release the data and code to support broader open mobile-agent research.

A Implementation Details of OpenMobile Framework

OpenMobile constructs global environment memory from exploration by deduplicating screens, annotating their functionalities, and indexing those descriptions for retrieval across screens.

  • Global environment memory is built through screen deduplication, functionality annotation, and semantic index construction.
  • Screens are deduplicated with perceptual hashing, clustering screenshots whose similarity exceeds τ = 0.95 and retaining one representative per cluster.
  • A strong language model extracts functionality descriptions from each screen using the preceding screen and action as context.
  • Per-app functionality embeddings support cross-screen retrieval, while greedy filtering keeps retrieved functionalities semantically distinct with pairwise cosine similarity below 0.8.

A.3 Memory-Augmented Task Synthesis

Memory-augmented task synthesis uses global environment memory to provide a vision-language model with screen, transition, and retrieved-functionality context for grounded multi-step instruction generation.

  • OpenMobile presents global environment memory to a strong vision-language model to generate grounded, multi-step task instructions.
  • For each candidate screen, the synthesis prompt combines its screenshot and functionality annotations with neighboring-screen context from the transition graph.
  • Algorithm 1 constructs memory by collecting screens from exploration trajectories, deduplicating them, extracting functionalities, and building per-app retrieval indexes.
  • Generated instructions are filtered by language-model scores for complexity, clarity, and reasonableness, then semantically deduplicated and capped per app.

A.4 Error-Intervention Policy Switching

OpenMobile’s prompting and rollout procedures are designed to produce executable, diverse, high-level tasks and trajectories enriched with error-recovery behavior through learner–expert intervention.

  • Error-Intervention Policy Switching: Error-intervention switching starts with learner execution and invokes an expert when a monitor detects deviation from the task objective.
  • Error-Intervention Policy Switching: The monitor evaluates recent action history and the last two screenshots, and its deviation analysis is provided to the expert to improve recovery signals.
  • Functionality Extraction: Functionality extraction analyzes only the resulting screenshot and records each element’s type, label, description, hierarchy, and device-level role.
  • Task Instruction Synthesis: Task prompts require clear, explicit, executable, diverse, long-range, high-level instructions that combine multiple functions without prescribing step-by-step actions.
  • Task Instruction Synthesis: Generated tasks should begin from the phone’s home screen and avoid temporary interface states such as popup dialogs.
  • Task Instruction Synthesis: Tasks must respect the offline virtual-device setting by avoiding internet or login requirements while using data already saved in apps.

B.1 Benchmark Evaluation Setup

The evaluation uses three dynamic mobile-agent benchmarks and accounts for execution randomness through repeated runs, variation estimates, and Pass@3. Policy-switching ablations use Qwen2.5-VL-7B-Instruct as learner and Gemini-3.1-Pro-Preview as expert.

  • Evaluation covers AndroidWorld, AndroidLab, and MobileWorld, with all models deployed using vLLM.
  • Each benchmark runs three times, reporting the mean and half-range to quantify variation across dynamic executions.The half-range is defined as (max − min) / 2.
  • Pass@3 reports success when any of three runs solves a task, indicating an empirical performance upper bound.
  • Policy-switching ablations use Qwen2.5-VL-7B-Instruct as learner π_l and Gemini-3.1-Pro-Preview as expert π_e.

C Exploration with Reinforcement Learning

The paper explores step-level and trajectory-level reinforcement learning on synthesized mobile-agent data. Both approaches improve in some settings but do not consistently surpass supervised fine-tuning on dynamic benchmarks.

  • Step-Level RL: Step-level GRPO initially improves AndroidWorld performance but saturates and ultimately fails to surpass the SFT baseline.
  • Step-Level RL: Step-level RL optimizes format, action-type, and grounding rewards, but can overfit single-step outputs rather than sustain long-horizon interaction.
  • Trajectory-Level Agentic RL: Trajectory-level agentic RL improves performance but cannot consistently surpass its fully SFT-trained counterpart.
  • Trajectory-Level Agentic RL: The trajectory-level experiments retain 244 instructions that an early checkpoint fails while the expert successfully completes.
  • Trajectory-Level Agentic RL: The paper attributes trajectory-level RL’s limitations to limited environment diversity and instability in the RL framework.

D Additional Experimental Results

Additional experiments examine scaling, comparisons with other synthesis methods, and similarity between synthetic and benchmark instructions. OpenMobile performs strongly with moderate data scale while showing limited task-level overlap with AndroidWorld tests.

  • Comparison with data synthesis methods: OpenMobile reaches 64.7 AndroidWorld task success with 2.8K instructions, exceeding the listed synthesis-method results.
  • Scaling to larger models: Larger Qwen2.5-VL-72B-Instruct models yield notably stronger performance when fine-tuned on the same OpenMobile data.
  • Comparison with data synthesis methods: Direct comparison with existing synthesis methods is imperfect because base models, experimental settings, and implementation transparency differ.
  • Similarity Between Synthetic and Test Instructions: Only 3.5% of synthesized instructions exceed cosine similarity 0.7 with AndroidWorld tests, indicating functionality-level relevance rather than task-level overlap.
  • Similarity Between Synthetic and Test Instructions: The appendix lists nearest synthetic–test instruction pairs for AndroidWorld test instructions.
  • Additional task examples: The additional examples cover browser tasks, camera actions, file management, contacts, expenses, notes, timers, and media operations.
Loading 2604.15093v1…