PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools

Chenyang Yan, Yingying Zhang, Kanzhi Cheng, Qiushi Sun, Zhengyuan Pan, Zheng Ma, Hang Yan, Yian Wang, Nuo Chen, Jialin Cao, Xingdong Gong, Wenpo Song, Han Chen, Zichen Ding, Fangzhi Xu, Shujian Huang, Xinyu Dai, Yichen Liu, Tiankuo Yao, Bo Wang, Ben Kao, Jianbing Zhang, Lewei Lu, Dahua Lin

arXiv:2610.14157v1cs.SEcs.CLcs.CV

TL;DR

Open mobile-agent training depends on interaction experience, but high-quality data remain largely proprietary and controllable commercial-app environments are difficult to scale. OpenMobile-2 builds an open stack of environments, data, and recipes, and reports competitive benchmark performance with transfer to real-device use.

  • Problem

    Mobile-agent progress depends on large-scale interaction experience, yet collecting it is costly and high-quality data remain largely proprietary, while controllable commercial-app environments are difficult to scale.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    OpenMobile-2 combines controllable commercial-style apps, open training trajectories and executable RL tasks, and app-native tools that agents can interleave with GUI actions.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    SPA-Bench performance improves from 31.9 to 59.6, with competitive results of 79.9 on AndroidWorld and 50.4 on MobileWorld.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that app-native tools are complementary: tool access alone brings limited gains, while hybrid training substantially amplifies their benefits.

    Source: §6 · p. 10

Abstract

from arXiv · show

We introduce OpenMobile-2, a near-frontier mobile agent with fully open training environments and recipes. We make three key advances: (1) Diverse environments with simulated commercial apps: We build MobileGym++, featuring 35 realistic, functionally rich commercial-style apps with cross-app workflows, while preserving full controllability for reset and verification. Together with newly configured apps in Android emulators, this yields a diverse playground spanning over 110 apps. (2) Open training data at scale: Building on this foundation, we curate nearly 12K mobile interaction trajectories for supervised fine-tuning and the largest open collection of verifiable RL training data for mobile agents, comprising over 2K executable tasks with automatic rewards. (3) Hybrid GUI and app-native tool use: We explore an experimental mobile-use setting where apps expose selected functionalities as app-native tools alongside their GUIs, allowing agents to interleave GUI actions and tool calls within a task. We implement this setting in MobileGym++ with over 300 carefully scoped tools across 50+ apps. Additionally, we introduce MobileGym++ Bench for complex, long-horizon commercial mobile scenarios, supporting both GUI-only and hybrid GUI-tool evaluation on a shared task suite. OpenMobile-2 performs competitively across established benchmarks, including AndroidWorld (79.9) and MobileWorld (50.4), while showing promising transfer to real-device mobile use, nearly doubling SPA-Bench performance from 31.9 to 59.6. All resources are available at https://os-copilot.github.io/OpenMobile2-Home/.

1 INTRODUCTION

OpenMobile-2 addresses the shortage of open mobile-agent training resources and the difficulty of scaling controllable environments for rich commercial workflows. It combines diverse environments, large-scale training data, and hybrid GUI–tool interaction, with competitive benchmark results and transfer to real devices.

Motivation

  • The work targets costly data collection and limited open resources, while existing commercial-app environments have struggled to combine rich workflows with controllability at scale.Real apps can lack reliable reset, while simulated environments have often been few and shallow.Source: §1 · p. 2

Open training stack

  • OpenMobile-2 combines 35 controllable commercial-style apps, a playground spanning over 110 apps, nearly 12K trajectories, and over 2K executable RL tasks.The tasks include reliable verifiers for checking outcomes.Source: §1 · pp. 2–3

Hybrid GUI and app-native tools

  • The paper explores hybrid mobile use by exposing over 300 app-native tools across 50+ apps for interleaving tool calls with GUI actions.The tools expose selected app functionalities alongside their GUIs.Source: §1 · pp. 2–3

Benchmark

  • MobileGym++ Bench provides 215 long-horizon cross-app tasks with persistent user states and programmatic verification, allowing GUI-only and hybrid agents to be compared on the same tasks.The task suite targets commercial mobile workflows.Source: §1 · pp. 2–3

Results

  • SPA-Bench performance improves from 31.9 to 59.6, alongside competitive results of 79.9 on AndroidWorld and 50.4 on MobileWorld.The same task suite supports GUI-only and hybrid-agent evaluation.Source: §1 · p. 3

2 RELATED WORK

Related work frames interaction experience as a central resource for training mobile agents, while noting that existing open environments trade realism, controllability, or application coverage. OpenMobile-2 combines controllable commercial-style apps with app-native tools alongside the GUI.

Mobile Agents

  • Mobile agents have progressed from proprietary pipelines to end-to-end action models and interactive reinforcement learning, making interaction experience central to scaling.The experience includes trajectories, resettable environments, and verifiable RL tasks.Source: §2 · p. 3

Trajectory Data and Interactive Environments

  • Human-recorded trajectories are costly and lack executable success criteria, while interactive environments trade realism against clean reset, parallelization, and application coverage.Existing simulated applications are described as shallow, with no scalable way to capture commercial workflows.Source: §2 · p. 3

Hybrid GUI and Tool Use

  • Prior mobile hybrids typically attach external tools or system interfaces, whereas MobileGym++ exposes app-native functions alongside the GUI over shared application state.GUI actions can reach visible controls, while structured calls are limited to exposed functions.Source: §2 · pp. 3–4

Environment comparison

  • MobileGym++ is presented as providing diverse, high-fidelity commercial-style apps with full controllability and an experimental app-native tool interface.Table 1 compares mobile environments across scale, interaction, and controllability.Source: Table 1 · p. 4

3 MOBILEGYM++ PLAYGROUND

MobileGym++ addresses the shortage of controllable commercial-app environments with realistic workflows, persistent business state, and programmatic verification. Its benchmark supports long-horizon tasks and controlled comparison of GUI-only and hybrid interaction.

3.1 A GAP IN CONTROLLABLE COMMERCIAL APPLICATIONS

  • The environment targets the gap between realistic commercial workflows and controllable training, where real-device setups make reset costly and prior simulated apps cover narrow domains.The replicas are structurally designed around source-app schemas and interaction graphs, then iteratively refined against live applications.Source: §3.1 · p. 4

3.2 35 HIGH-FIDELITY COMMERCIAL APPLICATIONS

  • 35 high-fidelity commercial app replicas bring the environment to 63 installable clients, with persistent business state, deterministic reset, and programmatic task verification.New apps reach 1.48× the interactive actions and 1.91× the page transitions of the 12 prior MobileGym business apps.Source: §3.2 · pp. 4–5, §3.1 · p. 4, Figure 3 · p. 5

3.3 REALISTIC CROSS-APPLICATION WORKFLOWS

  • MobileGym++ supports cross-app workflows through clipboard transfer, sharing, payment redirection, and media selection on a shared resettable device.These mechanisms let information and actions flow between apps rather than restricting tasks to isolated applications.Source: §3.3 · p. 5

3.4 HYBRID GUI AND APP-NATIVE TOOL USE

  • MobileGym++ exposes app-native tools alongside GUIs, with tools and GUI counterparts operating on the same persistent state and terminal condition.Over 300 tools span more than 50 apps, and tasks remain solvable through GUI-only interaction.Source: §3.4 · p. 5

3.5 MOBILEGYM++ BENCH

  • MobileGym++ Bench evaluates intended changes in application business state, with GUI-only and hybrid agents using the same task suite, initial states, and checks.It comprises 215 tasks across seven domains, including single-app and multi-app workflows; success rate is the primary metric.Source: §3.5 · pp. 5–6

4 OPENMOBILE-DATA-V2

OpenMobile-Data-v2 provides demonstration trajectories and executable RL tasks collected across Android emulators and MobileGym++. Its construction uses generated goals, verified teacher rollouts, and filtering, and the resulting data span diverse domains, goals, and workflows.

4 OPENMOBILE-DATA-V2

  • OpenMobile-Data-v2 contains 11.6K demonstration trajectories and over 2K executable RL tasks, including 1.7K demonstrations that interleave GUI actions and app-native tools.Each RL task includes a goal, initial state, and programmatic reward, and can run in GUI-only or hybrid mode.Source: §4 · p. 6

4.1 CONSTRUCTION PIPELINE

  • The construction pipeline generates executable user instructions from application state–action views, then retains teacher-agent rollouts that complete tasks and pass runtime-specific verification.Step-level filtering removes redundant interactions; demonstration trajectories preserve observation–action sequences, while RL tasks retain the instruction, initial state, and verifier.Source: §4.1 · p. 6

4.2 DATA STATISTICS

  • The full dataset has a domain Gini coefficient of 0.43, mean pairwise task-goal cosine similarity of 0.43, and 30% cross-app trajectories.The emulator subset has Gini 0.52 and 40% cross-app trajectories, while simulated clients have Gini 0.25 and 11% cross-app trajectories.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

5 EXPERIMENTS

Experiments evaluate OpenMobile-Data-v2 across emulator apps, physical devices, and simulated apps, using GUI-only and hybrid interaction settings. The results show gains across benchmarks and settings, while analysis finds broader application coverage and hybrid training improve transfer and tool use.

5.1 TRAINING AND EVALUATION DETAILS

  • Training uses five-epoch GUI-only and hybrid SFT runs; RL starts from the hybrid checkpoint and trains on 512 tasks sampled from over 2K executable tasks.Evaluation spans emulator apps, physical devices, and simulated apps, with both GUI-only and tool-augmented settings.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • 19.7 points (54.3 to 74.0) for 9B and 12.8 points (67.1 to 79.9) for 27B on AndroidWorld; OpenMobile-Data-v2 also improves MobileWorld and MobileGym++.The table covers established mobile-agent benchmarks and real-device evaluation.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • 57.4 to 59.6 on SPA-Bench for the 27B model after RL, while gains on other benchmarks remain limited.The authors suggest RL benefits depend on alignment between training environments and evaluation scenarios.Source: §5.2 · p. 8
  • Hybrid SFT raises MobileGym++ Bench hybrid success from 41.9 to 44.7 for 9B and from 57.7 to 65.6 for 27B, with comparable GUI-only success.The same tasks are evaluated in GUI-only and hybrid settings.Source: §5.2 · p. 8

5.3 ANALYSIS

  • At a fixed 2K-trajectory budget, expanding coverage from 15 to 53 applications raises held-out Pass@1 from 60.7 to 67.5, while InD Pass@1 remains largely stable at 73.7 to 71.4.The InD share drops from 100% to 39.1%.Source: §5.3 · p. 9, Figure 5 · p. 9, Table 3 · p. 9
  • After Hybrid SFT, tool-enabled gains reach 5.6 and 10.3 points, and trajectories are 4.8 and 5.8 steps shorter.Tool calls occur in 113 and 128 of 215 tasks, covering 110 and 115 distinct tools for the 9B and 27B models, respectively.Source: §5.3 · p. 9, Table 4 · p. 9
  • Before Hybrid SFT, tool calls occur in only 20–102 of 215 tasks, with limited gains of 2.8–4.2 points for Qwen models and inconsistent results for GUI-Owl-1.5.GUI-Owl-1.5 can degrade in this setting.Source: §5.3 · p. 9, Table 4 · p. 9

6 CONCLUSION

OpenMobile-2 combines diverse mobile environments, open training data, and agent models. Its resources support benchmark performance and real-device transfer, while hybrid training makes app-native tool access more useful.

6 CONCLUSION

  • The environment stack spans over 110 applications, including high-fidelity commercial-style apps with clean reset and verification.MobileGym++ contributes 35 commercial-style applications with accessible state.Source: §6 · pp. 9–10
  • The release includes nearly 12K demonstration trajectories and over 2K executable RL tasks with automatic rewards.Source: §6 · p. 10
  • Tool access alone brings limited gains, while hybrid training substantially amplifies tool benefits; the trained models also transfer to real-device evaluation.The paper releases environments, data, benchmarks, and trained models.Source: §6 · p. 10

A TRAINING AND EVALUATION DETAILS

Training combines two supervised fine-tuning runs with reinforcement learning initialized from the hybrid checkpoint. The setup uses GUI-only or hybrid interaction, then trains and evaluates with executable simulator tasks and a defined rollout procedure.

A.1 SUPERVISED FINE-TUNING

  • The two 5-epoch SFT runs differ in data: GUI-only uses GUI trajectories, while hybrid adds 1,774 simulator trajectories with app-native tool calls.Both runs share hyperparameters; RL is initialized from the hybrid checkpoint.Source: §A.1 · p. 14, Table 6 · p. 15
  • Each SFT step-level sample includes the instruction, prior interaction history, a Thought paragraph, and one tool call.The three latest screenshots remain images; older screenshots are collapsed to text.Source: §A.1 · p. 14

A.2 REINFORCEMENT LEARNING

  • RL starts from the hybrid SFT checkpoint and samples groups of eight rollouts per task using GRPO-style advantages and the DAPO objective.Updates use asymmetric clipping and a KL loss toward the SFT reference policy.Source: §A.2 · pp. 14–15
  • RL training uses 512 tasks with bound initial states and programmatic verifiers, while validation runs 256 official MobileGym tasks before training and every 10 steps.Each validation task receives one rollout.Source: §A.2 · p. 15
  • Hybrid rollouts retain screenshot history, expose foreground-app tool catalogs, and return structured receipts for tool calls.Episodes end at termination or the smaller task-specific or global step limit.Source: §A.2 · p. 15
  • The rollout reward uses a relaxed state verifier and doubles the tool multiplier when a hybrid trajectory includes an app-native call.Reported accuracy instead uses a strict verifier requiring a clean state; the reward also accounts for length, malformed turns, and false success claims.Source: §A.2 · p. 15
  • RL training runs on 32 H800 GPUs across four nodes, with LoRA adapters, a frozen vision tower, and colocated vLLM rollout generation.Each node hosts a simulator; rollouts use 64 browser pages concurrently.Source: §A.2 · p. 15

A.3 EVALUATION BENCHMARKS

The evaluation spans five benchmarks across emulator apps, physical devices, and simulated apps, with differences in available actions and completion checks. Together, these settings cover reproducible device-state tasks, longer workflows, simulator tasks, and real-device transfer.

A.3 EVALUATION BENCHMARKS

  • Five benchmarks vary app runtime, permitted actions, and completion criteria, broadening evaluation beyond a single environment.The suite includes emulator, physical-device, and simulated-app settings.Source: §A.3 · p. 15
  • AndroidWorld evaluates 116 programmatic tasks across 20 real-world apps using deterministic device-state evaluation and task success rate.The tasks run in reproducible Android environments.Source: §A.3 · p. 15
  • MobileWorld contains 201 tasks across 20 apps, emphasizes long-horizon and cross-application workflows, and is evaluated on GUI-only and MCP-enabled tasks.Evaluation uses the official task checker.Source: §A.3 · pp. 15–16
  • MobileGym uses resettable mock apps and programmatic state-based evaluation, reporting success rate and partial-completion progress rate.Its tasks require application interactions and state transitions.Source: §A.3 · p. 16
  • MobileGym++ Bench contains 215 single-app and cross-app tasks evaluated on fixed initial states with a checker over application business state.GUI-only and hybrid interactions use the same tasks, initial states, and checker.Source: §A.3 · p. 16
  • SpaBench evaluates agents on real Android devices to assess transfer beyond controlled emulator and simulator environments.It provides a real-device evaluation setting.Source: §A.3 · p. 16

A.4 OVERLAP WITH EVALUATION BENCHMARKS

Training-goal overlap is assessed by maximum embedding similarity, with no exact matches or similarities above 0.9 reported for RL goals against MobileGym++ Bench. The appendix also describes how simulated apps, playground inventories, trajectory construction, and executable RL tasks are built.

A.4 OVERLAP WITH EVALUATION BENCHMARKS

  • 6.1% (151) of the 2,475 RL goals exceed 0.7 maximum cosine similarity to MobileGym++ Bench instructions; none is an exact match or exceeds 0.9.The mean maximum similarity is 0.562 and the maximum is 0.862.Source: §A.4 · p. 16

B MOBILEGYM++ APP CONSTRUCTION

  • App workflows are specified as page nodes and meaningful UI states, with each state-changing interaction documenting its action, expected effect, and state check.The specification serves as the contract for implementing and checking routes, actions, and state changes.Source: B MOBILEGYM++ APP CONSTRUCTION · p. 17
  • GUI actions and app-native tools operate on the same app-owned state, with coverage checks and runtime replay used to verify implementation behavior.Each delivered app includes its interface, state model, seeded data, and reusable checks for core flows.Source: B MOBILEGYM++ APP CONSTRUCTION · p. 17

C PLAYGROUND INVENTORY

C.1 MOBILEGYM++ APPLICATIONS

  • The 63-client MobileGym++ inventory spans lifestyle, productivity, entertainment, shopping, travel, social, and finance apps.The inventory tables list clients by domain.Source: §C.1 · p. 18

C.2 EMULATOR APPLICATIONS

  • The Android emulator inventory covers 62 applications grouped by domain, with AndroidWorld and MiniWoB task-support apps included.The emulator inventory is not divided by source batch.Source: Table 10 · p. 19

D.1 CONSTRUCTION DETAILS

  • Task synthesis uses app functions and reachable states, while rollouts are retained only when runtime-specific checks show task success.A model may select GUI actions or foreground-app tools, both acting on the same app state; unchanged-state transitions are removed.Source: §D.1 · p. 19

D.2 EXECUTABLE RL TASKS

  • Each executable RL task comprises an instruction, a fixed snapshot of application state, and a verifier that checks the requested state change after an episode.Because verification reads app state rather than the screen, it applies to both GUI and app-native tool actions.Source: §D.2 · p. 19

D.3 DATA COMPOSITION

The SFT mixture combines emulator and simulator trajectories, including a hybrid subset that uses app-native tools alongside GUI actions. MobileGym++ Bench composition is described by the number of apps tasks touch and by domain incidences, which can overlap.

D.3 DATA COMPOSITION

  • The SFT mixture contains 11,643 trajectories: 7,725 from emulator apps and 3,918 from simulated clients.The sources contribute about two thirds and one third of the mixture, respectively.Source: §D.3 · p. 19
  • Hybrid trajectories average 6.8 steps, compared with 11.9 for AndroidWorld, 10.7 for MobileGym, and 8.8 for MobileWorld.The 1,774 hybrid trajectories combine app-native tools with GUI actions.Source: §D.3 · p. 19

E MOBILEGYM++ BENCH

  • MobileGym++ Bench composition is shown by task app-count and domain incidences; domain counts are non-exclusive because a task can touch multiple domains.The section also documents the task split and domain distribution, with worked examples of the evaluation protocol.Source: Figure 8 · p. 20, Table 11 · p. 20, E MOBILEGYM++ BENCH · p. 20

E.1 TASK COMPOSITION

The 215 tasks span one to four apps, and 106 require cross-app interaction.

E.1 TASK COMPOSITION

  • 106 of 215 tasks require cross-app interaction; 109 use one app, 74 use two, 27 use three, and 5 use four.Domain incidences overlap because a cross-domain task is counted in each domain it touches.Source: §E.1 · p. 20

E.2 EXAMPLE TASKS

The examples cover GUI-only and hybrid tasks, including app-native tools that share state with the GUI. The evaluation prompt specifies how agents select actions and format tool calls.

E.2 EXAMPLE TASKS

  • Task 202 tests GUI-only transfer of the two most-liked RedNote search-result titles into the system Notes app, verified against the initial results.The task specifically requires the full titles, one per line, in the latest Notes entry.Source: §E.2 · p. 20
  • Task 111 requires a Taobao purchase using a saved address and Huabei installments, with tools for search, item details, and cart actions.Checkout prepares only a draft; the agent must finish payment in the GUI, and the checker verifies the resulting paid order and details.Source: §E.2 · pp. 20–21

F APP-NATIVE TOOL EXAMPLES

  • App-native tools share the GUI’s app state and action slot, and are available only while their app is foregrounded.The Taobao tools read catalog data, update the shared cart, or open a checkout confirmation draft without creating an order or payment.Source: F APP-NATIVE TOOL EXAMPLES · p. 21

G SYSTEM PROMPT

  • The evaluation harness serves Qwen3.5-9B, Qwen3.6-27B, and OpenMobile-2 checkpoints in both GUI-only and hybrid settings; only hybrid prompts include foreground-app tool catalogs.Other models marked with an asterisk in Table 2 use their own action formats and system prompts.Source: G SYSTEM PROMPT · p. 21
  • The prompt requires one tool call per assistant turn in the specified XML format, with reasoning before the call and no text after it.Agents should use only currently listed app tools, inspect returned screenshots, and terminate through mobile_use when the task is complete or cannot be completed.Source: G SYSTEM PROMPT · p. 22

Related papers

Loading 2610.14157v1…