Source-linked AI summary
PhoneWorld: Scaling Phone-Use Agent Environments
Yuxuan Liu, Xin Lai, Junyi Li, Pengyuan Lyu, Jason, Yiduo Guo, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Huawen Shen, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Chengquan Zhang, Han Hu, Benyou Wang, Ji-Rong Wen, Rui Yan, Zhengyang Tang
TL;DR
Phone-use agents lack scalable, reproducible environments for realistic mobile behavior. PhoneWorld converts real GUI trajectories and screenshots into controllable environments, tasks, verifiers, and rollouts, and broad supervision improves all four evaluation benchmarks, including HYMobileBench by 17.7 points.
Problem
Phone-use-agent progress is constrained by the limited supply of reproducible environments for visually rich, stateful interactions across many mobile apps.
Method
PhoneWorld uses real GUI trajectories and screenshots to construct controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts.
Results
Under a matched training budget, broad PhoneWorld supervision improves all four evaluation benchmarks, raising HYMobileBench by 17.7 points.
Takeaways & Limitations
Scaling app coverage under a fixed PhoneWorld budget yields the broadest gains and is the paper’s strongest scaling signal.
Takeaways & Limitations
Generated apps selectively abstract real apps rather than fully replicating their feature coverage or system fidelity, and cross-app workflows remain difficult.
Abstract
from arXiv · showhide
A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Existing mobile-agent benchmarks have made important progress on evaluation, but they do not by themselves provide a scalable way to construct many new phone-use environments. We present PhoneWorld, a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts. Rather than hand-building one mobile benchmark at a time, PhoneWorld uses real trajectories to recover which screens matter, how screens connect, which interactions must change environment state, and which user goals admit automatic verification. From these signals, it builds runnable mock Android apps backed by read-only app content and mutable state, then derives executable tasks, rule-based verifiers, and training rollouts from the same environments. In its current instantiation, PhoneWorld covers 34 apps across 16 domains, spanning common consumer mobile behaviors such as search, browsing, shopping, booking, media, and social interaction. Under a fixed training budget, replacing 10K steps from an auxiliary AndroidWorld corpus in an AndroidWorld-based baseline with broad PhoneWorld supervision improves all four evaluation benchmarks at once, raising HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points. We then study two additional scaling questions: increasing the amount of PhoneWorld supervision strongly improves PhoneWorld performance, and under a fixed PhoneWorld budget, expanding app coverage yields even larger gains. Overall, PhoneWorld shifts the focus from building one mobile benchmark at a time to scaling the supply of phone-use environments themselves.
1 INTRODUCTION
PhoneWorld addresses the bottleneck of scaling controllable phone-use environments by converting real GUI trajectories into reusable environments, executable tasks, automatic verifiers, and training rollouts. Its 34-app instantiation supports both evaluation and training, and scaling results show consistent gains from broader supervision and app coverage.
- Pipeline: PhoneWorld converts real GUI trajectories and screenshots into controllable environments, executable tasks, automatic verifiers, and training rollouts.Trajectories guide recovery of important screens, navigation flows, persistent state changes, and automatically verifiable user goals.
- Pipeline: Recovered screen inventories, transition graphs, and state-changing interactions drive resettable, inspectable mock Android apps with read-only content and mutable state.The same environments yield executable tasks, programmatic verifiers, and successful training rollouts while remaining grounded in real mobile behavior.
- Empirical scaling: Replacing 10K auxiliary AndroidWorld training steps with broad PhoneWorld supervision improves HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points.The improvement occurs under a matched total training budget; PhoneWorld supervision is strong but complementary to AndroidWorld data.
- Empirical scaling: More PhoneWorld supervision and broader app coverage both improve performance, with app coverage the strongest scaling signal under fixed PhoneWorld budgets.The scaling studies separately vary supervision amount and app coverage to examine their effects.
- Reinforcement learning: PhoneWorld is RL-ready because its automatic verifiers provide mock-app rewards, and a real+mock RL case study improves real-phone evaluation and AndroidWorld beyond real-app RL alone.This extends the pipeline’s use from evaluation and supervised rollout generation to reinforcement learning.
2 RELATED WORK
Prior mobile-agent benchmarks enable rigorous online evaluation in built environments, while scalable synthesis work addresses automated environment construction. PhoneWorld combines these directions for phone use by building runnable apps that also produce executable tasks, automatic checks, and training rollouts.
- Mobile-agent benchmarks: AndroidWorld, MobileWorld, and MobileBench-OL advance rigorous mobile-agent evaluation through real-world Android tasks, including online checking and increasingly realistic interactions.AndroidWorld provides programmatic task initialization, success checking, and reset logic; MobileWorld emphasizes longer-horizon, cross-app tasks.
- Mobile-agent benchmarks: These benchmarks primarily evaluate environments that have already been built, whereas PhoneWorld targets scalable construction of many new phone-use environments for mainstream consumer apps.PhoneWorld addresses environment creation rather than only evaluation.
- Scalable environment construction: Related scalable-construction work generates functional websites, finite-state interactive websites, and other environments for more general agents.InfiniteWeb uses task-centric specifications and verifiable evaluators, while AutoWebWorld models web environments as finite-state machines.
- Evaluation and training data: PhoneWorld extends the evaluation-and-data-generation view to phone use by producing executable tasks, automatic checks, and successful supervised training rollouts from runnable apps.Its pipeline supports both evaluation goals and training data generation.
3 METHOD
PhoneWorld combines real-app screenshots and exploratory usage episodes to recover prioritized screens, navigation, and interactions, then builds audited mock Android apps with executable tasks, deterministic verification, and training rollouts. Its controllable environments support repeatable evaluation and scalable supervision, including a 7,936-task pool that produced 3,354 verifier-confirmed episodes.
- Input recovery: Screenshots reveal app appearance and content, while real usage episodes reveal actual interactions, page priorities, and functional behavior.PhoneWorld uses both sources to create environments that are visually grounded while preserving how the app is used.
- Structure recovery: A page taxonomy, trajectory-derived priorities, and a transition graph recover the functional skeleton that guides environment construction.The pipeline concentrates effort on the screens and navigation relationships most relevant to phone-use agents.
- Environment construction: Per-page PRDs specify layout, interactive elements, transitions, and visual attributes for an iterative Kotlin/Jetpack Compose build-and-review loop.The coding agent compiles APKs, runs self-review checks, and fixes navigation, data-loading, and rendering issues across iterations.
- Environment construction: Human-audited smoke tests validate core flows after autonomous construction, including launch, tab switching, search, navigation, and representative write operations.Automatic checks catch common issues, while human reviewers focus on high-impact discrepancies against the real app.
- Task synthesis and rollouts: 7,936 generated tasks yielded 3,354 verifier-confirmed successful episodes totaling 36,193 interaction steps for the PhoneWorld training corpus.Tasks are grounded in read-only content, database schema, and page specifications, enabling achievable goals and deterministic verification; resettable environments also support repeated training and evaluation.
4 EXPERIMENTAL SETUP
The experiments study PhoneWorld scaling through controlled supervised fine-tuning, matched-budget corpus composition, and evaluation across four benchmarks covering transfer and in-domain performance. Models use a common multimodal setup and standardized evaluation protocols.
- Scaling questions: The study varies PhoneWorld supervision under matched total training budgets, including a full-replacement control testing complementarity with auxiliary AndroidWorld data.The shared AndroidWorld base corpus remains fixed while the remaining training steps vary in composition.
- Training setup: Qwen3.5-9B is fine-tuned for two epochs with the same training setup, hyperparameters, and multimodal input format across conditions.Inputs contain a system prompt, current screenshot, user instruction, and textual summary of previous actions; outputs are the next thought-and-action prediction.
- Training data: 36,193 interaction steps form each of the shared and auxiliary AndroidWorld corpora, while verifier-confirmed PhoneWorld rollouts provide a third training corpus.The shared corpus comes from Gemini 3.1 Pro rollouts, the auxiliary corpus from Seed 2.0 Pro, and the PhoneWorld corpus from Seed 2.0 Pro on generated PhoneWorld tasks.
- Training data: The Baseline uses 72,386 AndroidWorld steps, whereas the 10K PhoneWorld replacement model substitutes 10,000 auxiliary steps with PhoneWorld successful-rollout steps.Both retain the shared AndroidWorld base corpus; the replacement model keeps 26,193 auxiliary AndroidWorld steps.
- Evaluation: Four benchmarks measure offline transfer, real-app online transfer, and in-domain performance: HYMobileBench, AndroidControl, AndroidWorld, and PhoneWorld.HYMobileBench and AndroidControl report step SR; AndroidWorld and PhoneWorld report task SR, with PhoneWorld using held-out audited tasks.
- Evaluation: PhoneWorld evaluation uses Android 13 Pixel 6 emulators with the same runner, app release, and model-serving setup for all compared models.The current setup uses six emulators and three vLLM instances.
5 RESULTS
PhoneWorld improves phone-agent performance under matched training budgets, with gains driven especially by broader app coverage. Its mock-app environments also improve downstream reinforcement learning and transfer to external real-app evaluation.
- Matched-budget replacement: 17.7 points on HYMobileBench, 6.0 on AndroidControl, 14.7 on AndroidWorld, and 52.5 on PhoneWorld result from replacing 10K auxiliary AndroidWorld steps.Both models use the same Qwen3.5-9B backbone, training setup, 72,386-step budget, and shared AndroidWorld base corpus.
- Matched-budget replacement: Full PhoneWorld replacement yields +60.8 on PhoneWorld, +17.7 on HYMobileBench, and +5.6 on AndroidControl, while AndroidWorld falls by 10.3 points.The result shows PhoneWorld is powerful alone, while the two supervision sources are complementary rather than interchangeable.
- Supervision scaling: 14.2 to 64.2, 70.0, and 73.3: PhoneWorld task success rises monotonically as supervision increases from 0 to 10K, 20K, and 36,193 steps.The largest gain occurs in the first 10K steps, with smaller but positive returns thereafter.
- App-coverage scaling: Under a fixed 10K-step budget, expanding coverage from 5 to 34 apps raises PhoneWorld by 18.3 points, HYMobileBench by 18.3, and AndroidWorld by 10.4.AndroidControl remains roughly stable overall, indicating that breadth of environment coverage is the strongest scaling signal.
- Downstream reinforcement learning: 40.67% to 45.33% on real-phone evaluation and 77.2% to 83.2% on AndroidWorld: adding PhoneWorld mock-app RL improves over real-app RL alone.The real-phone evaluation contains 150 human-judged tasks, with 50 tasks per category.
6 DISCUSSION AND LIMITATIONS
The discussion identifies PhoneWorld’s empirical strengths, its complementarity with AndroidWorld, and limitations arising from selective abstraction, compact scope, and incomplete fidelity.
- Empirical findings: Five findings define the empirical story, including gains from partially replacing auxiliary AndroidWorld training steps with broad PhoneWorld supervision under a fixed budget.The discussion also covers full replacement, two scaling analyses, and a downstream reinforcement-learning case study.
- Relationship to AndroidWorld: PhoneWorld provides strong supervision and controllable infrastructure for mainstream consumer phone-use behaviors, while AndroidWorld supplies a distinct real-app transfer signal.The strongest matched-budget supervised fine-tuning setting therefore retains most of the auxiliary AndroidWorld contribution.
- Limitations: PhoneWorld’s generated apps are selective abstractions rather than full replicas, preserving important screens, state changes, and interaction paths without complete feature coverage or perfect system fidelity.The benchmark is intentionally compact and manually audited, improving stability while not exhausting the behavior space of the full PhoneWorld suite.
7 CONCLUSION
PhoneWorld is a reusable pipeline that converts real mobile GUI data into controllable environments, executable tasks, verification rules, and successful rollouts for evaluation and training. Experiments show that broad PhoneWorld supervision improves all four benchmarks under a matched budget, works best with AndroidWorld, and benefits from scaling supervision and app coverage.
- Contribution: PhoneWorld builds controllable phone-use environments from real GUI trajectories and screenshots, rather than handcrafting one benchmark at a time.The pipeline converts real mobile usage data into runnable environments.
- Contribution: The pipeline derives executable tasks, verification rules, and successful rollouts from the same environments for both evaluation and training.This makes PhoneWorld useful as both a benchmark and infrastructure for generating supervision.
- Experimental conclusions: Under a matched training budget, replacing auxiliary AndroidWorld steps with broad PhoneWorld supervision improves all four benchmarks simultaneously.The comparison uses partial replacement of steps from the auxiliary AndroidWorld corpus.
- Experimental conclusions: Full replacement shows that PhoneWorld supervision is strong independently but performs best when combined with the auxiliary AndroidWorld corpus.This conclusion follows the full-replacement experiment described in the passage.
- Scaling conclusions: Scaling PhoneWorld supervision mainly strengthens PhoneWorld performance, while scaling app coverage under a fixed PhoneWorld budget is also studied.The passage identifies both supervision amount and app coverage as central scaling dimensions.
AVAILABILITY
PhoneWorld plans to release its code, benchmark tasks, runner, and documentation, while distributing mocked APKs through a gated dataset for non-commercial academic research.
- PhoneWorld plans to release its code, benchmark tasks, runner, and documentation on GitHub.The planned release is available at the specified PhoneWorld repository URL.
- The mocked APKs will be distributed through a gated Hugging Face dataset for non-commercial academic research.Users must independently accept the dataset terms rather than receiving redistributed APK mirrors.
A SUPPLEMENTARY ADD-ONLY ANALYSIS FOR FULL REPLACEMENT
This supplementary analysis reports AndroidWorld and PhoneWorld results when increasing amounts of PhoneWorld supervision are added to a fixed AndroidWorld base corpus. Existing AndroidWorld data is retained throughout.
- Add-only supervision scaling: The runs add increasing amounts of PhoneWorld supervision while keeping the shared AndroidWorld base corpus fixed and removing no existing AndroidWorld data.Table 6 reports the resulting AndroidWorld and PhoneWorld performance.