Source-linked AI summary
PhoneBuddy: Training Open Models for Agentic Phone Use
Zhengyang Tang, Xin Lai, Pengyuan Lyu, Xinyuan Wang, Tianyi Bai, Chenxin Li, Yiduo Guo, Huawen Shen, Yuxuan Liu, Junyi Li, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Ji-Rong Wen, Rui Yan, Chengquan Zhang, Han Hu
TL;DR
Reliable phone-agent training is difficult because deployment uses real devices and authentic apps that are stateful, side-effectful, and costly to reset or verify. PhoneBuddy combines real-app training with PhoneWorld mock-app training through shared supervised fine-tuning and reinforcement learning across both environments. Mixed training improves task success on both real-phone and AndroidWorld evaluations, while cross-app workflows remain a key limitation.
Problem
Training open models for reliable real-phone task completion remains unresolved because real-app environments are realistic but expensive to scale, reset, and verify, whereas mock environments may not transfer fully.
Method
PhoneBuddy uses a shared supervised fine-tuning stage followed by reinforcement learning across complementary real-app and PhoneWorld mock-app environments.
Results
Mixed real-app and mock-app RL improves task success from 36.67% to 45.33% on a 150-task real-phone evaluation and from 60.3% to 83.2% on AndroidWorld.
Takeaways & Limitations
Mock-app training complements rather than replaces real-app RL by providing scalable, resettable, and automatically verifiable interaction grounded in realistic GUI structure.
Takeaways & Limitations
Cross-app workflows remain a major limitation because current PhoneWorld coverage lacks direct multi-app information handoff, artifact transfer, and persistent cross-app state dependencies.
Abstract
from arXiv · showhide
Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67\% after supervised fine-tuning to 40.67\% after real-app RL and 45.33\% after mixed RL. On AndroidWorld, the same progression rises from 60.3\% to 77.2\% to 83.2\%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.
1 Introduction
PhoneBuddy addresses the difficulty of training open models for reliable real-phone task completion by combining realistic real-app interaction with scalable mock-app training. Its mixed training recipe improves success on real-phone and AndroidWorld evaluations, while cross-app workflows remain a major challenge.
- Problem setting: Real phone tasks are stateful, permission-rich, and side-effectful, involving app logic, device settings, user data, and server-side behavior.These conditions make reliable task completion harder than static GUI grounding.
- Problem setting: The paper frames agentic phone use as a training problem for open models rather than only a GUI grounding problem.The unresolved question is how to improve real-phone task completion instead of merely local action imitation or benchmark-specific interaction.
- Training recipe: PhoneBuddy combines a real-app environment with PhoneWorld, a mock-app environment reconstructed from real GUI usage structure.The real-app setting provides realism, while PhoneWorld supports scalable, resettable, and automatically verifiable interaction.
- Results: 36.67% to 45.33%: mixed real-app and mock-app RL improves task success on a 150-task real-phone human evaluation.The evaluation spans apps, mini-apps, and cross-app workflows; real-app RL alone reaches 40.67%.
- Results: 60.3% to 83.2%: the same progression improves task success on AndroidWorld.The intermediate real-app RL checkpoint reaches 77.2%.
- Limitations: PhoneWorld-driven gains are strongest on app and mini-app tasks, whereas cross-app workflows remain a major open challenge.The paper identifies long-horizon state tracking, information handoff, and runtime verification as remaining needs.
2 Background
Phone-agent research has progressed from static screen understanding toward multi-step operation of real software, but training still faces a realism–scalability tradeoff. Runtime execution is also part of deployable agent capability because agents must convert model outputs into controlled, checked interaction.
- GUI-agent foundations: Web and desktop GUI-agent benchmarks show that software tasks require grounding, planning, tool use, and robust execution beyond isolated perception.These developments motivate training models to complete real phone tasks rather than only predict plausible next actions.
- Environment tradeoffs: Real applications provide high-fidelity behavior, but collecting trajectories, resetting state, and verifying outcomes are expensive.Synthetic or reconstructed environments reduce interaction cost but must preserve enough structure to transfer to real software.
- Runtime execution: A deployable agent needs a runtime harness defining observations, actions, parsing, execution, budgets, logging, and completion checks.The harness may coordinate GUI actions, tool calls, CLI commands, or other execution channels.
3 Method
PhoneBuddy trains phone-use agents across complementary real-app and mock-app environments, using shared supervised fine-tuning followed by alternative reinforcement-learning branches. Real apps provide deployment fidelity, while PhoneWorld provides scalable, resettable, and verifiable interaction.
- Training recipe: PhoneBuddy compares a shared SFT baseline, real-app RL, and mixed RL across real-app and mock-app environments.All checkpoints share the same backbone, SFT initialization, action interface, and evaluation protocol.
- Real-app environment: The real-app environment exposes authentic app behavior, device state, timing variation, permissions, and user-facing side effects.It also reveals account-dependent behavior, app instability, and gaps between apparent progress and completed tasks.
- Environment trade-offs: Real-app training is slower and harder to reset or verify, while exploration can produce real and sometimes irreversible side effects.PhoneBuddy therefore uses real-app interaction selectively rather than relying on it alone.
- Mock-app environment: PhoneWorld provides runnable mock apps reconstructed from real GUI traces, with writable state, resettable interaction, and rule-based task verification.Its pipeline recovers relevant screens, transitions, actions, and state changes before building executable Android apps and verifiers.
- Training recipe: The shared SFT stage uses trajectories from both environments before branching into real-app-only RL and mixed real-plus-mock RL.This common starting point isolates the effect of the final reinforcement-learning branch.
4 Experimental Setup
PhoneBuddy is evaluated on three 50-task real-phone categories and AndroidWorld using fixed interaction and evaluation settings. The protocol reports task success rate while holding the action space, prompts, execution harness, and step budgets consistent across checkpoints.
- Benchmarks: The real-phone evaluation contains 50 Single-App, 50 Cross-App, and 50 WeChat Mini-App tasks, totaling 150 tasks.AndroidWorld is the fourth evaluation setting.
- Controls: The action space, prompt template, step budget, evaluation harness, inference setup, and execution setup remain fixed across compared checkpoints.Training recipe is the only changed experimental factor.
- Interaction protocol: The action space is a shared phone-control API using normalized coordinates in the [0, 1000] range and fixed phone actions.Actions include clicking, typing, scrolling, dragging, navigation buttons, app opening or closing, and waiting.
- Metrics: Task success rate is the metric across all four settings, with real-phone tasks counted successful only when fully completed.The main table also reports an unweighted mean across the four settings.
- Compared models: The comparison includes PhoneBuddy-4B-SFT, PhoneBuddy-4B-Real, and PhoneBuddy-4B-Real+Mock, alongside representative closed-source systems.The three internal checkpoints differ by supervised fine-tuning, real-app RL, or mixed RL training.
5 Main Results
PhoneBuddy-Real+Mock delivers the strongest overall performance, with gains concentrated in single-app, mini-app, and AndroidWorld settings while cross-app tasks remain difficult.
- Overall: 54.8 average task success rate makes PhoneBuddy-4B-Real+Mock the best overall internal model, 12.2 points above SFT and 5.0 points above real-app RL.It surpasses GPT-5.4 and Seed 2.0 Pro on this average but remains below Gemini 3.1 Pro overall.
- RL Branches: The second RL delta measures the additional effect of mock-app RL on top of real-app RL, with improvements highlighted for Single-App, WeChat Mini-App, and AndroidWorld settings.Cross-App tasks remain difficult.
- Single-App Tasks: 62.0% on Single-App tasks exceeds 34.0% for SFT and 54.0% for real-app RL, surpassing all compared closed models.The full recipe produces the clearest gain in this setting.
- Cross-App Tasks: 18.0% on Cross-App tasks trails the SFT and real-app RL checkpoints at 22.0% and 20.0%, respectively, showing no meaningful improvement.The current mock-app task pool does not directly model cross-app information handoff and persistent dependencies.
- WeChat Mini-App Tasks: 56.0% on WeChat Mini-App tasks exceeds the SFT baseline of 54.0% after real-app RL alone drops performance to 48.0%.PhoneWorld is especially helpful for multi-step workflows with stable structure and verifiable state changes.
- AndroidWorld: 83.2% on AndroidWorld follows 60.3% after SFT and 77.2% after real-app RL, establishing the cleanest monotonic trend.AndroidWorld is outside the real-phone human evaluation suite, supporting transfer beyond the internal benchmark.
6 Qualitative Examples
Representative trajectories show that mixed-environment RL improves execution of task constraints and information transfer beyond aggregate success-rate gains.
- Constraint Following: In Tongcheng Travel, PhoneBuddy-Real+Mock applies the budget constraint and reduces the hotel budget to 150 yuan, unlike SFT.The task involves finding budget-friendly hotels near Shanghai Disneyland in a WeChat mini-app.
- Information Transfer: In Tencent Docs, PhoneBuddy-Real+Mock correctly transfers Yuanbao’s generated leave request note, whereas SFT inserts stale clipboard content.The workflow requires generating the note with Yuanbao and saving it in a newly created document.
- Interpretation: Mixed-environment RL supplies supervision for following task constraints and performing complex information-transfer operations.These qualitative cases extend the aggregate success-rate comparison with concrete execution behavior.
7 Discussion and Limitations
PhoneBuddy’s results support combining real-app RL with PhoneWorld because each addresses a different training constraint. Cross-app execution remains limited, and deployable agents additionally require runtime, privacy, and safety systems outside this paper’s empirical focus.
- Why Real+Mock Works: Real-app RL and PhoneWorld are complementary: real apps provide realism and side effects, while PhoneWorld adds scale, reset, and automatic verification.The combination is especially effective when workflows are stable and end states are easy to check.
- Why Cross-App Still Lags: Cross-app execution remains a major weakness because PhoneWorld primarily covers single-app tasks and lacks direct modeling of information handoff, artifact transfer, and persistent cross-app state.Cross-app tasks also stress long-horizon memory and runtime coordination, so broader environment coverage alone may not suffice.
- What This Paper Does Not Solve: A deployable phone agent also needs a runtime system and clear privacy and safety boundaries, which are outside this paper’s empirical center.PhoneBuddy focuses on training rather than the full deployment stack.
- Scope: PhoneBuddy is the training layer in a broader research stack that separately addresses mock environments, runtime execution, privacy, and safety.This paper therefore fits into a larger phone-agent matrix rather than standing alone.
8 Conclusion
PhoneBuddy studies how to train open models for real-world agentic phone use by combining real-app and mock-app interaction. Its strongest recipe uses shared supervised fine-tuning followed by mixed RL, improving both real-phone and AndroidWorld task success, while cross-app execution remains unresolved.
- Training Recipe: PhoneBuddy’s strongest recipe uses shared supervised fine-tuning from both environments followed by mixed RL across real-app and mock-app environments.The paper frames real-app training and mock-app training as complementary rather than interchangeable.
- Results: Task success improves on both real-phone human evaluation and AndroidWorld under the mixed-environment recipe.The conclusion reports improvement across both evaluation settings without reducing the result to a single benchmark.
- Takeaway: Mock-app interaction can transfer when grounded in realistic GUI structure, but environment scaling alone does not solve long-horizon state tracking, information handoff, or runtime coordination.The conclusion identifies stronger execution harnesses, intermediate verification, and safety-aware deployment boundaries as future needs.