Source-linked AI summary
PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions
Chenxin Li, Zhengyao Fang, Zhengyang Tang, Pengyuan Lyu, Xingran Zhou, Xin Lai, Fei Tang, Liang Wu, Yiduo Guo, Weinong Wang, Junyi Li, Yi Zhang, Yang Ding, Huawen Shen, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Chengquan Zhang, Han Hu
TL;DR
Mobile-agent research largely treats phone use as screen navigation, although real workflows span GUIs, commands, tools, and verifiable side effects. PhoneHarness combines these action surfaces with trace-backed evaluation, achieving a 12.9-percentage-point improvement over the strongest non-PhoneHarness settings.
Problem
Mobile-agent research mostly frames phone use as screen navigation, limiting evaluation of workflows spanning app interaction, external tools, and verifiable task completion.
Method
PhoneHarness combines device-side CLI execution, delegated GUI interaction, and host-side tools in an auditable phone-agent loop, with PhoneHarness Bench verifying observable side effects.
Results
12.9 percentage points: PhoneHarness improves pass rate over MobileClaw and Seed2.0-Pro, supporting mixed-action routing rather than screen-navigation-only evaluation.
Takeaways & Limitations
PhoneHarness makes realistic mixed-action phone workflows executable, while PhoneHarness Bench measures whether agents complete them reliably with observable side effects.
Takeaways & Limitations
The benchmark’s stable subset is smaller than its candidate pool, and real-app evaluation is brittle because apps, networks, login states, and permissions vary.
Abstract
from arXiv · showhide
Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
1 Introduction
PhoneHarness frames reliable phone-use as mixed-action workflow completion, requiring GUI, CLI, and tool actions plus evidence of real side effects. It introduces an execution harness and benchmark that jointly enable and measure verifiable mobile automation.
- Motivation: Modern language-model agents are evaluated by whether they complete tasks and leave evidence that the intended work occurred, not merely generate plausible answers.This shift targets the full perception–planning–action loop, including changing files, updating records, and sending messages.
- Problem: Existing mobile-agent systems largely frame phone use as screenshot or accessibility-tree navigation through GUI actions.The cited mobile-agent environments have advanced GUI control, but the surrounding stack remains fragmented.
- Novelty: Unlike GUI benchmarks and general tool-use benchmarks, PhoneHarness treats the phone, device state, app UI, and mobile side effects as first-class evidence.Its central evaluation question is whether phone workflows succeed, rather than whether an agent can tap the next button.
- Contributions: PhoneHarness unifies device-side CLI execution, high-level GUI delegation, and host-side MCP-style tool calls within one phone-agent loop.PhoneHarness is the execution harness, while PhoneHarness Bench is constructed on top of it as the evaluation artifact.
- Contributions: PhoneHarness Bench organizes evaluation around 14 mock-app tasks, 45 real-app tasks, 30 exploratory safety tasks, and a 124-task annotated split from 181 candidates.The benchmark is designed to measure whether the underlying harness executes realistic mixed workflows successfully.
- Contributions: The framework verifies mobile workflows through trace-backed checks of tool calls, artifacts, system settings, sent messages, confirmations, and unintended data egress.Its evaluation protocol also diagnoses failures across reasoning, action routing, GUI grounding, tool use, environment instability, and verifier mismatch.
2 Related Work
Prior work spans mobile GUI control, execution-based computer-use evaluation, tool-use benchmarks, and agent safety. PhoneHarness is positioned as complementary to these efforts by evaluating phone agents that combine GUI control with broader phone-use actions.
- Mobile GUI agents and Android benchmarks: Mobile-agent research includes app-specific exploration and reuse, multi-agent planning and reflection, and Android benchmarks for GUI control.AppAgent studies app-specific operation knowledge, while Mobile-Agent-v2 separates planning, decision, and reflection components.
- Mobile GUI agents and Android benchmarks: PhoneHarness evaluates phone agents that may need GUI control without reducing phone-use to GUI control alone.This positions PhoneHarness as complementary to mobile GUI-agent research.
- Computer-use and execution-based benchmarks: Computer-use benchmarks evaluate agents in realistic environments using execution-based rewards, grounded interaction, and task validators rather than only static demonstrations or final-text judgments.The cited work covers OSWorld, web-interaction benchmarks, and realistic enterprise web tasks.
- Tool-use and MCP-augmented agents: Tool-use benchmarks study tool selection, API calling, and consistency across multi-step interactions, with recent mobile work extending toward MCP-augmented environments.Related embodied benchmarks connect language agents to interactive environments.
- Agent safety and side effects: Safety research evaluates unsafe actions, hidden side effects, prompt-injection-like failures, defenses, and privacy behavior in tool-using or phone-use agents.The cited benchmarks include ToolEmu, AgentDojo, Agent-SafetyBench, SafeArena, and MyPhoneBench.
3 PhoneHarness
PhoneHarness is a host-device execution harness for realistic phone-use evaluation, combining GUI, CLI, and host-side tools on real Android devices or emulators. Its deterministic-first routing, progressive skill disclosure, and layered execution traces support reliable action selection and auditable mobile side effects.
- Design requirements: The harness runs the agent loop against a real Android device or emulator, supports deterministic execution alongside GUI control, keeps heavy tools off-device, and produces auditable traces.These requirements target observable mobile side effects while avoiding an all-on-device dependency stack.
- Architecture: PhoneHarness uses a host-device architecture in which the device runs the agent loop and host proxies provide model, GUI, and MCP services.The GUI proxy translates high-level actions into Android Debug Bridge operations including screenshots, taps, swipes, text input, launches, and UI-tree retrieval.
- Auditable execution: Each run records an outer device-agent trace and nested GUI traces containing tool calls, results, screenshots, actions, outcomes, and timing for benchmark evidence.The separation helps distinguish wrong-tool, wrong-argument, underspecified-delegation, and GUI-execution failures.
- Mixed-action affordances: Its mixed-action affordance modes combine GUI, CLI, and MCP-style tools, allowing alternative tasks to use deterministic operations and GUI-primary tasks to reduce brittle navigation or retrieve auxiliary state.GUI-only fallback remains available for visually grounded subtasks where other action surfaces are insufficient.
- Action routing: The deterministic-first routing principle prefers reliable CLI commands or structured tool calls, delegating bounded GUI subtasks only when app-specific visual navigation is required.The paper frames choosing the appropriate action surface as a core research question for phone agents.
- Skill access: Progressive skill disclosure exposes a compact index of skill families and loads detailed instructions and examples only when the agent requests a specific capability.Skill families include device operations, environment utilities, email, file handling, web search, maps, social, and document workflows.
4 PhoneHarness Bench
PhoneHarness Bench evaluates mobile workflows by requiring agents to route subtasks across GUI, CLI, and host-side tools while producing verifiable execution evidence. Its annotated tasks are organized by execution structure and safety requirements, with mock-app tasks serving mainly as diagnostics.
- Benchmark scope: 181-task candidate pool underlies PhoneHarness Bench, whose public evaluation emphasizes real-app workflows and safety behavior.The mock-app subset contains 14 controlled-application tasks used mainly to check harness and verifier alignment.
- Task organization: Tasks are grouped by execution structure rather than a single difficulty axis because deterministic CLI or MCP paths can simplify some workflows while GUI tasks remain brittle under permissions, logins, or unstable search.The benchmark uses four task types based on execution structure.
- Mixed-action evaluation: Agents choose among CLI commands, GUI delegation, host tools, or combinations without being told which action surface to use.The runner records streamed traces, collects artifacts, and applies task-specific verifiers during execution.
- Verification: Rule-based and trace-backed verifiers check tool calls, recipient correctness, device settings, generated artifacts, calendar events, and required final-answer information.Composite verifiers combine multiple conditions when workflow success cannot be established by one check.
- Verification: The verifier layer treats traces and environments as first-class scoring evidence, including device settings, app states, device files, host-tool logs, and GUI traces.This adapts execution-based evaluation to mobile workflows.
- Safety policy: Safety tasks distinguish SAFE_COMPLETE, CONFIRM_FIRST, and NEVER_AUTO operations, requiring direct execution, confirmation, or refusal/deferment respectively.CONFIRM_FIRST tasks require warning the user and obtaining confirmation before sensitive actions.
5 Evaluation Setup
PhoneHarness Evaluation Setup measures verified task completion while comparing flexible model pairings and analyzing how failures arise across agents, harnesses, environments, and safety behavior. The setup combines pass-rate evaluation with trace-based diagnostics of actions, efficiency, routing, and verification outcomes.
- Evaluation configurations: The delegated configuration pairs an outer model for planning, routing, tool use, and CLI/MCP operations with a GUI model for bounded screen-interaction subtasks.This supports pairing text-strong and vision-strong models and comparing model pairs rather than only individual models.
- Metrics and failure attribution: Task pass rate under each task’s verifier is the primary metric, supplemented by subset pass rates, tool-use patterns, step counts, runtime, GUI delegation frequency, verifier failures, and safety violations.Mixed-action analysis distinguishes model-level failures from harness-level and environment-level failures, such as choosing GUI when a deterministic CLI path exists versus brittle real-app UI flow.
- Failure analysis: Trace evidence assigns failed runs to categories including wrong action-surface routing, missing tool knowledge, incorrect parameters, GUI grounding failure, premature termination, hallucinated completion, instability, and verifier mismatch.Safety tasks additionally track correct refusal, late confirmation, unnecessary sensitive-data access, and hidden side effects.
6 Experimental Findings
PhoneHarness improves verifiable phone-task reliability by routing among GUI, CLI, and tools, with the largest gains on device/system and tool-assisted workflows. Its efficiency, model-combination, and safety findings further show that routing and conservative refusal behavior matter alongside completion rate.
- Overall reliability: 75.0% pass rate: PhoneHarness improves over MobileClaw and Seed2.0-Pro by 12.9 percentage points and over AutoGLM-Phone by 37.9 percentage points.The result supports evaluating phone agents on observable side effects rather than screen navigation alone.
- Task type: 96.7% on device/system operations and 74.3% on tool-assisted workflows make PhoneHarness strongest where deterministic operations and verifiable side effects are central.Seed2.0-Pro instead reaches 76.7% on visually grounded single-app GUI tasks, while the two systems tie at 65.5% on cross-app workflows.
- Action-space label: 97.0% on GUI-and-CLI alternative-route tasks and 67.6% on GUI-primary tasks with optional CLI support indicate gains from action-surface routing.Optional CLI support can retrieve device state, prepare artifacts, or reduce brittle screen navigation; it is not required in every successful trajectory.
- Execution steps: 23 mean execution steps per attempted task: PhoneHarness uses slightly fewer steps than Seed2.0-Pro’s 24 and avoids unnecessary GUI exploration on device/system and tool-assisted workflows.Lower step counts are consistent with routing rather than simply spending more actions to complete tasks.
- Model combinations: 74.8% overall pass rate: DeepSeek V4 flash with Seed2.0-Pro as GUI worker is strongest among the reported same-harness pairings.Under the DeepSeek controller, Seed2.0-Pro exceeds AutoGLM-Phone overall, on GUI-or-CLI alternatives, and on GUI-primary tasks with optional CLI support.
- Safety behavior: 90.0% dangerous-action refusal rate: HY3-preview paired with either GUI model is strongest on safety-oriented tasks.The paired results indicate stable refusal behavior across both GUI workers, separating safety from task-completion performance.
7 Discussion and Limitations
PhoneHarness and PhoneHarness Bench are evolving together, while real-app evaluation remains brittle and some capabilities depend on a host proxy. Virtual display support points toward background mobile assistance that could reduce disruption to the user’s active phone session.
- Limitations: PhoneHarness and PhoneHarness Bench evolve together, with the stable subset smaller than the candidate pool because tasks require verifier alignment and human validation.The passage describes the current stable subset as smaller than the full candidate pool.
- Limitations: Real-app evaluation is more brittle than mock-app evaluation because apps change, network conditions vary, and login or permission states affect task feasibility.These factors can change whether a task is feasible during evaluation.
- Limitations: The host proxy makes tool access practical, but some capabilities are not purely on-device.This creates a boundary between device-side execution and host-mediated access.
- Future Directions: Virtual display support could enable phone agents to operate on an independent background display without taking over the user’s foreground screen.If robust, this would shift phone agents from demonstration-style screen control toward concurrent mobile assistance.
8 Conclusion
The paper presents PhoneHarness and PhoneHarness Bench as complementary artifacts for reliable phone-agent evaluation. The harness executes verifiable workflows across CLI, GUI, and MCP-style host tools, while the benchmark measures observable side effects with trace-backed verification.
- PhoneHarness: PhoneHarness unifies CLI, GUI, and MCP-style host tools inside a phone-agent loop for verifiable execution.It makes realistic phone workflows executable.
- PhoneHarness Bench: PhoneHarness Bench evaluates whether agents complete mobile workflows with observable side effects and trace-backed verification.It is built on top of PhoneHarness and tests agent reliability on realistic workflows.
- Conclusion: Together, the harness enables realistic phone workflows, while the benchmark tests whether agents can complete them verifiably.This complementary relationship is central to the paper’s conclusion.