Source-linked AI summary
PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
Yangqin Jiang, Lingrui Xu, Chao Huang
TL;DR
Mobile GUI agents use a slow, costly, and brittle screenshot–VLM loop, while much everyday app navigation is repetitive and static. PhoneCLI compiles that navigation into callable commands and uses a VLM for open-ended interaction or compiled-path failures; the authors report improved AndroidLab success with lower costs and efficiency gains on AndroidWorld.
Problem
Mobile GUI agents rely on a costly, brittle screenshot–VLM loop, while training-based approaches require costly per-app adaptation and do not directly help with unseen apps.
Source: §1 · p. 2Method
PhoneCLI explores an app externally and compiles its navigation into deterministic per-screen commands, verifying and replaying matches while handing open-ended interactions and failures to a VLM.
Source: §1 · p. 2, §2.1 · p. 3Results
PhoneCLI improves task success while reducing step and token costs on AndroidLab, and transfers to AndroidWorld’s official M3A agent with efficiency gains.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
Because PhoneCLI compiles app navigation rather than a single run, its commands can support unseen tasks, not only repeated ones.
Source: §1 · p. 2The compiled layer covers static navigation rather than open-ended interaction, which remains with the runtime VLM.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.
1 Introduction
Mobile agents are confined to a costly screenshot-and-VLM loop, while everyday navigation is repetitive and training for each app is costly. PhoneCLI compiles app navigation into callable commands and reports gains on two benchmarks.
1 Introduction
- Mobile GUI agents rely on a screenshot–VLM–action loop, but mobile apps lack the programmatic interfaces available to desktop agents, making navigation slow, costly, and brittle.The passage describes steps taking seconds and requiring one API call per step.Source: §1 · p. 1
- Everyday app navigation follows repetitive, ordered patterns that trace a small set of static paths, motivating offline enumeration and deterministic replay.Examples include opening a settings page, selecting a tab, and locating an item.Source: §1 · pp. 1–2
- Per-app training is costly and inflexible, and the authors argue that training for millions of apps is infeasible in principle.A model that has not encountered an app has no direct shortcut for navigating it.Source: §1 · p. 2
- PhoneCLI compiles an app’s screens, UI elements, and navigation edges into deterministic commands, then verifies and replays them while leaving open-ended tasks to the VLM.The method requires no app-internal API, instrumentation, or model updates.Source: §1 · p. 2
- Unlike methods that compile an agent’s trajectory or feed mined paths back as hints [15] [16] [17] [18], PhoneCLI compiles app navigation offline so commands can serve unseen tasks without runtime perception.The authors characterize the distinction as compiling what the app is rather than what an agent has done.Source: §1 · p. 2
- The authors report state-of-the-art AndroidLab performance with lower step and token costs on successful tasks, plus efficiency gains when transferring to AndroidWorld’s official agent.A controlled three-way ablation attributes improvements independently to map information and deterministic replay.Source: §1 · p. 2
2 Methodology
PhoneCLI compiles static app navigation offline into a map and commands, then routes tasks to verified deterministic replays while using a VLM for interactions or failures outside the compiled paths. The method formalizes the map and commands, explores and annotates app structure, and checks execution before handing control to its runtime interpreter.
2.1 Design Overview
- PhoneCLI compiles static, reusable navigation offline and leaves variable interactions, such as form filling or dynamic content, to the runtime VLM.The app map formalizes screens, interactive elements, navigation edges, and one deterministic replay command per screen; every compiled-path failure falls back to the pure VLM agent.Source: §2.1 · pp. 3–4
2.2 Offline Compilation: Building App Maps
- Offline compilation explores a black-box app’s navigation skeleton, semantically annotates screens and elements, and derives one executable replay command per screen.The crawler traverses from a clean launch in breadth-first order; exploration is bounded by depth and screen budgets.Source: §2.2 · p. 4
2.3 Online Invocation: Routing and Replaying Commands
- Online invocation conservatively routes a task to a symbolic command, verifies its target before replay, and checks the landing screen afterward.Replay uses no perception or model call and completes in sub-second time; mismatches are restarted to a clean state before VLM fallback.Source: §2.3 · pp. 4–5
2.4 Runtime Interpretation: VLM Fallback and Graceful Degradation
- The runtime interpreter handles interactions absent from the partial map and every unavailable or failed compiled path, using the pure VLM agent from the current or restarted state.It maintains structured state across rounds and can provide targeted hints when behavior is unproductive or an action fails.Source: §2.4 · pp. 5–6
3 Evaluation
On AndroidLab, PhoneCLI improves success while reducing steps and token use compared with VLM-only agents. Ablations and case analyses examine the roles of map information, deterministic replay, map scale, and navigation failures.
3.1 Experimental Setup
- Evaluation covers AndroidLab (138 tasks, 9 apps) and AndroidWorld (116 tasks, 20 apps), comparing against AndroidWorld’s official M3A agent.The evaluation also uses three backbone models and a range of general-purpose and GUI-specialized baselines.Source: §3.1 · p. 6
3.2 Overall Performance
- PhoneCLI improves success by 12.3 points on Qwen3.7-Plus and 0.8 points on Kimi-K3, while Qwen3.7-Plus steps fall 11% and tokens 14%.Table 1 reports success, steps, and tokens; Table 2 situates success rates against additional AndroidLab models.Source: Table 1 · p. 6, §3.2 · pp. 6–7, Table 2 · p. 7
3.3 Ablation Studies
- On nine apps, PhoneCLI reaches 63.0% success versus 50.7% for VLM-only and 55.1% without replay; replay contributes the larger gain.The ablation compares the full system with map-only and VLM-only conditions.Source: §3.3 · pp. 7–8, Table 3 · p. 8
- Every tested mechanism removal costs at least 12 success-rate points; element classification is the costliest single removal at −15.9%.Other reported drops are −15.2% for semantic enrichment, −12.3% without a landing check, −13.0% without a landing hint, and −14.4% without command completion.Source: Table 3 · p. 8, §3.3 · p. 8
3.4 Deep Analysis of Map Scale
- Success peaks at a 50-screen budget then falls at 100: Setting goes 87.0%→95.7%→78.3%, while Clock goes from 77.8% to 66.7%.Small maps under-cover apps; oversized maps enlarge and noisify the command set, making mis-routing more likely.Source: Figure 3 · p. 8, §3.4 · pp. 8–9
3.5 Error Analysis
- Failures fall from 68 to 51, with 27 saved across all four failure modes and 10 lost, chiefly to routing costs and single-run variance.The analysis attributes the broad reductions to preventing the search for an entry point rather than repairing only one failure mode.Source: Figure 4 · p. 9, §3.5 · p. 9
3.6 Case Study
- For the Calendar task, PhoneCLI reaches the event form by replay and finishes in 17 rounds, while VLM-only gets stuck and fails after 25.The task is to add a 5PM event titled “work”; both agents understand the requested event, but locating its entry point is the contrast examined.Source: Figure 5 · p. 9, §3.6 · pp. 9–10
4 Related Work
Prior work improves mobile GUI agents through training or programmatic execution, and explores apps to produce readable knowledge, graph guidance, or replay programs. PhoneCLI distinguishes itself by compiling an app’s navigation offline into deterministic commands, one per screen.
4.1 Mobile GUI Agents and Programmatic Execution
- Mobile GUI agents commonly use screen-based perception and actions, with capabilities learned through grounding or task-completion training [1] [2] [3] [28] [29] [30] [31] [32].New apps can require new data or retraining [11, 12].Source: §4.1 · p. 10
- Programmatic-execution agents use code, APIs, or device-side commands, but assume an execution surface that mobile apps may not expose [5] [6] [8].PhoneCLI instead synthesizes callable commands from GUI navigation without an app API, instrumentation, or training.Source: §4.1 · p. 10
4.2 App Exploration and Structure Mining
- App exploration has produced operation documents, executable scripts, mined trajectories, datasets, and retrieved tutorials, but these approaches leave each action to the model [17] [18] [35] [36].Android GUI testing crawlers target coverage and crash discovery rather than agent performance.Source: §4.2 · p. 10
- UI-KOBE uses an offline app graph to guide a lightweight agent that remains in the navigation loop [16].PreAct compiles a successful agent run online into a state-machine program for replaying that same task [15].Source: §4.2 · p. 10
- PhoneCLI compiles offline, before a task, into one deterministic navigation command per screen, rather than compiling a whole task run [15].Its commands execute the explored structure instead of asking a model to read or follow it.Source: §4.2 · p. 10
5 Conclusion
PhoneCLI compiles static, repetitive mobile navigation into deterministic commands that execute without a VLM, while falling back to the embedded interpreter for open-ended interaction or compilation failures. On AndroidLab, it improves task success while reducing step and token costs.
5 Conclusion
- PhoneCLI compiles static, repetitive navigation offline into deterministic, zero-VLM commands and replays a matched command online in sub-second time.Open-ended interaction and compilation failures fall back to the embedded VLM interpreter.Source: §5 · p. 10
- On AndroidLab, PhoneCLI improves task success rate while reducing step and token costs.The conclusion describes a stronger agent at lower cost.Source: §5 · p. 10
A.1 Experimental Setup
The experimental setup builds reusable app maps offline and evaluates agents on controlled Android emulator runs. All conditions share the same observation and model setup, with differences limited to injected context.
A.1 Experimental Setup
- Across nine AndroidLab apps, offline maps contain 320 screens, 2,822 elements, and 1,089 navigation edges, compiling into 2,017 operations.A map takes about ten minutes to build on one emulator, is reused across tasks, and uses no task or evaluation information.Source: §A.1 · p. 14
- 138 tasks are attempted once on a clean Android 13 emulator with a 25-round cap; success is averaged over all tasks, while steps and tokens are averaged over successful tasks.94 tasks use deterministic XML checkers and 44 use an LLM judge; all conditions are re-judged with one model.Source: §A.1 · p. 14
- All conditions use the same observation format, prompt, action space, and off-the-shelf backbone, differing only in injected context.Compiled operations issue ADB commands without a model call; token accounting separates interpreter, routing, and verification calls.Source: §A.1 · p. 14
A.2 App Map Data Structure
The app map is stored as YAML and pairs screen information with a deterministic command for reaching each screen.
A.2 App Map Data Structure
- The Settings app map excerpt records each screen’s elements with normalized centers and its navigation edges.Each screen is associated with a deterministic replay command that reaches it.Source: §A.2 · p. 14
- The map is stored as a YAML file.Figure 6 shows a simplified excerpt from the Settings app map.Source: §A.2 · p. 14
- Each screen has a deterministic replay command that reaches it.The command is associated with the screen’s elements and navigation edges in the map.Source: §A.2 · p. 14
A.3 From App Maps to Callable Commands
The section turns app-map records into path-keyed operations and a compact navigation catalog, which the router uses to select deterministic replay commands.
A.3 From App Maps to Callable Commands
- Each operation pairs an element path with its parent-screen macro, required scroll steps, and a tap; navigation elements are marked NAV and other actions ACT.Operations are keyed by path rather than destination, and only the path that first expanded a screen carries that screen’s children.Source: §A.3 · pp. 14–15
- The app map records elements and navigation edges from which the compiler derives a deterministic replay command; invoking it costs no VLM call.The catalog is compiled once per app, cached, and contributes one routing-call prefix per task.Source: Figure 6 · p. 15, §A.3 · p. 16, Figure 7 · p. 16
- For Settings, the navigation-only catalog compresses 288 operations to 98 NAV entries in 9 groups and roughly 6.2k tokens to 1.9k.Shared path prefixes are merged, and each entry includes element text, semantic type, an alias, and a stable operation ID.Source: §A.3 · p. 15
- The router selects among four decisions, and a selected command resolves to a fixed-coordinate macro replayed without a model call.For Bluetooth pairing, the example replays force_stop, launch, and two taps before handing control to the VLM on the Bluetooth page.Source: §A.3 · p. 15
A.4 Generalization to AndroidWorld
On AndroidWorld, PhoneCLI augments the official M3A agent with map-based routing while leaving its prompts, action space, and perception unchanged. The augmented agent scores higher and shows consistent efficiency gains.
A.4 Generalization to AndroidWorld
- PhoneCLI is added to the official AndroidWorld M3A agent by subclassing it and adding only map-based routing.M3A’s prompts, action space, and perception remain untouched; when routing declines, execution follows the original M3A.Source: §A.4 · p. 16
- The authors describe M3A as mature and near the capability ceiling of Qwen3.7-Plus, leaving efficiency rather than capability as the remaining gain.They present the AndroidWorld transfer as evidence that the efficiency enhancement does not depend on AndroidLab-specific design.Source: §A.4 · p. 16
- On AndroidWorld, map-augmented M3A scores higher, uses 9% fewer tokens overall, and needs 7.7 versus 8.4 steps on the 70 tasks both agents solve.The authors report these gains on a different benchmark and agent framework, with M3A untouched apart from map-based routing.Source: Table 4 · p. 16, §A.4 · p. 16