Source-linked AI summary
PalmClaw: A Native On-Device Agent Framework for Mobile Phones
Hongru Cai, Yongqi Li, Ran Wei, Wenjie Li
TL;DR
Existing mobile agents largely rely on long, interface-dependent GUI action sequences that cannot directly access device capabilities or define clear execution boundaries. PalmClaw is a native on-device framework that exposes mobile capabilities through explicit, structured, bounded tools, improving task success by 11.5% relative and reducing completion time by 94.9% versus the strongest baseline.
Problem
Existing mobile agents primarily use GUI actions that create long, layout-sensitive sequences, lack direct access to device capabilities, and provide unclear execution boundaries.
Method
PalmClaw runs the agent loop and core state on-device while exposing mobile resources through tools with explicit arguments, structured results, and tool-specific boundaries.
Results
11.5% relative improvement in task success and 94.9% reduction in completion time were achieved on representative mobile tasks versus the strongest baseline.
Takeaways & Limitations
The results show that mobile phones can serve as environments for agent frameworks using direct and controlled access to device capabilities beyond screen operation.
Takeaways & Limitations
Deployments should clarify provider-side data flow because remote LLM providers may receive prompts, tool results, or task context.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action. Most agent systems run on desktops or servers, which support tool use and task automation. Mobile devices are also important agent environments because they are widely accessible and contain users' data, sensors, and daily-use applications. Existing mobile agents mainly operate smartphones through graphical user interface (GUI) actions such as tapping, swiping, and typing, which often form long, interface-dependent sequences, cannot directly access device capabilities, and make execution boundaries difficult to define. We present \textbf{PalmClaw}, an open-source agent framework that runs natively on mobile phones and manages the sessions, memory, skills, tools, and agent loop directly on the device. PalmClaw exposes device capabilities as device tools with explicit arguments, structured results, and clearly defined execution boundaries. This design enables agents to use mobile capabilities directly while keeping each action explicit and controlled. Experiments show an 11.5\% relative improvement in task success and a 94.9\% reduction in completion time over the strongest baseline, with lower setup burden and traces illustrating how execution boundaries are applied. Code is available at https://github.com/ModalityDance/PalmClaw.
1 Introduction
PalmClaw addresses the limitations of GUI-driven mobile agents by running the agent framework natively on phones and exposing device capabilities through explicit, structured tools with bounded execution. It reports an 11.5% relative improvement in task success and a 94.9% reduction in completion time, alongside lower setup burden and trace-based execution-boundary analysis.
- Agent context: LLM agents decompose complex instructions, call tools, observe results, and iteratively select actions to complete tasks.This extends agents beyond response generation toward autonomous task execution.
- Motivation: Mobile-native agents are important because phones are widely available personal computers and store users’ highly personal data.The passage states that four out of five people aged 10 or older worldwide own a mobile phone.
- Limitations: GUI-driven mobile agents rely on tapping, swiping, and typing, creating long layout-sensitive sequences, lacking direct access to device resources, and leaving execution boundaries broad.These limitations concern interface dependence, access to resources such as local files and sensor data, and control over action scope.
- PalmClaw: PalmClaw runs the agent loop, memory, skills, tools, and session state directly on the mobile device.The framework is open-source and reduces dependence on external desktop or server devices for agent orchestration.
- Device tools: Device tools expose mobile resources and capabilities through explicit arguments, structured results, and tool-specific execution boundaries.This design lets agents use mobile capabilities more directly while keeping each mobile action explicit and bounded.
- Evaluation: 11.5% relative improvement in task success and 94.9% reduction in completion time were achieved, with lower setup burden and trace-based execution-boundary analysis.These results are reported from PalmClaw’s evaluation on mobile tasks.
2 Related Work
Prior agent frameworks package tool use, observations, and multi-step action loops for execution on computers or cloud servers, while mobile-agent research targets smartphone task completion. PalmClaw differs by hosting the agent framework directly on the phone and exposing device capabilities through device tools.
- General agent frameworks: General agent frameworks combine tool use, intermediate observations, and multi-step action loops, with systems such as AutoGen, AgentScope, and OpenClaw targeting real tasks on computers or cloud servers.ReAct connects reasoning traces with actions, while Toolformer studies API calls as part of language modeling.
- Mobile agents: Mobile-agent research studies how LLM-based systems can complete tasks on smartphones.
- Mobile agents: Existing mobile agents and benchmarks include AndroidWorld, MobileAgentBench, Mobile-Bench, AgentCPM-GUI, and Mobile-Agent-v3.5, while related systems explore phone-side execution through GUI actions.Examples of related systems include ApkClaw, MobileClaw, and ClawMobile.
- PalmClaw's distinction: PalmClaw focuses on hosting the agent framework on the phone and exposing device capabilities through device tools.
3 Framework
PalmClaw runs sessions, memory, skills, and device tools directly on mobile devices, while coordinating them through context assembly and a bounded agent loop. Remote LLM inference supplies reasoning, whereas context management, tool execution, and session state remain on-device.
- Framework components: PalmClaw maintains sessions, memory, skills, and device tools directly on the mobile device.These components support multi-step agent execution on the phone.
- Sessions: Sessions organize local, remote-channel, and scheduled inputs while preserving conversation history, tool traces, attachments, and workspaces.Tasks can continue across multiple turns while concurrent tasks remain separate.
- Memory: Memory uses shared long-term storage and per-session summaries, updated when unconsolidated messages reach a configurable window.A separate LLM call summarizes older messages and updates both memory layers.
- Skills: Skills provide reusable task instructions, with always-active and relevant skills loaded selectively to avoid loading every skill into context.Relevant skills are matched against recent user messages, names, descriptions, and keywords.
- Agent loop and tool execution: The bounded agent loop repeatedly rebuilds session context, queries a remote LLM API, executes returned tool calls on-device, and appends results for subsequent rounds.Device tools use model-facing JSON schemas and return structured results or errors after execution.
4 Evaluation
PalmClaw’s evaluation shows effective and efficient mobile task execution, reduced deployment burden through its on-device design, and explicit execution boundaries for permissions, workspace access, and unsupported operations.
- Evaluation setup: PalmClaw was evaluated on MobileTask’s 70 tasks across mobile capabilities and on AssistantBench using its official accuracy metric.MobileTask uses final-state checks, evidence-grounded judging, or manual review depending on task type.
- Task completion: PalmClaw runs agent components on the phone and uses structured device tools, unlike baselines relying on GUI actions, mixed API-GUI control, or desktop-hosted runtimes.Compared systems include ApkClaw, MobileClaw, and ClawMobile.
- Task completion: 11.5% relative improvement in task success and 94.9% reduction in average completion time over the strongest baseline were achieved on MobileTask, with fewer actions and tokens.PalmClaw also obtained the highest accuracy on AssistantBench.
- Deployment and operation: PalmClaw requires no separate computer, CLI workflow, or bridge, and needs two setup steps and about two minutes before the first instruction.Its agent components and execution flow are integrated within the mobile application.
- Execution boundaries: Each device-tool call passes registry, schema, permission or confirmation, and workspace checks before execution, preventing an open command channel to the phone.Trace cases show user involvement for calendar permissions and manual settings access for workspace-boundary violations.
5 Conclusion
PalmClaw is an open-source, on-device agent framework designed for mobile environments beyond phone-interface operation. It exposes device capabilities through explicit tools and improves MobileTask performance while reducing completion time and setup burden.
- Framework: PalmClaw runs directly on mobile phones as an open-source agent framework beyond operating phone interfaces.The framework is designed specifically for the mobile environment.
- Framework: Device tools provide explicit arguments, structured results, and tool-specific execution boundaries for mobile capabilities.This design makes device-capability use explicit within the framework.
- Results: 11.5% relative improvement in task success and 94.9% reduction in completion time were achieved on MobileTask, alongside lower setup burden.These results are reported for PalmClaw on MobileTask.
Ethics Statement
PalmClaw keeps mobile actions explicit and bounded through registered tools, validated arguments, permissions or confirmation for sensitive actions, and workspace-limited file operations. Deployments should clarify provider-side data flows, while the reported experiments use public or constructed tasks without private user data.
- Action boundaries: PalmClaw bounds mobile actions through registered tools, validated arguments, Android permissions or user confirmation, and workspace-limited file operations.These mechanisms make execution explicit and controlled.
- Data flow: Deployments should clearly explain when remote LLM providers receive prompts, tool results, or task context.Provider-side data flow remains a user-facing transparency requirement.
- Evaluation data: The experiments use public or constructed evaluation tasks and do not require collecting private user data.The evaluation setting avoids private-data collection.
A Additional Details · A.1 Agent Context Composition
PalmClaw composes each agent turn from instructions, context, memory, skills, conversation history, tool traces, and available tools. Figure 5 illustrates this assembled context in a shortened example.
- A.1 Agent Context Composition: Each agent turn includes system instructions and runtime context as part of its assembled context.
- A.1 Agent Context Composition: The context incorporates shared long-term memory and active skill instructions.
- A.1 Agent Context Composition: It also includes an available-skill summary, recent conversation history, tool traces, and available tools.
A.2 Datasets and Metrics
The evaluation uses MobileTask, a filtered collection of rewritten tasks from three mobile-agent benchmarks, and AssistantBench as a manually reviewed secondary information-seeking set. MobileTask measures mean success over 70 tasks using oracle checks or a fixed-model judge, while AssistantBench reports official accuracy over its retained subset.
- MobileTask: MobileTask adapts tasks from AndroidWorld, MobileAgentBench, and Mobile-Bench into user-goal instructions with identifiers, source labels, expected artifacts, and evaluation notes.Each candidate is rewritten with structured metadata for evaluation.
- MobileTask: MobileTask retains tasks that avoid fixed GUI paths, third-party app state, and GUI-only advantages, while removing unstable or duplicated cases.Removed cases include relative dates, live weather, and duplicated calendar variants.
- AssistantBench: AssistantBench retains 19 of 33 development tasks after manual review and excludes unstable live-state questions or unverifiable reference answers.The retained tasks form a secondary information-seeking evaluation set.
- Metrics: 70 tasks define MobileTask’s mean success rate, with deterministic oracle checks for inspectable outcomes and DeepSeek-V4-Flash judging answer-based tasks.The judge receives the instruction, final answer, and fixed reference evidence, then returns binary success or failure.
- Metrics: AssistantBench reports official accuracy averaged over the retained subset.The supplied passage introduces this metric but does not provide the completed averaging procedure.