Source-linked AI summary
X-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction
Xiaoming Ren, Ru Zhen, Chao Li, Yang Song, Qiuxia Hou, Yanhao Zhang, Peng Liu, Qi Qi, Quanlong Zheng, Qi Wu, Zhenyi Liao, Binqiang Pan, Haobo Ji, Haonan Lu
TL;DR
Mobile agents need deeper user-defined control and customization within dynamic smartphone contexts. X-OmniClaw unifies multimodal perception, memory, and action for Android, with demonstrations showing efficient, reliable execution across diverse mobile scenarios.
Problem
Existing mobile agents offer cross-app orchestration but limited user-defined customization, while OpenClaw remains centered on PC execution rather than dynamic mobile contexts.
Method
X-OmniClaw unifies multimodal ingress, scene-grounded intent understanding, runtime and long-term memory, and hybrid structural-visual action grounding for Android.
Results
Demonstrations show X-OmniClaw executing complex mobile tasks efficiently and reliably, including direct access across e-commerce, local services, short-video, and search scenarios.
Takeaways & Limitations
X-OmniClaw provides a practical architectural blueprint for mobile-native personal assistants that integrate smartphone context into task execution.
Takeaways & Limitations
The current replay pipeline does not directly reproduce users’ step-by-step actions, reserving trace-level replay for future complex scenarios.
Abstract
from arXiv · showhide
Inspired by the development of OpenClaw, there is a growing demand for mobile-based personal agents capable of handling complex and intuitive interactions. In this technical report, we introduce X-OmniClaw, a unified mobile agent designed for multimodal understanding and interaction in the Android ecosystem. This unified architecture of perception, memory, and action enables the agent to handle complex mobile tasks with high contextual awareness. Specifically, Omni Perception provides a unified multimodal ingress pipeline that integrates UI states, real-world visual contexts, and speech inputs, leveraging a temporal alignment module to decompose raw data into structured multimodal intent representations. Omni Memory leverages multimodal memory optimization to enhance personalized intelligence by integrating runtime working memory for task continuity with long-term personal memory distilled from local data, enabling highly context-aware and personalized interactions. Finally, Omni Action employs a hybrid grounding strategy that combines structural XML metadata with visual perception for robust interaction. Through Behavior Cloning and Trajectory Replay, the system captures user navigation as reusable skills, enabling precise direct-access execution. Demonstrations across diverse scenarios show that X-OmniClaw effectively enhances interaction efficiency and task reliability, providing a practical architectural blueprint for the next generation of mobile-native personal assistants.
1 Introduction and Related Work
X-OmniClaw is presented as a unified mobile-native framework that integrates multimodal perception, memory, and action for reliable Android task execution. It builds on OpenClaw-inspired mobile automation while addressing the need to connect real-world context with application control.
- System motivation and design: X-OmniClaw unifies on-screen UI state, real-world visual context, and audio input into a cross-modal perception-to-action framework.The prefix “X” emphasizes cross-modal integration, while “Omni” denotes the three sensing domains.
- System motivation and design: X-OmniClaw aims to bridge architectural execution and mobile-native autonomy through coordinated Omni Perception, Omni Memory, and Omni Action components.The supplied context describes these components as jointly supporting contextual awareness, personalized memory, and robust interaction.
- System motivation and design: The framework treats the smartphone as a persistent multimodal interface and combines environmental sensing with application-level control for complex mobile tasks.This design enables real-world context to inform digital-task execution across diverse mobile-centric environments.
- Related work: Prior mobile-agent research explored screenshot-based visual interaction, coordinate-level grounding, and scalable Android automation under dynamic GUI conditions.Mobile-Agent and AppAgent investigate purely visual interaction, while Doubao Phone demonstrates cross-app orchestration feasibility.
- Related work: Existing cloud-centric mobile-agent systems run virtualized Android instances in remote data centers, separating agent operation from the user’s physical device.RedFinger, Wuying, and Tencent Cloud Phone exemplify this paradigm, which reduces local computational demands.
- Related work: The project was inspired by OpenClaw and AndroidClaw, extending this open mobile-automation baseline with distinctive core capabilities.OpenClaw emphasizes layered control, explicit skills, persistent storage, and atomic tool abstractions for agent engineering.
2 Frameworks of X-OmniClaw
X-OmniClaw presents perception, memory, and action as a co-designed Android mobile-agent stack. Its local execution architecture unifies multimodal understanding, persistent context, and reliable interaction through three integrated modules.
- System Architecture: X-OmniClaw co-designs perception, memory, and action as one stack for continuous perception and reliable Android device execution.The architecture integrates Voice, Screen, and Camera inputs with an agent loop for on-device execution.
- System Architecture: Core perception and execution reside on the user’s Android device, while a cloud-based LLM supplies on-demand support for high-level reasoning.The local design enables direct manipulation of authentic applications and system settings without maintaining a cloud phone.
- Omni Perception: Omni Perception integrates UI states, real-world visual contexts, and speech inputs, decomposing streaming data into structured intents that drive reasoning and execution.
- Omni Memory and Omni Action: Omni Memory unifies runtime working memory with long-term personal knowledge, while Omni Action combines XML and visual information with behavior cloning and trajectory replay for precise reusable skills.Memory maintains task continuity and personalized context; Action translates high-level intent into hardware-level actions while preserving state consistency.
3 Omni Perception
X-OmniClaw’s Omni Perception unifies diverse user, scheduled, and external inputs into a common ingress pipeline, then jointly models UI, visual, and audio signals. A vision-language model grounds these multimodal inputs in the current scene to derive structured intent or answer directly when possible.
- Multimodal Entry and Unified Ingress: X-OmniClaw funnels direct user, scheduled, and external ecosystem requests through one unified multimodal ingress pipeline.Inputs include in-app UI interactions, floating widgets, microphone input, scheduled tasks, Feishu, Discord bots, and other remote gateways.
- Integrated Multimodal Perception: Its first-person perception system jointly models on-screen UI state, real-world visual context, and audio input from camera, screen, and speech channels.Camera streams and screen projection capture visual information, while speech recognition transcribes microphone input in real time.
- Scene-Grounded Intent Understanding: A vision-language model interprets the visual scene with the user query, answers directly when possible, and otherwise converts the result into structured intent.The system expands raw input into a more complete semantic representation before triggering downstream actions.
4 Omni Memory
Omni Memory combines multimodal working memory for runtime continuity with persistent long-term memory distilled from local personal data. Skill–Tool coordination supports memory maintenance and consumption, while filtering, redaction, and user controls constrain what is stored and used.
- Working Memory and Long-Term User Memory: Working memory preserves multimodal runtime context across turns, foreground changes, and app switches by combining screenshots, compressed observations, and execution state.These signals record visual evidence, distilled semantic context, and task progress.
- Working Memory and Long-Term User Memory: Long-term memory distills personal media, interaction trajectories, and task-relevant metadata into persistent memory artifacts and user-profile representations.The resulting multimodal memories can be injected into downstream reasoning and interaction context.
- Working Memory and Long-Term User Memory: Gallery photos are transformed into structured semantic records capturing objects, scenes, events, and user-relevant cues for question answering, retrieval, and personalized media selection.This replaces reliance on raw images alone with compact semantic representations for later automation workflows.
- How Memory Is Built, Used, and Secured: Skill–Tool coordination separates memory-maintenance workflows from memory-consumption workflows, with tools executing their concrete operational steps.Maintenance includes synchronization, update, and rebuild, while consumption includes question answering, retrieval, and memory-grounded operations.
- How Memory Is Built, Used, and Secured: A unified filtering and redaction step precedes memory storage, while users control gallery-memory enablement and derived-profile injection.These mechanisms reduce the chance that sensitive information is stored in long-term memory and provide explicit user controls over downstream use.
5 Omni Action
Omni Action organizes mobile interaction as an observation–reasoning–execution loop that combines structural and visual evidence for heterogeneous Android interfaces. It extends one-shot actions with behavior cloning, deeplink-based fast entry, and trajectory replay while preserving traces for future complex scenarios.
- Hybrid grounding: Dynamic balancing of structural and visual evidence addresses Android applications’ heterogeneous rendering styles, interface exposure, and interaction logic.The strategy avoids relying on a single source of interface evidence.
- Agent loop: The agent loop builds a unified multimodal observation stack, reasons over page state and prior-step status, selects skills or memory, and executes the resulting action.The loop can return either the next action or a direct response before execution.
- Behavior Cloning: Behavior Cloning records observable UI-layer interactions and summarizes their purpose as named reusable skills rather than literal action sequences.Examples include finding a reward-claim entry or jumping directly to a specific video-editing template.
- Direct entry: Deeplink and intent extraction via dumpsys activity introspection bypasses redundant UI replays, using keyword filtering first and full parsing as fallback.UI-tree parsing captures paths while the two-stage recovery strategy balances rapid target-activity location with completeness.
- Trajectory Replay: Directly replayable or fast-entry routes span four categories—e-commerce, local services, short-video platforms, and search—while preserved traces support future trace-level replay under interface drift.The current pipeline does not depend directly on step-by-step reproduction, but retains execution traces for complex scenarios lacking stable direct-entry routes or requiring finer-grained semantics.
6 Use Cases and Demo Scenarios
Section 6 demonstrates three application tracks: real-world copilot assistance, proactive personalized services, and behavior cloning with trajectory replay. Across these scenarios, X-OmniClaw combines multimodal perception, memory, deeplink-based action, and reusable navigation skills to automate context-aware mobile tasks.
- 6 Use Cases and Demo Scenarios: The demos span camera-informed execution, ScreenAvatar assistance, memory-based one-tap video generation, and instant portals created through cloned navigation paths.These scenarios correspond to the three tracks of real-world copilot assistance, proactive personalized services, and behavior cloning with trajectory replay.
- 6.1 Real-world Copilot Assistant: Camera-informed execution uses live visual observations and spoken queries to infer product intent, resolve the target application and parameters, and enter the task through a deeplink.After entry, the agent repeatedly scrolls, captures screenshots, and applies VLM-based reading to extract structured fields such as prices, sales, ratings, and distances.
- 6.1 Real-world Copilot Assistant: For screen-projected tasks, a lightweight floating ScreenAvatar follows the active interface and starts complex, long-chain execution after a microphone trigger.The ScreenAvatar acts as a digital surrogate on the display, executing tasks on the user’s behalf.
- 6.2 Proactive Personalized Services: Memory-based one-tap video combines multimodal perception, long-term memory, and deeplink action to proactively organize themed photos and automate CapCut composition.Gallery memories can be generated during scheduled idle-time tasks or by explicit voice or manual invocation, then retrieved for high-level requests such as creating a themed highlight album.
- 6.3 Behavior Cloning and Trajectory Replay: Behavior cloning captures a user’s navigation as a reusable skill, while trajectory replay restores bookmarked pages through progressively simpler launch methods, including task-stack restoration when full replay fails.This enables precise restoration even for apps without public deeplinks, such as reopening an exact video or specific product detail.
7 Conclusion and Future Work
X-OmniClaw presents an edge-native omni-modal Android agent that unifies perception, memory, and action on the smartphone. Future work targets self-evolving execution, adaptive device-cloud collaboration, and continued open-source development.
- Conclusion: X-OmniClaw treats the Android smartphone as a unified substrate for perception, memory, and action rather than isolated screenshot automation or remote control.The architecture integrates on-screen UI state, real-world context, speech input, and personalized information on-device.
- Future Work: Future work will add self-evolving execution that refines trajectories and distills complex reasoning chains to reduce token consumption and response latency.The goal is to improve system intelligence and efficiency through compact trajectory representations.
- Future Work: The architecture is moving toward device-cloud synergy, retaining lightweight on-device processing while selectively offloading intensive open-domain reasoning through secure, intent-aware gateways.This direction is intended to support a more resource-efficient and private agent experience.
- Research Scope and Release: The project is intended strictly for academic research and will release its code, assets, and related artifacts as open source while continuing updates.It does not target specific commercial applications or platforms.