PaperlayerView exact revision on arXiv

Source-linked AI summary

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

Yunxin Li, Jinchao Li, Shibo Su, Zhenran Xu, Chenrui Zhao, Tongshu Bian, Xiaoman Liang, Meishan Zhang, Baotian Hu, Min Zhang

arXiv:2607.12625v2cs.CLcs.CV

TL;DR

OpenClaw-style agents lack robust cross-platform GUI interaction and native self-evolution for complex personal-assistant tasks. KnowAct-GUIClaw combines routed GUI execution with experience memory and self-evolving skills, achieving the highest reported MobileWorld success rate of 64.1%.

  • Problem

    OpenClaw-style agent systems lack sufficient cross-device GUI interaction capacity and native self-evolution mechanisms for complex user tasks.

  • Method

    KnowAct-GUIClaw uses a Know–Route–Act–Reflect pipeline with attribution-aware memory, self-evolving skills, and typed blackboard information transfer.

  • Results

    64.1 SR is achieved on MobileWorld with Kimi-K2.6, the highest SR in the reported table, while distilled experience memory and skills transfer to a Qwen3.5 executor at 41.0 SR.

  • Takeaways & Limitations

    The framework supports cross-model transfer of textual experience rationales and state-validated executable behaviors in the tested Kimi-to-Qwen direction.

  • Takeaways & Limitations

    Reported HarmonyOS failures include inability to settle an inertial wheel picker and premature termination before submitting an order.

Abstract

from arXiv · show

OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience. To resolve these issues, we propose the Know Deeply, Act Perfectly paradigm for personal assistants, which holds that accumulated user interaction and task-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution. Based on this paradigm, we introduce KnowAct-GUIClaw, a novel Know-Route-Act-Reflect framework designed to address OpenClaw's GUI manipulation deficits and break through its cross-platform and recursive self-improvement constraints. First, the host agent leverages accumulated interaction experience and task-relevant knowledge for long-horizon task decomposition and allocation (Know). Second, a pluggable GUI subagent with an experience-attributable memory system (Know) and self-evolving skill library (Act), enabling seamless cross-platform migration and fast-path integration. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls. Extensive experiments across Android, iOS, HarmonyOS and Windows show that KnowAct-GUIClaw achieves superior efficiency, accuracy and cross-platform adaptability. Especially, the GUIClaw with open-source Kimi-2.6 models achieves the best performance (64.1%) on the long-horizon MobileWorld benchmark, beating all agentical frameworks and closed-source agentical models, e.g., Seed-2.0-Pro and GPT-5.5. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8.5% with Kimi-2.6.

1. Introduction

The introduction frames GUI interaction as a central challenge for long-running personal assistants and identifies inefficiency and fragility in simply integrating a standalone GUI agent into OpenClaw. It presents KNOWACT-GUICLAW as an OpenClaw-style framework combining personalized memory and a self-evolving skill library under the principle “Knowing Deeply, Acting Perfectly.”

  • Background: LLM agents are evolving into stateful, long-running personal assistants that preserve workspace state, invoke tools, coordinate subagents, and resume interrupted workflows.The introduction contrasts this direction with ReAct-based agents, which unify reasoning and environmental action execution.
  • Problem: Real-world assistant tasks often require GUI interaction across mobile, web, and desktop environments instead of standardized APIs.Examples include inspecting applications, migrating data, handling permission dialogues, and executing workflows in login-protected environments.
  • Problem: Integrating a standalone GUI agent into OpenClaw causes severe inefficiency and fragility for long-horizon personal-assistant tasks.The introduction attributes these problems to four drawbacks, including cross-application instructions whose intermediate data may be lost in concise summaries.
  • Contribution: KNOWACT-GUICLAW augments an OpenClaw-style host runtime and GUI-centric execution engine with structured knowledge and executable skills.Its two core components are an attribution policy-enhanced personalized memory system and a self-evolving skill library supporting rapid invocation and iterative optimization.
  • Contribution: The framework is organized around the principle “Knowing Deeply, Acting Perfectly” and contributes four connected mechanisms.The supplied introduction identifies this principle and announces the framework’s four-mechanism contribution structure.

2. Related Work

Related work spans LLM-agent runtimes, GUI/computer-use agents, and memory- or skill-based adaptation. KnowAct-GUIClaw builds on these directions while targeting cross-platform deployment, resumable host–executor collaboration, and transferable memory and skills.

  • KnowAct-GUIClaw: KnowAct-GUIClaw uses two-tier host–executor collaboration with structured, resumable GUI tasks, standardized outcomes, progress updates, and reusable checkpoints.The host can resume stalled runs or rearrange subsequent workflows instead of issuing one fuzzy command.
  • KnowAct-GUIClaw: KnowAct-GUIClaw attains SOTA performance on MobileWorld while enabling cross-platform deployment and memory and skill transfer.The supplied passage summarizes these capabilities as outcomes of the model’s experimental evaluation.
  • LLM-Agent Runtimes: LLM-agent runtimes combine language models with action interfaces, memory, and control loops, with OpenClaw and Nanobot emphasizing configuration-centric local-first assistants.These systems expose message channels, external tools, MCP servers, conversation history, and session state.
  • GUI and Computer-Use Agents: GUI and computer-use agents enable language models to operate through visual interfaces, supporting long-horizon web navigation with HTML, accessibility-tree, screenshot, and page-structure observations.Representative systems include Mind2Web, WebArena, VisualWebArena, SeeAct, and WebVoyager.
  • Memory and Skill Reuse: Agent memory and skill reuse improve behavior without weight updates across language-only and embodied settings.Examples include Reflexion’s verbal feedback, Generative Agents’ reflective memory stream, Voyager’s executable skill library, and KnowAgent’s action knowledge base.

3 Preliminaries

GUI automation is modeled as a partially observable Markov decision process, while multi-agent systems distribute problem solving across coordinated, specialized agents. LLM frameworks apply this by pairing a central orchestrator with environment- or tool-facing agents for delegated subtasks.

  • GUI Automation Model: GUI automation is naturally modeled as a partially observable Markov decision process (POMDP).The model captures GUI control as sequential decision-making under incomplete state information.
  • GUI Automation Model: The POMDP is defined by M = (S, A, O, T, Ω, R, γ), with hidden device state and observations such as screenshots, metadata, foreground apps, and bounded action history.The agent observes only a projection of hidden state rather than S directly.
  • Multi-Agent Systems: Multi-agent systems distribute problem solving across agents with distinct roles, capabilities, and local information through communication and shared environments.Recent LLM frameworks pair a central orchestrator maintaining conversation state, memory, and tool access with specialized agents for delegated subtasks.

4 KnowAct-GUIClaw

KnowAct-GUIClaw organizes long-horizon execution as a Know–Route–Act–Reflect loop, combining persistent memory and skills with selective GUI delegation. Its routing decomposes tasks into typed single- or multi-app workflows, while GUIClaw executes subtasks through validated hybrid actions and structured information transfer.

  • Execution loop: KnowAct-GUIClaw organizes long-horizon task execution as a four-stage Know, Route, Act, and Reflect loop.Persistent memory/history and skill/shortcut stores provide advisory context throughout the loop.
  • Host-centric control: The host agent delegates GUI subtasks only when live app state, visual grounding, cross-app manipulation, text entry, or device verification is required.Other subtasks can be answered directly using context, recalled memory, or non-GUI tools, reducing hand-offs and token-heavy GUI traces.
  • Task decomposition: Routing emits either a single GUI task or an ordered multi-app workflow whose subtasks use goal-level tuples (gi, hi, Ii, Oi) and retrieve relevant memory and skills afresh.The router predicts app-scoped goals rather than screen sequences, enabling task-specific retrieval across subtasks.
  • Information transfer: The blackboard exposes declared cross-app inputs and outputs, writing back only trajectory-derived values so downstream subtasks consume typed evidence.Missing required inputs or outputs cause the workflow to fail closed rather than infer or fabricate state.
  • Hybrid action space: GUIClaw executes routed subtasks over a hybrid action space of GUI primitives, reusable skills, validated Android deeplinks and intents, and intervention actions.Skills and shortcuts are validated against the current device state before use, trading efficiency against reliability.
  • Skills: Each stored skill specifies app and platform scope, parameters, reliability counters, and ordered steps, with stable fields fixed in advance and task-dependent values grounded at runtime.App-scoped candidates are retrieved before each subtask for reuse across repeated interactions.

5. Experiments … 5.3 Ablation and Efficiency Analysis

KnowAct-GUIClaw is evaluated on long-horizon mobile GUI tasks using success-rate and execution-cost metrics, with ablations showing that host coordination, experience memory, skills, and direct host participation improve accuracy and efficiency. Its memory and skills also transfer across base models, while skill reuse reduces trajectory costs on applicable tasks.

  • 5.1 Benchmarks and Metrics: MobileWorld provides the primary evaluation: 117 GUI-Only tasks selected from 201 tasks spanning 20 applications, scored by deterministic evaluators.AndroidDaily complements it with tasks grouped by type, complexity, and ambiguity.
  • 5.1 Benchmarks and Metrics: The default setup pairs a Qwen3.5-397B-A17B host with a Qwen3.5-35B-A3B GUI executor, while Kimi-K2.6 configurations use Kimi-K2.6 for both roles.All MobileWorld configurations use the same 117-task GUI-Only subset.
  • 5.1 Benchmarks and Metrics: Evaluation reports success rate alongside GUI steps, GUI task invocations, executed GUI-trace tokens, and host tokens to measure execution cost.Total measures executed GUI traces, while host total measures routing, coordination, external-tool calls, and eligible-subtask resolution.
  • 5.2 Main Results of MobileWorld: The Qwen3.5-35B-A3B executor improves from 24.8 SR alone to 34.5 with the host and experience memory, then 37.9 with skills enabled.Using memory and skills distilled from Kimi-K2.6 trajectories reaches 41.0 SR, 16.2 points above the base executor and 3.1 points above the standard configuration.
  • 5.3 Ablation and Efficiency Analysis: Adding the host, router, and experience memory raises SR from 24.8 to 34.5 for the 35B executor and from 40.7 to 43.3 for the 397B executor.Total tokens remain nearly flat for the 35B executor, 281k vs. 279k, but rise about 7% for the 397B executor, 254k to 273k.
  • 5.3 Ablation and Efficiency Analysis: Skills reduce GUI steps and total tokens while increasing SR; for the 397B host, steps fall from 26.8 to 23.7 and total tokens decrease by about 5%.The savings come from shorter trajectories and fewer screenshot-heavy observations.
  • 5.3 Ablation and Efficiency Analysis: The 397B host achieves the best SR, 46.2, by resolving eligible information-query subtasks directly instead of delegating each through the GUI executor.This lowers GUI task invocations to 1.7 versus 2.5, total tokens to 260,516 from 278,289, and host total to 10,982 tokens.
  • 5.3 Ablation and Efficiency Analysis: Among tasks invoking a skill, reuse removes about three GUI steps per task, −3.3, cuts total and prompt tokens by roughly 6%, and improves SR by +4.8 and +1.9 points.This focused analysis isolates the efficiency and accuracy effect where skills actually apply.

5.4 AndroidDaily Results … 5.5 Case Study

AndroidDaily results show that KNOWACT-GUICLAW benefits especially on analysis-heavy and complex tasks through decomposition, blackboard transfer, and retrieved memory. The case study illustrates how routing, corrective experience memory, navigation-compressing skills, and host intervention support reliable cross-app execution.

  • 5.4 AndroidDaily Results: AndroidDaily evaluates long-horizon cross-application tasks requiring analysis and explicit answers rather than a single visual goal.The resolved-versus-all distinction reflects unavailable entries and their treatment as failures in the overall score.
  • 5.5 Case Study: The case study presents routing, experience-memory correction, navigation compression through skills and shortcuts, and host recovery with direct participation.Screenshots depict actual executions, with readability overlays added for host cards, step labels, thoughts, and blackboard pills.
  • 5.5 Case Study: For long-horizon cross-app requests, the framework decomposes work into app-scoped GUI tasks and transfers only declared outputs through the blackboard.The Email-to-Clock and Email-to-Messages-to-Maps workflows demonstrate explicit handoffs, including a resolved address passed to downstream tasks.
  • 5.5 Case Study: Retrieved experience memory corrects unproductive app and route selection by directing an invite-link task from Mastodon’s mobile settings to the web administration panel.Without memory, GUIClaw remains in the mobile settings and follows a nonproductive path; with memory, it uses the productive interface.
  • 5.5 Case Study: Validated skills and shortcuts compress navigation by landing directly on task-relevant pages or prepopulating messaging fields.A JD shortcut reaches product results directly while Taobao requires five ordinary GUI steps; a messaging shortcut opens SMS composition with recipient and body populated.
  • 5.5 Case Study: The host recovers from failed or contract-violating GUI execution by recording failures, retrieving missing information, replanning, and participating directly.Examples include recovering a phone number from a resume after a missing-contact lookup and resolving an ambiguous hotel name returned by an Email GUI task.

5.5 Case Study … 5.7 Reproducibility

The case studies show KnowAct-GUIClaw using attributed experience, host-mediated recovery, and typed delegation for multi-step GUI workflows. Cross-platform checks demonstrate usability on HarmonyOS and Windows while exposing control, grounding, and reproducibility constraints.

  • 5.5 Case Study: Attributed experience supports workflow recovery by passing observed time through the blackboard and replanning failed contact lookups.The email-to-alarm case subtracts one hour before setting the alarm, while the contact workflow recovers a phone number from a resume before delegating Messages.
  • 5.5 Case Study: Host-mediated recovery resolves incomplete GUI evidence with external web search before delegating typed Messages and Maps subtasks.In the conference-location task, the host expands a hotel name into a full address, and Maps reports a 13-minute walk.
  • 5.6 Cross-Platform Checks: The same interface is tested on HarmonyOS using MobileWorld-derived tasks, including manually initialized cases weakly tied to Android-specific mirrored states.Outcomes are judged with qwen3.5-flash.
  • 5.6 Cross-Platform Checks: Windows usability is assessed with a manually designed desktop set spanning browser, file, office, terminal, and system-control workflows.Windows instructions specify one concrete application when source instructions offered alternatives or named generic applications.
  • 5.7 Reproducibility: HarmonyOS failures reveal low-level control and state-semantic mismatches, including inability to settle an inertial wheel picker at 25 minutes for an 8:25 a.m. alarm.The temporary-tattoo case also stops at order confirmation instead of submitting or returning control.
  • 5.7 Reproducibility: Windows failures expose grounding bottlenecks in mapping natural-language affect labels to unlabeled icons and coordinating sustained observation with immediate clicks.In the WeChat case, the requested laughing emoji is replaced by a visually similar face from the built-in panel.
  • 5.7 Reproducibility: Reproduction uses deterministic MobileWorld evaluators on 117 GUI-Only tasks with a 50-step cap and separately logged GUI-executor traces and host calls.AndroidDaily scoring uses qwen3.5-flash for GUI-only tasks and two human experts applying the 1.0/0.5/0 rule for answer-returning tasks.

6 Conclusion

KnowAct-GUIClaw is proposed as a cross-platform GUI agent framework built on the “Know Deeply, Act Perfectly” paradigm. It addresses limited cross-device GUI interaction and missing native self-evolution through a four-stage Know–Route–Act–Reflect pipeline separating high-level orchestration from low-level visual device manipulation.

  • 6 Conclusion: KnowAct-GUIClaw targets insufficient cross-device GUI interaction and the absence of native self-evolution in mainstream OpenClaw-style agent systems.These are identified as the framework’s two core limitations to address.
  • 6 Conclusion: The framework is built on the “Know Deeply, Act Perfectly” paradigm for cross-platform GUI agents.The paradigm provides the conceptual foundation for KnowAct-GUIClaw.
  • 6 Conclusion: A four-stage Know–Route–Act–Reflect closed-loop pipeline separates high-level task orchestration by the host agent from low-level visual device manipulation by the GUI subagent.This decoupling structures orchestration and device interaction within the framework.

7 Contributors

The paper lists Yunxin Li, Jinchao Li, Baotian Hu, and Min Zhang as contributors. Their affiliations include Harbin Institute of Technology, Shenzhen, and Shenzhen Loop Area Institute.

  • The listed contributors are Yunxin Li, Jinchao Li, Baotian Hu, and Min Zhang.
  • The contributors are affiliated with Harbin Institute of Technology, Shenzhen, and Shenzhen Loop Area Institute.

A Supplementary Case Studies

Supplementary case studies illustrate GUIClaw’s routing, navigation compression, and host-mediated recovery across messaging and map tasks. The examples show validated shortcuts, host grounding through web search, and downstream GUI execution.

  • Routing and navigation compression: A TaoDian task extracts product names, an order number, and a recipient phone number as explicit outputs.These outputs support a validated messaging shortcut for the next step.
  • Routing and navigation compression: A validated messaging shortcut opens SMS composition with the recipient and message body populated, while GUIClaw verifies and sends the message.
  • Host-mediated recovery: An underspecified GUI output is routed back to the host, grounded through web search, and consumed by downstream Messages and Maps GUI tasks.Figure 9 presents this control flow as an annotated Chinese companion to Figure 7.

B Action Space

GUIClaw exposes a unified action space with shared core actions across mobile and desktop, while platform-specific additions and runtime extensions support broader task execution.

  • Unified action space: GUIClaw provides shared pointer, text, navigation, and task-control actions across mobile and desktop platforms.These actions realize Agui ∪ Aask and let one host drive both platforms through the same GUI task interface.
  • Platform-specific actions: Desktop adds key combinations, explicit pointer swipes, and application launch and close, while mobile adds a dedicated enter key.These additions are platform-specific extensions to the shared action surface.
  • Runtime extensions: Skills, Android deeplink/intent shortcuts, and external MCP tool calls further extend GUIClaw’s runtime action set.Together with shared and platform-specific actions, these extensions realize the broader runtime interface described for GUIClaw.

C Skill Extraction and Prompt Contracts

KnowAct-GUIClaw uses execution and reflection prompt contracts to select reusable GUI skills and convert trajectories into structured Python skills. The contracts constrain action choice, skill composition, state grounding, and parameterization for reliable execution.

  • Prompt Contracts: At runtime, GUIClaw retrieves top-k relevant skills into a compact catalog, allowing the executor to select one or continue with ordinary actions.Reflection uses additional contracts to turn GUI experience into reusable knowledge rather than unrestricted trace replay.
  • Execution-Time Contracts: The decision process prioritizes a listed skill when it clearly matches the requested app or workflow, and otherwise selects the most appropriate valid GUI action.A skill may open or navigate the target app internally, so manual app launching is unnecessary when the match is clear.
  • Skill Extraction: Skill extraction produces ONE cohesive reusable GUI skill as Python code rather than multiple tiny functions.The target format uses the OpenGUI skill API, while fixed and dynamic values distinguish static UI parameters from trajectory-dependent inputs.
  • Skill Extraction: Extracted skill bodies contain only await action(...) statements, use approved action types, and forbid control flow, computations, helper calls, and invented actions.Action arguments must use literal strings or {{param}} placeholders, while fixed_values may contain executable action fields but not selectors.
  • Skill Contracts: Every required interactive step receives a natural-language target and valid_state, and all app-launch steps collapse into one initial open_app step.Grounding hints should describe visible user-facing elements rather than raw class or resource identifiers.

D Skill and Shortcut Examples

The examples show a unified skill abstraction spanning parameterized actions, state-validated multi-step workflows, and validated deeplink or intent shortcuts. These entries support Know’s retrieval and Act’s application during GUI execution.

  • Skill and Shortcut Forms: The skill abstraction covers parameterized click-then-type patterns, multi-step skills with per-step state validation, and validated deeplink or intent shortcuts.The examples are drawn from Android’s skill and shortcut store and are abbreviated for readability.
  • Execution Integration: These stored skills and shortcuts are retrieved by Know and applied by Act during GUI task execution.The store therefore links experience-backed retrieval with operational execution.
  • Parameterized Pattern: A click-then-type skill taps a bounded coordinate and then types supplied text.Its skill identifier is compact:action:click_then_type, with the coordinate constrained to [0,999].
  • Multi-Step Validation: The Pinduoduo search skill validates app identity, selector properties, and visible or enabled states before performing input and tapping the search button.The shown signatures include the Pinduoduo package, a search content description, an EditText class or resource ID, and a focused-state requirement for the later step.
  • Validated Shortcut: A JD shortcut specifies the package, component, MIME type, and text extra, with valid_state set to “No need to verify.”The shortcut targets SearchBridgeActivity and substitutes the query into android.intent.extra.TEXT.
Loading 2607.12625v2…