Source-linked AI summary
UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li
TL;DR
Lightweight GUI agents are attractive for efficient, privacy-conscious deployment but remain unreliable when handling GUI tasks through screenshot-based end-to-end reasoning. UI-KOBE autonomously constructs reusable app knowledge graphs and guides runtime decisions with local graph context. Experiments report improved GUI task performance across model scales, with particularly strong gains for lightweight models, while current deployment remains limited by app-version dependence, an external embedding model, and mobile-only evaluation.
Problem
Lightweight GUI agents are desirable for lower-cost, privacy-conscious deployment but often struggle with long-horizon reasoning and planning.
Method
UI-KOBE autonomously explores mobile applications to construct reusable knowledge graphs and uses graph-supported local options to guide lightweight agents during task execution.
Results
UI-KOBE improves GUI task performance across different model scales, with particularly strong gains for lightweight models.
Takeaways & Limitations
Reusable app knowledge is a promising direction for efficient and deployable GUI agents.
Takeaways & Limitations
The system is app-version dependent, still relies on an external embedding model, and has only been evaluated on mobile applications.
Abstract
from arXiv · showhide
Recent advances in mobile GUI agents have shown strong potential for automating mobile tasks, but most effective systems still depend on large vision-language models for screenshot understanding and long-horizon planning. Small GUI agents that can be deployed directly on mobile devices are more attractive for practical use, offering lower inference cost and better protection of sensitive on-device information. However, due to limited model capacity, such lightweight agents remain unreliable when planning and executing GUI tasks end-to-end from screenshots alone. We propose Knowledge-Oriented Behavior Exploration (\textbf{UI-KOBE}), a framework that improves lightweight mobile GUI agents with reusable app-specific graph knowledge. UI-KOBE first autonomously explores a mobile application and constructs an app knowledge graph, where nodes represent distinct UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph as external guidance: given a user task and the current screenshot, it identifies the current graph node and selects among self-loop actions, neighboring transitions, task completion, or fallback free actions associated with that node. By supporting runtime decisions with app-specific graph guidance, UI-KOBE reduces the burden of end-to-end GUI planning and helps lightweight models perform mobile GUI tasks more effectively, offering a practical step toward efficient, interpretable, and privacy-conscious on-device GUI agents.
1 Introduction
UI-KOBE addresses the difficulty of deploying reliable lightweight GUI agents by separating app knowledge acquisition from runtime task execution. It constructs reusable app-specific graph knowledge and uses local graph guidance to reduce end-to-end planning demands.
- Lightweight GUI agents offer lower inference cost and local handling of sensitive data but often struggle with long-horizon reasoning and planning.
- UI-KOBE autonomously explores applications to construct reusable knowledge graphs whose nodes are UI states and edges are executable transitions.
- At runtime, the agent identifies the current graph node and selects among constrained local options, including self-loop operations and neighboring-state transitions.
- Graph guidance replaces open-ended runtime planning with step-by-step local decisions, while fallback planning preserves robustness when graph guidance is unavailable.
- The framework is designed to improve small-model capability and reliability while grounding actions in explicit graph structures and supporting reuse across tasks.
2 Related Work
Prior GUI-agent systems commonly use end-to-end screenshot-based control, while related work also reuses app knowledge, transition graphs, memory, or interaction histories. UI-KOBE differs by constructing reusable app-specific graph knowledge and using it to guide lightweight agents’ runtime decisions.
- Most recent GUI systems formulate control end to end, predicting actions directly from screenshots or UI representations and task instructions.
- Lightweight-agent variants demonstrate potential for efficient deployment, but UI-KOBE specifically reduces runtime reasoning through reusable app-specific graph guidance.
- Related approaches reuse app knowledge, UI transition graphs, memory, or historical interaction experience to improve GUI task execution.
3 UI-KOBE: Knowledge-Oriented Behavior Exploration
UI-KOBE builds a reusable directed graph of semantic UI states and executable interactions through iterative exploration, then refines the graph through auditing, normalization, and coverage-oriented re-exploration.
- 3.1 Graph Representation: UI-KOBE constructs a reusable mobile-app knowledge graph capturing UI states, executable transitions, navigation, and interaction knowledge.
- 3.1 Graph Representation: Nodes represent distinct semantic UI states rather than individual screenshots, allowing dynamic content to vary across visits when functional roles remain consistent.
- 3.1 Graph Representation: Edges store executable actions and observations, encoding both cross-screen navigation and within-screen state-transforming operations.
- 3.2 Graph Construction: The exploration loop observes screens, identifies or creates nodes, plans unexplored interactions, executes one grounded action, and records the resulting transition.
- 3.3 Graph Refinement and Re-Exploration: Graph refinement addresses duplicate nodes, incorrect transitions, and uneven coverage through auditing, edge-template normalization, and re-exploration of under-explored nodes.
4 Graph-Guided GUI Agent
The runtime agent uses an app knowledge graph to replace open-ended screenshot-based planning with guided local decisions. It identifies the current state, selects graph-supported or fallback options, and grounds each choice into one executable action.
- State Identification: The agent identifies the current UI state in a fixed app graph before selecting an action.Visual retrieval narrows candidates, while semantic verification reduces errors between visually similar but functionally different screens.
- Graph-Guided Decision Making: For the identified node, the agent builds a local option list containing task completion, self-loop actions, neighboring transitions, and free actions.Edges provide instructions, target observations, and optional schema deltas describing expected effects.
- Graph-Guided Decision Making: The runtime policy conditions option selection on the user task, screenshot, identified node, local outgoing edges, and runtime memory.The selected option may be graph-supported or a fallback free action.
- Action Execution: An action grounding model converts each selected instruction into an executable device action, separating high-level choice from low-level interaction.Supported actions include tapping, typing, swiping, and pressing system buttons.
- Runtime Memory: Runtime memory records completed instructions, task-relevant information, and recent observations to prevent repeated execution of the same graph edge.This memory tracks progress across steps and helps the agent determine whether results or confirmations have appeared.
- Fallback Planning: When graph guidance is unavailable, a fallback planner produces only the next immediate instruction, preserving the same action-grounding interface.Once the app returns to a known screen, graph-guided decision making resumes.
5 Experiments
UI-KOBE is evaluated on AndroidWorld and A3 with runtime models spanning lightweight to stronger scales. Across both benchmarks, graph guidance improves task performance, with gains especially pronounced for lightweight models, while requiring a one-time per-app construction cost.
- Benchmarks: UI-KOBE is evaluated on AndroidWorld using task success rate and on A3 using ESAR and Overall SR.A3’s ESAR measures fine-grained progress, while Overall SR measures full task completion.
- Models: The runtime evaluation uses Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-Plus, covering lightweight to stronger model scales.Exploration and graph construction use separate models for grounding, description, planning, verification, auditing, and retrieval.
- Graph Statistics: Each app is explored for 300 steps to construct a compact graph that can be reused across runtime tasks.The graph construction reports audited nodes, edges, average cost, and time; its one-time overhead is amortized over repeated executions.
- A3 Results: 71.5 ESAR and 61 Overall SR with Qwen3.5-4B exceed 43.7 ESAR and 26 Overall SR for the original model on A3.UI-KOBE also reaches 75.7 ESAR and 67 Overall SR with Qwen3.5-9B, and 84.8 ESAR and 78 Overall SR with Qwen3.5-Plus.
- Effect of Graph Guidance: Across both benchmarks, graph guidance improves performance across model scales, with particularly strong relative gains for lightweight models.The exploration cost averages $6.2 and 6.4 hours per app, paid once and amortized across future tasks.
6 Conclusion
UI-KOBE autonomously builds reusable app knowledge graphs containing UI states and executable transitions, then uses them to guide runtime GUI agents. Experiments on AndroidWorld and A3 show improved performance across model scales, particularly for lightweight models.
- Contribution: UI-KOBE autonomously explores mobile apps to capture UI states, executable transitions, and interaction knowledge in reusable graphs.The graphs are later used to guide GUI agents during task execution.
- Runtime Guidance: The graph-guided runtime agent turns end-to-end GUI planning into step-by-step decisions supported by app-specific knowledge.The conclusion identifies reusable app knowledge as a direction for efficient and deployable GUI agents.
- Conclusion: Experiments on AndroidWorld and A3 show improved GUI task performance across different model scales, with particularly strong gains for lightweight models.The reported conclusion covers both benchmark evaluations and multiple runtime model scales.
Limitations
UI-KOBE’s graph is app-version dependent, still relies on an external retrieval model, and has only been evaluated on mobile applications. These constraints motivate graph repair, fully local deployment, and broader GUI-environment evaluation.
- Scope Boundaries: Major UI or navigation changes can make the constructed graph partially outdated, requiring incremental repair or re-exploration.The limitation is explicitly tied to app-version dependence.
- Deployment: External embedding is still required for graph retrieval and node matching, preventing fully local deployment.This limits the current system’s on-device locality despite its lightweight-agent goal.
- Evaluation Scope: The experiments focus on mobile applications, leaving effectiveness on websites and PC applications unverified.Extending graph construction, maintenance, and execution to broader GUI environments remains future work.
A.1 Empirical Study of Design Choices
UI-KOBE evaluates alternative graph retrieval and construction choices, finding that screenshot-based node matching and one-step edges are preferable for stable identification and faithful transition recording.
- Graph Retrieval: Screenshot-based node matching is preferred because text-only retrieval can become unstable when graph construction and runtime models differ.The text-only alternative generates a screenshot description and compares it with stored node descriptions using text embeddings.
- Graph Construction: Compound-action edges make graphs more compact but skip intermediate UI states and may be executed incorrectly across multiple precise steps.The alternative stores multi-step instructions as single edges instead of recording each low-level action separately.
- Graph Construction: One-step edge construction preserves intermediate observations and reduces the risk of incomplete transitions caused by compound-action grounding failures.The paper therefore adopts one-step edge construction after examining the limitations of compound edges.
A.2 Error Study and Additional Analysis
UI-KOBE’s remaining errors arise mainly from imperfect graph construction and incomplete graph coverage, with fallback planning becoming critical when required actions are unexplored.
- Error Sources: UI-KOBE remains below ideal success rates because failures primarily come from graph construction errors and incomplete graph coverage.These two sources are identified through failed-trajectory analysis.
- Graph Construction Errors: Incorrectly grounded actions can create misleading edges that appear locally plausible and are difficult to remove through post-hoc auditing.The planned target may differ from the clicked element while the recorded source, target, and observation still look reasonable.
- Graph Construction Errors: Overly detailed descriptions can leave duplicate UI-state nodes unmerged, fragmenting outgoing transitions and weakening local runtime context.Different dynamic page contents can cause visits to the same screen template to be treated as distinct states.
- Incomplete Graph Coverage: Limited-step autonomous exploration can omit useful edges, forcing tasks outside graph coverage to use the fallback free-action planner.Larger runtime models perform better in these cases because they handle uncovered execution regions more effectively.
- Incomplete Graph Coverage: When a required action is absent from a node’s outgoing edges, Qwen3.5-4B can produce an incorrect fallback, whereas Qwen3.5-9B and Qwen3.5-Plus can recover to graph-guided execution.The example indicates that graph guidance reduces planning burden in covered states, but fallback planning remains a bottleneck outside coverage.
A.3 Qualitative Graph Visualization
The eboox visualization presents UI-KOBE’s audited knowledge graph as semantic UI-state nodes connected by executable transitions discovered during exploration.
- Graph Scope: Figure 3 depicts an audited eboox graph with around 30 nodes, chosen because its relatively small size keeps the full visualization readable.Other evaluated applications contain substantially larger graphs that are difficult to display clearly.
- Visual Encoding: Each node represents a semantic UI state, while each directed edge represents an executable transition discovered during exploration.The graph captures high-level navigation structures as well as executable connections between states.
- Deployment Caveat: Deployment requires safeguards because GUI automation may trigger unintended actions during sensitive operations such as payments, messaging, account changes, or data deletion.The paper recommends avoiding private or sensitive screens when possible and using user confirmation for high-impact actions.