Source-linked AI summary
Subgoal-aware Memory Graph for GUI Agent
Dezhao Luo, Daniel Beechey, Derek Yuen, Weilin Luo
TL;DR
Long-horizon GUI tasks require agents to reuse experience, while action-level and whole-trajectory memories have mismatched granularity for reusable task progress. The paper builds a subgoal-aware memory graph from past interactions; on SPA-Bench, it improves success by 11.11% over a memory-free baseline. Its approach assumes screenshots can be clustered into a compact set of shared states, an assumption limited in highly dynamic interfaces.
Problem
Action-level memories carry little task-progress meaning, while whole trajectories are tied to specific paths and are difficult to reuse across tasks.
Source: §1 · p. 1Method
The method represents contiguous subgoal action sequences as annotated transitions between visually similar shared states in a graph, which supplies guidance from past experience.
Source: §1 · pp. 1–2, §5 · p. 4Results
11.11% improvement over a memory-free baseline with the same Qwen3-VL-8B-Instruct backbone on SPA-Bench’s English single-app level-3 split.
Source: §1 · p. 2Takeaways & Limitations
At inference time, the graph provides subgoal guidance by showing what was tried from similar states and the outcome of each subgoal.
Source: §5 · p. 4The method assumes screenshots can be clustered into a compact shared-state space, limiting its suitability for highly dynamic interfaces where visual similarity is a noisier proxy for semantic equivalence.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
GUI agents aim to automate complex user tasks by interacting with mobile interfaces, but ef- fective decision-making in long-horizon set- tings requires reusing prior experience rather than relying solely on the environment ob- servations. Existing memory-based methods typically store experience either at the single- action level or as entire trajectories. Single- step actions are often too fine-grained to carry meaningful semantics, since the success of an individual action does not necessarily imply progress towards the overall goal. In contrast, whole trajectories are often too coarse-grained and difficult to reuse across different tasks. We propose a subgoal-aware memory graph that distils past trajectories into structured, reusable evidence at the level of subgoals. We define a subgoal as consecutive actions shar- ing a single intent, the granularity at which progress toward the task becomes meaning- ful. To reduce redundancy and support cross- task reuse, we organise subgoal experiences as nodes in a graph, where related memories can be merged, extended, and retrieved based on the environment observation. On SPA-Bench, our method improves success rate by 11.11% over the baseline.
1 Introduction
Long-horizon GUI tasks require reuse of prior experience, but action-level and whole-trajectory memories are respectively too fine-grained and too coarse-grained. The paper proposes subgoal-level memory organized as a graph, reporting an 11.11% improvement over a memory-free baseline on SPA-Bench.
1 Introduction
- Long-horizon tasks can require more than 30 GUI actions, and recurring subgoals make reusing prior experience preferable to re-exploring workflows from scratch.Source: §1 · p. 1
- Action-level memories lack meaningful task-progress semantics, whereas whole trajectories are tied to specific paths and are difficult to reuse across tasks.Source: §1 · p. 1
- Subgoals are contiguous action sequences expressing a coherent sub-intent, capturing goal progress while transferring across related tasks.Figure 1 illustrates that the same subgoal can succeed or fail depending on the underlying behavior.Source: §1 · p. 1, Figure 1 · p. 2
- The graph represents each subgoal as a transition between boundary screenshots, annotated with its intent and outcome-related information, and merges visually similar states across trajectories.The graph can be incrementally extended with new trajectories.Source: §1 · pp. 1–2
- 11.11% improvement over a memory-free baseline with the same Qwen3-VL-8B-Instruct backbone on SPA-Bench’s English single-app level-3 split.Source: §1 · p. 2
2 Related Work
Prior GUI-agent work includes trained end-to-end models and frozen-LLM systems, as well as memories stored at action or trajectory granularity. This paper focuses on reusable subgoal-level guidance organized around shared visual states.
2 Related Work
- End-to-end GUI agents can achieve strong in-distribution accuracy at the cost of substantial training, while frozen-LLM agents rely on supplied context rather than weight updates.Source: §2 · p. 2
- Existing GUI memories include per-step observation-action-outcome triples and whole-episode retrieval, but these representations offer little reusable task structure or remain tied to one task path.GUI-Explorer, HyMEM, and EchoTrail-GUI exemplify these approaches.Source: §2 · p. 2
- The paper targets the underexplored intermediate granularity of contiguous subgoal segments, organized in a graph whose nodes are shared visual states.Source: §2 · p. 2
3 Method
The method extracts annotated subgoals from trajectories, connects their boundary states in a shared-state graph, and retrieves guidance from visually similar states during task execution.
3 Method
- A trajectory corpus is converted into a directed graph of visually similar GUI-state clusters and subgoal edges, queried with the current screenshot for guidance items.Source: §3 · pp. 2–3
3.1 Subgoal Extraction and Annotation
- An LLM segments each trajectory into consecutive actions expressing a coherent sub-intent; only the first and last screenshots are retained as visual anchors.Subgoal annotations include boundary screenshots, intent, visual effect, behavior summary, and an LLM-judged success label.Source: §3.1 · p. 3
3.2 Subgoal-aware Memory Graph Construction
- Visually similar boundary screenshots are merged into shared nodes, and each annotated subgoal becomes a directed edge carrying its intent, effect, outcome, and behavior.Duplicate outgoing edges are removed only when target, outcome, and intent match; distinct evidence is retained.Source: §3.2 · pp. 3–4
3.3 Memory Retrieval
- At each decision step, the system retrieves the top-k nearest memory nodes and returns their successful and failed outgoing subgoals, tagged with outcomes, in a prompt memory block.Source: §3.3 · p. 4
4 Experiments
On SPA-Bench’s English single-app level-3 split, the subgoal-graph memory improves success over a memory-free baseline, while the flat-memory ablation scores 30.77%. The evaluation uses cloud-based Android instances and excludes some apps that cannot run there.
4.1 Setup
- Evaluation uses cloud-based Android instances and SPA-Bench’s English single-app level-3 split, with 13 tasks after excluding apps that cannot run on cloud phones.Qwen3-VL-8B-Instruct is the base model.Source: §4.1 · p. 4
- The visual and text similarity thresholds are both 0.9, retrieval uses the nearest one memory node, and each retrieval adds approximately 0.5 seconds of latency.Memory construction requires 4 GPU-hours.Source: §4.1 · p. 4
4.2 Results
- 11.11% improvement over the memory-free baseline with the same backbone and setup; neither action-level nor task-level memory yields meaningful gains over baseline.Table 1 reports per-attempt success rate averaged over three independent runs.Source: §4.2 · p. 4, Table 1 · p. 4
- Replacing the graph with a flat memory store reduces SR to 30.77%; the authors argue shared-state merging keeps retrieved context precise.Source: §4.2 · p. 4
5 Conclusion
The paper proposes a mobile GUI memory graph that stores experience as subgoals and retrieves it to guide current tasks. It reuses visually similar states as shared nodes linked by subgoal transitions.
5 Conclusion
- The paper proposes a subgoal-aware memory graph for mobile GUI agents, storing experience at the level of subgoals rather than raw actions or whole trajectories.Source: §5 · p. 4
- Visually similar environment states are fused into shared memory nodes, with subgoal transitions attached to make experiences reusable across trajectories.Source: §5 · p. 4
- At inference time, the agent retrieves subgoals from the graph as guidance for the current task, including what was tried from similar states and its outcome.Source: §5 · p. 4
Limitations
The method assumes GUI environments can be approximated by a discrete state space. This fits relatively stable mobile layouts better than highly dynamic interfaces such as video players.
Limitations
- The method assumes GUI environments can be approximated by a discrete state space, with screenshots clustered into shared memory nodes.Source: Limitations · p. 5
- The authors consider this assumption reasonable for typical mobile applications with relatively stable layouts.Source: Limitations · p. 5
- Highly dynamic interfaces such as video players are a boundary: continuous state changes make visual similarity a noisier proxy for semantic equivalence.Source: Limitations · p. 5