Source-linked AI summary
Hybrid Self-evolving Structured Memory for GUI Agents
Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, Biwei Huang
TL;DR
GUI agents still struggle with long-horizon tasks, while existing memories lack the structured organization and continual evolution of human memory. HYMEM combines symbolic strategy nodes with continuous trajectory embeddings in an evolving graph, supporting multi-hop retrieval, node updates, and inference-time refreshing. Across benchmarks, it consistently improves open-source agents, including a +22.5% gain for Qwen2.5-VL-7B and performance above strong closed-source models.
Problem
Existing GUI-agent memories rely on flat discrete summaries or continuous embeddings and lack structured organization and continual evolution for difficult long-horizon tasks.
Method
HYMEM couples discrete high-level symbolic nodes with continuous trajectory embeddings in a graph supporting multi-hop retrieval, self-evolution, and working-memory refreshing.
Results
+22.5% improves Qwen2.5-VL-7B from 12.5% to 35.0%, while HYMEM consistently improves open-source agents and can match or surpass strong closed-source models.
Takeaways & Limitations
HYMEM substantially narrows—and in some settings surpasses—the performance gap between open-source GUI agents and strong closed-source systems.
Takeaways & Limitations
Memory updates rely on heuristics rather than full optimization, and scalability to larger 32B or 70B backbones remains unevaluated.
Abstract
from arXiv · showhide
The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.
1 Introduction
GUI agents struggle with long-horizon, diverse computer-use tasks because intermediate errors and missed conditions remain common. HYMEM addresses limitations of flat memory by combining structured symbolic knowledge with multimodal trajectory evidence and continual updates.
- Long-horizon and diverse real-world computer-use tasks remain difficult because GUI agents make intermediate errors or overlook critical conditions.
- Human-inspired memory motivates preserving fine-grained multimodal episodes while associating them with higher-level concepts for efficient retrieval and reasoning.
- HYMEM merges trajectories into discrete symbolic nodes while retaining continuous trajectory embeddings to preserve detailed multimodal evidence.
- Its graph evolution mechanism adds, updates, and replaces nodes as new trajectories arrive, avoiding uncontrolled memory growth.
- +22.5% improves Qwen2.5-VL 7B from 12.5% to 35.0%, exceeding Gemini2.5-Pro-Vision by 5.4% and GPT-4o by 15.3%.
- HYMEM consistently improves open-source GUI agents and enables 7B/8B backbones to match or surpass strong closed-source models.
2 Related Work
GUI-agent memory methods span discrete, continuous, and hybrid forms, but existing approaches often sacrifice either visual detail or explicit structure. HYMEM places symbolic strategies and continuous multimodal evidence in an evolving graph.
- GUI agents shifted from DOM-based grounding toward visually capable VLMs, but consistency across long-horizon open-ended tasks remains challenging.
- Discrete memories are interpretable but can miss fine-grained GUI visual nuances, whereas continuous memories preserve sensory detail but hinder explicit reasoning through information bottlenecks.
- HYMEM combines high-level textual strategies with continuous trajectory embeddings inside a graph, providing both strategic abstraction and precise perceptual grounding.
- Figure 1 organizes HYMEM around graph construction and four inference phases: retrieval, working-memory initialization, execution, and refresh.
3 Methodology
HYMEM is an evolving graph memory that combines symbolic strategies, semantic attributes, and continuous trajectory embeddings. It supports structured retrieval, VLM-guided memory updates, hybrid working-memory initialization, and phase-sensitive refreshing during execution.
- Hybrid Structured Memory: HYMEM uses a continuous pathway for latent trajectory embeddings and a discrete pathway for symbolic schemas representing high-level concepts.
- Hybrid Structured Memory: Each trajectory node represents a successful interaction sequence and contains a high-level strategy, middle-level semantic attributes, and low-level trajectory embeddings.
- Hybrid Structured Memory: Edges connect trajectory nodes sharing middle-level attributes, creating an associative topology for multi-hop structured search.
- Memory Construction via Self-Evolving: New trajectories enter a three-stage pipeline of relevant-node retrieval, VLM redundancy checking, and structured updating.
- Memory Construction via Self-Evolving: ADD creates nodes for new strategies or attributes, MERGE consolidates complementary evidence, and REPLACE substitutes strictly superior trajectory evidence.
- Memory Construction via Self-Evolving: Graph connectivity is refined after updates so recurring regularities become more coherent, less redundant, and easier to retrieve.
- Inference-time Working Memory: Working memory combines VLM-synthesized strategy guidance with continuous trajectory embeddings, then refreshes after detected phase shifts as GUI states evolve.
4 Experiments
The experiments evaluate HyMEM across multimodal web-agent benchmarks, memory configurations, and scaling settings. Results show consistent gains from hybrid memory, self-evolution, larger memory graphs, and balanced retrieval.
- Experimental setup: The agent uses ReAct-style reasoning with GUI and semantic tools, plus SOM-based labels and UI-INS-7B fallback grounding.The two-stage interface supports screenshot-based action references and location resolution when generated labels are invalid.
- Experimental setup: HyMEM is evaluated on WebVoyager, Multimodal-Mind2Web, and MMInA using task success as the primary metric.The study compares open-source and closed-source base models alongside textual, discrete, continuous, and hybrid memory strategies.
- Main results: HyMEM achieves the best results across open-source models by combining discrete concepts with continuous multimodal representations.Qwen2.5-VL-7B + Hybrid reaches 35.0% (+22.5%), while Qwen3-VL-8B and UI-TARS-1.5-7B also obtain their highest scores with HyMEM.
- Memory self-evolution: Global evolution raises success by ∼25% on Amazon and ∼16.6% on Google Maps through strategy consolidation and UPDATE operations.The reported changes are 48.8% → 61.0% on Amazon and 43.9% → 51.2% on Google Maps; 52% of Google Maps success trajectories trigger updates.
- Memory self-evolution: Local evolution produces a ∼15% Amazon gain by refreshing working context when workflows enter new interaction phases.Dynamic Relevance Assessment was triggered 39 times on Google Maps to detect phase shifts during execution.
- Memory scaling: Scaling from 500 to 1,000 Amazon trajectories increases success from 19.5% to 31.7%, reaching 63.4% with 8,000 examples.Retrieved-item scaling shows steady returns, while graph nodes grow sublinearly as raw trajectories increase through semantic merging.
- Retrieval strategy: The |e Ninitial| = 5 retrieval strategy achieves the highest overall success across Amazon, Coursera, and GMap.Its task success rates are Amazon 63.4%, Coursera 54.8%, and GMap 53.7%, indicating a similarity–diversity trade-off in retrieval.
5 Conclusion
HYMEM addresses flat, non-evolving GUI-agent memory with structured symbolic and trajectory-based retrieval, self-evolution, and working-memory refresh. Across benchmarks, it strengthens open-source agents and narrows or sometimes surpasses the gap to closed-source systems.
- HYMEM combines symbolic strategy nodes with continuous trajectory embeddings in a graph-based memory.The design supports multi-hop seed-and-expand retrieval, node add/merge/replace updates, and on-the-fly working-memory refreshing during inference.
- HYMEM substantially narrows—and in some settings surpasses—the gap to strong closed-source systems.
- Future work includes richer strategy abstractions, improved update policies under distribution shift, and memory evolution for continual learning.
6 Limitations
The authors identify heuristic memory updating and limited evaluation scale as key limitations. They leave principled update learning and evaluation on larger backbones for future work.
- The memory update mechanism relies on heuristics rather than being fully optimized.Reinforcement learning is suggested as a way to learn when and how to use or update memory more effectively.
- Computational constraints prevented evaluation on larger-scale models such as 32B or 70B backbones.Scalability to more powerful backbones remains an important direction for future work.
7 Ethical considerations
The study uses public websites and datasets without login-protected content, private user data, or terms-of-service violations. Real-world deployment still requires safeguards against harmful or unauthorized actions.
- All experiments use publicly accessible websites and publicly available datasets, without login-protected content or private user data.
- Scaled deployment of autonomous web-interface agents requires careful safeguards.The authors call for robust safety mechanisms, access controls, and ethical oversight beyond controlled research settings.
- Without proper constraints, autonomous agents could unintentionally perform harmful or unauthorized actions.
A Implementation Details
The method is evaluated on three public GUI benchmarks using trajectories from existing training data and agent rollouts. Parameter-efficient encoder training updates 1.2% of model parameters, and one epoch is sufficient for strong performance.
- Evaluation Benchmarks: The evaluation covers WebVoyager, Multimodal-Mind2Web, and MMInA.These benchmarks assess dynamic-web multimodal reasoning, out-of-distribution generalization, and real-world multistep planning, respectively.
- Continuous Encoder Training: Training uses 1,009 Mind2Web trajectories and 1,000 successful trajectories collected through agent rollouts.The trajectories come from publicly accessible websites including Amazon, Coursera, and Google Maps.
- Continuous Encoder Training: Only 1.2% of total model parameters are updated through parameter-efficient fine-tuning of Q-Former and LoRA layers.The training completes within 20 hours on a single NVIDIA H100 GPU.
- Continuous Encoder Training: A single training epoch is sufficient to achieve strong performance across Qwen2.5-VL-7B, Qwen3-VL-8B, and UI-TARS-1.5-7B.
- Visualization of Hybrid Memory: Figure 4 visualizes how discrete memory is generated, updated, and used during agent execution.The workflow retrieves structured takeaways, action traces, and visual references before distilling them into actionable guidance.
B Additional Qualitative Results
HYMEM supports GUI execution by retrieving structured memory, refreshing it during task phases, and grounding decisions in both semantic and procedural knowledge. Qualitative evidence shows successful filtering with hybrid memory, while the base model loses context and exhausts its step limit.
- Hybrid Memory Workflow: HYMEM retrieves structured takeaways, action traces, and visual references, then uses refined memory to guide subsequent actions.The workflow includes retrieval, digestion, memory-update decisions, and incorporation of new knowledge during execution.
- Embedding Visualization: Continuous trajectory embeddings form meaningful domain clusters, with Amazon, Coursera, and Google Map separated while Travel and Shopping remain closer.The t-SNE visualization uses one point per stored trajectory and colors points by domain.
- Failure Case: Without hybrid memory, Qwen2.5-VL-7B fails to apply the necessary filters, repeatedly opens similar products, loses context, and reaches the maximum step limit.The base model locates the search bar and sorts results but does not make meaningful progress toward a suitable product.
C Prompts Used in HYMEM
HYMEM uses prompts for inference-time guidance digestion, conservative local working-memory refresh, and global graph evolution. Together, these prompts convert retrieved experience into navigation cues and deterministic memory updates.
- Guidance Digestion: Guidance digestion converts the current task intent, screenshot, and retrieved summaries into compact task-specific navigation cues.The prompt emphasizes key actions and filters while avoiding termination-related instructions.
- Local Self-Evolution: Local self-evolution refreshes working memory after genuine phase shifts while preserving reusable takeaways and ignoring operational errors.The prompt decides whether to refresh after each environment transition and identifies takeaways to preserve during reretrieval.
- Global Evolution: Global evolution deduplicates successful trajectories and chooses ADD, UPDATE, or REPLACE using textual metadata and screenshots.The judge returns a strict JSON decision for downstream graph updates.
D LLM Usage Statement
The authors state that LLMs were used only for grammatical refinement and language polishing. They report no substantive LLM edits to the scientific content, conceptual development, or experimental results.
- Scope of LLM Use: LLMs were used solely for grammatical refinement and language polishing to improve manuscript readability.The statement limits their use to language refinement.
- Author Responsibility: The authors state that LLMs made no substantive edits to scientific content, conceptual development, or experimental results.The authors retain responsibility for the final content and accuracy.