Source-linked AI summary
MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai, Rogerio Feris, Jiebo Luo
TL;DR
Long-horizon GUI agents struggle to preserve decision-useful multimodal state across many interface transitions. MementoGUI adds learned active memory control to frozen backbones and consistently improves performance over history-based and text-only baselines.
Problem
Long-horizon GUI agents remain brittle because passive histories or text-only memories may omit sparse, localized visual evidence needed for decision-making across interface transitions.
Method
MementoGUI augments frozen GUI backbones with MementoCore, which actively writes, compresses, and retrieves multimodal working and episodic memory.
Results
MEMENTOGUI consistently improves frozen GUI backbones across GUI-Odyssey, Multimodal-Mind2Web, and MEMENTOGUI-Bench; on GUI-Odyssey, AMS rises from 54.58 to 68.32 and trajectory success from 1.29 to 3.57.
Takeaways & Limitations
The results support active multimodal memory control as a plug-in alternative to raw history replay or text-only memory for long-horizon GUI agents.
Abstract
from arXiv · showhide
Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce \textbf{MementoGUI}, a plug-in agentic memory framework that equips MLLM-based GUI agents with \textbf{MementoCore}, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce \textbf{MementoGUI-Bench} for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.
1 Introduction
The introduction frames long-horizon GUI control as an active multimodal memory-management problem rather than a single-step visual-understanding problem. MEMENTOGUI addresses this with a learned controller over working and episodic memory, scalable trajectory-derived supervision, a dedicated benchmark, and consistent gains over memory baselines.
- Motivation: Long-horizon GUI agents fail when missed visual cues accumulate, causing forgotten constraints, lost progress tracking, and repeated ineffective actions.This failure occurs across cross-app mobile and multimodal web settings, indicating that active long-term multimodal-state management is the central bottleneck.
- Motivation: Passive history replay and text-only memory can overwhelm agents with redundant context while discarding sparse task constraints, completed subgoals, and localized visual evidence.Useful information is unevenly distributed across long GUI trajectories, making selective memory control necessary.
- MEMENTOGUI framework: MEMENTOGUI augments frozen GUI backbones with a learned controller that actively manages working memory for evolving state and episodic memory for reusable prior experience.The plug-in framework transforms relevant interaction history into decision-useful multimodal state without finetuning the underlying action policy.
- Training and evaluation: 54.58 to 68.32 action matching and 1.29 to 3.57 trajectory success are achieved by MEMENTOGUI with UI-Venus-1.5-8B on GUI-Odyssey, outperforming no-history, history-replay, and text-only memory baselines.The paper also introduces MEMENTOGUI-BENCH and memory-aware metrics for semantic action matching, task progress, and memory consistency, reporting consistent improvements across mobile and web environments.
- Training and evaluation: An automatic curation pipeline converts PSAI computer-use trajectories into supervision for step processing, working-memory compression, episodic writing, and episodic selection.The pipeline provides scalable memory-control training signals with minimal trajectory-level annotation.
2 Related Work
Related work advances GUI agents through compact within-task history management, cross-task trajectory reuse, and broader autonomous-agent memory mechanisms. In parallel, vision-language models and open-source systems have improved GUI grounding and navigation across desktop, web, and mobile environments.
- Memory Systems for Autonomous Agents: GUI-agent memory systems compact long trajectories for within-task state tracking and store reusable cross-task experience as reasoning chains, retrievable traces, or structured knowledge.Examples include MGA and adaptive history modeling for state tracking, alongside Chain-of-Experience, EchoTrail, and HybridAgent for cross-task reuse.
- Memory Systems for Autonomous Agents: Computer-use agents accumulate reusable knowledge through online interaction, demonstrations, or self-improvement, while autonomous-agent memory includes streams, verbal replay, skill libraries, and procedural memory.The cited systems include AppAgentX, MobileGPT, ScaleCUA, UI-Explorer, EvoCUA, and AppAgent; broader approaches also study self-updating memory, retrieval-augmented refinement, and learned memory control.
- Long-Horizon Challenges in GUI Agents: Vision-language models have advanced GUI automation from visual grounding to cross-platform foundation action models, while recent systems improve grounding and navigation across desktop, web, and mobile settings.Examples include MAI-UI, GUI-Owl-1.5, Step-GUI, and UI-Venus-1.5.
3 Data Curation
The data-curation pipeline transforms raw computer-use trajectories into hierarchical annotations and operator-specific supervision for MementoCore. It also creates preference data for online memory operators through rule-based corruption and VLM-judged filtering.
- Preference Data: DPO preference pairs for the Step Processor and WM Compressor use rule-based corruptions to generate negatives, followed by VLM-judged filtering.These operators are selected because they are most directly tied to online memory quality.
- Trajectory Annotation: Raw trajectories are converted into frame-level and subgoal-level annotations that capture interface transitions, changed-region ROIs, and chronological task progress.Frame annotations include actions, event descriptions, input types, applicable key sequences, and ROI boxes; subgoal annotations segment metadata and interaction logs.
- Operator Supervision: Four supervised datasets, Dstep, Dcmp, Dwrite, and Dsel, train the Step Processor, WM Compressor, Episodic Writer, and Episodic Selector.Each example pairs the task goal and relevant multimodal context with a structured target matching its memory operation.
- Operator Supervision: Step-processing supervision combines adjacent-frame annotations with subgoal context to target importance scores, event summaries, ROI boxes, and episodic-retrieval activation tags.Compression supervision simulates working-memory buffers and summarizes older entries while preserving representative visual information.
- Episodic Memory: Episodic-writing examples compress completed trajectories into reusable memories, while episodic-selection examples filter retrieved candidates for relevance to the current task state.These datasets support reusable episodic memory construction and task-conditioned retrieval selection.
4 Methodology
MementoGUI augments a frozen GUI action backbone with MementoCore, which controls multimodal working and episodic memory online. Its methodology combines learned memory operators, ROI-based context construction, structured training, and dedicated long-horizon evaluation.
- MementoCore Architecture: MementoCore augments a frozen GUI backbone with four learned operators for step processing, working-memory compression, episodic writing, and episodic selection.The controller uses four task-specific LoRA adapters on a shared frozen Qwen3-VL backbone.
- Working Memory: Working memory retains salient interface events as summaries and ROI evidence, consolidating older entries when capacity is exceeded instead of replaying the full history.The backbone receives selected ROI crops as ordinary images, with at most Kroi ROI references from compressed blocks and recent entries.
- Episodic Memory: Episodic memory stores reusable completed-episode summaries, metadata, representative ROI crops, and embeddings, using coarse retrieval followed by multimodal selection when retrieval is triggered.Retrieval is initialized at the first step and refreshed only when γt = 1; working memory gates later retrieval.
- Controller Training: The four LoRA adapters are trained as structured memory-control tasks from curated datasets, with DPO applied to step processing and compression while writing and selection use SFT only.DPO addresses the informativeness-versus-context-budget tradeoff for the Step Processor and Working-Memory Compressor.
- Evaluation: MEMENTOGUI-BENCH contains 200 trajectories and 6,953 steps, evaluating memory-dependent GUI decisions with VAM, TPS, and MCS alongside reference-based metrics.The benchmark averages 34.8 steps per trajectory and evaluates reported results on 80 trajectories.
5 Experiments
Experiments across GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI improves frozen GUI backbones through working and episodic memory. Ablations further indicate that ROI-level visual grounding and learned episodic selection are both necessary, while larger controllers generally strengthen long-horizon support.
- Main Results: MementoGUI consistently improves frozen GUI backbones over no-history, predicted-history, and text-summary baselines across the evaluated benchmarks.The evaluation covers GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench.
- Main Results: On GUI-Odyssey with UI-Venus-1.5-8B, working memory raises AMS from 54.58 to 67.69 and trajectory success from 1.29 to 2.69.These gains are further increased when episodic memory is added.
- Main Results: Adding episodic memory further increases UI-Venus-1.5-8B AMS to 68.32 and trajectory success to 3.57.The combined working-plus-episodic configuration also performs better across trajectory-length bins than history-based and text-only memory baselines.
- Ablations: Removing ROI reference images or learned episodic selection consistently degrades performance, showing that visual grounding and filtered episodic experience both contribute to memory control.The visual-memory ablation removes ROI images while retaining learned text memory writing and compression; the retrieval ablation compares single-stage and two-stage selection.
- Controller Scaling: The 8B MementoCore generally achieves the strongest results as controller capacity increases, particularly with working-plus-episodic memory, while the frozen GUI action backbone remains fixed.The scale study compares 2B, 4B, and 8B controllers and reports GUI performance alongside end-to-end trajectory latency.
6 Conclusion
MEMENTOGUI is a plug-in online multimodal memory-control framework that reframes long-horizon GUI control as active memory control. It enables frozen GUI backbones to selectively update, preserve, compress, and retrieve decision-relevant multimodal state across interface transitions.
- Conclusion: MEMENTOGUI provides plug-in online multimodal memory control for long-horizon GUI agents.The framework is designed as an alternative to raw history replay and longer context windows.
- Conclusion: MEMENTOGUI reframes long-horizon GUI control as active memory control rather than raw history replay or longer context windows.This formulation focuses memory operations on decision-relevant multimodal state across interface transitions.
- Conclusion: Frozen GUI backbones can selectively update, preserve, compress, and retrieve decision-relevant multimodal state across interface transitions.These operations allow memory augmentation without changing the GUI backbone.