Source-linked AI summary
MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, Zhengxi Lu, Gao Wu, Hao Wang, Liang Liu, Yong Liu
TL;DR
Existing mobile GUI benchmarks underrepresent memory-intensive tasks and do not evaluate cross-session learning. MemGUI-Bench addresses this gap with memory-centric pass@k evaluation and staged LLM-as-judge assessment, finding substantial memory deficits and benefits from explicit long-term memory.
Problem
Current evaluation platforms include only 5.2-11.8% memory-related tasks, lack multi-attempt pass@k protocols for long-term learning, and face scalability and accuracy constraints in judgment methods.
Method
MemGUI-Bench provides a memory-centric benchmark with pass@k evaluation and staged LLM-as-judge, supported by a taxonomy of short-term and long-term memory.
Results
4-10× capability gaps appear on memory-intensive tasks, while short-term memory is mandatory and long-term memory yields +21.9 pp; the evaluation also identifies 5 failure modes and 5 design implications.
Takeaways & Limitations
MemGUI-Bench establishes empirical baselines for advancing memory-enhanced GUI agents toward more capable and human-like mobile automation.
Takeaways & Limitations
Agent-S2 requires 27.5 seconds per step versus 5.3s for M3A, showing a substantial computational cost for memory-equipped operation.
Abstract
from arXiv · showhide
Reliable mobile GUI agents must retain and reuse information across actions, applications, and repeated interactions. However, current benchmarks systematically underrepresent these memory demands: only 5.2-11.8 percent of their tasks are memory-related, and none evaluates cross-session learning. We introduce MemGUI-Bench, a comprehensive memory-centric benchmark that assesses both short-term information retention and long-term experience accumulation through pass@k protocols and staged LLM-as-judge evaluation. Our contributions include: (1) a systematic taxonomy of short- and long-term memory based on 11 agents across 5 architectures; (2) a snapshot-based suite of 128 tasks across 26 applications, organized into 64 mirror pairs, where 89.8 percent require cross-temporal and cross-spatial retention; (3) MemGUI-Eval, an automated 3-stage Progressive Scrutiny pipeline with 7 hierarchical metrics spanning memory fidelity, learning effectiveness, and execution efficiency; and (4) an assessment of 11 state-of-the-art agents guided by 6 research questions. Our experiments reveal substantial memory deficits across all evaluated systems, including 4-10x capability gaps on memory-intensive tasks. They further show that short-term memory is indispensable, while explicit long-term memory improves cross-session learning by 21.9 percentage points, with cross-application transfer and computational cost remaining major bottlenecks. We additionally identify 5 distinct failure modes and synthesize 5 actionable design implications for future memory-enhanced agents. All resources, including code, benchmark, and evaluation results, will be fully open-sourced and continuously maintained at https://memgui-bench.github.io/.
1 Introduction
MemGUI-Bench addresses the lack of standardized evaluation for mobile GUI-agent memory by testing short-term retention and long-term learning in realistic interactions. It combines a memory-centric task suite, pass@k protocols, and automated staged judgment to expose substantial deficits across current agents.
- Evaluation Gap: Current benchmarks contain only 5.2-11.8% memory-related tasks and lack multi-attempt pass@k protocols for long-term learning.Existing judgment methods also face scalability and accuracy constraints.
- Benchmark Scope: MemGUI-Bench evaluates both short-term contextual retention and long-term cross-session learning in mobile GUI interactions.The benchmark is motivated by memory demands involving intermediate results, UI changes, and cross-application data transfer.
- Benchmark Scope: The benchmark includes 128 tasks across 26 applications, with 89.8% requiring cross-temporal and cross-spatial information retention.Its snapshot-based framework supports pass@1 and pass@k evaluation protocols.
- Evaluation Pipeline: MemGUI-Eval applies Progressive Scrutiny across 3 stages and 7 hierarchical metrics covering memory fidelity, learning effectiveness, and execution efficiency.The evaluator is designed to automate assessment of memory-intensive tasks.
- Findings: Evaluation of 11 agents reports 4-10× capability gaps, mandatory short-term memory, a +21.9 pp long-term-memory benefit, and 5 distinct failure modes.The study also synthesizes 5 actionable design implications.
2 Memory in Mobile GUI Agents
The paper defines mobile GUI-agent memory as retaining, processing, and using contextual information and experiential knowledge to improve decisions and task performance over time. It distinguishes short-term in-session memory from persistent, transferable long-term learning.
- Memory Definition: Mobile GUI-agent memory is the ability to retain, process, and use contextual information and experiential knowledge across tasks over time.The definition links memory to decision-making and task performance.
- Short-Term Memory: Short-term memory temporarily retains prior actions, intermediate results, and UI-state changes during a task session.It supports coherent decision-making across sequential interaction steps.
- Short-Term Memory: Memory-intensive tasks require extracting, retaining, and recalling specific information across extended sequences such as cross-application transfer and multi-step collection.The paper identifies five short-term architectures, including memory agents, action-thought patterns, multi-turn context, rule-based aggregation, and no historical context.
- Long-Term Memory: Long-term memory accumulates knowledge from successful and failed interactions into reusable, persistent, and transferable skills.The taxonomy separates success-based learning from failure-based learning.
3 Memory-Centric Benchmarking Environment
MemGUI-Bench combines a memory-intensive task suite with a snapshot-based plug-and-play framework for scalable, reproducible evaluation. Its tasks span applications, interaction complexity, and paired sessions to assess both retention and learning.
- Task Suite: The suite contains 128 tasks across 26 real-world applications, spanning four app-crossing complexities and task lengths from 3 to 160 golden steps.Average task length is 36.2 steps, and 78.1% require cross-application information transfer.
- Task Suite: 115 memory-intensive tasks require extracting, retaining, and recalling information across extended interaction sequences, while 13 standard tasks provide baselines.Standard tasks support MTPR computation and pass@k-based long-term memory assessment.
- Task Suite: Tasks implement information-transfer patterns from single-app scenarios to four-app workflows, including extraction, search, computation, and recording across applications.AnalyzeApartmentCommute exemplifies the highest cross-application complexity.
- Long-Term Learning: The 128 tasks form 64 mirror pairs with similar application combinations and cognitive demands but distinct requirements for measuring cross-task learning.The paired design tests whether agents transfer knowledge and strategies from earlier attempts.
- Framework: The snapshot-based framework supports efficient, scalable, reproducible evaluation through parallel execution, rapid environment recovery, and native pass@k support.Automatic reset and retry attempts enable systematic long-term learning assessment.
4 An Automated Evaluation Pipeline with Memory-Specific Metrics
MemGUI-Eval combines seven memory- and efficiency-specific metrics with a three-stage Progressive Scrutiny pipeline that escalates evidence only when needed. Validation shows strong accuracy, cost effectiveness, and cross-application performance.
- Metrics: The evaluation framework uses 7 metrics across short-term memory fidelity, long-term learning capabilities, and execution efficiency.The dimensions cover retention, learning across attempts, path length, time, and cost.
- Metrics: Short-term assessment uses pass@1 success rate, information retention rate, and MTPR to measure task success, recalled information, and memory-specific proficiency.MTPR compares memory-intensive with standard-task performance.
- Metrics: Long-term assessment uses pass@k success rate and failure recovery rate to quantify experience accumulation and learning from failed attempts.Failure recovery applies harmonic decay weighting to reward faster recovery.
- Progressive Scrutiny: Progressive Scrutiny begins with minimal-evidence triage, adds semantic trajectory analysis when inconclusive, and performs targeted visual verification using requested historical screenshots.The staged design seeks a cost-accuracy balance without overwhelming the judge with complete trajectories.
- Validation: MemGUI-Eval reaches 95.9% F1-score at $0.031 per trajectory in its M2 configuration and achieves 94.1-100% cross-app F1 where baselines reach 40-61.5%.Validation covered SPA-Bench and MemGUI-Bench trajectories against human annotations.
5 Benchmarking GUI Agent Baselines
Evaluation of 11 GUI agents reveals substantial memory deficits, with short-term memory required for functionality and long-term memory improving cross-session learning. Cross-application complexity and deployment compute constraints remain major bottlenecks.
- Overall performance: 4-10× capability gaps show that standard benchmarks hide substantial deficits on memory-intensive tasks.M3A reaches 32.8% single-attempt success, while framework-based agents outperform end-to-end models and performance declines on harder tasks.
- Memory ablations: Short-term memory is mandatory, whereas long-term memory is beneficial but optional for functional GUI agents.Removing short-term memory collapses performance and IRR, while removing Agent-S2’s long-term memory reduces pass@3 SR by 20.0 percentage points but leaves the agent functional.
- Cross-application complexity: 16-40 percentage point drops from single-app to four-app tasks identify cross-application transfer as the primary memory bottleneck.M3A drops 16.4 pp and Agent-S2 drops 40 pp in SR, while most agents collapse on four-app tasks.
- Long-context capability: +18.8 pp improvement raises M3A from 32.8% to 51.6% when switching from single-turn to multi-turn context.UI-TARS-1.5-7B remains at only 3.1-6.2% success with a five-turn sliding window.
- Long-term learning and deployment: 21.9 percentage points of cross-session improvement raises Agent-S2 from 27.3% to 49.2%, while computational cost remains substantial.Agent-S2 requires 27.5 seconds per step versus 5.3 seconds for M3A, and high-token agents degrade sharply under token constraints.
6 Failure Pattern Analysis
Failure analysis of 1,265 executions identifies five memory-related failure modes among non-timeout failures. Memory hallucination dominates the observed failures, motivating targeted architectural responses.
- Failure taxonomy: Five memory-related failure modes were identified among 343 non-timeout failures in 1,265 task executions.The categories are Partial Memory Hallucination, Process Memory Hallucination, Output Memory Hallucination, Knowledge Deficiency, and Intent Misunderstanding.
- Failure distribution: 58.9% of non-timeout failures are memory hallucinations on average.Figure 7 summarizes the failure distribution across all evaluated agents.
- Design implications: The proposed design directions include multi-granularity memory buffers, persistent goal tracking, strategic long-context use, and explicit long-term memory mechanisms.These directions target partial and process memory hallucinations while supporting cross-session learning.
7 Conclusion
MemGUI-Bench provides a memory-centric benchmark and staged evaluation framework for mobile GUI agents. Its assessment exposes substantial memory deficits and identifies failure modes and design implications for future architectures.
- MemGUI-Bench combines pass@k evaluation with staged LLM-as-judge assessment for memory-centric mobile GUI benchmarking.
- 4-10× capability gaps appear on memory-intensive tasks, while short-term memory is mandatory and long-term memory is optional but beneficial.
- The study identifies 5 memory-related failure modes and synthesizes 5 design implications for future architectures.
Impact Statement
The work positions memory-centric benchmarking as a way to advance mobile GUI agent evaluation while acknowledging potential misuse risks in autonomous automation.
- Improved GUI agents may enhance accessibility and automate repetitive tasks, but autonomous automation also carries potential misuse risks.
A.1 Related Work
The appendix identifies evaluation environment and evaluation pipeline limitations as related-work topics.
- The appendix includes evaluation environment limitations as a related-work topic.
- The appendix includes evaluation pipeline limitations as a related-work topic.
A.3 Detailed Memory Implementations
The appendix organizes detailed memory implementations into short-term and long-term memory categories.
- The appendix covers short-term memory implementations.
- The appendix covers long-term memory implementations.
A.4 Details of Task Suite Design
Appendix A.4 covers application selection, task-suite characteristics, memory-intensive task design, information-retention pathways, and mirror task pairs for long-term learning.
- A.4.1 Application Selection Strategy: Application selection strategy is detailed in Appendix A.4.1.
- A.4.2 Task Suite Characteristics: Task-suite characteristics are detailed in Appendix A.4.2.
- A.4.3–A.4.5: Memory-intensive task design, information-retention pathways, and mirror task pairs for long-term learning are detailed in Appendices A.4.3–A.4.5.
A.5 Details of Snapshot-based Plug-and-Play Architecture
Appendix A.5 covers parallel experiment implementation, multi-attempt support for long-term memory, comprehensive agent integration, and advantages over existing approaches.
- A.5.1 Parallel Experiment Implementation: Parallel experiment implementation is detailed in Appendix A.5.1.
- A.5.2 Long-Term Memory Support Through Multi-Attempt Mechanism: Multi-attempt mechanisms support long-term memory evaluation in Appendix A.5.2.
- A.5.3–A.5.4: Comprehensive agent integration and advantages over existing approaches are detailed in Appendices A.5.3–A.5.4.
A.6 Details of Memory-Specialized Metrics
Appendix A.6 covers metrics for short-term memory, long-term memory, execution efficiency, and computational considerations.
- A.6.1 Short-Term Memory Assessment Metrics: Short-term memory assessment metrics are detailed in Appendix A.6.1.
- A.6.2 Long-Term Memory Assessment Metrics: Long-term memory assessment metrics are detailed in Appendix A.6.2.
- A.6.3–A.6.4: Execution-efficiency metrics and computational considerations are detailed in Appendices A.6.3–A.6.4.
A.7 Details of Evaluation Pipeline Validation
The validation section identifies gaps in existing memory evaluation and presents MemGUI-Bench and MemGUI-Eval as responses, with strong accuracy across benchmark and memory-intensive tasks at configurable cost.
- Existing Benchmark Limitations: Current benchmarks underrepresent memory-intensive tasks and often omit cross-application or historical dependencies.SPA-Bench contains 40 memory tasks out of 340 total tasks (11.8%), while LlamaTouch and MobileAgentBench contain zero memory tasks.
- Environment Scalability Constraints: Existing environments require manual script writing for new tasks, limiting scalable memory-intensive evaluation.
- Evaluation Pipeline Limitations: Rule-based and existing LLM-as-a-Judge methods face accuracy, context, information-overload, and automation limitations on memory-intensive tasks.
- Metrics Limitations: Current metrics rely on single-round success rate and lack pass@k protocols for assessing long-term learning.
- MemGUI-Bench Response: MemGUI-Bench addresses these gaps through memory-intensive tasks, a snapshot-based plug-and-play framework, Progressive Scrutiny, and memory-specific metrics.
- Validation Results: 99.0% F1-score is achieved by M1 on SPA-Bench trajectories, while M2 reaches 95.9% F1-score at $0.031 versus $0.055 for G1.
- Validation Results: 93.1% F1-score is achieved by M1 on MemGUI-Bench trajectories, with evaluation maintaining high performance across memory-intensive scenarios.
- Detailed Performance Breakdown: 98.8% F1-score versus 92.5% for the best baseline is reported on single-app SPA-Bench tasks, while M1 achieves perfect performance on cross-app tasks.