MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments

Introduction

• Existing platforms have only 5.2-11.8% memory-related tasks, lack multi-attempt pass@k protocols for long-term learning, and face scalability and accuracy constraints in judgment.Mobile GUI agents show promise on basic tasks but struggle with memory-intensive scenarios [13] [14].Source: §1 · pp. 1–2
• The memory taxonomy distinguishes temporary short-term buffering from cross-session long-term learning and analyzes 11 agents across five architectures.Source: §1 · p. 2
• The benchmark contains 128 snapshot-based tasks across 26 applications; 89.8% require cross-temporal and cross-spatial retention, with pass@1 and pass@k protocols.Source: §1 · p. 2
• MemGUI-Eval uses a three-stage Progressive Scrutiny pipeline and seven hierarchical metrics spanning memory fidelity, learning effectiveness, and execution efficiency.Source: §1 · p. 2
• 4-10× capability gaps on memory-intensive tasks, indispensable short-term memory, +21.9 percentage points from explicit long-term memory, and five failure modes with five design implications characterize the assessment.The assessment evaluates 11 agents through six research questions.Source: §1 · p. 2

Defining Memory for Mobile GUI Agents

• Memory is the ability to retain, process, and use within-task context and cross-task experience to improve decision-making and performance over time.This definition spans contextual information within tasks and experiential knowledge accumulated across tasks.Source: §2 · p. 3

Short-term (in-session) memory

• Short-term memory supports coherent decisions during a task by retaining prior actions, intermediate results, and UI changes across interaction steps.Memory-intensive tasks, including cross-application information transfer and multi-step data collection, require accurate recall of specific information across extended sequences; the taxonomy identifies five architectures: Memory Agent, Action-Thought Pattern, Multi-turn Context, Rule-based Aggregation, and No Historical Context.Source: §2 · p. 3

Long-term (cross-session) memory

• Long-term memory accumulates successful and failed interaction experience into persistent, transferable skills and knowledge for improving efficiency across tasks.The taxonomy distinguishes Success-Based Learning, which extracts reusable shortcuts and tips from successful executions [1] [22], from Failure-Based Learning, which analyzes failed attempts to avoid prior errors [1].Source: §2 · p. 3

3.1 Memory-Intensive Task Suite Design

• Across 128 tasks and 26 applications, task lengths range from 3 to 160 golden steps (average 36.2), 78.1% require cross-application transfer, and difficulty coverage is 37.5% easy, 32.8% medium, and 29.7% hard.Tasks span four app-crossing complexities.Source: §3.1 · p. 3, Figure 2 · p. 4
• The suite contains 115 memory-intensive tasks and 13 standard tasks, which provide baselines for MTPR and support long-term memory assessment with pass@k.Memory-intensive tasks require extracting, retaining, and recalling information across extended interaction sequences.Source: §3.1 · pp. 3–4
• The 128 tasks form 64 mirror pairs with similar app combinations and cognitive demands but distinct requirements, enabling assessment of cross-task transfer of knowledge and strategies.Transfer patterns range from single-app tasks to four-app workflows.Source: §3.1 · p. 4

3.2 A Snapshot-Based Plug-and-Play Framework

• A five-stage pipeline schedules tasks, delivers them through a unified interface, supports emulator interaction, evaluates progress automatically, and resets environments for retries.Retries are triggered when tasks fail or reach maximum step limits.Source: §3.2 · pp. 4–5
• The framework supports scalable parallel execution, rapid snapshot-based environment recovery, and persistent agent state across pass@k attempts, unlike single-attempt benchmarks.Port-based isolation enables concurrent agent evaluation; the default pass@k limit is k = 3.Source: §3.2 · p. 5

4.1 Memory-Specialized Metrics with Hierarchical Assessment

• Seven metrics span short-term fidelity, long-term learning, and execution efficiency: SR, IRR, MTPR, pass@k SR, FRR, Average Step Ratio, Average Time Per Step, and Average Cost Per Step.Short-term measures use pass@1; long-term measures use pass@k, and efficiency measures use both.Source: §4.1 · pp. 5–6

4.2 MemGUI-Eval: A Progressive Scrutiny Evaluator

• MemGUI-Eval’s Progressive Scrutiny pipeline uses four specialized agents to deepen evidence review only when needed, aiming to avoid rigid matching and full-trajectory LLM judging.The agents are the Triage Judge, Step Descriptor, Semantic Judge, and Visual Judge.Source: §4.2 · p. 6

Stage 1: Cost-Effective Triage.

• The Triage Judge screens cases using task goals, raw action logs, and the final three screenshots, marking success only when limited evidence irrefutably meets all requirements; ambiguous cases advance.This conservative first pass is designed to reduce evaluation costs while preserving high precision.Source: Stage 1: Cost-Effective Triage · p. 6

Stage 2: Full Semantic Analysis.

• When triage is inconclusive, the Step Descriptor enriches each action with semantic descriptions, and the Semantic Judge checks task-critical information against text and the final three screenshots.For failed multi-unit memory tasks, the IRR Analyzer quantifies retention; unresolved cases require a list of historical screenshots needed for adjudication.Source: Stage 2: Full Semantic Analysis · p. 6

4.2 MemGUI-Eval: A Progressive Scrutiny Evaluator

• The final visual stage supplies only the historical screenshots requested by the Semantic Judge, requires a definitive success-or-failure decision, and treats missing critical evidence as failure rather than guessing.For failed memory tasks, the Visual Judge triggers the IRR Analyzer to compute final IRR from available visual and textual evidence.Source: Stage 3: Targeted Visual Verification · p. 7

4.3 Validation of the Evaluation Pipeline

• 99.0% F1 on SPA-Bench for M1 exceeded the best baseline’s 92.5%; M2 achieved 95.9% F1 at $0.031 per trajectory, while cross-app F1 was 94.1–100% versus baselines’ 40–61.5%.M2 was selected for subsequent experiments as the optimal quality-cost balance.Source: Table 1 · p. 7, §4.3 · p. 7

5.1 Experimental Setup

• The evaluation covers 11 agents, including two with explicit long-term memory, across 128 tasks run up to three times on Android simulators.Results are assessed with MemGUI-Eval.Source: §5.1 · p. 8

5.2 Main Results

• M3A leads single-attempt success at 32.8%, while Agent-S2 leads multi-attempt performance at 49.2%.Framework-based agents achieve 22.7–32.8% success versus 0.0–6.2% for end-to-end models; MTPR is below 0.1 for most agents, indicating 4–10× capability gaps. Figure 5 also shows drops from AndroidWorld to MemGUI-Bench: Agent-S2 54.3% to 27.3%, GUI-Owl-7B 66.4% to 6.2%, and UI-Venus-7B 49.1% to 5.5%.Source: Table 2 · p. 8, §5.2 · pp. 8–9, Figure 5 · p. 9
• Removing short-term memory collapses M3A’s success rate from 32.5% to 2.5% and Agent-S2’s from 27.5% to 5.0%.Both agents’ information retention rate falls to 0.0%; removing Agent-S2’s long-term memory instead lowers pass@3 success from 45.0% to 25.0%, while agents remain functional.Source: Table 3 · p. 9, §5.2 · p. 9
• Cross-application complexity reduces top-agent success by 16–40 percentage points from single-app to four-app tasks.M3A falls from 46.4% to 30.0%, while Agent-S2 falls from 50.0% to 10.0%; most agents’ information retention collapses on four-app tasks.Source: §5.2 · pp. 9–10, Table 4 · p. 10
• M3A’s multi-turn configuration raises success from 32.8% to 51.6%, a gain of 18.8 percentage points.It leverages Gemini-2.5-Pro’s 1M-token context window; UI-TARS-1.5-7B, limited to a sliding window of the last five turns, achieves 3.1–6.2% success.Source: Figure 6 · p. 10, §5.2 · p. 10
• Agent-S2 improves cross-session success by 21.9 percentage points, from 27.3% to 49.2%, while Mobile-Agent-E improves by 4.7 points.Agent-S2’s failure recovery rate is 21.5%, compared with 0.8–4.4% for agents without explicit memory; Agent-S2 takes 27.5 seconds per step versus M3A’s 5.3 seconds.Source: §5.2 · pp. 10–11
• Under token constraints, Agent-S2’s SR@3 falls from 49.2% to 0.0%, while M3A’s declines from 47.7% to 21.9%.Agent-S2 uses 41,760 tokens per step, and sophisticated memory architectures consume 4.4–5.9× baseline tokens; M3A consumes 31% of Agent-S2’s tokens, while lightweight agents’ performance is unchanged.Source: Table 5 · p. 11, §5.2 · p. 11

• Across 1,265 task executions, the analysis identified five failure modes among 343 non-timeout failures: PMH, ProcMH, OMH, KD, and IM.These are Partial Memory Hallucination, Process Memory Hallucination, Output Memory Hallucination, Knowledge Deficiency, and Intent Misunderstanding.Source: §6 · p. 11
• Memory hallucination accounts for 58.9% of non-timeout failures on average, as shown in the cross-agent failure-pattern heatmap.Source: §6 · p. 11, Figure 7 · p. 12
• The proposed remedies include multi-granularity memory buffers for PMH, hierarchical task decomposition with persistent goal tracking for ProcMH, and strategic long-context use beyond naive history concatenation.Source: §6 · p. 12
• Explicit long-term memory for cross-session learning is evidenced by Agent-S2’s 21.5% FRR versus 0.8-4.4% for agents without dedicated memory; hybrid architectures are proposed to balance memory management and efficient end-to-end models.Source: §6 · p. 12

7 Conclusion

• MemGUI-Bench is a comprehensive memory-centric benchmark using pass@k evaluation and staged LLM-as-judge.Source: §7 · p. 12
• 4–10× capability gaps on memory-intensive tasks, 16–40 pp degradation from cross-app complexity, +18.8 pp from long-context, and +21.9 pp from long-term memory were observed across 11 agents and 128 tasks.The evaluation also found severe computational trade-offs.Source: §7 · p. 12
• Systematic failure analysis identified 5 memory-related failure modes and synthesized 5 design implications for future architectures.The benchmark establishes empirical baselines for advancing memory-enhanced GUI agents.Source: §7 · p. 12

• Memory-centric benchmarking advances the evaluation of mobile GUI agents.Source: Impact Statement · p. 12
• Improved GUI agents can enhance accessibility and automate repetitive tasks.Source: Impact Statement · p. 12
• Potential misuse of autonomous automation makes safeguards important alongside capability improvements.Source: Impact Statement · p. 12

A Appendix

• The appendix lists subsections titled “Evaluation Environment Limitations” (A.1.1) and “Evaluation Pipeline Limitations” (A.1.2).Source: §A.1 · p. 15

Appendix Table of Contents

• The appendix includes a table of contents.Source: §A.1 · p. 15

A.1 Related Work

• A.1 is titled Related Work.Source: §A.1 · p. 15
• A.1.1 is titled Evaluation Environment Limitations.Source: §A.1 · p. 15
• A.1.2 is titled Evaluation Pipeline Limitations.Source: §A.1 · p. 15
• The listed limitations are divided into environment and pipeline categories.Source: §A.1 · p. 15

A.4.1 Application Selection Strategy

• The task-suite-design section includes an application selection strategy.Source: §A.4 · p. 15

A.4.2 Task Suite Characteristics

• The task-suite-design section covers task suite characteristics.Source: §A.4 · p. 15

A.4.3 Memory-Intensive Task Design

• The task-suite-design section covers the design of memory-intensive tasks.Source: §A.4 · p. 15

A.4.4 Information Retention Pathways

• The task-suite-design section addresses information retention pathways.Source: §A.4 · p. 15

A.4.5 Mirror Task Pairs for Long-Term Learning

• The task-suite-design section covers mirror task pairs for long-term learning.Source: §A.4 · p. 15

A.4 Details of Task Suite Design

• These five topics are listed as subsections of the task suite design section.Source: §A.4 · p. 15

A.7.1 Experimental Setup Details

• The section includes experimental setup details.Source: §A.7 · p. 15

A.7.2 Detailed Results Analysis

• The section includes a detailed results analysis.Source: §A.7 · p. 15

A.7.3 Detailed Performance Breakdown

• The section includes a detailed performance breakdown.Source: §A.7 · p. 15

A.7.4 Key Validation Insights

• The section presents key validation insights.Source: §A.7 · p. 15

A.8 Analysis of Failure Cases

• The section is organized around defining failure modes, analyzing their agent-specific distributions and cross-agent patterns, and outlining design implications for future memory-enhanced GUI agents.Source: §A.8 · p. 15

• The section is organized around five analyses: detailed memory performance tables, long-term learning, cross-application complexity, memory ablations, and test-time-compute-normalized evaluation.Source: §A.9 · p. 15

A.1.1 Evaluation Environment Limitations

• Existing benchmarks do not systematically assess mobile-GUI memory: SPA-Bench has 40 of 340 memory tasks (11.8%), while LlamaTouch and MobileAgentBench have none.Cross-application tasks are also limited or absent in most benchmarks.Source: §A.1.1 · p. 16, Table 6 · p. 16

A.1.2 Evaluation Pipeline Limitations

• Rule-based success checks require expert-crafted rules, LLM judges risk overload or fragmented context, and existing metrics omit memory-specific assessment and pass@k.MemGUI-Bench addresses these gaps with a snapshot-based plug-and-play framework, Progressive Scrutiny, and memory-specific metrics.Source: §A.1.2 · pp. 17–18

A.3.2 Long-Term Memory Implementations

• Mobile-Agent-E and Agent-S2 derive reusable procedural knowledge from successful tasks and use lessons from failed attempts to guide later execution.The described long-term memory systems remain early-stage, relying mainly on simple experience aggregation.Source: §A.3.2 · p. 19

A.4.1 Application Selection Strategy

• Applications were chosen for research representativeness, emulator compatibility, and login-free operation; tasks are human-annotated, cross-validated, and rated by step-count difficulty.Difficulty levels are Easy (1–20 steps), Medium (21–40), and Hard (41+).Source: §A.4.1 · pp. 19–20, §A.4.2 · p. 20

A.4.3 Memory-Intensive Task Design

• The suite includes 115 memory-intensive and 13 standard tasks, organized into 64 mirror pairs across 128 tasks to assess cross-task learning.Tasks span single-app to four-app workflows and require retaining information across temporal and spatial scales.Source: §A.4.3 · p. 20, §A.4.4 · pp. 20, 22, §A.4.5 · p. 22

A.4.5 Mirror Task Pairs for Long-Term Learning

• Tasks require comparing category results, remembering the best-value item from each, and then choosing between them, as in Meesho toy and baby-gear searches.The task selects the highest rating-to-price ratio among the first five results in each category, then navigates to the better-value product.Source: §A.4.5 · p. 23
• Other cross-app tasks require collecting product specifications, calculating value scores, and recording the results in Joplin, or checking compatibility using a Bing search.Examples include scoring webcams with (star rating * number of reviews) / price and checking RAM clock speed against a motherboard’s supported maximum.Source: §A.4.5 · pp. 23–24
• Some tasks test retention of long text by having agents reproduce ten Amazon reviews in Joplin or summarize six AP News articles in a section-organized digest.The review task prohibits using the copy function; the news task asks for each article’s title and an approximately 50-word summary.Source: §A.4.5 · p. 24
• Cross-app tasks transfer retrieved information into Joplin, including ranking ten Austin apartments by square footage and recording their addresses and rents.Other examples collect Kindle bestseller data for a note, showing that the task designs span different apps and information types.Source: §A.4.5 · pp. 23–24
• Agents must collect ten Denver rental listings’ addresses, rents, and bedroom counts, then record them in a Joplin note ordered by rent.The task requires carrying details from Apartments.com into a titled note and sorting the listings from lowest to highest rent.Source: §A.4.5 · p. 24
• A Germany AutoUncle task requires comparing Mercedes-Benz C-Class and BMW 3 Series price-to-mileage ratios, then calculating 80% financing for the better-value car.The task transfers listing data into a Calculator workflow across two apps.Source: §A.4.5 · p. 25
• A Citymapper task requires recording time, cost, and transfers for four NYC-to-JFK travel modes, then identifying the fastest, cheapest, and least-transfer option.The requested comparison is to be written in a Joplin note titled “JFK Transit Analysis.”Source: §A.4.5 · p. 26
• Net-a-Porter tasks require recording prices and primary material or color for four designer items across categories, then listing them together in Joplin.The Jimmy Choo example covers Boots, Heels, Sandals, and Sneakers.Source: §A.4.5 · p. 26
• Agents must remember Wi-Fi control and picture-in-picture app permissions, then list each set under its own heading in Joplin.The information is gathered in Settings and reported in a note titled “App Access Permissions.”Source: §A.4.5 · p. 26
• Tasks span multi-step workflows that gather information in one app and reuse it elsewhere, such as recording permission counts in a Joplin note.Source: §A.4.5 · p. 26
• Examples require collecting and organizing sports stories or product reviews, including summaries, ratings, and translations, in notes across apps.Source: §A.4.5 · p. 27
• Other workflows combine product search with calculations or compatibility checks, then record component prices, ratings, totals, or compatible parts.Source: §A.4.5 · pp. 27–28
• Apartment tasks require filtering listings, retaining rents, areas, amenities, or addresses, and comparing value or commute details for a final note or message.Source: §A.4.5 · p. 28
• Flight tasks combine web searches and route planning with remembered details used to create departure or arrival events and transit-related reminders.The London-to-Rome task sets a reminder two hours beyond the journey duration; the Tokyo-to-Seoul task creates an arrival event and a following transit event using the remembered duration.Source: §A.4.5 · p. 29
• Shopping and finance tasks carry item prices or exchange rates across apps into calculated totals, asset accounts, expense transactions, or messages.Examples include discounted and taxed Wish purchases sent by message, and imported electronics converted to another currency before recording an expense.Source: §A.4.5 · p. 29
• Other examples transfer retrieved information into notes or checklists, including comparing artworks and translating the comparison before sending it.The tasks require recalling details from multiple paintings, producing a short comparison, and translating it into Spanish or Italian for messaging.Source: §A.4.5 · pp. 29–30

A.5.1 Parallel Experiment Implementation

• Independent emulator instances, unique ADB ports, and sequential mirror-task execution support parallel experiments while preserving isolation and long-term learning assessment.The multi-attempt protocol permits up to k attempts (default k = 3), retaining agent state while snapshot resets maintain environment consistency; a unified interface supports 12 agents.Source: §A.5.1 · p. 30, §A.5.2 · p. 31, §A.5.3 · p. 31

A.5.4 Advantages Over Existing Approaches

• The snapshot-based framework enables extensible recovery without expert reset scripts, instant recovery for emulator-compatible, login-free apps, and pass@k evaluation with persistent agent state.The authors contrast its scalability with AndroidWorld [19] and AndroidLab [25], its recovery speed with SPA-Bench [5] and A3 [4], and its long-term memory support with single-attempt benchmarks.Source: §A.5.4 · p. 31

A.6.1 Short-Term Memory Assessment Metrics

• IRR measures correctly recalled and used information units, MTPR compares success on 115 memory-intensive versus 13 standard tasks, and pass@k SR and FRR assess multi-attempt success and recovery.IRR distinguishes partial retention from binary task success; FRR weights earlier recoveries more heavily.Source: §A.6.1 · pp. 31–32, §A.6.2 · p. 32

A.6.3 Execution Efficiency Assessment Metrics

• Execution efficiency is assessed using Average Step Ratio on successful tasks, Average Time Per Step across attempts, and AverageCostPerStep across executions.IRR is set to 100% for perfect-success tasks and 0% when early failure processes no information; pass@k success requires success in any attempt, while FRR focuses on initially failed tasks.Source: §A.6.3 · p. 33, §A.6.4 · p. 33

A.7.1 Experimental Setup Details

• Validation used 78 M3A trajectories from 26 SPA-Bench tasks and 256 pass@1 trajectories from all 128 MemGUI-Bench tasks, with three experts annotating success or failure.The SPA-Bench set comprised 18 single-app and 8 cross-app tasks, each executed three times; the MemGUI-Bench trajectories came from M3A and T3A.Source: §A.7.1 · pp. 33–34

A.7.2 Detailed Results Analysis

• On SPA-Bench, MemGUI-Eval M1 reaches 99.0% F1 versus G1’s 92.5% overall, and 100% on cross-app tasks versus baselines’ 40–61.5%.For single-app tasks, M1 achieves 98.8% F1 versus 92.5% for the best baseline.Source: §A.7.2 · p. 34, §A.7.3 · p. 34, Table 11 · p. 35
• On MemGUI-Bench trajectories, MemGUI-Eval M1 achieves 93.1% F1, while agent-specific M1 results span 92.7–93.9%; M3 results span 77.9–79.2%.M2 is described as the optimal balance of accuracy and efficiency and was selected for the main experiments; M3 has the lowest evaluation costs.Source: §A.7.2 · p. 34, Table 12 · p. 35, §A.7.3 · p. 36

A.8.1 Failure Mode Definitions

• Execution timeouts account for 72.3% of failures across 1,265 executions, and trajectory analysis identifies seven failure modes, including partial, process, and output memory hallucinations.Examples include character-by-character deletion exhausting a step limit, forgetting an observed stock price, losing track of remaining task steps, and incompletely transcribing observed permission lists.Source: §A.8 · p. 36, §A.8.1 · pp. 36, 38
• Case studies distinguish memory-independent failures: app-knowledge deficiency, intent misinterpretation despite a correct comparison, and SeeAct’s inability to wait for an app to load.The examples show UI-TARS-1.5-7B selecting Google Calendar instead of the requested N Calendar app and ending on the wrong Wikipedia edition; SeeAct terminates because its action space lacks a wait operation.Source: §A.8.1 · p. 38

A.8.2 Agent-Specific Failure Distribution Analysis; A.8.3 Cross-Agent Failure Pattern Analysis

• Memory hallucination accounts for 58.9% of non-timeout failures on average; Agent-S2 has the highest partial-memory hallucination rate at 66.7%, while model-based agents often show process-memory hallucination.The analysis recommends multi-granularity buffers for fact retention and persistent goal tracking to address these distinct patterns.Source: §A.8.3 · pp. 38, 44, §A.8.2 · p. 38

A.8.4 Design Implications for Future Memory-Enhanced GUI Agents

• The design recommendations include structured, multi-granularity memory buffers with verification and hierarchical task decomposition that preserves high-level goals across app boundaries.They draw on Agent-S2’s 66.7% partial memory hallucination rate and the degradation from single-app success of 42.9–50.0% to 0.0–30.0% in four-app scenarios.Source: §A.8.4 · pp. 44, 46
• The recommendations also call for strategic use of long context, explicit cross-session memory, and hybrid architectures balancing capability with efficiency.M3A-Multi-Turn reaches 51.6% success versus 32.8% for single-turn M3A, while Agent-S2’s 21.5% FRR and 21.9-point cross-session gain illustrate long-term-memory benefits; framework-based agents achieve 22.7–32.8% success at 27.5–38.7 seconds per step, versus 0.0–6.2% at 9.6–12.2 seconds for model-based agents.Source: §A.8.4 · p. 46

A.9.2 Long-Term Learning Analysis

• Agents with explicit long-term memory show 2–4× greater learning potential, while most agents without dedicated memory plateau after the first attempt.Only 2 of 11 agents use explicit long-term memory; the detailed tables report single-attempt IRR and MTPR and multi-attempt FRR for learning from prior failures.Source: §A.9.2 · pp. 47–48, §A.9.1 · p. 47, Table 14 · p. 47

A.9.3 Performance Analysis by Cross-Application Complexity

• Agent-S2’s pass@3 performance rises from 50.0% to 78.6% on single-app tasks and from 10.0% to 30.0% on four-app tasks, while GUI-Owl-7B falls from 21.4% to 0.0–2.9% on multi-app tasks.Agent-S2 maintains IRR of 33.3–51.7% across complexity levels; GUI-Owl-7B’s IRR is 4.0–11.7%.Source: §A.9.3 · pp. 48–49

A.9.4 Memory Ablation Study

• Removing short-term memory drops M3A’s SR from 32.5% to 2.5% and IRR from 35.1% to 0%; removing Agent-S2’s long-term memory lowers pass@3 SR by 20.0 pp.Agent-S2’s pass@3 SR falls from 45.0% to 25.0%, and its FRR decreases from 15.5% to 9.1%; short-term-memory removal also reduces its SR from 27.5% to 5.0% and IRR from 33.3% to 0%.Source: §A.9.4 · p. 49

A.9.5 Test-Time Compute Normalized Evaluation

• Under token constraints, Agent-S2’s SR@3 overall falls 49.2% →0.0%, while M3A retains 21.9% SR@3 overall, or 46% of its unconstrained capability.Agent-S2’s IRR also falls from 39.5% to 0.1%; M3A’s token-constrained SR@3 is 21.9% versus 47.7% unconstrained, while using 31% of Agent-S2’s tokens.Source: §A.9.5 · p. 50

A.10.1 Stage 1: Cost-Effective Triage

• Stage 1 can identify clear success efficiently from the final three screenshots and raw action logs, without detailed semantic analysis.The example confirms successful Amazon product filtering and illustrates cost-effective triage for straightforward cases.Source: §A.10.1 · p. 51, Figure 18 · p. 51

A.10.2 Stage 2: Full Semantic Analysis

• Stage 2 combines action descriptions with visual evidence to resolve complex tasks that Stage 1 marks uncertain, including both success judgments and memory-failure analysis.For a cross-app Amazon, Bing, and Joplin task, the Semantic Judge confirmed success. In a separate failure case, the IRR Analyzer reported 0.9 (9/10 information units correctly recalled).Source: §A.10.2 · pp. 51, 53, Figure 19 · p. 52, Figure 20 · p. 54

A.10.3 Stage 3: Targeted Visual Verification

• Stage 3 requests specific historical screenshots when semantic analysis is inconclusive, then uses a stitched composite and context to make a definitive judgment.The Semantic Judge identifies steps needing confirmation; the Visual Judge uses the requested evidence to confirm success or determine failure and compute final IRR.Source: §A.10.3 · p. 53, Figure 21 · p. 55, Figure 22 · p. 56

A.11.1 Stage 1: Cost-Effective Triage Prompts

• The Stage 1 prompt permits only Success or Uncertain and requires conservative, explicit verification of every task condition from limited evidence.It examines raw action logs and the last three screenshots, and cannot decide Failure at this stage.Source: §A.11.1 · p. 53

A.11.2 Stage 2: Full Semantic Analysis Prompts

• Stage 2 prompts turn before-and-after visual evidence and raw logs into step descriptions, then let the Semantic Judge combine those descriptions with screenshots or request more evidence.If required information remains unverified, the judge can return a required_steps list for additional screenshots.Source: §A.11.2 · pp. 57–58

A.11.3 Stage 3: Targeted Visual Verification Prompts

• The Stage 3 prompt requires a final success-or-failure decision based on requested screenshots synthesized with the full action descriptions, without further requests.Critical task information must be clearly described or visible in the screenshots; otherwise the task is marked failure.Source: §A.11.3 · p. 58

A.11.4 IRR Analyzer: Memory Failure Quantification

• The IRR Analyzer defines retention as correctly used information units divided by required units and applies scenario-specific rules to quantify memory performance.Successful tasks receive 100%; explicit-output partial failures use the correctly output proportion, while incorrect implicit decisions and early failures receive 0%.Source: §A.11.4 · p. 59