PaperlayerView exact revision on arXiv

Source-linked AI summary

MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management

Guangyi Liu, Gao Wu, Congxiao Liu, Pengxiang Zhao, Liang Liu, Mading Li, Qi Zhang, Mengyan Wang, Liang Guo, Yong Liu

arXiv:2606.19926v1cs.HC

TL;DR

Long-horizon mobile GUI agents struggle to retain intermediate facts across many steps and app transitions. MemGUI-Agent addresses this with Context-as-Action, and its zero-shot and 8B variants achieve strong performance across MemGUI-Bench and MobileWorld.

  • Problem

    Mobile GUI agents remain unreliable on long-horizon tasks requiring intermediate-fact retention, multi-step progress tracking, and cross-app information transfer.

  • Method

    MemGUI-Agent makes context management first-class through ConAct, which jointly selects UI or memory actions and maintains folded history, persistent UI state, and recent step records.

  • Results

    MemGUI-Agent achieves 62.5% Pass@3 on MemGUI-Bench and 29.1% SR on MobileWorld, while MemGUI-8B-SFT attains the best open-data 8B performance and generalizes across benchmarks.

  • Takeaways & Limitations

    Proactive, policy-level context management supports long-horizon mobile GUI control across different model scales and environments.

  • Takeaways & Limitations

    The experiments focus on Android-style mobile GUI environments, leaving extension to iOS, desktop, and web interfaces for future work.

Abstract

from arXiv · show

MLLM-based mobile GUI agents have made substantial progress on short-horizon tasks, yet remain unreliable on long-horizon tasks that require retaining intermediate facts across many steps and app transitions. We attribute this limitation to ReAct-style prompting, which passively accumulates per-step records, leading to prompt explosion and dilution of critical cross-app facts. To address this, we introduce MemGUI-Agent, an end-to-end long-horizon mobile GUI agent with proactive context management. MemGUI-Agent is built on Context-as-Action (ConAct), which casts context management as first-class actions emitted by the same policy that selects UI actions. Instead of passively appending history, ConAct maintains three structured context fields: folded action history, folded UI state, and recent step record, preserving critical UI facts while keeping context compact. To make proactive context management learnable across model scales, we construct MemGUI-3K, a 2,956-trajectory dataset with full ConAct annotations for supervised training and offline analysis. Training an 8B model on MemGUI-3K produces MemGUI-8B-SFT, an 8B MemGUI-Agent that achieves the best open-data 8B performance on MemGUI-Bench and generalizes to the out-of-distribution MobileWorld benchmark. Code, data, and trained models will be released at https://memgui-agent.github.io/.

1 Introduction

Long-horizon mobile GUI agents struggle to retain intermediate and cross-app facts as task horizons grow, because passive context strategies cause prompt explosion and information loss. MemGUI-Agent addresses this with ConAct, proactive policy-level context management, and MemGUI-3K supervision, achieving strong long-horizon results.

  • Motivation: 71.6% on AndroidWorld falls to 38.2% on MobileWorld for GUI-Owl-1.5-8B as average task length rises from 8.4 to 27.8 steps.This illustrates the reliability gap on long-horizon tasks requiring intermediate-fact retention, progress tracking, and cross-app information transfer.
  • Motivation: Passive strategies append, truncate, or mechanically discard records, causing prompt explosion and dilution, paraphrasing, truncation, or forgetting of critical cross-app facts.Examples of such facts include prices, identifiers, and copied text.
  • MemGUI-Agent: MemGUI-Agent uses ConAct to emit context-management actions alongside UI actions and maintain Folded Action History Ht, Folded UI State Mt, and Recent Step Record Lt.All three fields are predicted in a single forward pass, keeping context curation inside the end-to-end policy.
  • Results: 62.5% Pass@3 on MemGUI-Bench is achieved by MemGUI-Agent applied zero-shot to Qwen3-VL-235B-Thinking, surpassing agentic workflows built on Gemini-2.5-Pro.The introduction also reports flatter prompt growth than ReAct-style prompting.
  • Contributions: 2,956 trajectories comprise MemGUI-3K, which provides full ConAct annotations for supervised training and offline analysis across model scales.Training MemGUI-8B-SFT on MemGUI-3K yields the best open-data 8B performance on MemGUI-Bench and generalizes to the out-of-distribution MobileWorld benchmark.

2 MemGUI-Agent: End-to-End Mobile GUI Agent with ConAct

MemGUI-Agent treats context management as a first-class policy behavior through ConAct, jointly selecting UI or memory actions and context updates. Its structured context combines folded action history, folded UI state, and recent step records, with all decisions produced by one multimodal policy.

  • ConAct formulation: ConAct jointly decides which UI or memory action to execute, which history to fold, and how to describe the current interaction.This makes context maintenance a policy-level behavior rather than a passive log external to the model.
  • ConAct formulation: The working context contains Folded Action History, Folded UI State, and Recent Step Record for compressed trajectories, persistent UI facts, and latest-step details.Unlike ReAct-style prompting, ConAct updates these fields through model-emitted context actions instead of appending a growing task-progress string.
  • History folding: History folding compresses step spans or completed subtasks into reusable records, shifting context growth from linear accumulation toward sub-linear growth.Step-level distillation retains the latest step, while span-level abstraction summarizes an earlier range.
  • UI memory actions: Memory writes preserve complete task-relevant information, such as copied prices or phone numbers, so facts survive screen changes, delays, and app transitions.Memory items are structured triples with identifiers, descriptions, and complete content, and support add, update, and delete operations.
  • Step execution loop: A single multimodal policy emits folding, memory, action-intent, and UI decisions in one forward pass, letting context operations inherit task-level reasoning.Self-described screen facts and action intent provide grounded content for memory writes and future history folding.

3 MemGUI-3K Dataset

MemGUI-3K was created because ConAct alone benefits only the strongest tested backbone, motivating supervised learning for proactive context management. The dataset expands seed tasks, filters teacher rollouts, and provides verified ConAct training samples across diverse mobile-app trajectories.

  • Motivation: ConAct gains emerge only for Qwen3-VL-235B-Thinking, while smaller models and Qwen3-VL-235B-Instruct often regress under the same protocol.This scale dependence motivates constructing MemGUI-3K for supervision.
  • Task expansion: MemGUI-3K expands 128 MemGUI-Bench seed tasks through entity substitution, memory-operation augmentation, and task simplification into a 7,303-task pool.The expansion strategies preserve task structure, increase memory-action coverage, and decompose complex multi-app tasks.
  • Teacher rollouts: 5,293 tasks enter rollout after expansion, using Qwen3-VL-235B-Thinking as teacher with the full 5-part ConAct protocol in MemGUI-Bench’s Android environment.Rollouts terminate on terminate or at the task-specific budget 2.5g + 1, and trajectories undergo progressive scrutiny plus step-level reasonableness annotation.
  • Filtering and samples: 2,956 successful trajectories across 26 apps form MemGUI-3K after filtering and a 90/10 split, with zero overlap against the 128 MemGUI-Bench evaluation tasks.The dataset yields 64,430 SFT samples: 57,951 train and 6,479 test.
  • Dataset statistics: 28.8 average trajectory steps make MemGUI-3K about 1.9× longer than GUIOdyssey’s 15.3-step average, while the dataset spans 26 apps and 7 categories.The median trajectory length is 25 steps, and productivity, shopping, and information & news are the most frequent categories.

4 Experiments

Experiments show that ConAct improves long-horizon GUI performance across benchmarks, scales to an 8B model, and transfers beyond MemGUI-Bench. Offline analyses, ablations, and case studies attribute these gains to complementary context-control mechanisms that reduce context-induced hallucinations.

  • Benchmark results: MemGUI-Agent-235B achieves 37.5% Pass@1 and 62.5% Pass@3 on MemGUI-Bench, surpassing M3A on Gemini-2.5-Pro at 32.8% / 47.7%.It also improves over the same backbone by +13.3 Pass@1 and +16.8 IRR.
  • Benchmark results: MemGUI-Agent-235B reaches 29.1% SR on MobileWorld, improving by +14.6 over the Qwen3-VL-235B-Thinking baseline.This demonstrates transfer to a different environment and app set.
  • Benchmark results: MemGUI-8B-SFT reaches 23.4% Pass@1 on MemGUI-Bench versus 9.4% for Qwen3-VL-8B-Instruct, with larger gains on Medium and Hard tasks.The gains are +19.0 on Medium, +18.5 on Hard, and +6.2 on Easy; the learned skills also transfer beyond the source benchmark.
  • Offline analysis: Offline evaluation on 295 trajectories and 6,479 steps isolates action, memory, folding, and formatting decisions, finding the largest gain in promoting transient observations to durable memory.Each reasonable step uses the same SFT input and compares the generated response with the gold response.
  • Ablation study: Full ConAct reaches 40.0% Pass@1 / 62.5% Pass@3 on MemGUI-Bench-40, exceeding single-component variants whose Pass@1 scores range from 17.5% to 25.0%.UI memory actions, history folding, and self-describing outputs address persistent state, compression, and grounded context reuse respectively.
  • Error analysis: Full ConAct reduces total failures by 41% (99→58), mainly reducing process hallucination by −22 (−42%) and output hallucination by −17 (−57%).Knowledge deficiency, intent misunderstanding, and other errors remain roughly stable, indicating the remaining ceiling involves knowledge, intent understanding, and environment robustness.

5 Conclusion

MemGUI-Agent manages long-horizon mobile GUI context within its action policy through ConAct, unifying history folding, UI memory actions, and self-describing step outputs. Zero-shot ConAct achieves state-of-the-art performance on MemGUI-Bench, while MemGUI-3K enables MemGUI-8B-SFT to achieve the best open-data 8B performance and generalize to MobileWorld.

  • Conclusion: MemGUI-Agent manages context inside the action policy through ConAct, unifying history folding, UI memory actions, and self-describing step outputs.ConAct is the agent’s context-management mechanism for long-horizon mobile GUI tasks.
  • Conclusion: Zero-shot ConAct sets a new state of the art on MemGUI-Bench with Qwen3-VL-235B-Thinking.The result is reported for the zero-shot ConAct setting using Qwen3-VL-235B-Thinking.
  • Conclusion: MemGUI-3K enables MemGUI-8B-SFT to achieve the best open-data 8B performance and generalize to MobileWorld.MemGUI-3K provides the training basis for MemGUI-8B-SFT and supports its reported benchmark generalization.

Limitations · A Related Work

The paper limits its experiments to Android-style mobile GUI environments, while positioning MemGUI-Agent against modular agentic frameworks, end-to-end mobile GUI agents, and established context-management approaches.

  • Limitations: The experiments focus on Android-style mobile GUI environments, with extension to iOS, desktop, and web interfaces left for future work.This is identified as a limitation of the current evaluation scope.
  • A Related Work: Mobile GUI agents generally follow either modular agentic frameworks or end-to-end models.The related-work discussion divides existing mobile GUI agents into these two broad paradigms.
  • A Related Work: Agentic frameworks combine multimodal backbones with planning, memory, and grounding modules.The passage describes these systems as modular combinations of an MLLM and additional capabilities.
  • A Related Work: Many agentic frameworks use proprietary backbones such as Gemini-2.5-Pro, making them strong but complex and difficult to reproduce.The passage contrasts their performance with reproducibility and system complexity.
  • A Related Work: End-to-end models map screenshots directly to actions, offering simplicity and deployability but often relying on passive context mechanisms.Examples include Action-Thought traces, multi-turn prompts, rule-based aggregation, and no-history prompting.
  • A Related Work: Long-context LLM agents manage histories through external memory or prompt-internal curation.External memory includes retrieval-augmented databases, hierarchical memory, and experiential replay; prompt-internal curation includes truncation, summarization, and model-controlled compression.
  • A Related Work: Prior prompt-internal curation methods include MEM1, MemAgent, and AgentFold, including recent applications to web agents.These methods edit in-task context through truncation, summarization, or model-controlled compression.

B Benchmark and Online Evaluation Protocol … C.4 Training Data Example

The paper evaluates long-horizon mobile GUI agents on MemGUI-Bench and MobileWorld with benchmark-specific success protocols and memory-oriented metrics. It constructs MemGUI-3K by expanding seed tasks, filtering teacher trajectories and individual steps, and retaining reasonable ConAct actions as supervised targets.

  • B Benchmark and Online Evaluation Protocol: MemGUI-Bench contains 128 tasks across 26 Android apps, including 89.8% explicitly memory-intensive and 78.1% cross-app tasks averaging 36.2 steps.Tasks are divided into Easy (1–20, 48 tasks), Medium (21–40, 42 tasks), and Hard (41+, 38 tasks).
  • B Benchmark and Online Evaluation Protocol: MemGUI-Bench tasks allow up to 3 independent attempts with an adaptive budget, and results are reported as Pass@k for k∈{1, 3}.A task succeeds when any of its first k attempts is judged successful by MemGUI-Eval.
  • B Benchmark and Online Evaluation Protocol: MobileWorld GUI-Only uses a single attempt per task with a 50-step limit and reports Pass@1 success rate determined by its backend verifier.The benchmark contains 117 tasks averaging 27.8 steps, with 62.2% cross-app.
  • B Benchmark and Online Evaluation Protocol: MemGUI-Bench additionally reports IRR for retention and correct use of task-critical information units and MTPR for explicitly memory-intensive-task success.Both metrics are computed by the MemGUI-Eval pipeline.
  • C.2 Teacher Rollout and Trajectory Filtering: Teacher rollouts use Qwen3-VL-235B-Thinking with the full 5-part ConAct protocol, and MemGUI-Eval retains only trajectories receiving positive final decisions.Among 5,293 rollout trajectories, 2,959 receive positive decisions before removing one abnormal 321-step outlier and two low-frequency successful trajectories.
  • C.2 Teacher Rollout and Trajectory Filtering: The final trajectories are split 90%/10% at the trajectory level with seed 42 and have zero overlap with MemGUI-Bench, including its MemGUI-Bench-40 subset.Disjointness is checked using task identifiers and descriptions.
  • C.3 Step-Level Reasonableness Filtering / C.4 Training Data Example: Step-level reasonableness filtering converts only reasonable steps into SFT samples, while unreasonable steps remain in trajectory records for analysis but are excluded from training.Annotations include reasonableness, explanation, and positive, negative, or neutral impact; Figure 8 illustrates this training-data construction.

C.5 Dataset Statistics · C.6 Dataset Format · D Training Setup and Dynamics

MemGUI-3K comprises long-horizon, richly annotated mobile-agent trajectories, with structured statistics on length, folding, memory actions, and app coverage. It is released both as complete trajectory records for offline analysis and as step-level multimodal SFT samples.

  • C.5 Dataset Statistics: 28.8 average steps and 25 median steps characterize MemGUI-3K trajectories, while reasonable steps average 21.8, yielding a 75.7% reasonable-step ratio.These statistics are reported under all-step and reasonable-step views.
  • C.5 Dataset Statistics: MemGUI-3K includes step-level folds that distill the latest step and span-level folds that compress completed multi-step subtasks into reusable records.The dataset therefore provides annotations at both step and completed-subtask granularities.
  • C.5 Dataset Statistics: 1.17 memory_add, 0.03 memory_update, and 0.04 memory_delete actions occur per trajectory on average, with 65.1% of trajectories invoking at least one memory action.Memory actions are frequent enough to characterize proactive context management in the dataset.
  • C.5 Dataset Statistics: MemGUI-3K spans 26 Android apps across 7 functional categories, with app-name variants normalized before counting.Frequent apps include Joplin, Amazon, Bing, Calculator, Apartments.com, Messages, Citymapper, AutoUncle, Calendar, and AP News.
  • C.6 Dataset Format: The trajectory-level release preserves complete rollouts, evaluation metadata, IRR statistics, token usage, screenshots, and reasonableness annotations for offline analysis.Supported analyses include memory operations, folding granularity, token cost, and recovery patterns.
  • C.6 Dataset Format: The step-level release converts all reasonable steps into ms-swift-compatible SFT samples containing the shared system prompt and user messages with structured context.The SFT data follows a standard chat-style multimodal format with images and structured action-agent fields.

D.1 Training Setup · D.2 Training Dynamics

MemGUI-8B-SFT is trained with LoRA supervised fine-tuning on MemGUI-3K using structured context, screenshots, and gold ConAct responses. During the single-epoch run, token accuracy rises, loss falls, and the learning rate follows warmup and cosine decay.

  • D.1 Training Setup: MemGUI-8B-SFT starts from Qwen3-VL-8B-Instruct and fine-tunes on 57,951 reasonable-step MemGUI-3K samples using LoRA SFT with ms-swift.Training uses eight 80GB GPUs and takes 535 minutes.
  • D.1 Training Setup: Each training sample contains the shared system prompt, structured context state, current screenshot, and gold 5-part ConAct response.
  • D.1 Training Setup: The run uses bfloat16 precision, one epoch, the fused AdamW optimizer, cosine learning-rate scheduling, and a warmup ratio of 0.05.
  • D.1 Training Setup: The effective batch size is 128 across eight GPUs, with learning rate 1 × 10−4, weight decay 0.1, maximum gradient norm 1.0, and sequence length 32,768.Per-device batch size is 2 with gradient accumulation of 8.
  • D.1 Training Setup: LoRA uses rank 8, alpha 32, dropout 0.05, and applies to all linear language-model projection modules.The configuration also uses seed 42, four dataloader workers, and Weights & Biases for training logs.
  • D.2 Training Dynamics: During the single-epoch LoRA SFT run, token accuracy steadily increases while training loss decreases.
  • D.2 Training Dynamics: The learning rate follows the configured warmup and cosine decay schedule throughout training.

E Offline Skill Evaluation Protocol

The offline skill evaluation protocol runs gold-context, step-level inference on the MemGUI-3K test split, then computes automatic metrics and model comparisons. Evaluation covers action matching, memory actions, folding behavior, and output-format compliance.

  • Protocol overview: 295 trajectories and 6,479 reasonable-step samples comprise the MemGUI-3K test split used to evaluate three models.The pipeline runs offline inference, computes automatic metrics, and builds the model-comparison summary.
  • Offline inference protocol: Each test sample provides only the system prompt and user message with the task, structured context, and screenshot, while holding out the gold assistant message.Inference uses temperature 0.01, a 16,384-token generation limit, 32 concurrent workers, resumable JSONL output, and retry logic.
  • Action matching: Action evaluation separates type accuracy from match accuracy, with coordinate tolerances for clicks, long presses, and swipes and exact or token-F1 criteria for text actions.Click and long-press coordinates must fall within 14% of the screen diagonal; swipe endpoints use 1.5 times that threshold.
  • Memory-action metrics: Memory-action evaluation measures operation-type correctness and matches deletes by exact memory_id or adds and updates by description or content token F1 > 0.5.The description criterion accommodates numeric content whose token overlap may be low despite a semantically correct prediction.
  • Folding metrics: Folding evaluation measures presence, endpoint range accuracy within ±2 steps, and predicted deep-fold ratio, separating shallow from deep folds.A fold is deep when its range covers more than one step; otherwise it is shallow.
  • Format compliance: Format compliance checks matched tags for five ConAct output blocks and reports trajectory-level full-match rate as a diagnostic.The folding tag is optional at step 1, and full-match requires every evaluated step in a trajectory to achieve action match success.

F Trajectory Completion Comparisons

Figures 10–13 compare paired long-horizon executions and show that MemGUI-Agent’s explicit context actions preserve task-relevant facts across trajectories and app transitions. These examples cover both 8B and Qwen3-VL-235B-Thinking backbones on MemGUI-Bench and MobileWorld.

  • Paired trajectory comparisons: Paired executions show explicit context actions helping preserve task-relevant facts across long trajectories compared with a ReAct-style base agent.The comparisons use the same task for both agents.
  • MemGUI-Bench: On MemGUI-Bench, MemGUI-8B-SFT completes a task that the 8B base agent fails after learning ConAct behavior from MemGUI-3K.This comparison uses an 8B backbone.
  • MemGUI-Bench: On MemGUI-Bench, MemGUI-Agent-235B changes only the prompting/action protocol while keeping Qwen3-VL-235B-Thinking weights unchanged.The paired execution uses the same task with the 235B backbone.
  • MobileWorld: On MobileWorld, the 8B trajectory comparison illustrates learned context-management behavior transferring beyond the MemGUI-Bench source environment.The example uses an 8B backbone.
  • MobileWorld: On MobileWorld, zero-shot ConAct helps maintain compact task state across app transitions without changing Qwen3-VL-235B-Thinking backbone weights.This comparison uses the 235B backbone.

G Failure Taxonomy and Annotation · H Full Prompt Templates

Section G defines a five-category taxonomy for failed MemGUI-Bench attempts and reports that Full ConAct mainly reduces process and output hallucinations. Section H provides the complete ReAct-style and MemGUI-Agent rollout prompts, including their action, context-management, observation, and response-format specifications.

  • G Failure Taxonomy and Annotation: Five primary categories annotate failed attempts by their main cause: process hallucination, output hallucination, knowledge deficiency, intent misunderstanding, or other.The taxonomy is applied to zero-shot Qwen3-VL-235B-Thinking failures on MemGUI-Bench-40.
  • G Failure Taxonomy and Annotation: Process hallucination covers goal drift, premature termination, irrelevant actions, skipped steps, and falsely believing an intermediate operation is complete.It describes losing track of the task objective or procedural workflow during execution.
  • G Failure Taxonomy and Annotation: Output hallucination occurs when the agent observes relevant UI information but later stores, recalls, writes, calculates, or reports incomplete or incorrect facts.The category includes selective retention failures over multi-item information and final-output transcription errors.
  • G Failure Taxonomy and Annotation: Full ConAct mainly reduces process and output hallucination, whereas knowledge deficiency and intent misunderstanding remain comparatively stable.This pattern supports attributing the primary gains to improved context management rather than unrelated model capabilities.
  • H.1 ReAct-Style Qwen3-VL Prompt: The ReAct-style prompt specifies touchscreen UI actions, JSON tool calls, and a three-part response ordered as thinking, tool_call, and conclusion.Its per-step prompt supplies the user goal, action history, and screenshot, while requiring concise reasoning and action summarization.
  • H.2 MemGUI-Agent Prompt: The MemGUI-Agent prompt treats context management as an explicit decision alongside UI actions, using folded UI state, folded action history, and a recent step record.The agent may emit UI or memory operations, but is restricted to one function action per step.
  • H.2 MemGUI-Agent Prompt: Its five-part response format requires thinking, a folding directive, tool_call, detailed UI observation, and action intent.Folding supports step-level distillation or span-level abstraction, with span-level abstraction preferred when intermediate details are irrelevant.
  • H.2 MemGUI-Agent Prompt: The MemGUI-Agent per-step prompt presents folded action history, the recent step record, folded UI state, and the current screenshot before requesting the next action.It requires detailed task-relevant observations, including exact visible text, numbers, prices, names, and counts, plus an action intent.
Loading 2606.19926v1…