PaperlayerView exact revision on arXiv

Source-linked AI summary

Mem-W: Latent Memory-Native GUI Agents

Guibin Zhang, Yaohui Ling, Fanci Meng, Kun Wang, Shuicheng Yan

arXiv:2605.09317v1cs.CLcs.CVcs.LG

TL;DR

GUI agents need to carry visual, procedural, and task evidence beyond the current screen, but existing memory systems impose externally designed structures that may not match policy representations. Mem-W unifies working and experiential memory in a shared latent context, and its benchmark results show consistent improvements over hand-engineered memory architectures.

  • Problem

    Existing GUI memory systems prescribe memory categories and representations that may not align with the structures demanded by downstream tasks.

  • Method

    Mem-W uses a shared trajectory-to-latent compressor to place retrieved historical trajectories and in-session segments into one continuous policy context.

  • Results

    Across four web and mobile benchmarks, Mem-W consistently improves diverse backbones and outperforms agents with hand-engineered memory architectures.

  • Takeaways & Limitations

    The results suggest latent context can serve as a practical foundation component for GUI agent memory.

  • Takeaways & Limitations

    Existing GUI memory designs remain constrained by prescribed taxonomies that may not coincide with downstream task needs.

Abstract

from arXiv · show

GUI agents are beginning to operate the web, mobile, and desktop as interactive worlds, where successful control depends on carrying forward visual, procedural, and task-level evidence beyond the fleeting present screen. Yet most agents still treat memory as an external, human-readable artifact: histories are summarized, categorized, retrieved, and reinserted as text or structured records before being encoded again by the policy. This creates a mismatch between the representational form in which experience is stored and the latent embedding sequence over which modern GUI policies actually act. We introduce Mem-W, a series of latent-memory-native GUI agents that treat memory as part of the agent's continuous context rather than as an auxiliary symbolic scaffold. Mem-W weaves both historical trajectories (as experiential memory) and in-session segments (as working memory) into compact memory tokens through a shared trajectory-to-latent compressor. These tokens are woven with the current GUI observation and local context into one continuous embedding sequence, allowing the agent to read successes, failures, and unfinished progress through the same machine-native interface. Mem-W is trained with self-distillation and outcome-aware supervision to preserve decision-relevant state while filtering memory toward evidence that truly supports task success. Across four web and mobile navigation benchmarks, Mem-W consistently improves diverse backbones and memory-enhanced baselines, with gains of up to $+30.0$, suggesting that latent-context-native memory can serve as a scalable foundation for long-horizon GUI agency.

1 Introduction

Mem-W frames GUI memory as a latent-context interface, addressing the mismatch between externally engineered memory structures and the continuous embeddings used by policies. It unifies working and experiential memory through a shared compressor and reports gains across four web and mobile benchmarks.

  • Motivation: Current GUI agents must preserve past interaction evidence because the present screen is insufficient, while full histories are too long, noisy, and redundant.The central challenge is making past evidence part of operational context rather than repeatedly translating detached records back into model inputs.
  • Method: A unified latent space lets task-relevant memory organization emerge through training instead of requiring hand-engineered categories or role-specific modules.The approach projects trajectory evidence into the policy’s continuous space, reducing format mismatch and allowing end-to-end gradients to determine retention and composition.
  • Motivation: Mem-W addresses the question of whether end-to-end latent memory can replace manually prescribed GUI memory hierarchies.Existing systems require designers to choose memory categories, representations, and interfaces, which may not match downstream task demands.
  • Method: Mem-W maps retrieved trajectories and online episode prefixes into fixed-size latent blocks with a shared trajectory-to-latent compressor.Both memory sources enter the policy as tokens of the same type, while working and experiential memory differ by provenance rather than representation.
  • Training: Mem-W trains memory with self-distillation from richer contexts and outcome-aware supervision distinguishing successful from failed evidence.The resulting latent context jointly organizes experience, state, and action for GUI control.
  • Empirical Evidence: +30.0 is the reported maximum gain across four web and mobile navigation benchmarks over mainstream GUI agents and hand-engineered memory architectures.The experiments also report favorable scaling as the experiential memory bank grows while inference delay remains well controlled.

2 Related Work

Related GUI memory methods separately address in-session history or cross-session experience, often using prescribed structures or single-role latent representations. Mem-W instead uses one learnable latent space and shared compressor for both memory roles.

  • GUI Agent Memory: GUI memory methods span working-memory techniques for retaining in-session visual history and experiential-memory techniques for reusing cross-session task experience.Working memory includes screenshot pruning, KV-cache compression, and structured state tuples, while experiential memory includes workflows, lessons, scripts, and hybrid strategies.
  • GUI Agent Memory: Prior systems commonly prescribe memory types, representations, filtering, or composition before learning, and these choices may not align with downstream task needs.This ontology-first design is the limitation Mem-W targets.
  • Latent Memory: Mem-W projects all memory evidence into a shared learnable latent space so task-relevant organization can emerge through end-to-end optimization.This differs from methods that use latent representations for only one memory role.
  • Comparison: Table 1 compares representative mechanisms by working memory, experiential memory, latent memory, learnability, and multimodal support.The table uses ✓, ✗, and ● for full, absent, and partial support.
  • Latent Memory: Latent memory is motivated by token efficiency, machine-native formatting, and end-to-end learnability across textual and multimodal agents.Related multimodal systems encode visual cues, trajectories, or historical experience as latent tokens or continuous embeddings.

3 Method

Mem-W equips a frozen GUI agent with a compact latent memory interface that converts historical trajectories and in-session experience into soft tokens consumed alongside current observations. A shared compressor, dual-scale memory weaving, and two-stage training preserve decision-relevant evidence while keeping memory in the agent’s native embedding space.

  • Overview: Mem-W replaces an auxiliary human-readable memory store with soft memory tokens in the agent’s input embedding space.Historical and current-episode evidence are exposed through the same continuous interface as native context.
  • Latent weaving: The frozen GUI agent receives ordinary current observations and context augmented with compact latent blocks from procedural and working memory.Each selected segment contributes a fixed K-token block, with source embeddings distinguishing the two memory types.
  • Unified compressor: A shared trajectory compressor maps retrieved historical trajectories and online episode prefixes into the same continuous latent space.The compressor uses structured observation–action segments, outcome embeddings, and a lightweight query-transformer design to produce fixed-size blocks without decoding text.
  • Procedural memory: Procedural memory retrieves a small number of trajectories from an external bank using frozen-encoder similarity, then compresses them in retrieval-rank order.Retrieved trajectories retain success or failure information through the compressor’s outcome conditioning rather than hand-designed filtering rules.
  • Working memory: Working memory compresses expired prefixes of the current episode into non-overlapping temporal chunks while retaining the most recent raw interaction window.Online chunks use an unknown-outcome marker because the current episode is unlabeled before termination.
  • Training: Training updates only the compressor through self-distillation and outcome-aware optimization while keeping the GUI agent and retrieval encoder frozen.Self-distillation transfers extended-context behavior into latent inputs, and policy-gradient feedback further optimizes memory encoding with a fixed KL reference.

4 Experiments

Mem-W is evaluated across four multimodal GUI benchmarks using frozen backbones and memory-enhanced baselines. It consistently improves navigation performance, with especially large web gains, while retaining favorable efficiency and benefiting from richer trajectory memories.

  • Experimental setup: Mem-W is trained on web and mobile trajectories with self-distillation followed by outcome-aware supervision, while freezing backbone parameters and training only the compressor.The evaluation uses three mainstream GUI backbones and a ReAct-style structured reasoning and tool-invocation paradigm.
  • Main results: Mem-W consistently improves frozen GUI backbones across mobile control and web navigation benchmarks, indicating that representation of interaction history matters beyond model scale.The evaluated benchmarks include AndroidControl-v2, GUI-Odyssey, MMINA, and Multimodal-Mind2Web.
  • Ablation and efficiency: Full Mem-W improves MMINA success rate from 18.5 to 48.5 for UI-Venus and from 11.5 to 40.5 for Qwen, while reducing Hit-Max from 70.5% to 42.2% and 99% to 22%.The full configuration performs best across the reported ablation settings, suggesting working and experiential memory are complementary.
  • Ablation and efficiency: Mem-W surpasses stronger open-source baselines in success rate while maintaining comparable or lower task-level runtime, and it is faster than HyMEM and CoMEM on Mind2Web.The added compression step remains within the same practical runtime range as representative GUI agents.

5 Conclusion

Mem-W unifies working and experiential GUI memory in a shared latent context interface, replacing prescribed memory taxonomies with learned latent organization. Across four web and mobile benchmarks, it consistently improves diverse backbones and hand-engineered memory baselines.

  • 5 Conclusion: Mem-W unifies working memory and experiential memory through a shared, end-to-end learnable trajectory-to-latent compressor.Both in-session history and cross-session experience are projected through the same mechanism.
  • 5 Conclusion: Task-relevant memory structure emerges from training rather than manual design, allowing latent tokens to organize experience, state, and action in one continuous context.
  • 5 Conclusion: Across four web and mobile benchmarks, Mem-W consistently improves diverse backbones and outperforms agents equipped with hand-engineered memory architectures.
  • 5 Conclusion: The results suggest that latent context can serve as a practical foundation component for future GUI agent architectures.The proposed substrate organizes perception, memory, procedural experience, and ongoing task state in the policy’s machine-native form.
  • 5 Conclusion: Mem-W frames long-horizon GUI competence around learned latent context interfaces rather than externally prescribed memory modules.

Contributions

The listed contributors include Guibin Zhang, Yaohui Ling, and Fanci Meng, with Kun Wang and Shuicheng Yan as corresponding authors.

  • Contributions: Guibin Zhang and Yaohui Ling are listed as core contributors.
  • Contributions: Fanci Meng is listed as a contributor.
  • Contributions: Kun Wang and Shuicheng Yan are listed as corresponding authors.

A Training Details

Training freezes the GUI agent backbone while learning a compressor that supplies procedural and working-memory latent blocks. Self-distillation trains action prediction, while stage-two outcome-aware optimization uses rollout rewards, RLOO advantages, and KL regularization.

  • A Training Details: The GUI agent parameters remain frozen, while only the compressor parameters receive updates in the two-stage training procedure.Gradients reach the learnable compressor through latent memory blocks.
  • A.1 Stage 1: Self-Distillation: Stage one constructs procedural memory from retrieved trajectories and working memory from expired context chunks, then trains against action and distillation losses.The student uses a latent-augmented context, while the teacher observes a larger finite raw window.
  • A.2 Stage 2: RLOO-Based Outcome-Aware Optimization: Stage two keeps both memory types active and optimizes the compressor with environment feedback from sampled rollouts.Each rollout receives a binary terminal reward indicating task success or failure.
  • A.2 Stage 2: RLOO-Based Outcome-Aware Optimization: RLOO estimates each rollout’s advantage using the other rollouts for the same instruction as a baseline, assigning that advantage across the trajectory.The per-instruction baseline is parameter-free and does not require a learned value function.
  • A.2 Stage 2: RLOO-Based Outcome-Aware Optimization: A KL penalty constrains the stage-two compressor relative to a frozen stage-one reference policy.The combined objective includes policy-gradient and KL terms, with β controlling the KL penalty.

B.1 Training Dataset

Training uses separate web and mobile datasets and memory banks, with evaluation kept on official test splits without overlap. The implementation freezes pretrained backbone parameters while updating LoRA adapters and compressor modules.

  • B.1 Training Dataset: The web training set contains 81,527 samples, including 11,176 successful and 70,351 failed trajectories, while successful trajectories are used as training examples.
  • B.1 Training Dataset: The web memory bank contains 22,346 successful memories spanning 13 domains for reusable planning, retrieval, and decision-making references.
  • B.1 Training Dataset: The mobile training set contains 2,610 samples across six task categories, including 2,486 successful and 124 failed trajectories.
  • B.1 Training Dataset: Training and memory banks use training-set trajectories, while evaluation uses official test sets with no overlap or data leakage.
  • B.1 Training Dataset: The mobile memory bank contains 4,972 successful memories covering the six mobile task categories and reusable cross-application operation patterns.
  • B.1 Training Dataset: The original language-model and visual-encoder parameters are frozen, while LoRA adapters and compressor-related modules are updated for parameter-efficient training.This design reduces training cost while preserving pretrained vision-language capabilities.

B.3 Evaluation Setup

Mem-W is evaluated across four web and mobile GUI benchmarks using interactive and offline settings with retrieved experience memory incorporated into action generation.

  • Mem-W is evaluated on MMInA, Mind2Web, GUI-Odyssey, and Android-Control-v2.
  • Interactive GUI Evaluation on MMInA and Mind2Web: Interactive MMInA and Mind2Web evaluation uses ReAct-style reasoning, combining screenshots, page text, task instructions, action history, and structured GUI function calls.
  • Interactive evaluation limits tasks to 15 interaction steps and falls back to standard vLLM inference when no relevant memory is retrieved.
  • The memory module retrieves the top five similar historical trajectories through a FAISS inner-product index and inserts them as contextual experience memory.
  • Benchmark Evaluation on GUI-Odyssey: GUI-Odyssey inference uses current GUI observations, task instructions, and top-five step-level retrieved experiences in an offline setting.

C Case Study

Qualitative case studies show Mem-W carrying task constraints and evolving progress across web and mobile interactions while retrieved trajectories supplement working memory.

  • Mem-W preserves task constraints and intermediate progress across multiple web page transitions in the website case.
  • The mobile case shows an evolving record of the current episode being used while the agent selects executable GUI actions.
  • Retrieved mobile trajectories provide procedural evidence that complements in-session working memory and guides more reliable action choices.

D Limitation

The paper identifies fixed compression capacity, untested desktop generalization, and limited tokenizer independence as open boundaries for Mem-W.

  • The trajectory-to-latent compressor uses a fixed latent-token budget, limiting adaptive allocation of capacity to trajectory complexity.The authors suggest adaptive compression could improve information retention.
  • Although evaluated across four web and mobile benchmarks, Mem-W's generalization to desktop applications remains open.
  • The proposed latent-memory interface is intended to be model-agnostic, but tokenizer-independent representations transferable across models remain future work.
Loading 2605.09317v1…