Auto-scaling Continuous Memory for GUI Agent

1 INTRODUCTION

• Prior GUI memories can span thousands or hundreds of thousands of tokens, while text representations miss precise widget size and position.These visual cues can be decisive for reliable GUI execution.Source: §1 · pp. 1–2
• Continuous memory encodes each GUI trajectory as eight embeddings injected directly into the VLM, preserving visual cues while reducing context length.Performance improves monotonically with memory size and retrieval depth, whereas text memories degrade beyond roughly ten retrieved items.Source: §1 · p. 2
• Qwen-2.5-VL-7B with continuous memory reaches performance comparable to closed-source models and significantly outperforms them on Webvoyager.The comparison includes GPT-4o and Claude-4 across real-world GUI benchmarks.Source: §1 · p. 2, Figure 1 · p. 2
• The method collects more than 100k trajectories for about $4k and fine-tunes only the memory encoder with 1,500 samples and 1.2% of parameters.The optimized components are LoRA and a Q-Former in the memory encoder.Source: §1 · p. 2
• The auto-scaling data flywheel discovers environments, synthesizes tasks, rolls out trajectories, and verifies quality without human annotation.It is designed to expand diverse, high-quality memory at low cost.Source: §1 · pp. 2–3

GUI Agents

• GUI-agent benchmarks cover web control, realistic multimodal interaction, desktop workflows, and cross-app mobile navigation.Examples include Mind2Web, WebArena, VisualWebArena, OSWorld, and GUIOdyssey.Source: §2 · p. 3

Memory for LLMs and Agents

• Memory research is moving from concatenated textual retrieval toward structured and continuous representations to reduce context cost and distraction.RAG and REALM exemplify textual memory, while VoCo-LLaMA and MA-LMM represent compact visual-memory approaches.Source: §2 · p. 3

LLM for Data Annotation

• LLM-driven synthesis and grading replace costly human supervision with scalable task generation and verification.Self-Instruct, ZeroGUI, and SEAgent provide related recipes for continual data growth in GUI or agent settings.Source: §2 · p. 3

Problem Statement

• At each step, the agent observes a screenshot and selects CLICK, TYPE, SCROLL, WAIT, or STOP, producing a trajectory through environment transitions.The environment returns the next screenshot after executing each structured action.Source: §3 · p. 4
• A task is successful when its last s page states satisfy the goal under LLM-as-Judge evaluation, with performance reported as benchmark success rate.The interaction horizon is capped at T_max.Source: §3 · p. 4

Our Target

• The proposed memory contains compressed embeddings of useful trajectories, retrieves the top-k relevant items, and concatenates their vectors into the agent context.The policy conditions the next action on both the current observation and retrieved memory items.Source: §3 · p. 4

Memory Auto-Scaling Data Flywheel

• A four-phase data flywheel discovers environments, synthesizes tasks, rolls out agent trajectories, and verifies successful completion before expanding the memory pools.It searches for stable, accessible websites, uses a VLM to create tasks, records actions and observations, and retains positive trajectories with their tasks and environments.Source: §4.1 · pp. 4–5

Continuous Memory Integration

• The system stores diverse environments and successful trajectories, then uses a VLM-based encoder to convert retrieved trajectories into compact continuous embeddings for inference.The framework overview identifies the memory store and encoder as complementary components of the memory-augmented VLM agent.Source: §4 · p. 4, Figure 2 · p. 5, §4.2 · p. 6

Continuous Memory Encoder

• A Q-Former compresses each multimodal trajectory into a small embedding set that is prepended to the agent’s input embeddings without architectural changes or full retraining.This plug-and-play integration lets the agent attend to contextual memory while preserving the backbone architecture.Source: §4.2 · p. 6

Memory Retrieval

• At inference, CLIP-based multimodal keys and FAISS retrieve the top-k nearest prior experiences from the stored trajectory collection.Screenshots, actions, and queries are mapped into pooled multimodal keys before nearest-neighbor retrieval.Source: §4.2 · p. 6

Efficient Fine-tuning

• The memory encoder is adapted efficiently with LoRA across Q-Former layers using 1,500 trajectories while updating only 1.2% of parameters.The training setup uses rank-16 shared LoRA and augments each trajectory step with its top-3 retrieved memories.Source: §4.2 · p. 6

5.2 MAIN RESULTS

• 46.20% on MMInA, 21.28% on Mind2Web, and 54.5% on WebVoyager make CoMEM the best-performing setting across all reported memory comparisons.Qwen2.5-VL-7B with CoMEM outperforms the corresponding text-memory results and other settings.Source: §5.2 · p. 7
• 23.8% overall versus 6.6% shows that adding CoMEM substantially improves UI-TARS-1.5, whose baseline lacks interactive web-navigation exposure.The improvement is attributed to structured, multimodal task experiences that guide decision-making.Source: §5.2 · p. 7

5.3 FURTHER ANALYSIS

• Continuous memory improves as memory size and retrieved samples increase, whereas text memory peaks and deteriorates beyond roughly ten retrieved items.CoMEM shows a sustained upward trend with retrieval depth, while text memory is associated with longer sequences and accumulated noise.Source: §5.2 · p. 8
• Text-based Memory negatively affects out-of-distribution performance, while continuous memory generalizes by encoding abstract multimodal experience.The comparison uses Action Matching Score for GUIOdyssey and Success Rate for OSWorld.Source: §5.3 · p. 9
• Memory integration does not add inference latency, and Shopping completion time differs from the baseline by at most 0.5 minutes.For Wikipedia, both memory variants complete faster than the baseline; overall, memory preserves time efficiency.Source: §5.3 · p. 9
• With only 1,500 high-quality trajectories, the model reaches peak performance on Wikipedia at 47.40% and Shopping at 45.00%.The training-efficiency experiment varies the amount of data used to align compressed memory embeddings with the VLM representation space.Source: §5.3 · p. 9

6 CONCLUSION

• Continuous memory encodes prior GUI trajectories as compact embeddings, preserving critical visual cues while avoiding ballooning context length.The framework retrieves and injects these embeddings during inference.Source: §6 · p. 9
• Performance improves monotonically as memory size and retrieval depth increase, and the memory-augmented 7B model attains accuracy competitive with leading closed-source systems.Source: §6 · pp. 9–10
• The autonomous data flywheel collected more than 100k trajectories for about $4k, while LoRA plus Q-Former adaptation used 1.2% of parameters and 1.5k samples.Source: §6 · p. 9

Technical limitations

• Retrieval can drift under extreme UI shifts, screenshot-only inputs omit non-visual states, and larger memory banks increase latency and GPU-memory pressure.Freshness, deduplication, provenance, and benchmark non-stationarity also complicate scaling and reproducibility.Source: LIMITATION AND ETHICS STATEMENT · p. 10

Data flywheel risks

• The data flywheel risks judge errors, bias toward popular or easy layouts, data poisoning, prompt-injection-style UI content, and restrictive content licenses.The passage proposes ensemble judging, balanced crawling, source filtering, safety scans, trust scoring, and license-aware releases.Source: LIMITATION AND ETHICS STATEMENT · p. 10

Ethics

• Autonomous rollouts require sandboxing and fail-safes, screenshots require consented collection and privacy controls, and large-scale indexing carries environmental costs.Source: LIMITATION AND ETHICS STATEMENT · p. 10

Reproducibility Statement

• Complete source code has been uploaded as a supplementary zip, with plans to open-source the codebase and generated dataset.Source: REPRODUCIBILITY STATEMENT · p. 10
• Model architecture, training procedures, and hyperparameters are described in Section 4.2 of the main paper.Source: REPRODUCIBILITY STATEMENT · p. 10
• Environment setup, dataset details, and the dataset-construction format are further documented in Section 5.1 and the supplementary materials.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SHOPPING SCENARIO

• Figure 3 depicts Qwen2.5-VL-7B + CoMEM understanding the goal, retrieving relevant memory, and selecting the correct product.Source: Figure 3 · p. 15
• The base model selects the first product without rigorously checking price, color count, or suitability for canvas painting.Source: §A.1 · pp. 15–16
• CoMEM improves the shopping behavior by leveraging compressed memory representations from multiple prior shopping trajectories.These embeddings provide richer context for following a more deliberate strategy.Source: §A.1 · p. 16

• CoMEM-enhanced Qwen2.5-VL-7B accurately understands the goal, retrieves relevant memory, and finds the answer in the successful information scenario.Source: Figure 5 · p. 16, §A.2 · p. 16
• The base model fails after issuing an overly general “Shanghai” query and repeatedly scrolling through irrelevant Wikipedia sections.This inefficient exploration causes it to miss the visual and textual cues needed to answer the question.Source: §A.2 · p. 16, Figure 6 · p. 17

Environment and page discovery

• The pipeline searches each sampled task query and collects the top 20 results, yielding 6,676 unique page links for task generation.Source: B TASK GENERATION PIPELINE · p. 17

Page processing

• Each link is reduced to its parent domain and first path level, opened with Playwright, and paired with a screenshot for VLM-based page extraction and accessibility checking.Blocked pages are skipped before detailed information is supplied to the task-generation model.Source: B TASK GENERATION PIPELINE · p. 17

Page and task prompts

• Page descriptions summarize site purpose, navigation, interactive elements, content types, and distinctive features to support web-automation task generation.The description is combined with five category-matched seed tasks as context for generating new problems.Source: B TASK GENERATION PIPELINE · p. 18

Initial task generation

• The first-pass prompt requests exactly 10 direct, diverse, multi-step tasks with clear objectives, achievable scope, measurable outcomes, and difficulty levels.The generated output follows a fixed one-task-per-line format containing the task, expected outcome, and difficulty.Source: B TASK GENERATION PIPELINE · pp. 18–19

Task refinement

• First-pass generation can produce overly specific tasks that do not resemble tasks undertaken by human users.Source: B TASK GENERATION PIPELINE · p. 19
• A second LLM pass rewrites tasks in more natural language, removes procedural hints, and preserves an unambiguous goal.Examples replace instructions to use named interface controls with outcome-oriented requests such as finding flights within a price range.Source: B TASK GENERATION PIPELINE · p. 19