PaperlayerView exact revision on arXiv

Source-linked AI summary

UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents

Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, Hongsheng Li

arXiv:2602.05832v1cs.CV

TL;DR

Online GUI RL is limited by sparse rewards, inefficient credit assignment, and repeated failures that do not transfer across tasks. UI-Mem introduces hierarchical parameterized memory, stratified guidance, and a self-evolving update loop to reuse experience during online learning. Experiments report improved success rates and cross-task generalization over traditional RL and static reuse strategies, while the authors note safety risks during live-environment exploration.

  • Problem

    Long-horizon GUI tasks suffer from inefficient credit assignment and repetitive cross-task errors because online RL lacks mechanisms for transferring experience.

  • Method

    UI-Mem stores workflows, subtask skills, and failure patterns as parameterized templates, applies stratified guidance across rollout trajectories, and continuously updates memory from successes and failures.

  • Results

    UI-Mem significantly outperforms traditional RL baselines and static reuse strategies, with strong cross-task generalization; on AndroidWorld, UI-Mem-4B reaches 58.2% success versus 45.3% for vanilla Qwen3-VL-4B and 49.1% for UI-Venus-7B.

  • Takeaways & Limitations

    Reusable, evolving experience transfer supports more efficient online GUI policies and generalization to unseen applications.

  • Takeaways & Limitations

    Online RL agents may perform unintended actions in live environments, so robust safe-exploration protocols are needed before real-world deployment.

Abstract

from arXiv · show

Online Reinforcement Learning (RL) offers a promising paradigm for enhancing GUI agents through direct environment interaction. However, its effectiveness is severely hindered by inefficient credit assignment in long-horizon tasks and repetitive errors across tasks due to the lack of experience transfer. To address these challenges, we propose UI-Mem, a novel framework that enhances GUI online RL with a Hierarchical Experience Memory. Unlike traditional replay buffers, our memory accumulates structured knowledge, including high-level workflows, subtask skills, and failure patterns. These experiences are stored as parameterized templates that enable cross-task and cross-application transfer. To effectively integrate memory guidance into online RL, we introduce Stratified Group Sampling, which injects varying levels of guidance across trajectories within each rollout group to maintain outcome diversity, driving the unguided policy toward internalizing guided behaviors. Furthermore, a Self-Evolving Loop continuously abstracts novel strategies and errors to keep the memory aligned with the agent's evolving policy. Experiments on online GUI benchmarks demonstrate that UI-Mem significantly outperforms traditional RL baselines and static reuse strategies, with strong generalization to unseen applications. Project page: https://ui-mem.github.io

1. Introduction

Online RL for GUI agents is hindered by sparse rewards, inefficient credit assignment, and repeated failures that do not transfer across tasks. UI-Mem addresses these problems with structured, evolving memory and stratified guidance during exploration.

  • Long-horizon GUI tasks make standard online RL ineffective because sparse rewards create inefficient credit assignment and blind trial-and-error.Correct intermediate actions may not be reinforced when a single final error produces negative feedback.
  • Agents repeatedly encounter similar failures across tasks because existing online RL lacks mechanisms to store and transfer experience.Confirmation-popup handling is one example of a failure that may need to be rediscovered in each task.
  • Experience Replay and dense rewards address sample efficiency or within-rollout credit assignment but do not support reusable experience across tasks and applications.Replay may struggle on novel tasks, while step-level rewards identify correct steps without enabling cross-task transfer.
  • UI-Mem stores workflows, subtask skills, and failure patterns as parameterized templates for hierarchical credit assignment and cross-application skill reuse.Templates can be instantiated with current task details, such as turning “Send email to {{recipient}}” into a concrete plan.
  • Stratified Group Sampling varies guidance across trajectories, preserving outcome diversity so unguided policies can internalize guided behaviors.A dynamic curriculum progressively reduces guidance as task success improves.

2. Related Work

Related work establishes GUI agents and GRPO-based reinforcement learning as active areas, while emphasizing that online GUI RL remains difficult because rewards are extremely sparse. Existing responses include step-level feedback and experience replay, but the proposed framework is presented as a structured-memory alternative.

  • Recent work applies multimodal language models to mobile and desktop GUI agents that interpret screenshots and generate sequential actions.
  • GRPO supports group-based advantage estimation without a critic network and has been explored for GUI reinforcement learning with rule-based rewards.
  • Online reinforcement learning remains challenging in GUI domains because rewards are extremely sparse.
  • Prior work mitigates sparse rewards through step-level feedback or stabilizes training through Experience Replay, motivating structured experience memory in the online RL loop.

3. Method

UI-Mem integrates structured hierarchical experience memory into online GUI reinforcement learning, combining reusable plans, skills, and failure patterns with guided exploration. Its retrieval and curriculum mechanisms support transfer across tasks while progressively reducing reliance on external guidance.

  • Framework overview: UI-Mem combines hierarchical experience memory with online RL to retain reusable workflows, subtask skills, and failure patterns as parameterized templates.The framework addresses GRPO's difficulty with sparse rewards, long horizons, and rediscovery of solutions for similar tasks.
  • Memory representation: The memory represents high-level workflows, reusable mid-level skills, and failure patterns that help agents plan, execute, and avoid repeated errors.Examples include ordered email-sending workflows, search or form-filling skills, and warnings against saving before entering a filename.
  • Template retrieval: Parameterized templates abstract variable values such as filenames or dates, enabling one experience to transfer across semantically similar tasks and applications.Retrieved templates are matched to new instructions and instantiated with instruction-specific variables to produce concrete plans.
  • Template retrieval: UCB-inspired retrieval favors plans with high historical success while exploring less-used plans, and recency bias prioritizes recent failure diagnoses.This makes memory guidance adapt as the agent's policy and stored experience evolve.
  • Memory-guided exploration: Stratified group sampling assigns strong, weak, or no guidance within each GRPO group, preserving outcome diversity for advantage estimation.A dynamic dropout curriculum then progressively reduces guidance as task success improves, encouraging the policy to internalize retrieved experience.
  • Self-evolving loop: A self-evolving loop continuously extracts and integrates new experience from online trajectories so the memory captures novel strategies and failures.This keeps retrieved guidance aligned with the agent's evolving policy rather than treating the memory as static.

4. Experiments

Experiments evaluate UI-Mem on online GUI benchmarks, compare it with alternative training paradigms, and analyze component contributions, generalization, and training dynamics. Across these analyses, UI-Mem improves success, transfers to unseen applications, and benefits from hierarchical memory, memory updates, and stratified guidance.

  • Main Results: 71.1% success on AndroidWorld is achieved by UI-Mem-8B with memory retrieval, while UI-Mem-4B reaches 58.2% and UI-Mem-4B⋆ reaches 62.5%.UI-Mem-4B exceeds vanilla Qwen3-VL-4B at 45.3% and UI-Venus-7B at 49.1%; the 8B result surpasses the cited commercial APIs.
  • Main Results: On AndroidLab, UI-Mem substantially improves Sub-SR and UI-Mem-8B outperforms MobileRL at 42.5%.The reported explanation is that retrieved workflows and subtask plans support decomposition, while active memory guides new trajectories rather than only replaying past paths.
  • Component Analysis: Removing hierarchical structure, using raw experience, disabling memory updates, or applying full guidance without stratification reduces performance.Raw experience reaches 58.2%, and disabling memory updates reduces performance to 62.9%; the analysis attributes excessive guidance to over-reliance on memory.
  • Cross-Application Generalization: On five held-out apps, zero-shot UI-Mem matches or exceeds the baseline, while inference-time memory retrieval adds consistent gains, especially on Bluecoins and Maps.me.The held-out apps are Bluecoins, Cantook, Maps.me, Pi-Music, and Zoom.
  • Training Dynamics: UI-Mem improves more stably and converges faster than standard GRPO by maintaining within-group reward variance through guided and exploratory trajectories.The reported mechanism is that standard GRPO often samples all-failure groups with near-zero variance and vanishing gradients, whereas stratified sampling supplies informative gradients.

5. Conclusion

UI-Mem addresses online RL inefficiencies in GUI environments through reusable hierarchical memory, stratified GRPO sampling, and continuous memory refinement. Experiments report improved sample efficiency, success rate, and cross-task generalization through experience transfer.

  • Conclusion: UI-Mem decomposes raw experiences into reusable workflows, subtask skills, and failure patterns within a hierarchical, self-evolving memory.The memory is used through stratified group sampling and continuously refined to track the current policy.
  • Conclusion: UI-Mem balances memory-guided exploitation with exploration to support effective advantage estimation and internalize external memory into the policy.The framework uses stratified group sampling tailored for GRPO.
  • Conclusion: Experiments demonstrate improved sample efficiency and success rate, with strong cross-task generalization enabled by reusable experience transfer.This conclusion summarizes the reported benchmark findings without specifying a single benchmark value.

Impact Statement

The work targets more efficient and generalizable autonomous GUI agents, while identifying safety and privacy risks for real-world deployment. Online exploration can cause unintended actions, and screenshots may contain personal data requiring strict handling.

  • Risks: Live-environment exploration may perform unintended actions such as deleting data or initiating financial transactions.The paper states that robust safe-exploration protocols are needed before real-world deployment.
  • Risks: Although memory abstraction replaces sensitive text with placeholders such as {{password}}, screenshots containing personal data still create privacy concerns.The paper calls for strict handling procedures for such data.

A. Preliminaries

GUI interaction is modeled as a finite-horizon MDP with sparse terminal rewards, and GRPO optimizes grouped trajectories without a separate critic. Essential-state templates support rule-based verification of critical intermediate states.

  • GUI Task Formulation: The GUI environment is a finite-horizon MDP whose states combine screenshots and instructions, actions are atomic GUI operations, and rewards are binary and terminal.The trajectory reward is provided only at termination, reflecting the sparse nature of real-world GUI tasks.
  • Group Relative Policy Optimization: GRPO samples a group of trajectories for each instruction and computes each trajectory’s advantage by normalizing its reward within that group.This avoids requiring a separate critic model.
  • Group Relative Policy Optimization: The policy objective uses clipped probability ratios and a KL-divergence penalty relative to a reference policy.The clipping hyperparameter is δ, while β controls the KL penalty to prevent policy collapse.
  • Hierarchical Memory Representation: An essential-states template defines key intermediate states that characterize valid task completion and enables rule-based reward verification.These states provide structured milestones beyond the raw task instruction.

C. Reward Model Details

UI-Mem replaces direct whole-history visual judging with two-stage text-based verification, using paired screen descriptions and action descriptions to compute more reliable rewards. The resulting pipeline achieves an F1 score of 0.902 in the reported Qwen2.5-VL-72B-Instruct+DeepSeek-V3 setting.

  • Motivation: Directly feeding entire screenshot histories to an MLLM judge can cause severe hallucinations, motivating text-based state verification.The proposed approach addresses this limitation by separating visual description from subsequent verification.
  • Two-Stage Verification: The first stage describes each screen state and executed action from paired before-action and after-action screenshots.Descriptions capture relevant UI elements, visual state changes, and the operation performed.
  • Two-Stage Verification: The textual history is used for state-based verification rather than directly judging subtask or task completion.This provides a structured basis for evaluating whether required UI states were reached.
  • Evaluation: 0.902 F1 Score is achieved by the Qwen2.5-VL-72B-Instruct+DeepSeek-V3 configuration, compared with 0.837 for Gemini 2.5 Pro.The reported evaluation compares prediction accuracy, Precision, Recall, and F1 Score across backbone settings.

D. Experience Extraction Details

UI-Mem extracts structured experience from successful and failed trajectories at subtask and workflow levels, then parameterizes and ranks it for reuse. The representation combines plans, failure diagnoses, variables, and retrieval metadata.

  • Experience Extraction: Seed1.8 extracts success plans and failure patterns in separate subtask-level and high-level workflow phases.Successful subtasks yield plans, while failed subtasks yield diagnoses and correction guidance.
  • Experience Extraction: The extraction process preserves entities such as phone numbers so concrete details can later map accurately to template variables.Entity preservation is imposed as a constraint during subtask experience abstraction.
  • Experience Extraction: Figure 9 illustrates textual UI-state and action descriptions generated by comparing screens before and after operations such as text entry and button clicks.These descriptions support the trajectory analysis used for experience extraction.
  • Experience Parameterization: Experience parameterization converts concrete experiences into abstract experiences containing variables, enabling transfer to new tasks.DeepSeek-V3 performs this transformation from task-specific experiences to reusable knowledge.
  • Experience Ranking: UCB ranking balances exploitation of high-success workflows or skills with exploration of less-tested plans, while failure diagnoses use time decay.The time-decay rule prioritizes more recent errors as interfaces and policies evolve.
  • Hierarchical Retrieval: The hierarchical retrieval algorithm constructs structured guidance containing a plan plus tips or warnings from high- and mid-level memory.It retrieves task templates, workflows, and subtask-level guidance before instantiating the final guidance.

PHASE 1: TASK MATCHING & VARIABLE EXTRACTION

The memory update pipeline tracks task competence, abstracts and deduplicates subtask experience, and consolidates successful trajectories into prioritized workflows. It updates long-term structured knowledge from executed traces and feedback.

  • PHASE 1: TASK MATCHING & VARIABLE EXTRACTION: The retrieval pipeline embeds the current instruction, selects similar task templates, and uses an LLM to choose a match and extract task-specific variables.Qwen3-Embedding-8B performs semantic retrieval, while DeepSeek-V3 determines the best template match.
  • PHASE 1: TASK MATCHING & VARIABLE EXTRACTION: The selected abstract workflow is instantiated by injecting extracted variables into predefined placeholders, producing a context-specific executable plan.This converts parameterized experience into an actionable workflow for the current task.
  • PHASE 1: TASK MATCHING & VARIABLE EXTRACTION: Pending subtasks are enriched with semantically retrieved success plans and failure-correction guidelines from mid-level memory.The resulting guidance combines workflow-level planning with subtask-level support.
  • Memory Update: The self-evolving update mechanism derives structured long-term knowledge from raw trajectories using success or correction feedback.Its phases include global statistics, mid-level memory updates, and high-level workflow consolidation.
  • Memory Update: Task difficulty is tracked through an exponential moving average of task success rate, dynamically reflecting agent competence.The update uses the task’s observed success signal to maintain a global statistic.
  • Memory Update: Subtask feedback is parameterized and semantically deduplicated to replace instance-specific values with generic placeholders and reduce redundancy.This limits overfitting to values such as filenames while maintaining a compact knowledge base.
  • Memory Update: Successful trajectories are compressed into workflows and merged into the high-level library, with success counts updated to prioritize robust strategies.Unstable strategies are progressively discarded as workflow statistics are maintained.

H. Qualitative Examples

Qualitative examples show UI-Mem improving long-horizon execution through memory guidance and failure diagnosis, while visual grounding errors remain a practical failure mode.

  • Evaluation Scope: The qualitative evaluation presents examples of long-horizon planning, failure correction, and unresolved visual-grounding challenges.The accompanying prompts and figures cover workflow extraction, subtask skills, failure analysis, template abstraction, and trajectory evaluation.
  • Memory Guidance: Full memory guidance completes the contact-creation task perfectly, whereas weak guidance yields partial completion and no guidance yields failure.The full-guidance trajectory fills both the name and organization fields; weaker settings leave information missing or make no progress.
  • Error Correction: Failure diagnosis converts an initial navigation mistake into a correction guideline that enables successful file sorting on the second attempt.The diagnosis identifies the erroneous back navigation and guides the agent to tap “Get started” and avoid switching applications.
  • Remaining Failure Modes: A fine-grained plan can still fail when the agent visually misgrounds the final action, despite correctly opening an Incognito tab.The planner intends to enter the URL, but the agent does not correctly select the address bar.
Loading 2602.05832v1…