Source-linked AI summary
MobileMem: Learning from a Year of Mobile Experiences
Xinle Deng, Yida Xue, Xiangyuan Ru, Haoming Xu, Shuofei Qiao, Mengru Wang, Yijun Chen, Buqiang Xu, Chen Jiang, Yuchen Eleanor Jiang, Lizhong Wang, Jianfeng Wang, Li Zeng, Haofen Wang, Guilin Qi, Huajun Chen, Ningyu Zhang
TL;DR
Existing mobile-memory benchmarks inadequately represent heterogeneous, multimodal, evolving personal experiences. MobileMem constructs temporally coherent long-horizon trajectories and shows that specialized memory-augmented frameworks outperform long-context and RAG approaches across most tasks.
Problem
Existing personalization approaches focus on adapting agents at a given moment, leaving persistent understanding of users’ evolving experiences insufficiently addressed.
Method
MobileMem benchmarks on-device memory using knowledge-grounded synthesis to create long-horizon, temporally coherent trajectories across heterogeneous mobile interactions and reasoning categories.
Results
Specialized memory-augmented frameworks consistently outperform long-context and RAG methods across almost all tasks and backbone models; EverMemOS reaches 39.41% versus 24.76%.
Takeaways & Limitations
MobileMem provides standardized evaluation of long-horizon memory preservation and adaptation in realistic mobile scenarios.
Takeaways & Limitations
Synthesized experiences may contain noise, inaccuracies, or inconsistencies, and current user modeling may not capture users’ evolving preferences and contextual factors.
Abstract
from arXiv · showhide
The next generation of AI agents is increasingly moving beyond systems that answer isolated questions toward persistent personal assistants that can understand, remember, and continuously learn from users' experiences. Such assistants require long-term memory to accumulate and leverage user-specific experiences over time, yet existing benchmarks remain inadequate for realistic mobile settings, where experiences are heterogeneous, multimodal, evolving, and deeply personal. We introduce MobileMem, a benchmark and framework for studying on-device long-term memory, grounded in a year-scale collection of mobile experiences. MobileMem employs a knowledge-grounded synthesis pipeline to construct coherent and temporally consistent long-horizon trajectories from user-app sessions. It provides complementary text and multimodal settings covering multi-hop and temporal reasoning, knowledge updating, and implicit preference inference. Specifically, MobileMem enables agents to remember the past, understand the present, and adapt to the future. By modeling experiences rather than isolated facts, MobileMem moves memory beyond information retrieval toward experiential intelligence for continuous personal learning.
1. Introduction
MobileMem addresses the need for persistent, secure, and durable on-device personal memory that integrates users’ evolving experiences across fragmented mobile data. It introduces a benchmark and synthesis framework for evaluating multimodal, long-horizon memory through realistic textual and multimodal scenarios.
- Motivation: Mobile agents require continuous user understanding grounded in past experiences, evolving preferences, routines, relationships, and behaviors rather than isolated interactions.This motivates persistent personal intelligence and personalized language-model systems.
- Challenges of Evolving Mobile Memory: On-device mobile memory must handle exploding event volumes, fragmented multimodal application data, and continual evolution under limited storage and bandwidth.These constraints distinguish mobile memory from cloud-based systems that assume centralized databases, cloud computation, and effectively unlimited storage.
- The MobileMem Benchmark: MobileMem fills a benchmark gap by evaluating on-device memory systems over heterogeneous, multimodal, continuously evolving mobile interactions.Its architecture represents entities, relations, and temporal dynamics in structured personal knowledge graphs, assessing transformation of fragmented events into coherent experiences.
- Long-Horizon Data Synthesis from User Prior Knowledge: KEME synthesizes realistic long-horizon, personalized, multimodal user trajectories from structured prior knowledge to address costly, privacy-sensitive, and data-limited collection.The synthesis framework models how user experiences unfold and evolve over time.
- The MobileMem Benchmark: MobileMem evaluates textual cross-session retrieval and inference across multi-hop reasoning, temporal reasoning, knowledge updating, implicit preference inference, and abstention, while MobileMem-Omni extends evaluation beyond text.The textual setting connects applications to a system-level assistant through predefined templates.
2. Preliminaries
MobileMem formalizes on-device personal-assistant memory over heterogeneous, multimodal mobile interaction trajectories and evaluates answers from the resulting memory state. It also frames mobile memory as a coordinated ecosystem of system-level and application-specific stores, instantiated through MobileMem and MobileMem-Omni.
- Benchmark formalization: MobileMem represents each benchmark instance as a trajectory T paired with question-answer pairs Q, whose answers are evaluated against reference sets.The system generates answers from the final memory state after processing the full trajectory, using an LLM-as-a-Judge protocol [Gu et al., 2024, Li et al., 2025a].
- Benchmark formalization: Unlike conversational benchmarks, MobileMem models mobile experiences as heterogeneous action streams spanning user and assistant actions, third-party applications, interface states, notifications, and multimodal records.The stream is segmented into chronological sessions and flattened into a single action sequence for memory processing.
- Benchmark formalization: The memory layer updates its state sequentially as each action arrives, producing M_t from M_t−1 and x_t through an update function.The final state integrates information from the entire flattened interaction trajectory before answering questions.
- On-device memory ecosystem: Mobile environments expose an observability gap because substantial user activity occurs in third-party applications and is only partially visible to conventional memory layers.Screenshot-based systems expand observability but treat applications mainly as passive targets, overlooking their existing historical records and retrieval mechanisms.
- On-device memory ecosystem: The proposed ecosystem combines a system-level memory layer for cross-application knowledge with application-specific memories that filter detailed interactions and provide specialized retrieval.The system layer acts as a shared blackboard, while application memories provide cognitive offloading; standardized protocols let users and developers control information exchange and sensitive-data exposure.
- Benchmark scenarios: MobileMem instantiates integrated applications that forward newly created records to system memory, whereas MobileMem-Omni uses user-shared screenshots as the observation channel for disconnected applications.The two settings provide simplified but representative initial scenarios for evaluating mobile memory capabilities.
3. Long-Horizon Data Synthesis from User Prior Knowledge
This section presents KEME, a knowledge-guided framework that synthesizes temporally coherent, evolving mobile-experience trajectories and QA pairs from user prior knowledge. It combines hierarchical planning, knowledge grounding, experience realization, and quality control to construct long-horizon multi-session data while supporting distinct text and multimodal settings.
- Framework Overview: The framework comprises user prior knowledge construction, interaction trajectory synthesis, QA-pair synthesis, and quality control applied throughout the pipeline.Figure 4 summarizes this progressive synthesis process from user knowledge to long-horizon trajectories and QA pairs.
- User Prior Knowledge: MobileMem and MobileMem-Omni construct prior knowledge differently, using interviewed profiles and distinct observation modalities, with Omni adding yearly events, future plans, virtual profiles, and relationship graphs.Real-user data are anonymized and desensitized, while MobileMem records app-usage statistics and metadata rather than raw interaction content.
- KEME Trajectory Synthesis: KEME synthesizes long-horizon interaction trajectories by hierarchically organizing knowledge anchors under persona and temporal constraints, then progressively refining experiences as they unfold.The process uses top-down knowledge guidance and bottom-up experience evolution rather than statically assembling trajectories.
- KEME Formalization: KEME formalizes synthesis in two stages: generating an evolving trajectory T from persona, knowledge anchors, and a time horizon, then constructing QA pairs Q from T and a question taxonomy.Validation logic and corresponding data models support robust interactions with the synthesis agents.
- KEME Formalization: Its generic knowledge-anchor interface transforms fragmented off-the-shelf data into coherent long-horizon, multi-session trajectories, addressing the scarcity of data for cross-session reasoning.This compositional design extends beyond the MobileMem series.
- Top-Down Knowledge Guidance: The planner recursively decomposes a time horizon into temporally valid, persona-compatible life events, finer-grained sub-event graphs, and concrete interaction sessions.This hierarchy moves from coarse life phases toward concrete interactions and depends on hierarchy depth and grounded sessions.
4. MobileMem Dataset
MobileMem evaluates diverse memory systems on heterogeneous, detail-sensitive mobile trajectories, finding that A-MEM and HippoRAG2 achieve the strongest overall performance across two LLM backbones. The dataset also exposes increasing difficulty with multi-evidence tasks, system-specific failure modes, and diminishing benefits from overly fine-grained profile schemas.
- Overall performance: A-MEM and HippoRAG2 achieve the best overall performance across both LLM backbones because they preserve conversational information and retain fine-grained trajectory details.Their advantage is attributed to avoiding aggressive memory compression, overwriting, or deletion, which benefits heterogeneous MobileMem trajectories.
- Overall performance: A-MEM and HippoRAG2 outperform NaiveRAG partly because their retrieval achieves substantially higher recall of target memories.A-MEM additionally uses extracted metadata, including keywords and tags, to distinguish similar memories.
- Task-wise performance: Performance generally declines from single-hop to multi-hop questions and further to query-focused summarization, which requires both evidence aggregation and concise answer synthesis.Construction costs also vary substantially: Long Context and NaiveRAG incur no construction cost, while A-MEM and EverMemOS are the most expensive among methods that invoke an LLM.
- Failure analysis: The dominant failure mode differs by system: Long Context suffers memory-update errors, NaiveRAG misses target evidence, Mem0 misextracts facts, and EverMemOS mainly fails during answer generation.All four systems nevertheless exhibit downstream answer-generation errors caused by distraction, missed evidence, or overgeneralization.
- Profile schema analysis: Medium- and fine-grained profile schemas produce more diverse trajectories than coarse schemas, but fine-grained schemas provide limited additional benefit because only about 44% of fields are mentioned after synthesis.The under-activation is attributed to limited trajectory length, which leaves insufficient opportunity for extra profile details to surface.
5. MobileMem-Omni Dataset
MobileMem-Omni is a multimodal, year-scale benchmark built from structured user trajectories to evaluate diverse long-term memory capabilities. Results show specialized memory frameworks generally outperform long-context and RAG methods, while performance remains constrained by visual, abstention, short-span, language, and efficiency challenges.
- Dataset construction and scope: MobileMem-Omni is generated with GPT-5.1 through hierarchical event, dialogue, persona, question, and image synthesis, yielding 16 trajectories, 19,060 images, and 7,415 question-answer pairs.The benchmark covers single-hop, multi-hop, knowledge-update, temporal-reasoning, abstention, implicit-preference, and visual-reasoning capabilities, with balanced English and Chinese users and long multimodal sessions.
- Overall performance: Specialized memory-augmented frameworks consistently outperform long-context and RAG methods across almost all tasks and backbone models, with GPT-5.4-mini generally surpassing Qwen3-VL-8B-Instruct.The comparison covers textual and multimodal methods evaluated on MobileMem-Omni, with Table 4 reporting task-wise and overall LLM-Judge and F1 results.
- Task-wise performance: Multimodal long context substantially outperforms caption-based long context on visual reasoning, while multi-hop, abstention, and visual reasoning are the most challenging tasks.Visual-reasoning scores are 21.9% versus 12.16% for GPT-5.4-mini and 13.5% versus 6.78% for Qwen3-VL-8B-Instruct; most abstention methods score below 20%.
- Token cost: EverMemOS achieves the best performance but incurs relatively high token costs, whereas LightMem reduces token consumption with reasonable performance.Figure 7 characterizes current memory methods as slow and resource-intensive for on-device scenarios.
- Performance variation: Performance improves with event span, while short-term events are hardest and longer events benefit from richer contextual information for retrieval and reasoning.Most methods also score lower on Chinese than English questions, suggesting weaker language-specific memory, semantic alignment, and retrieval quality.
- Caption effects: Adding image captions to NaiveRAG improves visual reasoning but harms text-oriented questions because visual information can introduce redundancy and interference.Captions provide useful contextual information for visual retrieval but may distract from purely textual reasoning.
6. Applications
MobileMem demonstrates practical long-term memory applications across heterogeneous mobile experiences, while standardizing evaluation of multimodal, temporal, preference, and reasoning capabilities. Its modular, measurable design supports integration with mobile AI ecosystems and future expansion toward broader personalized AI.
- Application Scenarios: MobileMem covers health management, personalized itinerary planning, media tracking, and work review across diverse long-term mobile memory sources.Its sources include Calendar, Photos, Notes, Documents, To-Do List, Voice Recorder, Breeno Memory, and Video Memo, capturing temporal, visual, textual, behavioral, and activity traces.
- Application Scenarios: These scenarios evaluate cross-source association, multimodal fusion, temporal reasoning, preference modeling, personalized planning, knowledge organization, and generative reasoning.They extend long-term memory beyond retrieval toward health-related reasoning, personalized travel planning, interest modeling, work analysis, and continuous learning.
- Standardization: MobileMem standardizes evaluation using approximately one year of authentic smartphone activities and heterogeneous records rather than synthetic logs or manually curated profiles.The framework unifies foundational, cognitive, preference, temporal, and visual memory across structured records, semi-structured documents, free-form text, images, and multimodal content.
- User Memory Value: MobileMem treats evolving multimodal user memory as the basis of personalization, evaluating continuous accumulation, retrieval, integration, and reasoning across extended temporal horizons.This differs from isolated factual recall by modeling authentic user trajectories, multilingual interactions, and dynamic preferences.
- Ecosystem and On-Device Intelligence: MobileMem integrates with mobile-agent modules and provides quantitative metrics for retrieval accuracy, temporal consistency, preference understanding, multimodal reasoning, and long-horizon knowledge integration.Future versions are planned to add memory from wearables, tablets, computers, smart-home devices, and other sources, alongside broader multilingual, population, planning, collaborative-memory, and cross-modal tasks.
7. Conclusion and Future Work
MobileMem introduces a comprehensive benchmark for on-device memory in realistic mobile environments, using privacy-preserving synthesis to create coherent long-horizon trajectories. Future work targets broader modalities, online evaluation, stronger privacy, and bottleneck-specific improvements, while limitations include reconstruction errors and coarse user modeling.
- Conclusion: MobileMem benchmarks on-device memory in realistic mobile environments using large-scale, temporally coherent trajectories generated by a knowledge-grounded synthesis pipeline.The benchmark covers heterogeneous interaction types and diverse reasoning categories while preserving privacy.
- Future Work: Future work will extend MobileMem to additional languages and modalities, including audio, video, and sensor data, to better represent real-world mobile usage.The proposed extensions aim to capture richer mobile experiences.
- Future Work: Future work will establish online evaluation for evolving memory systems, strengthen privacy through advanced synthesis, and investigate bottlenecks in construction, retrieval, updating, and answer generation.The paper also proposes developing targeted improvements for these distinct bottlenecks.
- Limitations: Long-horizon reconstruction may introduce noise, inaccuracies, or inconsistencies from incomplete prior knowledge despite quality-control mechanisms, and fully eliminating these errors remains challenging.This limitation particularly affects reconstructed experiences based on imperfect prior knowledge.
- Limitations: Current user modeling is relatively coarse and may miss the complexity of users’ evolving preferences, habits, goals, and contextual factors.More comprehensive user modeling is needed to capture these evolving characteristics.
Contributions
The report acknowledges partner organizations and contributors across authorship, data and application, legal and compliance, and technical leadership. It is intended as a reference for researchers and practitioners working on mobile AI agents and on-device memory systems.
- The authors thank the contributing units, organizations, and individuals and present the report as a reference for mobile AI agents and on-device memory systems.
- The report identifies Guangdong OPPO Mobile Telecommunications Corp., Ltd. and the OpenKG community as partner organizations.
- It credits core authors and contributors responsible for data and application, legal and compliance, and technical leadership.