Source-linked AI summary
LightMem-Ego: Your AI Memory for Everyday Life
Yijun Chen, Boyi Xiao, Yixian Zhao, Haoting Xia, Buqiang Xu, Jizhan Fang, Yanya Li, Yaqi Zheng, Xuehai Wang, Zirui Xue, Liuxin Zhang, Hui Li, Ningyu Zhang
TL;DR
Personal AI assistants need lightweight multimodal memory to organize and retrieve experiences from continuous egocentric streams. LightMem-Ego builds a hierarchical streaming memory with query-aware retrieval, achieving 74.1 overall R@3 and 0.627 MRR, alongside 51.9 LLM-judged and 55.6 human-judged QA accuracy.
Problem
Everyday-life assistants lack lightweight multimodal memory that continuously organizes and retrieves experiences from unsegmented visual-audio streams.
Method
LightMem-Ego aligns egocentric visual-audio streams into current, short-term, and long-term memory, then routes queries to relevant evidence for grounded answers.
Results
74.1 overall R@3 and 0.627 MRR were achieved for memory retrieval, while overall QA accuracy reached 51.9 LLM-judged and 55.6 human-judged.
Takeaways & Limitations
Explicit long-horizon multimodal memory is a promising systems direction for personal AI assistants across object finding, conversation recall, life summarization, and routine understanding.
Takeaways & Limitations
The prototype lacks a complete privacy-preserving pipeline, including automatic sensitive-content redaction, bystander consent management, access control, and retention/deletion policies.
Abstract
from arXiv · showhide
Personal AI assistants on mobile and wearable devices continuously perceive users' daily lives through visual and audio streams. However, answering queries about past experiences requires lightweight multimodal memory that can continuously accumulate, organize, and retrieve long-term experiences, which remains challenging. To address this challenge, we present LightMem-Ego, a lightweight streaming multimodal memory system for everyday-life assistance. The system continuously captures egocentric visual and audio streams, aligns them on a shared timeline, and organizes them into a hierarchical memory consisting of current, short-term, and long-term memory. Given a user query, LightMem-Ego dynamically routes retrieval to the appropriate memory level and generates answers grounded in multimodal evidence. The demonstration can be deployed on smartphones and AI glasses, supporting object finding, conversation recall, life summarization, routine discovery, and personalized assistance. Code is available at https://github.com/zjunlp/LightMem-Ego.
1 Introduction
LightMem-Ego addresses the need for lightweight multimodal memory in everyday-life assistants by continuously organizing egocentric visual-audio streams into a hierarchical memory. It routes queries to temporally appropriate evidence for grounded assistance across present context, recent events, and long-term routines.
- Motivation: Smartphones and AI glasses enable continuous, hands-free capture of egocentric visual and audio streams, allowing assistants to act as everyday memory companions.The introduction motivates moving beyond isolated question answering toward memory-supported assistance for object locations, conversations, meetings, and changing routines.
- System: LightMem-Ego is a deployable streaming multimodal memory system connecting lightweight smartphone or AI-glasses-style clients to a backend.The backend ingests egocentric visual-audio streams and segments recent observations into events.
- Memory hierarchy: Its three-level hierarchy stores ongoing context in current memory, recent micro-events in short-term memory, and consolidated episodes and semantics in long-term memory.This organization supports continuous accumulation and structuring of everyday experiences.
- Retrieval and answering: A memory router selects evidence according to each query’s temporal scope and intent, enabling grounded answers about the present, recent past, and long-term routines.The system handles continuous frame streams, audio chunks, synchronized timestamps, and modular speech, vision-language, language-model, and retrieval components.
- Evaluation and applications: The system targets object finding, conversation recall, life summarization, and routine discovery, with quantitative evaluation of retrieval, question answering, and latency against application-oriented baselines.The baselines lack explicit multimodal long- and short-term memory.
2 Related Work
Related work spans persistent conversational memory, situated wearable and mobile assistants, and multimodal personal-memory systems for organizing and querying daily experiences. These directions motivate LightMem-Ego’s persistent, streaming, hierarchical multimodal memory while leaving a focus on language-centric interaction or separate long-context and lifelogging settings.
- Conversational Memory: Conversational-memory systems extend assistants with persistent, updateable, and retrievable user-specific information over time.The cited designs include personalized continuity and explicit hierarchical memory management with external storage, consolidation, and retrieval.
- Conversational Memory: These conversational systems motivate queryable memory but mainly emphasize language-centric interaction rather than streaming egocentric multimodal experience.This contrast directly frames LightMem-Ego’s multimodal, continuously captured setting.
- Wearable and Mobile AI Assistants: Wearable and mobile assistants use egocentric perception for real-time situated assistance, with benchmarks linking visual understanding to physical-world actions.The cited work includes wearable systems, smart-glasses agents, glasses-style benchmarks, and action-oriented prototypes.
- Multimodal Personal Memory: Multimodal personal-memory research organizes daily experiences through temporal video representations, egocentric retrieval, lifelogging archives, and semantic activity summarization.These studies motivate LightMem-Ego’s streaming hierarchical memory design.
3 LightMem-Ego
LightMem-Ego is a streaming multimodal memory system that captures and time-aligns everyday visual and audio experiences, organizes them hierarchically, and answers questions through explicit retrieval. Its lightweight edge design and backend processing support current-scene understanding, recent recall, and retrospective assistance on smartphones and wearable devices.
- System overview: LightMem-Ego captures multimodal life streams from smartphones or wearables and organizes them into hierarchical memory for explicit question answering.Online updates serve current and short-term memory, while asynchronous consolidation builds long-term episodic and semantic stores.
- Multimodal stream: Visual frames and audio chunks are uploaded to a shared session timeline, with timestamps preserved for event construction, transcript backfilling, and retrieval.The lightweight capture layer collects, normalizes, timestamps, and forwards inputs without heavy client-side reasoning.
- Event construction: The system incrementally constructs contiguous microevents using temporal continuity and crossframe changes, attaching representative frames, descriptions, and aligned or pending audio.Transcripts and refined descriptions are asynchronously added as more evidence arrives.
- Hierarchical memory: Current memory supports ongoing-scene responses, short-term memory stores recent event evidence, and long-term memory consolidates stable events for retrospective use.The hierarchy comprises current, short-term, and long-term memory, with long-term storage containing episodic and semantic information.
- Deployment: Mobile and AI-glasses deployment keeps edge processing lightweight by sampling, compressing, timestamping, and uploading low-rate frames and short audio chunks.Heavy VLM or LLM inference and costly operations such as ASR backfilling and event refinement are shifted to the backend.
- Query-time retrieval: A query router selects current memory for present-scene questions, short-term memory for recent recall, and long-term memory for retrospective or routine-level questions.Retrieval returns compact, event-centric multimodal evidence that is fused into an evidence view before answer generation.
4 Demonstration
LightMem-Ego demonstrates everyday assistance by retrieving temporally localized multimodal evidence for immediate help and conversation recall, while aggregating events for life summarization and routine discovery.
- Immediate Assistance: LightMem-Ego supports immediate assistance and short-horizon recall on mobile devices and AI glasses using recent observations and event segments.Example queries include finding a badge, recalling when keys were held, and identifying the current view.
- Conversation Recall: For conversation recall, the system combines short-term event memory with aligned transcript context to recover relevant spoken interactions.This supports questions about information given by a doctor or tasks requested by a colleague.
- Life Summarization and Routine Discovery: LightMem-Ego summarizes recent experiences by aggregating event-centered evidence across multiple segments.Supported queries include summarizing an afternoon or recounting what happened during lunch.
- Life Summarization and Routine Discovery: The system performs routine discovery by drawing on long-term semantic memory to identify recurring activities, preferences, and routines.It extends memory use from isolated event recall to habit-level reasoning.
5 Quantitative Evaluation
LightMem-Ego is quantitatively evaluated on object finding, conversation recall, and life summarization using retrieval, QA, and latency metrics. It achieves strong overall evidence retrieval and usable memory-grounded answers, while object finding and conversation recall remain more challenging.
- Evaluation setup: Evaluation covers object finding, conversation recall, and life summarization with annotated evidence, retrieval Recall@1/3/5 and MRR, QA accuracy, and P50/P90 latency.Latency is reported for retrieval, generation, and end-to-end QA on phone and glasses-style clients.
- Memory retrieval: 74.1 overall R@3 and 0.627 MRR show accurate retrieval across the three everyday-life scenarios.The results support evidence selection for downstream experience QA.
- Experience QA: 51.9 LLM-judged accuracy and 55.6 human-judged accuracy overall demonstrate usable memory-grounded answers when relevant evidence is retrieved.Life summarization performs best among the evaluated scenarios.
- Experience QA: Fine-grained object finding and conversation recall remain more challenging than life summarization in the current prototype.This limitation is reported for QA despite the system’s overall ability to produce usable answers.
P50 P90 P50 P90
LightMem-Ego organizes streaming visual-audio experience into current, short-term, and long-term memory stores, supporting both immediate assistance and retrospective experience QA. Short-term queries achieve near-interactive latency, while long-term queries require additional retrieval and evidence aggregation.
- Latency: Table 3 reports P50 and P90 latency breakdowns by memory scope and client, where P50 and P90 denote 50th and 90th percentile latencies.The short-term QA breakdown includes retrieval, answer generation, and end-to-end QA timings.
- Latency: 5.86s P50 end-to-end latency is achieved for short-term memory QA on phone, compared with 7.01s on the glasses-style client.Short-term memory QA is described as supporting near-interactive assistance.
- Latency: 14.87s P50 latency is required for long-term memory QA on phone, compared with 19.96s on the glasses-style client.Long-term queries incur higher latency because they require additional retrieval and evidence aggregation, while remaining practical for retrospective recall and life-summary tasks.
- Memory organization and capabilities: The system supports explicit multimodal memory capabilities spanning current, short-term event, long-term episodic, semantic routine, and timestamped evidence retrieval.The comparison covers publicly described capabilities rather than experimentally measured performance.
- Memory organization and capabilities: LightMem-Ego organizes everyday visual-audio experience into current, short-term, and long-term memory stores for immediate assistance and retrospective experience QA.Supported tasks include object finding, conversation recall, life summarization, and routine discovery.
6 Conclusion
LightMem-Ego is a streaming multimodal memory system for everyday-life assistance that connects mobile and wearable capture with hierarchical memory and memory-grounded question answering. It demonstrates how multimodal memory can help personal AI assistants capture, organize, and recall lived experience over time.
- LightMem-Ego connects mobile and wearable multimodal capture with hierarchical memory construction and memory-grounded question answering in a single backend.The system supports object finding, conversation recall, life summarization, and routine understanding.
- The system demonstrates support for object finding, conversation recall, life summarization, and routine understanding in everyday-life assistance.
- LightMem-Ego positions explicit long-horizon multimodal memory as a promising systems direction for personal AI assistants.The conclusion bases this direction on the demonstration and evaluation.
- Building real-world personal AI assistants requires capturing, organizing, and recalling human experience over time beyond language understanding alone.
- LightMem-Ego shows that multimodal memory can support AI systems that understand lived experience as well as utterances.
Limitations
LightMem-Ego is limited by dependence on external APIs, which makes accuracy and latency sensitive to model reliability, runtime variation, and rate limits. Upstream errors can propagate through memory construction and reduce answer faithfulness.
- System dependence: The system relies on upstream APIs for speech recognition, visual understanding, language generation, and retrieval-based question answering.This external dependence spans the main perception, generation, and retrieval components.
- System dependence: Accuracy and latency are sensitive to external model reliability, runtime variation, and rate limits.These factors can affect system behavior even when the memory pipeline itself is unchanged.
- Error propagation: Errors in transcripts, visual descriptions, or timestamp alignment may propagate into memory construction and affect answer faithfulness.The limitation arises because upstream multimodal and temporal errors can influence downstream memories and responses.
Ethical and Privacy Considerations
LightMem-Ego raises privacy concerns because it persistently converts sensitive egocentric visual and audio streams into queryable memories. Its research prototype lacks a complete privacy-preserving pipeline, motivating protections across the memory lifecycle.
- Privacy risks: Egocentric streams may expose sensitive information about users and bystanders, including conversations, faces, documents, locations, screens, and routines.Persistent queryable memory creates risks from both raw data and derived representations, including transcripts, summaries, embeddings, and long-term semantic memories.
- Prototype limitations: The research prototype does not automatically redact sensitive content, manage bystander consent, enforce fine-grained access control, or provide mature retention and deletion policies.The system is explicitly presented as a research demonstration without a complete privacy-preserving pipeline.
- Future protections: Future work will integrate on-device preprocessing, selective capture, sensitive-content filtering, encrypted storage, and user-controlled memory editing into the memory lifecycle.These measures are proposed as privacy protections for future versions of the system.