PaperlayerView exact revision on arXiv

Source-linked AI summary

According to Me: Long-Term Personalized Referential Memory QA

Jingbiao Mei, Jinghong Chen, Guangyu Yang, Xinyu Hou, Margaret Li, Bill Byrne

arXiv:2603.01990v1cs.AIcs.CLcs.CV

TL;DR

Existing long-term memory benchmarks largely omit multimodal, multi-source personal references grounded in lived experience. The paper introduces ATM-Bench and Schema-Guided Memory to evaluate and structure such memory, finding that current systems remain below 20% accuracy on the hard set while SGM improves performance over Descriptive Memory. The benchmark is intended as a non-commercial methodological testbed rather than a deployable personal assistant system.

  • Problem

    Existing long-term memory benchmarks focus primarily on dialogue history, leaving multimodal, multi-source personalized references and lived-experience grounding insufficiently evaluated.

  • Method

    The paper introduces ATM-Bench and formulates Memory QA as Memory Ingestion, Retrieval, and Answer Generation, using Schema-Guided Memory to represent heterogeneous multimodal sources.

  • Results

    Under 20% accuracy is achieved on ATM-Bench-Hard, while SGM improves performance over Descriptive Memory across QA and retrieval evaluations.

  • Takeaways & Limitations

    ATM-Bench exposes substantial performance gaps in current personalized memory systems and motivates research on robust and scalable personalized memory systems.

  • Takeaways & Limitations

    The benchmark and methodology are restricted to non-commercial research use and are not presented as a deployable personal assistant system.

Abstract

from arXiv · show

Personalized AI assistants must recall and reason over long-term user memory, which naturally spans multiple modalities and sources such as images, videos, and emails. However, existing Long-term Memory benchmarks focus primarily on dialogue history, failing to capture realistic personalized references grounded in lived experience. We introduce ATM-Bench, the first benchmark for multimodal, multi-source personalized referential Memory QA. ATM-Bench contains approximately four years of privacy-preserving personal memory data and human-annotated question-answer pairs with ground-truth memory evidence, including queries that require resolving personal references, multi-evidence reasoning from multi-source and handling conflicting evidence. We propose Schema-Guided Memory (SGM) to structurally represent memory items originated from different sources. In experiments, we implement 5 state-of-the-art memory systems along with a standard RAG baseline and evaluate variants with different memory ingestion, retrieval, and answer generation techniques. We find poor performance (under 20\% accuracy) on the ATM-Bench-Hard set, and that SGM improves performance over Descriptive Memory commonly adopted in prior works. Code available at: https://github.com/JingbiaoMei/ATM-Bench

1. Introduction

Existing memory QA benchmarks largely center on conversational history, while realistic personal memory requires multimodal, multi-source referential reasoning. ATM-Bench introduces this setting, formalizes its pipeline, and evaluates memory representations and systems, revealing substantial difficulty and benefits from Schema-Guided Memory.

  • Existing Memory QA benchmarks typically focus on conversational history, overlooking informative static sources such as photos and emails from lived experience.
  • Personalized references such as “the gift I bought for mum during our trip to Japan” require holistic understanding of personal experience and remain insufficiently evaluated.
  • ATM-Bench contains 1038 question–answer pairs with cross-validated ground-truth memory evidence for evaluating long-term, multimodal, multi-source personalized Memory QA.The benchmark spans approximately four years of privacy-preserving personal memory data, including emails, images, and videos.
  • Memory QA is formulated as Memory Ingestion, Retrieval, and Answer Generation, enabling comparisons among memory-system design variants.Ingestion includes choices of memory representation and organization.
  • Schema-Guided Memory represents heterogeneous, multimodal memory items structurally and improves QA and retrieval performance over Descriptive Memory.SGM supports metadata such as timestamps and GPS location.
  • Under 20% accuracy is achieved by current methods on the challenging ATM-Bench-Hard set, exposing a substantial performance gap.The evaluation implements five state-of-the-art memory systems and examines memory representation and organization methods.

2. Related Work

Prior long-term memory work mainly evaluates dialogue-based or media-centric settings, whereas personalized referential reasoning requires heterogeneous evidence and explicit temporal and location grounding. ATM-Bench addresses this broader retrieval-and-reasoning setting.

  • Long-context scaling does not solve long-term recall because models can exhibit context forgetting, motivating explicit episodic-memory evaluation.
  • Existing long-term memory benchmarks are predominantly text-centric and derived from human–assistant dialogues, including LongMemEval, LoCoMo, and MemoryBank.
  • Memory-management frameworks address long-term memory through explicit storage, context paging, or decoupling storage from inference.Examples include MemGPT, MemoryOS, Mem0, and Memobase.
  • RAG commonly retrieves passages to support knowledge-intensive generation and has expanded from single-pass retrieval to iterative, agentic workflows.
  • Personalized referential reasoning remains difficult because queries are underspecified and require separating retrieval from reasoning across heterogeneous data with temporal and location grounding.

3. ATM-Bench

ATM-Bench formalizes long-term personalized referential Memory QA over multimodal, multi-source personal memories, with human-annotated answers and evidence. It emphasizes implicit references, location, memory updates, multi-evidence composition, abstention, challenging temporal spans, and answer- and retrieval-aware evaluation.

  • Problem Formulation: ATM-Bench pairs a multimodal personal memory corpus with questions, ground-truth answers, and evidence sets for context-dependent Memory QA.The raw corpus includes image, video, and email data, while questions, answers, and evidence are human-annotated.
  • Comparison with Prior Work: ATM-Bench compares memory and long-context QA benchmarks across capabilities including PR, LA, MUT, ME, and ABS.The comparison also reports memory-item counts and accessible context, with media data counted as 300 tokens each.
  • Characteristics of ATM-Bench: The benchmark targets personalized referential reasoning, location awareness, memory updates over time, multi-evidence composition, and abstention.These settings require resolving implicit personal references, using metadata or visual and textual cues, revising earlier beliefs, combining heterogeneous evidence, and recognizing unanswerable queries.
  • Characteristics of ATM-Bench: Approximately 30% of ATM-Bench questions require multiple evidence items, compared with 6% in Memory-QA.ATM-Bench-Hard requires an average of 6.3 evidence items drawn from multiple sources, versus 1.6 for ATM-Bench overall.
  • Characteristics of ATM-Bench: The hard split requires evidence spanning an average of 226 days, with a maximum span of 933 days, and models can fail even with gold evidence.This setting models memory evolution over time through the MUT challenge.
  • Evaluation Metrics: The evaluation reports answer correctness with Question Type Score and retrieval quality with Recall@k and Recall@kGT.Question Type Score selects Exact Match, Jaccard similarity, or LLM-Judge according to whether the answer is numeric, a list, or open-ended; a joint metric combines answer correctness with retrieval quality.

4. Memory Assistant Methodology

The assistant processes heterogeneous personal artifacts into organized memory, retrieves relevant evidence, and generates grounded answers. Its memory design separates representation from organization, with Schema-Guided Memory encoding structured fields and optional links connecting items.

  • Personal Memory Assistant Formulation: The assistant has three stages: memory ingestion, retrieval, and answer generation.Ingestion normalizes heterogeneous artifacts, retrieval selects a small evidence set, and answer generation produces a grounded response.
  • Memory Ingestion: Memory ingestion splits into preprocessing, which constructs memory representations, and optional organization, which structures stored items.Organization can add graph-based linkages or update memory items across sources.
  • Memory Representation: Descriptive Memory represents each memory item as a natural-language description, whereas SGM uses text-based key–value fields under a fixed schema.SGM schemas can include time, location, entities, OCR, tags, email summaries, and bodies; both formats contain the same information.
  • Memory Organization: Linked Memory forms a graph by inferring adjacency relations among memory items, while Piled Memory stores items without explicit structure.The inferred links can be exploited during retrieval, and agentic systems may update items using newly inferred relations.
  • Retrieval and Answering: Embedding-based retrieval encodes queries and memory items in a shared vector space, then selects the top-k items using maximum inner product search.The answerer can either generate once from the query and evidence or iteratively reason and retrieve additional evidence.

5. Experiments

Experiments compare memory representations, organizations, retrieval configurations, and answer-generation strategies on ATM-Bench and ATM-Bench-Hard. SGM consistently improves over DM, while agentic answering is mixed and hard personalized QA remains challenging even with gold evidence.

  • Experimental Setup: The evaluation compares memory systems on ATM-Bench and ATM-Bench-Hard across ingestion, retrieval, and answer-generation designs.Reported measures include question-type accuracy, overall QS, retrieval recall, Joint@10, and memory encoding time.
  • Main Findings: SGM improves Oracle performance over DM by 20% on ATM-Bench-Hard.Across linked and piled settings, SGM also yields higher retrieval performance and QS scores than DM in HippoRAG, Self-RAG, and ATM-RAG.
  • Main Findings: Agentic answering improves ATM-Bench performance but does not transfer its gain to ATM-Bench-Hard.Self-RAG and ATM-RAG have comparable performance on ATM-Bench despite reflective versus single-pass answering.
  • Main Findings: 12.6h → 1.6h encoding time accompanies improved performance when A-Mem is modified from Linked to Piled Memory.The result suggests that the simpler organization can work well while substantially reducing encoding cost.
  • Answer Generation Bottlenecks: GPT-5 reaches 74.7 on ATM-Bench-Hard under Oracle retrieval, compared with 85.3 on ATM-Bench.Claude Opus 4.5 reaches 86.0 on ATM-Bench, while Gemini 2.5 Pro reaches 64.3 on the hard split.
  • Retrieval and Reranking: Scaling the retriever generally improves retrieval recall and downstream QA, but Qwen3-VL-Embedding-2B performs worse than text-based alternatives.The authors suggest that many visual tokens may dilute metadata signals such as timestamps and locations.

6. Conclusion

ATM-Bench evaluates personalized Memory QA over long-term, multimodal, and multi-source data, while Schema-Guided Memory unifies heterogeneous sources. Current systems show substantial performance gaps, with accuracy below 20% on the challenging ATM-Bench-Hard subset.

  • ATM-Bench targets personalized Memory QA over long-term, multimodal, and multi-source personal memory data.
  • Schema-Guided Memory unifies heterogeneous memory sources.
  • Accuracy remains below 20% on the challenging ATM-Bench-Hard subset.
  • The results highlight current limitations and motivate research on robust and scalable personalized memory systems.

Impact Statement

The work is presented as a privacy-aware, reproducible research resource for multimodal memory systems. Its data-processing pipeline combines consent, automated filtering, anonymization, synthetic replacement, and human verification, while excluding harmful or protected-attribute content.

  • ATM-BENCH aims to support privacy-aware, multimodal memory-system development in a controlled and reproducible setting.
  • All collected data had explicit informed consent and underwent automated filtering, anonymization, and human verification.The pipeline covered provenance, privacy preservation, content safety, and intended use.
  • Visual data were screened for sensitive content, with face blurring and fine-grained redaction applied during processing.Human reviewers verified blurring, manually corrected missed faces, and redacted private attributes such as names, ticket numbers, receipts, and computer-screen text.
  • Email data were sanitized by stripping metadata, rewriting text to remove linguistic fingerprints, and auditing rewritten content.The rewriting preserved core semantic information while removing identifying names, addresses, and other sensitive details.
  • Synthetic PII replacement preserves retrieval-task structure without exposing sensitive user information.Examples include replacing tracking numbers with consistent dummy formats and private addresses with fictitious realistic locations.
  • Approximately 150 human-verification hours supported the final privacy-processing stage.
  • The benchmark excludes offensive, harmful, inappropriate, and protected-attribute content.The stated exclusions include hate speech, harassment, sexual content, self-harm or violence, and attributes such as sexual orientation, detailed religious belief, political affiliation, and specific health diagnoses.

A.4. Intended Use and Limitations

ATM-BENCH is restricted to non-commercial research and is intended as a privacy-preserving methodological testbed rather than a deployable personal assistant. Its benchmark and code cannot be used for surveillance, profiling, sensitive-attribute inference, or consequential decisions about individuals.

  • ATM-BENCH is released strictly for non-commercial research under controlled, privacy-preserving conditions.
  • The dataset uses CC-BY-NC 4.0, while the accompanying code uses the MIT License.
  • The benchmark and code are explicitly prohibited for surveillance, monitoring, or profiling of individuals.
  • Use for inferring sensitive attributes or making consequential decisions about real people is prohibited.Examples include hiring, credit, insurance, and law-enforcement decisions.
  • The authors prohibit intrusive data collection beyond ethically acceptable and legally compliant practices.
  • The contribution is a privacy-preserving methodology testbed, not a ready-to-deploy personal-assistance system.

B.1. Failure Modes of Agentic Auto-Annotation

Agentic auto-annotation models can identify relevant evidence but generate questions that diverge from realistic human memory queries. ATM-Bench therefore uses human-centered annotation with cross-source merging, validation, and guidelines emphasizing entity-driven, temporally and spatially grounded recall.

  • The raw corpus contained approximately 13,000 images, videos, and emails, and models were used to generate question–answer–evidence triples.
  • Auto-generated questions often emphasized analytical properties instead of realistic human memory recall.Examples focused on exact temporal offsets or wildlife encountered on a specified date.
  • Human recall more often begins with salient entities or events and requires contextual grounding.The paper contrasts exact-date questions with recalling a deer sighting and asking when and where it occurred.
  • Fully automated annotation failed to capture the distribution of human-likely personal memory questions.ATM-Bench consequently adopts a human-centered strategy informed by prior HCI studies.
  • Five trained annotators produced annotations over images, videos, and emails in approximately 200 total hours.
  • Annotations were separately created by modality, temporally sorted and merged, and used to construct questions requiring reasoning across memory sources.Annotators reviewed memory collections grouped within temporal windows of approximately one year.
  • Each QAE pair received independent validation for evidence grounding, completeness, sufficiency, relevance, and temporal consistency.Disagreements were discussed, and only validated pairs were retained.
  • Annotation guidelines emphasize human-likely, entity- and event-driven questions with appropriate temporal and spatial grounding.

C.1. Statistics

ATM-Bench characterizes questions by evidence structure, temporal span, semantics, topics, modalities, and geographic coverage. The section also identifies the evaluated memory systems and experimental setup.

  • Question statistics: Questions are annotated by evidence count, maximum evidence-time span, and recall distance from an optional anchor date.Maximum time span is undefined with fewer than two evidence timestamps; recall distance is undefined without an anchor date or evidence timestamps.
  • Semantic taxonomy: The benchmark uses orthogonal topic and intent labels, while recording email, image, and video modality separately.Intents include Temporal, Geolocation, Recall, Identification, and Aggregation; topics include work, personal information, travel, activities, household, and everyday life logging.
  • Memory topics: Activities rely primarily on images and videos, emails dominate Work + Learning & Knowledge, and Travel is distributed more evenly across modalities.The distribution excludes entries labeled Unknown.
  • Geographic coverage: ATM-Bench spans countries across North America, Europe, Asia, and Oceania, supporting evaluation across diverse visual, cultural, and real-world settings.The geographic distribution is summarized in Figure 5.
  • Systems and setup: The experiments compare A-Mem, Mem0, HippoRAG2, Self-RAG, and retrieval variants using descriptive or schema-driven memory representations.The setup also reports oracle-retriever results and uses Qwen3-VL models in the implementation environment.

E.1. LLM Judge

The evaluation is robust to LLM judge selection, while retrieval-depth sweeps show stable performance across moderate K values and diminishing returns at larger depths. A temporal-update case illustrates a remaining answer-generation failure.

  • Judge robustness: Relative rankings and absolute scores remain highly consistent across two independent judges, answerer models, and difficulty splits.GPT-5-mini and GLM-4.7 produce small differences that do not alter the overall conclusions.
  • Judge robustness: 0.2% QS for No-Evidence closely matches the 0.3% abstention rate, because the baseline consistently abstains on those questions.The No-Evidence baseline is therefore aligned with the dataset’s abstention cases.
  • Retrieval depth: ATM varies only marginally from K = 2 to K = 20, with no monotonic improvement as retrieval depth increases.The results indicate that most relevant evidence is already captured at relatively small retrieval depths.
  • Retrieval depth: Larger K values show slight degradation, suggesting diminishing returns when additional irrelevant or weakly related memories increase contextual noise.The authors consequently motivate moderate retrieval depths for balancing coverage and answer stability.
  • Error analysis: The oracle case tests whether the answerer prioritizes an updated invoice over an outdated booking confirmation, exposing temporal memory-update limitations.GPT-5 reports C853.26 instead of the ground-truth C842.97 after relying on the earlier confirmation.

F.2. Case 2: Location Aliasing Under Geocoding Noise

Location aliasing under geocoding noise occurs when one event receives conflicting place names from erroneous metadata. Correct resolution requires reconciling metadata with temporal, visual, and semantic evidence.

  • Failure mechanism: Two photos from the same dining event are inconsistently mapped: the correct restaurant is Caf´e Boh`eme, while one image is labeled Pasha Kebab.The images share a time window and visual context, but one has an incorrect coordinate-to-POI mapping.
  • Observed failure: The model answers Pasha Kebab instead of the ground-truth Caf´e Boh`eme when asked to identify the restaurant in Sligo.The case demonstrates an incorrect answer under noisy or insufficiently discriminative GPS metadata.
  • Required reasoning: Correct resolution requires integrating food presentation, dining ambience, nearby timestamps, and visual similarity rather than over-trusting coarse location cues.The failure reflects limited evidence-based disambiguation when multiple plausible memories coexist.
  • Evaluation rule: The evaluator marks predictions false when they conflict with or fail to cover the ground-truth answer, even if they are otherwise reasonable.The judging prompt requires exact or near-exact agreement for numerical and temporal answers and coverage without contradictory extras for object sets.
Loading 2603.01990v1…