Source-linked AI summary
Personalizing LLM Agent Memory Using Biometrics
Yanhong Qian, Qingguo Meng, Shihao Ding, Xingbo Dong, Zhe Jin, Hanrui Wang, Isao Echizen
TL;DR
Shared-agent memory retrieval must distinguish semantic relevance from whether the requester matches the memory owner, because session or account identity can diverge from physical identity. Bio-Memory adds biometric embeddings to A-Mem notes and filters the retrieval candidate pool by biometric matching before semantic ranking. Across seven face benchmarks and ten palmprint protocols, it consistently separates owner from non-owner queries, supporting biometrics as a control signal for personalized retrieval in shared environments.
Problem
Shared environments can make account, session, and physical requester identities diverge, while semantic-only retrieval cannot reliably separate users’ private memories.
Method
Bio-Memory augments A-Mem’s atomic notes with biometric embeddings and performs biometric matching before semantic ranking to construct the retrieval candidate pool.
Results
Across seven face benchmarks and ten palmprint protocols, owner queries retain strong performance while non-owner queries consistently lose access to user-specific retrieval evidence.
Takeaways & Limitations
The results support biometrics as a practical retrieval-time control signal for personalized memory in shared-agent settings.
Takeaways & Limitations
The approach is built on A-Mem and addresses its semantic-only retrieval limitation by adding biometric identity gating rather than replacing the underlying memory organization.
Abstract
from arXiv · showhide
Personalized memory helps LLM agents deliver stable, tailored assistance by storing and reusing user-specific data across interactions. In multi-user scenarios, however, retrieval must consider not only semantic similarity but also whether the current requester matches the identity associated with the stored memory. We propose Bio-Memory, a biometric-aware memory architecture that conditions memory retrieval on both semantic similarity and biometric matching. Built on top of A-Mem, Bio-Memory augments each atomic memory note with a biometric embedding and uses biometric matching to form the retrieval candidate pool before semantic ranking. We evaluate Bio-Memory on LoCoMo in a 10-user shared-agent setting over 7 face benchmarks and 10 palmprint protocols. Across datasets, Bio-Memory consistently separates owner and non-owner queries. Under face-based personalization, the largest average gap reaches 27.29% / 21.15% in F1 / BLEU-1 on CALFW; under palmprint-based personalization, the corresponding gap is 25.75% / 19.22% on MS_Blue. These results support biometrics as a practical control signal for personalized memory retrieval in shared environments.
1 Introduction
LLM agents increasingly use persistent memory for personalized assistance, but shared environments require identifying the physical requester rather than relying only on semantic relevance or session identity. Bio-Memory extends A-Mem with biometric embeddings and filters memories by biometric match before semantic retrieval.
- Persistent memory lets LLM agents retain user preferences, schedules, interactions, and task-specific facts across sessions.
- Shared environments create a mismatch between the active session or account and the physical requester, complicating personalized memory retrieval.
- Bio-Memory preserves A-Mem’s structured notes while adding biometric embeddings that associate memories with physical users.
- At inference time, biometric matching forms the candidate pool, after which semantic retrieval operates only on matched memories.
- The evaluation uses LoCoMo in a 10-user shared-agent setting with seven face benchmarks and ten palmprint protocols.
2 Related Work
Prior memory systems improve long-term interaction but generally assume a single user or externally supplied identity, leaving shared-session requester ambiguity unresolved. Bio-Memory addresses this gap by inserting biometric identity conditioning into A-Mem’s retrieval pipeline.
- Memory-augmented LLM agents: Long-term memory research spans dialogue continuity, structured storage, memory evolution, and modular retrieval-generation infrastructure.
- A-Mem: A-Mem stores graph-connected atomic notes with metadata and continuously refines historical representations through new observations.
- A-Mem: A-Mem’s semantic-only retrieval cannot distinguish semantically similar private records belonging to different users in shared memory pools.
- Identity and biometrics: Shared devices expose a gap between logical credentials and physical identity, while prior multi-user work rarely uses physical identity to construct retrieval candidates.
- Identity and biometrics: Bio-Memory attaches biometric embeddings to notes and uses matching to determine which memories are eligible for semantic retrieval.
3 Methodology
Bio-Memory extends A-Mem with biometric identity conditioning, filtering memories by biometric match before chronological and semantic retrieval. The resulting context is passed to the LLM, with empty matched pools producing answers without personal memory support.
- 3.1 Overview: Bio-Memory augments each A-Mem atomic note with a biometric embedding while retaining its original memory mechanisms.The unified note includes semantic attributes, linked memories, and stored biometric identity evidence.
- 3.3 Memory Retrieval: Owner and Non-owner conditions differ through biometric matching, with matching memories included and non-matching memories excluded from the candidate pool.The protocol evaluates whether the probe embedding matches stored embeddings attached to target records.
- 3.3 Memory Retrieval: At query time, biometric matching constructs a requester-specific candidate pool before semantic similarity ranking.The pool is formed dynamically from records whose stored biometric embeddings satisfy the benchmark-specific verification threshold.
- 3.3 Memory Retrieval: The retrieval pipeline combines biometric filtering, chronological ordering, and top-K semantic selection before passing memories to the LLM.The matched pool is sorted chronologically, semantic retrieval is performed within it, and the selected records are serialized into temporary context.
- 3.3 Memory Retrieval: The evaluation uses LoCoMo in a 10-user shared-agent setting with face and palmprint recognition datasets.The supplied methodology identifies face and palmprint evaluation datasets through Tables 1 and 2.
- 3.4 Response Generation: When no biometric matches exist, Bio-Memory withholds personal memories and answers using the question without personal memory support.This fallback is especially relevant in the Non-owner setting, where the model may still respond from parametric knowledge.
4 Experimental Setup
The evaluation simulates a 10-user shared-agent memory store by linking LoCoMo conversations to biometric templates and comparing owner with non-owner retrieval. It uses Multi-Hop, Temporal, and Single-Hop questions, token-level F1 and BLEU-1, and matched evaluation conditions.
- Dataset and setting: LoCoMo conversations are paired with stored biometric embeddings, with each conversation’s memories sharing one biometric template.Independent same-identity and different-identity pairings are constructed for each biometric benchmark.
- Dataset and setting: A 10-user shared-agent memory store merges ten LoCoMo conversations into one biometric-linked memory space.Each conversation is assigned one biometric identity, and its memories share the corresponding gallery template.
- Evaluation protocol: Evaluation reports Multi-Hop, Temporal, and Single-Hop questions using token-level F1 and BLEU-1, with Average as their macro-average.The same fixed 10% test subset is reused across biometric datasets and Owner versus Non-owner conditions.
- Implementation and controls: Owner and non-owner trials share the same memory store, retrieval configuration, and evaluation subset, differing only in the query probe biometric embedding.Semantic indexing uses an all-MiniLM-L6-v2 sentence-transformer retriever.
5 Results
Across face and palmprint benchmarks, biometric conditioning consistently preserves personal evidence for owner queries while suppressing it for non-owner queries. The separation appears across question types and metrics, with the strongest reported gaps on CALFW and MS_Blue, and reflects filtering at candidate-pool construction rather than post hoc response adjustment.
- Overall results: Owner queries consistently outperform non-owner queries across all datasets, question types, and F1 and BLEU-1 metrics.Palmprint average gaps range from 20.13 to 25.75 in F1 and 14.97 to 19.53 in BLEU-1; face gaps range from 23.58 to 27.29 and 18.34 to 21.15.
- Overall results: 27.29 / 21.15 is the largest face average gap on CALFW, while 25.75 / 19.22 is the largest palmprint average gap on MS_Blue.Values are reported as F1 / BLEU-1 percentage-point gaps.
- Authorized utility: Owner averages remain strong while non-owner averages stay near the floor, indicating selective suppression of identity-mismatched evidence.Across palmprint protocols, owner averages range from 26.27 / 21.25 to 31.32 / 24.00; face owner averages range from 28.59 / 22.57 to 31.41 / 23.07.
- Question-type behavior: Temporal questions show the sharpest separation, with owner Temporal F1 far exceeding non-owner Temporal F1 in both palmprint and face results.Single-Hop and Multi-Hop owner performance remains stronger because the correct partition preserves key facts and related evidence.
- Cross-modal consistency: The owner advantage persists across face and palmprint modalities, although verification quality affects the sharpness of the memory boundary.The results support identity-aware retrieval as a distinction between matched and mismatched users across modalities.
- Main takeaway: Bio-Memory acts as retrieval-time access control by filtering the candidate pool before semantic ranking and answer generation.Qualitative examples show owners retrieving the needed notes while non-owners lose evidence for Multi-Hop, Temporal, and Single-Hop questions.
6 Conclusion
Bio-Memory frames personalized memory retrieval as both a relevance and user-alignment problem. Its biometric control signal supports user-aligned retrieval contexts in shared-agent settings while preserving structured memory.
- Biometric matching provides a control signal for personalized memory retrieval in shared-agent settings.
- Bio-Memory treats personalized memory as a user-alignment problem in addition to a relevance problem.
- Owner and non-owner queries follow different retrieval paths because their biometric embeddings align differently with the personal memory store.
- Bio-Memory preserves the strengths of structured memory systems such as A-Mem while introducing user-aligned retrieval.