Parametric Multimodal User Memory: Storing What Captions Cannot Carry

Bojie Li; Noah Shi

Source-linked AI summary

Parametric Multimodal User Memory: Storing What Captions Cannot Carry

Bojie Li, Noah Shi

arXiv:2608.28609v1Jul 8, 2026cs.CLcs.AIcs.CV

TL;DR

Existing text-based memories preserve captionable facts but lose perceptual identity, such as voice characteristics and cross-condition face appearance. The paper introduces grounded parametric multimodal memory, combining VLM localization, specialist encoding, and inline in-model storage. Its central conclusion is that perceptual identity belongs in a parametric bank while factual knowledge remains in a text store, with architectural advantages over external retrieval.

Problem
Text transcripts and captions preserve captionable facts but cannot retain perceptual identity, and caption-based re-identification can recover as little as 0.11 of a dedicated encoder’s recall.
Method
The method grounds a user-intended referent with a VLM, encodes it with a dedicated specialist, and stores the identity key as an inline attention-readable token in a frozen language model.
Results
The paper reports grounded recognition near the correct-region oracle across vision, audio, and video, while the training-free core reproduces specialist-encoder recall with O(1) registration cost.
Takeaways & Limitations
Perceptual identity is capacity-limited and belongs in a parametric memory bank, whereas factual knowledge is binding-limited and belongs in a text store.
Takeaways & Limitations
Compared with retrieve-and-reprompt, parametric memory does not claim faster or more accurate recognition; its advantages are architectural.

Abstract

from arXiv · show

A personalized agent needs a user memory: a persistent model of who its user is. Today it is almost always text -- transcripts and captions retrieved by similarity. This serves the captionable half of a person ("my cat is named Bibi"), but discards the perceptual half no caption can hold: how a voice sounds, how a face reads across age and lighting, how tired someone sounds. We measure this loss across five modalities: a strong caption-based re-identifier recovers as little as 0.11 of a dedicated encoder's recall, collapsing toward chance on non-nameable signals. We instead ground perceptual memory in the model, decomposing recall into two subproblems: a vision-language model grounds the referent in context (what and where), and a dedicated encoder extracts an identity key (who), stored as one inline token read by attention at generation with no external round-trip. Neither suffices alone -- the VLM identifies cross-age faces at only 0.54 recall where a face encoder reaches 0.81, and an ungrounded encoder recognizes a two-person-scene referent at 0.05 -- yet together they reach correct-region oracle (0.96), generalizing to multi-speaker audio and video. The recognition core is training-free: it reproduces the encoder's recall on any frozen model at O(1) registration cost. On PerceptMem (12 domains, 1,080 tasks) perceptual identity is capacity-limited while exact facts are binding-limited: identity belongs in a parametric bank, facts in a text store. The two memories compose cleanly: an agent with both can remember not only what its user said, but also what they are like.

1 Introduction

Current agent memories preserve captionable facts but discard perceptual identity. The paper proposes grounded, parametric multimodal memory that combines referent grounding, specialist identity encoding, and in-model storage.

Motivation: Text memories preserve facts such as “my cat is named Bibi” but cannot capture how voices sound or faces persist across age and lighting.Caption-based re-identification recovers as little as 0.11 of a purpose-built encoder’s recall.
Approach: The proposed pipeline grounds the intended referent, converts it into an identity key, and stores that key as one inline token in a frozen language model.Recall occurs inside generation through attention, without captioning or an external retrieval round-trip.
Approach: Neither component suffices alone: the VLM is weak at identity encoding, while a bare specialist encoder cannot select the referent in cluttered scenes.Grounding lets the components cover each other’s blind spots across visual and auditory conditions.
Results: The grounded combination recovers near-oracle recognition in cluttered scenes and generalizes across vision, audio, and video.The contribution is a division of labor between localization and identity extraction rather than caption-based memory.
Results: The recognition core is training-free, reproduces specialist-encoder performance across ten frozen model families, and supports O(1) insertion and recall.The design is presented as cheaper than retaining raw perceptions in context.
Benchmark: PerceptMem spans 12 perceptual domains and 1,080 tasks and motivates routing perceptual identity to a parametric bank while keeping factual knowledge in a text store.The benchmark includes text-only and random-chance baselines.

2 Decomposing perceptual recall: grounding and identification

Text-based routes discard perceptual signals, while raw embedding retrieval preserves identity but assumes the referent is already isolated. The paper therefore separates grounding from identification and places the resulting identity representation inside the model.

Problem: Recognizing a perception requires both localizing the intended referent in context and encoding its identity.The paper frames these as distinct subproblems that no single component performs well alone.
Text-based routes: Caption-and-search descriptions are too generic to distinguish speakers reliably, despite being factually accurate.The route discards the raw signal that differentiates voices across conditions.
Embedding retrieval: Embedding retrieval preserves raw perceptual information and reaches 0.95 recall at ten identities on a 2180-identity face pool, but performance degrades as the pool grows.Its recognition assumes a clean voiceprint or tight face crop.
Embedding retrieval: A bare encoder embedding an entire two-person scene achieves only 0.05 accuracy because it cannot isolate the intended referent.Thus, the strongest text-free route solves identification but not localization.
Proposed approach: The proposed system adds VLM grounding before encoding and stores the resulting key as a marker token read by attention inside the frozen model.Recognition remains tied to encoder cosine similarity while the memory additionally supplies grounding and in-model access.

3 Grounding perceptions in the model’s own representation

The method separates perceptual recall into grounding a referent, extracting an identity key, and reading it through an in-model attention memory. ATTMEM stores encoder keys with marker-token values on a frozen language model, enabling training-free, constant-cost registration and recall.

Grounding and identification: Perceptual recall is decomposed into grounding the referent, identifying it with a dedicated encoder, and storing the identity as a token inside a frozen language model.The VLM resolves what and where; the specialist encoder extracts who; ATTMEM supplies the in-model storage and read mechanism.
Grounding and identification: A VLM localizes context-dependent visual or auditory referents, optionally refined by a modality detector, before specialist encoding.Grounding returns a bounding box for images or a temporal span for audio; detectors such as RetinaFace can further align regions.
Grounding and identification: Dedicated perceptual encoders convert grounded regions into L2-normalized identity keys, using modality-specific models such as ArcFace, ECAPA-TDNN, CLIP, AST, and wav2vec2.The encoder choice follows the perceptual signal being remembered, including faces, speakers, styles, acoustic scenes, and tone.
In-model storage and reading: ATTMEM stores encoder keys alongside the language model’s marker-token embeddings and reads them by attention at the output head.The retrieved value produces a residual that biases the next-token prediction toward the matching marker.
Faithful read: The read uses no trainable parameters, while high attention sharpness, an output-head hook, and sufficient residual gain enable exact encoder-nearest-neighbor recovery.β and g are constants; the design choices make attention approach a hard bank lookup and deliver the residual directly to logits.
Cost and scaling: Recall latency remains ∼15 ms as bank size grows, insertion is O(1), and context-based storage grows linearly until exhausting a 32k context window at ten thousand identities.The constant-time lookup is dominated by the frozen model’s forward pass, whereas context feeding scales with the number of identities.

4 PerceptMem: a cross-condition perceptual-memory benchmark

PerceptMem evaluates perceptual memory across five caption-resistant sub-modalities using cross-session, cross-condition identity retrieval. Its design expands these concepts into a 12-domain, 1,080-task benchmark with fixed encoders and standardized recall@1 comparisons.

Benchmark scope: PerceptMem spans five sub-modalities across vision and audio, covering identity, style, physical appearance, and affective cues degraded or destroyed by captions.Each sub-modality is selected because captions cannot carry its discriminating signal.
Benchmark construction: The benchmark pairs standard datasets with fixed off-the-shelf encoders, including LFW and AgeDB with ArcFace, WikiArt with CLIP, and speech, scene, and emotion datasets with specialized encoders.The encoder pairings include LibriSpeech–ECAPA-TDNN, ESC-50–AST, and RAVDESS–wav2vec2 emotion encoding.
Evaluation protocol: PerceptMem measures memory rather than raw perception through register-and-recall tasks using cross-session samples, non-overlapping evaluation identities, and cross-condition queries.For tone of voice, an identity is a speaker–emotional-state pair, requiring paralinguistic matching across utterances.

5 Empirical evaluation

The evaluation shows that text loses perceptual identity, while grounding plus specialist encoding recovers oracle-level recognition across vision, audio, and video. The in-model read preserves encoder recall across domains, models, and recognition modes.

Text versus perceptual memory: Captions approach encoder recall for nameable concepts but fall toward chance for perceptual identity, reaching only 11% of speaker-encoder recall.The comparison uses identical draws across five modalities and traces the boundary between text memory and the parametric bank.
Grounding: 0.96 face recall and 0.36 painting recall match their correct-region oracles after grounding, whereas text-only recall reaches only 0.12 and 0.17.Because both grounded and store-only systems use the same nearest-neighbor matcher, the gain comes from isolating the referent before encoding.
Grounding: Grounded recall remains near oracle as visual clutter rises from two to six referents, with grounding accuracy staying at 1.00 through K=5 and reaching 0.99 at K=6.Whole-scene recall stays near the floor, while text-only recall falls from 0.12 at K=2 to 0.05 at K=3.
Audio and video: 0.92 grounded speaker recall at K=3 and 0.73 recall on VoxConverse show that the decomposition generalizes beyond vision, though temporal grounding degrades with complexity.On VoxCeleb, grounded recall is 1.00 at K=2 and 0.67 at K=4; on VoxConverse it nearly doubles whole-window recall of 0.38.
Recognition core: |∆| ≤0.001 from N=5 to N=1,000 confirms that the in-model read matches encoder retrieval across bank sizes.The match holds on random and adversarial banks, but under-sharpened reads lag cosine similarity at N ≥300.
Recognition core: The memory reproduces encoder recall across ten model families and supports verification and stranger rejection with AUROC up to 0.99 and rejection accuracy up to 0.99.Performance remains high for voice and face and lower for weaker encoders such as painting style.

5.5 Composition with in-context text memory

The system composes perceptual recognition with text-based fact lookup by recalling a name in-model and using it to retrieve an associated fact. This preserves exact fact storage while scaling identity memory with the encoder.

Composition: Attention over the memory bank forms a near-pure diagonal averaging 0.98, while encoder cosine similarities retain a softer diagonal averaging 0.46.The memory reads the encoder’s nearest neighbor as a token without sharpening the underlying similarity structure.
Composition: 4–10× higher end-to-end accuracy than the face-withheld chance baseline results from chaining recognition with in-context name-to-fact lookup.Accuracy closely tracks the product of recognition accuracy and in-context lookup reliability in a single forward pass.
Baseline comparison: Identical accuracy between in-model memory and retrieve-and-reprompt shows that both preserve the same encoder-based recognition quality.The external pipeline is cheaper for pure recognition, whereas the in-model read eliminates the external index and integrates recognition into generation.
Architectural benefit: The in-model read automatically consults memory during generation, enabling multiple perceptual mentions to be interleaved with ongoing reasoning in one stream.This contrasts with the pipeline’s two-pass orchestration, which must decide in advance to query an external index and re-prompt.
Text-memory compatibility: The populated perceptual bank leaves top-1 next-token predictions unchanged relative to the untouched model on every probe.The attention hook adds a residual only at perceptual positions, preserving ordinary text-only behavior.
Scaling: At 80 users, routed fact recall reaches 0.86, while binding all facts in context falls to 0.04 at 50 users.The contrast supports routing perceptual identity to the parametric bank and exact facts to the text store.

6 The encoder determines recall

Recognition quality is governed by the dedicated encoder rather than the memory read or host language model. Improving perceptual recall therefore requires improving the encoder.

Encoder ceiling: 0.95 versus 0.64 recall at ten identities and 0.75 versus 0.34 at one hundred show ArcFace outperforming the VLM’s native vision tokens on AgeDB.The comparison uses the same identities and demonstrates that the memory mechanism does not overcome weaker perceptual representations.
Implication: A stronger encoder—not a different language model or more sophisticated read mechanism—is the effective route to better perceptual recall.The in-model read faithfully reproduces the encoder’s recall, so encoder quality determines the system’s performance ceiling.

7 Related work

Prior personalized-memory systems primarily store captions or transcripts, while retrieval systems preserve text or perceptual embeddings in external stores. This paper adds native-modality perceptual memory through grounded, training-free in-model storage.

Text-first agent memory: Text-first conversational-memory systems store transcripts and captions in a text index retrieved by semantic similarity.They cover the captionable portion of user memory but do not retain perceptions in their native modalities.
Personalized VLMs: Personalized vision-language models typically learn an artifact per user-specific concept, whereas this approach appends a training-free O(1) memory row.The distinction is both mechanistic and cost-based.
Retrieval inside the model: ATTMEM repurposes attention-over-bank retrieval by storing persistent perceptual encoder embeddings rather than text hidden states or corpus chunks.Its goal is persistent user perception memory inside the model rather than language-model retrieval over a text corpus.
Perception-to-token bridges: Unlike learned perception-to-token bridges, the method injects a single marker token using the model’s own embedding, enabling training-free operation.Speaker-recognition and diarization provide the specialist recognition components used for identity keys.

8 Discussion

The discussion argues that perceptual identity and exact facts obey different memory capacity laws, motivating a parametric perceptual bank alongside a text store. The proposed system is training-free, constant-time, additive to existing text memory, and bounded by encoder, grounding, scale, and deployment constraints.

Core properties: The recognition core is training-free and reproduces specialist-encoder performance across ten frozen model families with constant-time insertion and recall.The memory is read by attention at the output head without gradient updates or an external round-trip.
System composition: The perceptual bank composes with text memory, preserving text-only behavior while activating for perceptual mentions.The combined system is intended to retain both what users said and what they are like.
Capacity laws: Perceptual identity is capacity-limited, while exact factual content is binding-limited, so identities belong in a parametric bank and facts in a text store.Recall scales with available identity slots, whereas exact multi-code retrieval collapses even when more latent tokens are added.
Limitations: The memory cannot exceed its encoder: recall stays within ±0.001 of encoder performance from N=5 to N=1000, declining to 0.78 at a thousand faces.No in-model mechanism recovers information absent from the underlying encoder.
Limitations: Evaluation reaches 1,000 registered users from a 2,180-identity pool, while constant-time behavior is confirmed to 10,000 rows but recall there remains untested.The paper also depends on external modality-specific encoders, inspection-set hyperparameters, and charitable baseline reimplementations.
Limitations: Grounding robustness declines under realistic stress, with pasted-face grounding accuracy falling to 0.63–0.81 and fourth-speaker audio grounding reaching 0.69.The system inherits the VLM’s referring-expression resolution limits.

9 Conclusion

The conclusion argues that text descriptions discard recognition-relevant perceptual information and that effective memory therefore combines grounding, dedicated encoding, and in-model storage. This division of labor gives each component the role the others cannot perform.

9 Conclusion: Text descriptions discard perceptual signals, so agents should retain grounded native perceptions alongside textual memory.The proposed pipeline grounds the referent, encodes its identity, and stores it as a content-addressable row read inside a frozen language model.
9 Conclusion: The VLM grounds but cannot reliably identify, the encoder identifies but cannot ground, and the in-model bank returns identity as a native generation token.The contribution is the training-free, constant-time, portable memory container rather than a new recognition encoder.

A Detailed results

Across PerceptMem’s 12 domains and 1,080 tasks, the training-free parametric read matches encoder retrieval across architectures, modalities, and memory sizes. Performance is limited by the identity encoder’s capacity and by grounding quality, while learned re-encodings add no meaningful discrimination.

Benchmark: 1,080 tasks across 12 domains evaluate five conceptual sub-modalities under dataset–encoder pairings, plus native VLM vision tokens as a weak control.Each domain uses N ∈{10, 20, 40} with 30 draws per setting.
Training-free read: The memory ties retrieval across random, style, and adversarial banks through N=1000, including when target slots are randomised.The paired protocol uses identical registrations and queries for both methods.
Training-free read: Every one of 25 architecture–modality cells reproduces encoder recall within one point, with a worst-case gap of 0.010.Most cells differ by ≤0.003 under the paired 20-draw protocol.
Capacity: 0.95, 0.99, 0.86, 0.52, and 0.43 are encoder recall@1 values at N=10 for face, speaker, acoustic, tone, and style modalities, respectively.These values are the one-slot-per-identity ceilings for the corresponding modality encoders.
Capacity: Capacity follows min(1, k/M): normalised error is ≤0.012 for strong encoders and ≤0.13 for weaker style and tone encoders.The ceiling itself varies by encoder, reaching 0.26–0.36 at M=32 for style and tone.
Baselines: A per-concept classifier matches raw cosine but costs 4.5 s per identity at 1,000 identities, while supervised contrastive projection trails raw cosine by 15–28 points.The proposed approach uses an O(1) tensor append instead of per-identity gradient descent.
Key spaces: 0.948/0.792/0.776 at N=10/100/1000 shows ArcFace-key recall, whereas native VLM tokens tie their own much lower retrieval ceiling.The comparison motivates using the VLM for localization and an external encoder for identity.
Grounding limits: Grounding accuracy falls from 1.00 on clean composites to 0.81 in rows and 0.63 in grids, making grounding and detection the practical bottleneck in cluttered scenes.Detector hit-rate declines to 0.09–0.43 on small, rescaled faces, reducing end-to-end recall below the oracle.

B Mechanics of a single recall

A single recall retrieves a registered identity by matching a new perceptual query against encoder keys and converting the result into a marker-token logit. Registration is training-free and requires only one appended key–value row.

Registration: Registration appends one row: a 512-dimensional normalised ArcFace embedding becomes the key, while the model’s marker-token embedding becomes the value.For Qwen2.5-3B, the value has hidden width 2048; no gradient step or fine-tuning is used.
Recall: At recall, the new ArcFace embedding is matched against all keys, softmaxed into attention weights, and used to blend the stored values.The resulting vector is added before the output head.
Recall: Because each value is a marker-token vector, the weighted read boosts the corresponding identity marker’s logit and makes it the next token.The marker token therefore names the retrieved identity directly inside generation.
Read sharpness: A 0.98-vs-0.46 diagonal gap measures the sharp attention read against raw encoder cosine on a held-out ten-identity bank.Sharp attention concentrates cosine similarity onto the top match instead of producing a diffuse blend.

C A discrete-codebook alternative

The discrete codebook alternative fails because quantization simultaneously merges different identities and splits the same identity across conditions. Continuous attention avoids this categorical bottleneck by preserving the encoder’s raw signal.

Observed failure: After 100,000 continual-pretraining steps, codebook recall remains 0.057–0.070 at 300 identities across K ∈{128, 256, 512, 1024}, versus 0.73 for encoder retrieval.Replacing ArcFace R50 with AntelopeV2 R100 did not improve the codebook result.
Collision trade-off: At K=16, inter-identity collisions reach 8.7%, making identities sharing a code permanently indistinguishable.This is the small-codebook side of the bottleneck.
Collision trade-off: At K=128, inter-identity collisions fall to 2.4% but same-identity cross-condition co-location drops from 0.33 to 0.20.Larger codebooks reduce collisions while making queries less likely to route to the registered identity’s cell.
Why it fails: Even when routing reaches the correct code about half the time, other identities sharing that cell remain equally good answers.The codebook’s categorical representation cannot preserve the continuous distinctions present in the encoder.
Design implication: Continuous attention over raw embeddings avoids quantization and its information-loss tax, whereas learned codes cannot recover discarded continuous identity signal downstream.The paper frames this as the learned analogue of captioning’s categorical bottleneck.