CUE-Mem: Benchmarking Long-Term User Memory via Implicit Cues in Multimodal Conversations

Motivation

• Multimodal interactions can carry incidental context beyond their main exchange content, including recurring objects in images or ambient sounds in voice messages.Long-term assistants can accumulate preferences, entities, habits, and personal context across sustained interactions.Source: §1 · p. 1
• Implicit multimodal cues are recurring visual or acoustic traces that remain peripheral or in the background rather than being directly stated or foregrounded.A single trace may be too weak for reliable memory, while recurring traces can inform interests, habits, household, or environment.Source: §1 · p. 1

Research gap

• Existing evaluations emphasize directly stated or foregrounded evidence, with ambient audio and cues that become informative through recurrence receiving limited attention.The open question is whether long-term systems can form memories from recurring background evidence across multimodal histories.Source: §1 · pp. 1–2

Benchmark contribution

• CUE-Mem evaluates long-term memory from recurring implicit multimodal cues using paired explicit and implicit evidence conditions.The conditions allow aggregate comparison, but individual examples are not constructed as one-to-one counterparts.Source: §1 · p. 2

Task design

• Its four tasks test entity and attribute recall, aggregation of recurring patterns, personalized assistance, and refusal when evidence is insufficient.Each task includes explicit and implicit variants for comparing direct evidence with weak multimodal cues.Source: §1 · p. 2

Evaluation findings

• Across evaluated systems, foregrounded evidence is easier to use than recurring implicit cues, while cue-aware textualization and native multimodal access do not close the gap.Textualization narrows the gap unevenly at growing input cost; native evidence use depends on the backbone, and multimodal indexing adds retrieval noise.Source: §1 · p. 2

2.1 Implicit User Memory Benchmarks

• CUE-Mem distinguishes its target from benchmarks on utterance-inferred preferences or personas, behavioral traces, procedural memory, and unstated roles or motivations.Its focus is recurring visual and acoustic cues peripheral to the current exchange.Source: §2.1 · p. 2

2.2 Multimodal Memory Benchmarks

• Multimodal memory benchmarks examine multi-session image-text memory, personalization, heterogeneous evidence, or cross-modal chains; CUE-Mem isolates recurring background visual and acoustic cues.Table 1 compares representative conversational and multimodal benchmarks across modalities and distinguishes implicit multimodal cues from semantic inference.Source: §2.2 · p. 2, Table 1 · p. 2

3.1 Overview

• CUE-Mem construction combines Script Synthesis, Multimodal Dialog Synthesis, and Data Review to create histories with direct, foregrounded, and recurring background evidence.The pipeline begins from structured user profiles and produces controlled text-image-audio conversations.Source: §3.1 · p. 2, Figure 2 · p. 3

3.2 Stage 1: Script Synthesis

• Profiles provide latent memory ground truth; attributes receive modality and evidence assignments, then chronological events distribute cues across sessions.Ambient-audio cues are selected from a reviewed pool of 96 categories, and later events are conditioned on earlier ones to support continuity and recurrence.Source: §3.2 · p. 3

3.3 Stage 2: Multimodal Dialog Synthesis

• Image and audio synthesis place explicit evidence in foregrounded or spoken content and implicit cues in cluttered backgrounds or unstated ambient sound.Dialog generation integrates the event timeline, profile, modality assignments, and media into multi-session exchanges while preserving each evidence setting.Source: §3.3 · p. 3

3.4 Stage 3: Data Review

• Automatic checks filter synthesis defects, cross-session inconsistency, and overly salient or text-leaked implicit cues; human review assesses naturalness, coherence, and evidence validity.Automatic filtering removes 42.80% of initial samples, and human review removes an additional 22.10% of the remainder.Source: §3.4 · p. 4

4.1 Task Design

• CUE-Mem tests entity recall, aggregation of recurring evidence into stable memories, personalized recommendations, and refusal when evidence is insufficient.Answer Refusal’s correct option indicates that the history lacks sufficient evidence to answer.Source: §4.1 · p. 4

4.2 Construction Pipeline

• Question construction selects target memories, generates answers and distractors, annotates and verifies clues, reviews quality, then filters questions answered correctly without history.An instance is discarded only if both evaluated backbones answer correctly in all three runs.Source: §4.2 · p. 4

4.3 Data Statistics

• 2,674 QA pairs comprise 1,401 explicit and 1,273 implicit questions; Entity Recall contributes 1,128, Answer Refusal 819, Personalized Recommendation 364, and Long Pattern 363.Excluding Answer Refusal, explicit and implicit questions average 3.14 and 3.34 supporting clues, respectively; Figure 4 summarizes task counts and clue-modality distributions.Source: §4.3 · p. 5, Figure 4 · p. 5

5.1 Experimental Setup

• The protocol reports accuracy by evidence setting and defines EI-Gap as explicit minus implicit accuracy; Memory Avg. covers the first three tasks, with Answer Refusal reported separately.Textualized systems caption images and audio before text-based processing; the study also evaluates native multimodal memory and two backbones for each paradigm.Source: §5.1 · pp. 5–6

5.2 RQ1: How Well Do Textualized Memory Systems Handle Implicit Multimodal Cues?

• Best implicit Memory Avg. reaches 38.5 with Qwen3.6-35B-A3B and 41.9 with GPT-5.4-mini, versus Oracle Evidence scores of 80.8 and 78.4, respectively.Human readers of full textualized histories score 83.2 and 82.8 on explicit and implicit samples, suggesting the gap is mainly preserving and retrieving dispersed evidence rather than answering once it is available.Source: §5.2 · p. 6
• Answer Refusal accuracy rises from 63.2 to 97.2 for Qwen3.6-35B-A3B and from 38.1 to 80.2 for GPT-5.4-mini as evidence shifts from explicit to implicit, reflecting missed cues rather than better calibration.Both settings use misleading adversarial choices; systems often retrieve and misapply explicit memories but miss implicit cues and default to refusal.Source: §5.2 · p. 6

5.3 RQ2: How Does Cue-Aware Textualization Affect Implicit Memory?

• Detailed image captions raise implicit Memory Avg. from 26.9 to 46.4 and reduce EI-Gap from 35.9 to 27.1 points; audio splitting raises it from 40.1 to 51.0 and reduces the gap from 29.4 to 19.4 points.The gains alleviate but do not resolve the textualization bottleneck.Source: Figure 5 · p. 6, §5.3 · p. 6
• Detailed captions improve implicit Entity Recall and Long Pattern by 36.9 and 13.0 points, but Personalized Recommendation by 8.8; they raise retrieved input from 1.9k to 7.0k tokens per QA.Implicit Answer Refusal drops by 16.9 points for images; over the Medium-to-Detailed change, implicit accuracy rises from 53.3 to 62.2 while explicit accuracy rises from 70.9 to 72.4.Source: §5.3 · p. 6

5.4 RQ3: Can Native Multimodal Access Alleviate the Textualization Bottleneck?

• MM-Use improves Qwen’s implicit Memory Avg. by 2.2 points with Text-Index and 7.4 with MM-Index, but reduces MiniCPM’s by 8.4 and 5.1 points, respectively.Oracle multimodal evidence leaves Qwen and MiniCPM 15.7 and 34.0 points below their text-oracle scores; MM-Index also lowers implicit Memory Avg. for both backbones as reduced precision offsets recall gains.Source: Table 3 · p. 7, §5.4 · p. 7, Figure 6 · p. 7

6 Conclusion

• Implicit-cue performance remains below explicit evidence: captioning narrows the gap at growing token cost, while native-access benefits depend on the backbone and indexing adds noise.The conclusion says no setting closes the gap and that native multimodal oracle evidence remains difficult for evaluated models to interpret and integrate.Source: §6 · p. 7

A.1 Evidence Taxonomy and Assignment Prompts

• Evidence type is assigned per target memory: explicit evidence is stated or foregrounded, while implicit evidence is unstated peripheral visual or ambient non-speech evidence without a dialogue source.Implicit evidence is contextual and separable from the current conversational topic, not ordinary inference from an explicit sentence.Source: §A.1 · p. 10

A.2 Multimodal Synthesis Details

• Synthesis repeats profile-grounded targets across sessions, placing implicit visual cues peripherally among distractors and recurring ambient events behind unrelated speech.Recurring entities use reference-guided assets for visual consistency; ambient tracks are mixed into selected turns while speech remains dominant and the background cue stays audible.Source: §A.2 · pp. 11–12

A.3 Automatic and Human Quality Control

• Quality control combines automatic checks for leakage, target placement, and media validity with independent annotator review and expert adjudication; average Fleiss’ κ was 0.73.Samples are directly accepted only when all three annotators approve dialogue naturalness, multimodal coherence, and evidence validity.Source: §A.3 · p. 12

B.1 QA Generation and Annotation

• Question generation creates a correct option with matched distractors, then identifies and verifies supporting history clues; image-option questions add visual-description and image-synthesis stages.The clue identifiers are intersected with identifiers in the matched events, and image captions are evaluation-time textualizations rather than substitutes for option images in native multimodal evaluation.Source: §B.1 · p. 12
• Refusal lures use eight single-slot transformations, and code verifies the transformation label, four options, exactly one refusal option, and a valid answer letter.Implicit overclaim turns peripheral-image or ambient-sound inference into an unsupported claim that the user explicitly stated, explained, or invariably performed something.Source: §B.1 · pp. 12–13
• Question-only filtering removes a non-refusal item only when all six predictions are correct; 525 non-refusal questions were removed, while refusal questions were exempt.The resulting task counts are 1,128 Entity Recall, 363 Long Pattern, 364 Personalized Recommendation, and 819 Answer Refusal questions.Source: §B.1 · p. 13

C.1 Models and Inference

• RQ1–RQ3 result trees contain one artifact per configuration, so the repository cannot establish three-run means without missing provenance or new repeated experiments.Common inference concatenates dialogue text with selected image and voice captions and asks the model to choose A–D at temperature 0.Source: §C.1 · pp. 13, 18

C.2 Memory-System Configurations

• Seven systems share ordered dialogue-round observations and a cosine-ranked text retriever, with top–10 retrieval and 4,096-word truncation unless otherwise noted.The checked-in configuration has no hyperparameter sweep logs; cache adaptations affect storage and provenance while the common answer backbone makes the final choice.Source: §C.2 · p. 18

C.3 Textualization Configurations

• The exact caption prompt templates, generation logs, output budgets, and tried alternatives are absent, and the Qwen3-ASR decoding command and parameters are also unavailable.The realized Brief, Medium, and Detailed histories and option captions are present.Source: §C.3 · pp. 18–19
• Brief captions retain dominant subjects, Medium captions add relations and scene context, and Detailed captions enumerate foreground, background, and peripheral content.In one design-workspace example, only the Detailed caption retains the peripheral Wallpaper* magazine that supports the implicit preference.Source: §C.3 · pp. 18–20
• Audio variants retain different evidence: ASR transcribes speech, Hint describes speech and background sound together, and Split pairs speech transcription with a separately captioned background track.The background-audio prompt requests a sentence limited to audible content.Source: §C.3 · p. 19

C.4 Native Multimodal RAG Implementation

• Text-Index embeds textualized dialogue rounds, whereas MM-Index separately embeds dialogue content and each original image or audio attachment in a unified matrix.Text-Use renders retrieved turns and image-option captions as text; MM-Use restores associated original media, falling back to captions when a media path is missing.Source: §C.4 · p. 20
• Both index types use the same composed-query mechanism, cosine similarity, reciprocal-rank fusion, and a final retrieval cutoff of five.The candidate pool is max(20, 4k), and reciprocal-rank fusion combines turn-level ranks using 1/(60 + r).Source: §C.4 · p. 20

C.5 Code and Reproducibility Package

• The reproduction package covers construction, evaluation, retrieval, token analysis, statistical testing, and table and figure generation, with machine-specific paths, credentials, and model-service access required.It includes checkpointing and resume support for computationally expensive generation, ingestion, and RQ3 stages.Source: §C.5 · p. 21

D.1 Statistical Significance Analysis

• The audit uses 10,000 bootstrap replicates that resample questions within fixed profile-by-task-by-evidence strata, with profiles equally weighted within task.RQ1 independently resamples explicit and implicit sets; RQ2 and RQ3 comparisons reuse the same question keys, and the 13 prespecified contrasts receive Holm adjustment.Source: §D.1 · pp. 21–22
• Explicit Memory Avg. exceeds implicit Memory Avg. by 34.8 points (pHolm = 0.0026) in the audited subset.The audit treats five profiles selected with seed 42 as fixed rather than as a random sample from a population.Source: §D.1 · pp. 21–22
• Detailed image captions increase implicit accuracy by 20.2 points and narrow EI-Gap by 12.3 points relative to Brief captions (pHolm = 0.0026 and 0.0060).Both effects remain significant after Holm correction.Source: §D.1 · p. 22
• Audio Split increases implicit accuracy by 11.0 points and narrows EI-Gap by 12.0 points, with both effects significant after Holm correction (pHolm = 0.0306).The audit compares the same sampled question keys across RQ2 conditions.Source: §D.1 · p. 22
• Under MM-Index, MM-Use increases Qwen3-Omni implicit accuracy by 8.9 points relative to Text-Use (pHolm = 0.0026).The reported result is specific to this backbone and index condition.Source: §D.1 · p. 22

D.2 Additional RQ2 Results

• Implicit Entity Recall rises from 10.3 to 47.2 across Brief to Detailed image captions, while implicit Answer Refusal falls from 82.0 to 67.6.Long Pattern improves monotonically on average, whereas Personalized Recommendation changes more modestly.Source: §D.2 · p. 22, Table 13 · p. 23
• For implicit audio Long Pattern, splitting speech and background raises average accuracy from 33.0 (ASR) to 50.6; explicit Long Pattern changes negligibly.Implicit Personalized Recommendation rises from 47.1 to 51.4, while implicit Answer Refusal falls slightly as adversarial strength increases from 95.9 to 94.2.Source: §D.2 · p. 22, Table 14 · p. 24
• Detailed captions increase total memory input by 104.4% and retrieved input per question by 377.1% relative to Brief captions.The image-caption component grows 27.1×, from 1,326 to 35,966 tokens per profile, and retrieval costs vary substantially across examples.Source: §D.2 · p. 22, Table 15 · p. 24

D.3 Additional RQ3 Results

• The multimodal index improves implicit recall at every reported cutoff, reaching 29.7 versus 23.1 at k = 25, but implicit Precision@10 is 8.6 versus 16.2 for the text index.For explicit evidence, multimodal indexing has higher Recall@1–5 but lower Recall@10–25.Source: §D.3 · p. 22, Table 16 · p. 24

D.4 Effect of Cue Recurrence

• All 152 implicit targets occur in at least three event groups, with a 3.62-occurrence average; their median first-to-last span is 354 days.The construction also includes 202 explicit targets, each occurring in at least two groups, over 671 planned event groups and 648 realized events.Source: §D.4 · pp. 22, 25
• Recurrence counts and spacing were not independently randomized, and no matched one-versus-several-occurrence evaluation was run, so these construction statistics do not establish a causal accuracy gain.The authors frame recurrence as grounding answers in longitudinal evidence and spacing sessions so questions are not reducible to adjacent-session clusters.Source: §D.4 · p. 25

E.1 Failure Taxonomy

• Textualization misses lose or distort target cues; cross-session association failures fail to consolidate recurrences; retrieval misses omit evidence, while retrieval noise introduces plausible distractors.These diagnostic categories can overlap: an omitted visual detail can produce a retrieval miss and then invite unsupported inference.Source: §E.1 · p. 25
• Evidence-use failures occur when a clue is retrieved but overlooked or mapped incorrectly; refusal-calibration errors include both unsupported claims and refusals despite adequate evidence.The selected cases were not sampled to estimate category frequencies.Source: §E.1 · p. 25
• Figure 16 shows stored top-five turn IDs from Qwen3-Omni RQ3 runs; Figure 17’s multimodal retrieval restores peripheral bookshelf evidence distinguishing two plausible venues.The figure caption identifies the retrieval contrast but does not report a frequency estimate.Source: Figure 16 · p. 25, Figure 17 · p. 26
• Figure 18 illustrates an image-textualization failure in which retaining one edge detail changes an underdetermined entity match into a supported answer.Source: Figure 18 · p. 27

• Synthetic identities and media reduce direct exposure of real conversations, faces, and voices, but generated faces or profiles may coincidentally resemble real people.Source: F Ethical Considerations · p. 27
• Implicit cues can reveal household composition, routines, interests, and living conditions without being stated, so deployed systems need user control and conservative thresholds for persistent memories.Source: F Ethical Considerations · p. 27
• Generated profiles may reproduce stereotypes, and ambiguous objects or sounds can support competing explanations; benchmark checks do not establish that real-world inferences are fair.Source: F Ethical Considerations · p. 27
• Consistent synthetic portraits and voices could be detached to fabricate statements, motivating provenance for released assets; CUE-Mem does not identify real people from images or voices.Source: F Ethical Considerations · pp. 27–28

• Controlled synthetic histories may contain generation artifacts or overly coherent patterns, and stable one-year preferences leave preference drift, contradictions, corrections, and longer-term forgetting unevaluated.Source: G Limitations · p. 28
• The benchmark covers 20 synthetic users, Chinese conversations, six profile domains, and text, images, and environmental audio; results may not transfer beyond these settings.Source: G Limitations · p. 28
• Visual and acoustic implicit cues use different construction controls and are not difficulty-matched, so explicit–implicit results support aggregate comparisons rather than paired counterfactual claims.Source: G Limitations · p. 28
• Four-way single-answer questions do not test open-ended memory behavior, and results depend on evaluated models and retrieval choices; the experiments also cover only one text and one multimodal index.Source: G Limitations · pp. 28–29
• Significance intervals come from a fixed five-profile sample and quantify uncertainty for that audit, not for all users, models, or deployment environments.Source: G Limitations · p. 29