AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue

1 Introduction

• Shared assistants can confuse residents, for example by shaping a response to one person’s medication question with another person’s gaming history.The paper characterizes this as a reliability risk that can erode utility and trust.Source: §1 · p. 1
• Existing dialogue agents often assume a single generic user or fixed persona profiles, leaving dynamic multi-user settings less addressed.Prior personalized-dialogue work cited here includes PersonaChat [20] and multi-session chat [19].Source: §1 · p. 2
• AFA combines speaker identification, personalized memory retrieval, and LLM response generation to support multiple users in shared environments.It retrieves and updates each user’s personalized memory over time.Source: §1 · p. 2
• PAT contains 58,289 persona-grounded dialogue turns spanning 133 user profiles and 12 real-world scenarios.Source: §1 · p. 2
• Identity-aware routing improves Persona Attribution Accuracy (PAA) from 35.7% to 61.3% in an interleaved multi-user evaluation.The protocol controls for topic bias, addressing a gap in evaluations based on surface-level overlap.Source: §1 · p. 2

2.1 Multi-User Personalization

• AFA differs from PAChat by targeting persistent per-user memory across sessions and evaluating interleaved user switches with Persona Attribution Accuracy.PAChat addresses identity-aware response generation within individual conversations; AFA’s PAA controls for topic bias.Source: §2.1 · p. 2

2.2 Memory-Augmented Conversational Models

• MemoryBank [22] and MemoChat [12] use external memory for long-term context, while AFA retrieves relevant context from a separate memory store for each user.Source: §2.2 · p. 2

2.3 Persona and Profile-Grounded Dialogue

• PersonaChat [20] and related datasets explore persona-based dialogue, while most persona-chat research assumes static profiles and single-user settings.AFA extends this paradigm to multi-user, voice-driven dialogue with simultaneous personalization across speakers.Source: §2.3 · p. 2

2.4 Speaker Identification

• Speaker identification determines who is speaking from a group of known voices and serves as AFA’s entry point for retrieving the corresponding memory and persona.ECAPA-TDNN [3] is identified as a speaker-embedding architecture, and SpeechBrain [16] provides pretrained models for identification and verification.Source: §2.4 · p. 3

3 PAT Dataset

• PAT contains persona-tailored query-response pairs intended to support training and evaluating personalized dialogue agents in multi-user settings.Source: §3 · p. 3

3.1 Data Generation

• PAT generation uses MSC conversational context and GPT-4o-extracted persona profiles to create persona-grounded human–AI dialogue turns.Figure 2 depicts persona attributes and context extracted from MSC with GPT-4o, then used with LLaMA-405B to generate pairs across 12 scenarios.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Persona Extraction

• GPT-4o extracts personality traits and organizes them into cohesive, structured persona descriptions.Source: §3.2 · p. 3

3.3 Personalized Query Generation

• The dataset generation process creates 40 linked persona-driven questions per scenario for each persona, producing 58,289 question-response pairs across 133 personas.Prompts include persona description, scenario context, past dialogue, and the current query; the database evolves as dialogue accumulates.Source: §3.3 · p. 3

3.4 Dataset Validation

• GPT-4o rated 300 stratified PAT samples from 4.21 for helpfulness to 4.99 for fluency on a five-point scale.The evaluation also reports relevance at 4.72 and coherence at 4.85; Table 2 presents the evaluation scope.Source: §3.4 · p. 3, Table 2 · p. 3

3.5 Illustrative Example

• Unlike PersonaChat [20], MSC [19], and PAChat, PAT targets assistant dialogues with distinct users, diverse personas, multiple scenarios, and sessions.The paper presents PAT as supporting human–AI interaction, multi-user testing, and multi-session continuity; Table 3 illustrates a persona-consistent response.Source: §3.5 · pp. 3–4

4 AFA Architecture

• AFA processes voice input, maintains per-user memory, and generates personalized text responses; Table 3 presents an example of persona-aligned response generation.Table 4 identifies PAT as uniquely combining multi-user testing, human-AI interaction, and multi-session continuity.Source: §4 · p. 4, Table 3 · p. 5, Table 4 · p. 5
• PAT uniquely combines multi-user testing, human-AI interaction, and multi-session continuity among the compared persona-grounded dialogue datasets.Source: Table 4 · p. 5

4.1 Speaker Identification

• AFA identifies speakers from voice embeddings and links recognized voices to user IDs, providing the entry point for identity-aware routing.It compares embeddings against stored embeddings using cosine similarity and assigns a new ID when no match is found.Source: §4.1 · p. 5

4.2 Dynamic User Profile Store

• Each user has a temporary table of the last ten conversations and a permanent table of summarized session histories.After ten conversations, GPT-4o summarizes the sessions, stores the summary permanently, and clears the temporary table.Source: §4.2 · p. 5

4.3 Persona Synchronizer

• After each turn, the Persona Synchronizer adds newly extracted attributes or replaces contradicted profile entries, and uses the updated persona for the next response.Persona updates are skipped in the constant setting and continue in the adaptive setting.Source: §4.3 · p. 5

4.4 Adaptive Response Generator

• The response generator retrieves the top three historical data points by embedding similarity and combines them with the user persona and current query in a contextual prompt.Source: §4.4 · p. 5

5.1 Speaker Identification Validation

• 100% identification accuracy was achieved across 50 held-out test samples.Source: §5.1 · p. 5
• 0.858 was the average cosine similarity across the speaker identification test samples.The authors report this result as confirming reliable routing for downstream persona retrieval.Source: §5.1 · p. 5
• Five synthetic speakers were enrolled with five utterances each and tested on ten held-out utterances using ECAPA-TDNN.Source: §5.1 · p. 5

5.2 Response Quality Evaluation

• The evaluation included fine-tuned LLaMA-2-70B and zero-shot GPT-4o, GPT-3.5, Claude, and Gemini-2.0 across no-persona, constant-persona, and adaptive-persona settings.Source: §5.2 · pp. 5–6
• Response quality was measured with BLEU, ROUGE, Distinct-1, P-Cover, and A-Cover; top-three retrieval was selected after top-five and top-eight showed no significant quality improvement.Source: §5.2 · p. 6
• 0.806 BLEU-1 and 0.512 ROUGE-1 were the fine-tuned LLaMA-70B’s highest scores under adaptive persona; its P-Cover was 0.463 and A-Cover 0.425.Source: §5.2 · p. 6
• GPT-4o BLEU-1 increased from 0.748 to 0.798 (∼6.7%) with persona adaptation.Source: §5.2 · p. 6
• PAT helped fine-tuned LLaMA-70B approach larger proprietary models on key personalization dimensions.The authors attribute this to PAT’s diverse, well-structured persona-query-response pairs.Source: §5.2 · p. 6
• This benchmark compares individual responses with PAT ground truth and does not test multi-user routing; the separate robustness evaluation addresses user-context separation.Source: §5.2 · p. 6

5.3.1 Evaluation Protocol.

• The protocol interleaves 15 distinct-persona user pairs for 300 turns, comparing pooled no-persona history with fixed and interaction-updated per-user profiles.Each user contributes 10 turns per sequence; constant and adaptive conditions begin with the same profile, but only adaptive updates it after each interaction.Source: §5.3.1 · p. 6

5.3.2 Metrics.

• PAA subtracts ground-truth similarity to control topic bias, then counts responses whose persona lift favors the correct over the wrong user.The section also reports Persona Margin and semantic A-Cover alongside lexical P-Cover and A-Cover.Source: §5.3.2 · p. 6

5.3.3 Results.

• PAA rises from 35.7% (no persona) to 60.3% (constant) and 61.3% (adaptive), while BLEU-1 and ROUGE-1 remain comparable across settings.Table 6 reports a 25-percentage-point PAA gain; BLEU-1 ranges from 0.300–0.312 and ROUGE-1 from 0.417–0.429.Source: §5.3.3 · p. 6, Table 6 · p. 7
• The no-persona condition pools users’ histories, and its 35.7% PAA falls below the 50% chance level, indicating drift toward the wrong persona.The reported example includes details from another user’s earlier conversation.Source: §5.3.3 · pp. 6–7
• Constant and adaptive persona reach 60.3% and 61.3% PAA, respectively; the small difference suggests accurate initial profiles matter more in short interaction windows.The authors describe adaptation’s role as maintaining and enriching profiles over extended use as preferences evolve.Source: §5.3.3 · p. 7

• Speaker identification determines who is speaking and retrieves that user’s persona and conversation history before generating a response.Source: §5.4 · p. 7
• With routing, User A’s response references a music career and British culture interests, while User B’s connects to their coffee shop, details absent from their questions.The example shows personalization beyond what the questions alone reveal.Source: Table 7 · p. 7
• Without routing, both interleaved users receive generic responses rather than responses grounded in their distinct profiles.The table contrasts this with routed responses tied to each user’s profile.Source: Table 7 · p. 7, §5.4 · p. 7

• Two annotators independently rated 50 randomly sampled responses: 25 no-persona and 25 adaptive-persona.Ratings used a scale from 1 (generic) to 5 (deeply personalized).Source: §5.5 · p. 7
• Annotators saw the user profile, question, and response, but not the producing setting; presentation order was randomized.They assessed whether each response connected to the profile beyond what the question stated.Source: §5.5 · p. 8
• Adaptive-persona responses received a mean personalization rating of 3.70, versus 2.32 for no-persona responses, a +1.38 difference on the five-point scale.Welch’s t=5.52, p<0.001; agreement within one point was 72%.Source: §5.5 · p. 8, Table 8 · p. 8

6 Discussion

• PAA rises from 35.7% with no persona to 60.3% with a constant persona, while adaptive persona reaches 61.3%.The authors describe the adaptive-versus-constant gain as marginal in short interaction windows, making correct user routing the primary bottleneck.Source: §6 · p. 8
• After 20 interaction turns, cold-start PAA reaches only 41%, compared with 60% when an onboarding profile is available from the first interaction.The authors conclude that an initial profile is necessary for reliable personalization in communal living facilities where intake information is available.Source: §6 · p. 8
• Speaker identification achieves perfect accuracy on synthetic voice profiles, and PAT fine-tuning lets LLaMA-70B approach or exceed larger proprietary models on personalization metrics.The speaker-identification module can be upgraded independently or supplemented with facial recognition, proximity-based identification, or login.Source: §6 · p. 8
• Evaluation uses synthetic voices and LLM-generated PAT, which may differ distributionally from natural conversation.Real-world performance with natural speech and acoustic variability remains to be validated.Source: §6 · p. 8

7 Conclusion and Future Work

• Identity-aware routing improves persona attribution accuracy from 35.7% to 61.3%, and is identified as the critical factor for preventing persona confusion.The conclusion describes evaluation across five LLM backends, a multi-user interleaved protocol, and human assessment.Source: §7 · p. 8
• Future work includes natural-speech deployment, multi-user conflict resolution, graceful degradation under identification errors, and evaluation in other multi-user settings.Stable preferences may persist indefinitely if they become outdated without being explicitly contradicted; periodic review and user-initiated corrections are proposed.Source: §7 · p. 8
• AFA uses per-user vector databases to isolate conversational histories and personal information, and speaker recognition triggers personalized responses for the identified individual.The study uses synthetic PAT data to avoid privacy risks during development; real-world deployment will require informed consent.Source: §7 · p. 8