Source-linked AI summary
From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents
Yucong Zhang, Cancan Li, Bo Xiao, Ke Zeng, Ming Li
TL;DR
Personalized spoken agents must identify who is speaking before accessing user-specific history, while many memory benchmarks assume participant attribution is already available. This paper introduces EvoSpeakerMem to construct persistent user entities as acoustic evidence evolves and builds LoCoMo-ID to evaluate speaker-sensitive spoken memory. Across the four-speaker test, non-fixed policies outperform Fixed in full QA, with Gated Update highest.
Problem
Conversational-memory benchmarks principally evaluate retrieval from histories with available participant attribution, while persistent user-memory construction requires resolving each incoming speaker and routing content to the appropriate memory.
Source: §1 · pp. 1–2Method
EvoSpeakerMem causally routes turns to persistent user entities, selectively maintains acoustic evidence, and partitions memory writes and retrieval by entity.
Source: §1 · p. 2Results
In the four-speaker interleaved test, ECAPA-only routing yields 71.99–73.60% full QA for the three non-fixed policies versus 67.83% for Fixed, with Gated Update highest.
Source: §5 · p. 8Takeaways & Limitations
The findings support maintaining speaker evidence beyond the initial voiceprint when constructing and querying personalized memory.
Source: §5 · p. 8LoCoMo-ID uses synthetic voices and simulated recording shifts, and its four-speaker test does not represent naturally recorded, overlapping four-party dialogue.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Personalized spoken agents must identify speakers before accessing user-specific memory, yet memory benchmarks typically assume correctly attributed histories. We argue that voiceprints are observations of identity, not identities themselves, and formulate online speaker-memory construction to maintain persistent user entities as acoustic evidence evolves. We introduce EvoSpeakerMem, a training-free layer for causal speaker routing, selective acoustic-state updates, and identity-partitioned memory writes and retrieval through a replaceable text-memory backend. We also construct LoCoMo-ID, a spoken-memory benchmark with hidden speaker identities, speaker-sensitive queries, simulated recording-condition shifts, and four-speaker identity-confusion tests. On longitudinal SMIIP-TV, genuine-probe rejection under a frozen near-enrollment threshold rises from 29.8% to 55.6% over approximately one year. On LoCoMo-ID speaker-sensitive queries, gated updating improves identity-gated QA over fixed representations by 3.86 and 11.62 percentage points under clean and shifted conditions, respectively, with no wrong-user authorizations. The advantage persists in the four-speaker test. Yet reducing fragmentation and improving QA can coexist with incorrect cross-user links, revealing a fragmentation--contamination trade-off that QA alone cannot capture.
1 Introduction
Personalized spoken agents need to attribute each incoming observation before adding it to a user’s history, but existing memory benchmarks largely start with attributed histories. The paper frames voiceprints as evolving evidence about persistent user entities and introduces EvoSpeakerMem and LoCoMo-ID to study online attribution and its consequences.
1 Introduction
- Speaker attribution errors can fragment one user’s history or contaminate another user’s profile, affecting later personalized responses.Conversational-memory benchmarks principally evaluate retrieval from histories with available participant attribution, leaving online history construction less understood.Source: §1 · pp. 1–2
- Acoustic variation across sessions, environments, and user states challenges fixed representations, while linking voiceprints to writable memory makes false rejection and false acceptance consequential.False rejection can duplicate a profile; false acceptance can misattribute information to another user.Source: §1 · p. 2
- EvoSpeakerMem treats voiceprints as observations of persistent user entities and provides causal CRE-ATE/LINK routing with selective speaker-state updating.Attributed content is written to a replaceable textual-memory backend.Source: §1 · p. 2
- LoCoMo-ID evaluates hidden-speaker, speaker-sensitive memory with online routing and identity-gated retrieval under recording shifts and multi-speaker identity-confusion scenarios.The paper also evaluates Audio LLM speaker-identity tasks, longitudinal acoustic evidence, and the full speaker-memory pipeline in three stages.Source: §1 · p. 2
2 Method
The method maintains causal user entities that combine acoustic evidence with attributed textual memory. It separates routing from acoustic-state updating and uses entity assignments consistently for memory writes and reads.
2.1 Problem Formulation
- Each causal user entity holds an acoustic representation and attributed textual memory, while routing assigns each arriving turn to an existing entity or NEW.The state is observed chronologically without speaker labels or access to future turns.Source: §2.1 · p. 2
2.2 Online Acoustic Entity Routing
- The router links a turn to its highest-scoring entity or creates a new one when its score falls below the fixed τmerge threshold.Unrestricted updating can make later errors self-reinforcing, so GATED UPDATE requires a stricter τupdate condition before changing a center; accepted links still increment the observation count.Source: §2.2 · pp. 2–3
2.3 Speech Support Layer and Memory Construction
- For each turn, the system transcribes speech, extracts acoustic identity evidence, routes to an entity, and writes attributed facts to that entity’s partition in a replaceable text-memory backend.At query time, read-only voice matching selects an entity and restricts retrieval to its memory partition; transcript semantics do not influence acoustic routing or center updates.Source: §2.3 · p. 3
3 LoCoMo-ID Benchmark Construction
LoCoMo-ID adapts conversational histories into spoken, speaker-sensitive queries and tests memory construction under acoustic variation and multi-speaker interference. Its evaluations use synthesized voices and simulated recording-condition shifts.
3 LoCoMo-ID Benchmark Construction
- LoCoMo-ID combines query adaptation, identity-consistent speech synthesis, and stress conditions for acoustic variation and multi-speaker interference.It retains LoCoMo’s ten histories, 5,882 turns, 1,986 QA items, and reference answers.Source: §3 · p. 3, §3.1 · p. 3
3.1 Text Adaptation for Spoken Queries
- Rule-based transformations replace explicit participant references with first-person forms so answering requires the spoken requester’s identity while preserving reference answers.The benchmark rewrites 1,371 single-person questions and 22 shared-experience questions; 593 questions remain unchanged, with manual checks of rewritten items.Source: §3.1 · pp. 3–4
3.2 Voice Assignment and Speech Synthesis
- Each conversation’s two participants receive distinct synthesized voices maintained across history turns, sessions, and corresponding spoken queries.Three voice assignments per conversation yield 30 clean collections, 17,646 history audio turns, and 5,958 query audio items.Source: §3.2 · p. 4
3.3 Recording-condition Shifts
- Simulated room, device, and environmental variations alter history audio while preserving content and speaker assignments, and no post-augmentation quality or reference-similarity filtering removes difficult observations.Source: §3.3 · p. 4
3.4 Four-Speaker Interleaved Stress Test
- The four-speaker test interleaves two fact-conflicting stories with shared character names but disjoint voice pairs, requiring identity-specific retrieval across alternating sessions.Queries target the original story even when same-named speakers in the parallel story have conflicting facts.Source: §3.4 · p. 4
4 Experiments
The experiments test whether maintaining speaker evidence over time improves online identity assignment and downstream personalized QA. Results show that fixed voiceprints degrade with time, while adaptive policies improve entity structure and identity-gated QA, with trade-offs in cross-user authorization.
4.3 MSU-Bench: Motivating Explicit Speaker State
- The MSU-Bench configuration with an ECAPA speaker tool achieves 96.61% speaker retrieval and 79.35% speaker verification accuracy, exceeding reported end-to-end Audio LLM results.The tool processes segments chronologically and maintains a persistent speaker registry.Source: §4.3 · p. 6
4.4.2 Fixed-Template Degradation
- 55.6% of genuine probes fall below the frozen near-enrollment threshold after approximately one year, up from 29.8% within one day.EER increases from 9.7% to 14.0%; a stronger encoder reduces but does not eliminate the fixed-template degradation.Source: §4.4.2 · p. 6, Table 2 · p. 6
4.4.3 Causal Speaker Maintenance
- All-exemplar KNN reaches 65.31% macro-averaged B3 F1 across six large-span conditions, versus 55.99% for online mean updating and 46.82% for First exemplar.KNN has lower LINK precision than First exemplar (52.56% vs. 66.25%), exposing a trade-off between fragmentation and incorrect linking.Source: §4.4.3 · p. 6
4.5.1 History Routing and Entity Structure
- Across both evidence modes and speech conditions, non-fixed policies reduce FNUR and Clusters/Gold and generally improve B3 F1 over Fixed.Under shifted ECAPA-only routing, B3 F1 rises from 74.18% for Fixed to 92.73–96.62% for non-fixed policies.Source: Table 1 · p. 5, §4.5.1 · p. 7
4.5.2 All-Memory Retrieval and QA
- All-memory QA is 73.38% for ECAPA gated updating versus 72.31% for Fixed on clean speech, and 72.04% versus 69.27% under shifted conditions.This retrieval evaluates the same 4,113 sensitive queries without identifying the requester or filtering by speaker.Source: §4.5.2 · p. 7, Table 3 · p. 7
4.5.3 Identity-Gated Retrieval
- With ECAPA-only routing, identity-gated QA rises from 71.41% for Fixed to 73.43–75.27% on clean audio and from 60.15% to 70.24–72.40% under shifted condition.Source: Table 4 · p. 7, §4.5.3 · p. 8
- All-exemplar KNN has slightly higher shifted-condition identity-gated QA than Gated Update (72.40% vs. 71.77%), but causes 116 wrong authorizations versus none.The comparison illustrates that QA alone does not capture cross-user access errors.Source: Table 4 · p. 7, §4.5.3 · p. 8
4.5.4 Four-Speaker Interleaved Stress Test
- 72.06%, 73.60%, and 71.99%: Update, Gated Update, and All-exemplar KNN exceed Fixed’s 67.83% full QA in the four-speaker test.Gated Update also has the highest four-speaker B3 F1 at 97.77%.Source: §4.5.4 · p. 8, Table 5 · p. 8
5 Conclusion
The paper frames speaker attribution as part of persistent memory construction, with EvoSpeakerMem routing turns and writing attributed content to user entities. Across LoCoMo-ID conditions, maintaining speaker evidence improves identity-gated QA over fixed representations, though improved QA can coexist with incorrect cross-user links.
5 Conclusion
- EvoSpeakerMem places speaker attribution in the memory write path as part of online speaker-memory construction.The approach treats voiceprints as observations of persistent user entities.Source: §5 · p. 8
- Across clean and shifted LoCoMo-ID experiments, Update, Gated Update, and All-exemplar KNN improve identity-gated QA over Fixed in both acoustic evidence modes.In the four-speaker interleaved test, the three non-fixed policies achieve 71.99–73.60% full QA versus 67.83% for Fixed, with Gated Update highest.Source: §5 · p. 8
- The four-speaker interleaved test preserves the advantage: ECAPA-only routing yields 71.99–73.60% full QA for non-fixed policies versus 67.83% for Fixed.Gated Update attains the highest score.Source: §5 · p. 8
- Reducing fragmentation and improving QA can coexist with incorrect cross-user links, exposing a fragmentation–contamination trade-off that QA alone cannot capture.The conclusion supports maintaining speaker evidence beyond the initial voiceprint when constructing and querying personalized memory.Source: §5 · p. 8
Limitations
The evaluation relies on synthetic voices and simulated recording-condition shifts, with limited coverage of natural multi-user dialogue. Several evaluations also have narrow scope, and incorrectly written memories are not automatically corrected.
Limitations
- LoCoMo-ID uses synthetic voices and simulated recording shifts rather than naturally recorded, overlapping multi-user conversations.Its standard clean and shifted collections each contain two participants; the four-speaker test interleaves synthetic fact-conflicting stories and covers only ten clean collections.Source: Limitations · p. 9
- SMIIP-TV evaluates entity maintenance over repeated recordings, not dialogue-memory QA, while the Parakeet comparison uses the same ECAPA cross-session registry rather than an independently evaluated published system.Source: Limitations · p. 9
- The merge/lookup threshold 0.489 has no independent sensitivity sweep, and the operational encoder and backend remain fixed.Only the center-update threshold is varied, on clean speech.Source: Limitations · p. 9
- Multi-exemplar KNN stores every accepted voiceprint without bounding memory growth; incorrectly written memories are not automatically corrected.Source: Limitations · p. 9
Ethical Considerations
The paper highlights privacy risks from persistent voice linkage and identity-routing errors, and recommends consent, access controls, data minimization, and user identity-management mechanisms. The synthetic voice bank uses anonymous generated references and is not intended to imitate or identify real individuals.
Ethical Considerations
- Raw audio, embeddings, transcripts, and extracted memories should be processed with consent and access controls, retained only as long as necessary, and protected from unauthorized reuse.Source: Ethical Considerations · p. 9
- A local anonymous identifier does not make a person’s voice anonymous, and long-term voice linkage could be misused beyond the intended interaction context.Source: Ethical Considerations · p. 9
- The synthetic voice bank uses anonymous generated references and is not intended to imitate or identify real individuals.Synthetic voices and derived conversational data should follow source-asset licenses and intended uses.Source: Ethical Considerations · p. 9
- A false merge may expose one user’s information to another, while fragmented profiles can make correction and deletion incomplete.Observed wrong authorizations under the simulated protocol illustrate the risk of exposing another user’s memory.Source: Ethical Considerations · p. 9
- The authors recommend transparent identity management with inspection, correction, deletion, and withdrawal, and caution against security-critical use of the router.They characterize the experiments as a research prototype rather than a deployment-ready biometric system.Source: Ethical Considerations · p. 9
A Detailed Online Routing Metrics
The routing metrics distinguish creation and linking decisions for new and previously observed speakers, and report measures of creation, linking, and known-user resolution. Table 6 reports MSU-Bench speaker retrieval and verification accuracy.
A Detailed Online Routing Metrics
- The causal first-seen categories distinguish new-speaker CREATE and LINK outcomes from known-speaker CREATE, correct LINK, and wrong-target LINK outcomes.A speaker is new if not seen earlier in the causal stream; a LINK is correct only when it matches the immutable first-creation binding.Source: A Detailed Online Routing Metrics · p. 11
- Table 6 reports MSU-Bench speaker retrieval (SR) and verification (SV) accuracy in percent.Source: Table 6 · p. 11
- The reported routing statistics include CREATE precision and recall, LINK precision, and known-user resolution recall; FNUR is a causal fragmentation indicator.Known-user resolution recall penalizes erroneous CREATE and wrong-target LINK decisions, while FNUR measures returning-speaker turns assigned to newly created entities.Source: A Detailed Online Routing Metrics · p. 11
B Additional MSU-Bench Results
The tool-augmented persistent speaker state outperforms the listed end-to-end Audio LLMs on MSU-Bench speaker recognition and verification. The static protocol freezes a threshold calibrated on near-enrollment probes for later time-gap analyses.
B Additional MSU-Bench Results
- 96.61% on SR and 79.35% on SV exceed every listed end-to-end model on the respective MSU-Bench tasks.The comparison attributes strong speaker-memory performance to an explicit acoustic identity module and persistent state, rather than direct answer generation alone.Source: B Additional MSU-Bench Results · pp. 11–12
C.1 Static Verification Protocol
- SMIIP-TV uses text-independent utterances from five daily periods, with three utterances per period and metadata on recording environments, noise, physical states, and time of day.An approximately one-year extension includes 118 matched speakers but lacks the original release’s full context annotations.Source: §C.1 · p. 12
- The static protocol calibrates τ2% = 0.4751 from near-enrollment impostor scores, then freezes it across later time-gap analyses.Below-τ is the rate of genuine probes rejected by the frozen template, distinct from false-new user rate during causal replay.Source: §C.1 · p. 12
C.2 Matched-Set ReDimNet2-B6 Static Robustness Check
On matched utterances, ReDimNet2-B6 improves absolute fixed-template verification and online entity maintenance relative to ECAPA, but does not eliminate deterioration at longer time gaps. The additional replay analyses describe context variation and trade-offs across policies.
C.2 Matched-Set ReDimNet2-B6 Static Robustness Check
- ReDimNet2-B6 EER rises from 3.34% at 0–1 days to 6.35% at 365–450 days, while ECAPA rises from 9.71% to 14.09%; ReDimNet2-B6 Below-τ also rises from 5.19% to 12.96%.These are matched-set results with encoder-specific frozen thresholds; stronger embeddings reduce, but do not eliminate, longitudinal mismatch.Source: §C.2 · p. 12, Table 7 · p. 13
C.3 Matched-Set ReDimNet2-B6 Causal Replay Check
- Mean B3 F1 rises from 65.45% with ECAPA to 73.68% with ReDimNet2-B6 for all-exemplar KNN, an 8.23-point improvement.ReDimNet2-B6 improves B3 F1 in five of six conditions and reduces false-new decisions and fragmentation in most conditions; alternative maintenance policies were not compared under this encoder.Source: §C.3 · p. 12, Table 8 · p. 14
C.4 Time-Gap and Score Diagnostics
- The speaker-level view indicates that aggregate Below-τ change is not attributable only to a small number of outlying users.The controlled observational comparison reduces measured context differences but does not establish a causal effect of time.Source: §C.4 · p. 14, Figure 5 · p. 14
C.5 Complete Context Breakdowns
- The complete context breakdowns show EER, mean genuine similarity, fixed-threshold Below-τ, and impostor acceptance, but remain descriptive rather than controlled causal comparisons.Some categories have few speakers or probes, so those breakdowns warrant caution.Source: §C.5 · p. 15, Figure 7 · p. 15, Figure 9 · p. 16
C.6 Online Replay Conditions
- Large-span replay consumes each speaker’s probes chronologically after enrollment, with speaker streams interleaved while preserving per-speaker order.Six conditions are summarized across five-seed means; the supplied table caption identifies the scope but provides no policy-specific values.Source: §C.6 · pp. 15–16, Table 10 · p. 17
C.7 Online Policy Diagnostics
- Low FNUR alone is insufficient to establish correct identity linking, so decision metrics should be read alongside B3 results.A policy can link returning turns frequently while still selecting the wrong target.Source: §C.7 · p. 17, Figure 11 · p. 17
D.1 Full-Set All-Memory QA
- The full-set all-memory QA scores pool 5,958 query audios per condition and use unrestricted retrieval without speaker-based filtering.The pooled scores combine 4,113 sensitive-query results and 1,845 non-sensitive-query results.Source: §D.1 · p. 17
D.2 Full Causal Decision and Private-Access Results
The section evaluates routing, authorization, and answer utility together, because reduced fragmentation or improved QA alone does not establish correct private access. It also reports a clean-speech update-threshold choice and cautions against generalizing from sparse short-turn observations.
D.2 Full Causal Decision and Private-Access Results
- Entity counts alone cannot establish correct access: fragmented histories may retain only part of a requester’s evidence, while fewer entities can still be consistent with incorrect cross-speaker associations.The proposed fragmentation mechanism is not an isolated causal test, and all-memory and strict identity-gated QA impose different authorization requirements.Source: §D.2 · pp. 17–18
D.3 Four-Speaker Interleaved Query Results
- The four-speaker results distinguish full strict QA, which requires correct authorization and a judge-correct answer, from targeted raw-judge QA, which does not require correct authorization.The targeted strict set contains 19 identity-gated questions; the raw-judge set evaluates 20 questions.Source: §D.3 · p. 18
D.4 Update-Threshold and Utterance-Duration Diagnostics
- 0.58 is the lowest tested clean-speech update threshold in the 0.58–0.66 range where online accuracy and B3 F1 are nearly stable and no wrong authorizations occur.Lower thresholds 0.50 and 0.54 produce 115 and 32 wrong authorizations, respectively.Source: §D.4 · p. 19
- All first-appearance turns fall in the ≥2 s bin, and the 11 of 13 shifted subsecond gated-turn errors are too few to establish a general short-speech failure rate.The duration analysis is post hoc, not a controlled intervention.Source: §D.4 · p. 19
E Backend Models and Decoding Configuration
The appendix describes the LoCoMo-ID models and deterministic acoustic routing, then details voice construction, query transformations, and recording-condition augmentation. It also limits what synthesis checks and embedding timing can establish about routing and full-pipeline performance.
E Backend Models and Decoding Configuration
- The reported 65.6 ms per turn covers ECAPA embedding extraction only, not routing, transcription, or textual-memory construction.It therefore does not establish full-pipeline real-time performance.Source: E Backend Models and Decoding… · p. 20
F LoCoMo Acoustic Tool-Use Prompt
- The deterministic acoustic tool extracts an ECAPA embedding and uses the configured registry threshold and update policy to link a segment or create a local speaker entity.The model invokes the tool but does not select the speaker identity; transcription output is not used to decide identity.Source: F LoCoMo Acoustic Tool-Use Prompt · pp. 20–21
F.1 Audio Transcription Prompt
- Qwen3-Omni transcribes each history or query clip, and the resulting transcript is passed to the unchanged LightMem/LoCoMo components.The transcription instruction requests a faithful transcript without additional commentary.Source: §F.1 · p. 21
F.2 Deterministic Acoustic Tool-Use Prompt
- For each audio segment, the prompt requires exactly one embedding-tool call and forbids the model from inferring or overriding speaker identity.The tool handles matching against the persistent registry and applies its configured threshold and update policy.Source: §F.2 · p. 21
G LoCoMo-ID Construction Details
- LoCoMo histories and reference answers are retained without LLM generation, while deterministic rules handle turn selection and query rewriting.Synthesis QC checks perceptual quality and reference-voice similarity, but passing QC does not guarantee correct online routing; the condition shifts are applied afterward without further QC filtering.Source: G LoCoMo-ID Construction Details · pp. 21–22
G.1 VoiceDesign Instruction and Enrollment
- ECAPA-TDNN diversity filtering retains 50 of 120 candidate voices, since descriptive voice attributes do not necessarily yield distinct acoustics.Each candidate reads the same canonical enrollment sentence, controlling lexical variation in embedding comparisons.Source: §G.1 · p. 22
G.2 Voice-Cloning Inputs
- Voice-cloning mode uses VoiceDesign-generated enrollment audio to synthesize history turns and queries.The supplied passage identifies the cloning inputs but does not specify them further.Source: §G.2 · p. 22
G.3 Deterministic Query Transformations
- Rule-based query rewriting converts identity-gated questions to first person and shared questions to first-person plural; unsafe transformations remain verbatim.The construction pipeline’s natural-language model instruction is the VoiceDesign prompt for the anonymous voice bank.Source: §G.3 · p. 22