Source-linked AI summary
VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction
Zhifei Xie, Jiaqi Lang, Ze An, Yifan Zhao, Dongchao Yang, Kai Li, Ziyang Ma, Mingbao Lin, Chunyan Miao, Shuicheng Yan
TL;DR
Real-time conversational systems lack streaming, accurate, and empathetic memory. VoiceMem addresses this gap with parallel informational and emotional memory, streaming retrieval, and a complete training, evaluation, and decoupled deployment pipeline. It reports strong accuracy and persona results while completing retrieval in 134 ms.
Problem
Conversational systems still lack a unified memory system that is streaming, accurate, and empathetic for real-time spoken interaction.
Method
VoiceMem combines a schema–entity informational left brain, an affective right brain, streaming memory I/O, and decoupled memory-aware SLM training and deployment.
Results
VoiceMem advances information, persona, and long-horizon audio memory benchmarks while achieving 91.2 at K=5 with 134 ms retrieval.
Takeaways & Limitations
VoiceMem provides a practical memory foundation for real-time, personalized, and emotionally aware speech interaction.
Takeaways & Limitations
Prior emotion-aware methods remain information-centric and have limited capacity for emotional association and long-term affective accumulation.
Abstract
from arXiv · showhide
Conversational systems, such as duplex speech language models (SLMs), still lack a streaming, accurate, and empathetic memory system as their soul. We introduce VoiceMem, a simple memory architecture with a parallel informational left brain, an emotional right brain, and streaming memory I/O mechanisms. We further build a complete pipeline for memory-aware SLM training, long-horizon evaluation, and decoupled deployment with interchangeable memory backends. Experiments and real-world deployment show three advantages: i) Accuracy: under top-5 retrieval, the left brain outperforms classical systems such as Mem0 at top-200 by nearly 30 points; ii) Emotional & Personal: the right brain, with short- and long-horizon affective attribution and dual-node persona modeling, achieves state-of-the-art performance across three persona benchmarks and improves the aggregate score by 4.29 points over the previous best system; and iii) Real-Time & Cheap: VoiceMem completes retrieval in 134 ms, well within standard VAD latency, adding no extra conversational delay while maintaining high accuracy and low cost. These results show that VoiceMem provides a practical memory foundation for real-time, personalized, and emotionally aware speech interaction.
1 Introduction
VoiceMem addresses the open challenge of combining informational and emotional memory with increasingly natural real-time conversational systems. Its dual-brain architecture, streaming retrieval, and decoupled training and deployment pipeline target accuracy, latency, and infrastructure constraints.
- Challenges: 2–3 s retrieval breaks the 500 ms budget of real-time conversation, while top-100 outputs exceed speech-model context capacity.The stated requirements are streaming retrieval, almost no added latency, and accuracy under a top-5 budget.
- Architecture: VoiceMem uses parallel left and right brains for information, emotion, and persona in real-time spoken interaction.The left brain organizes information through a two-level schema–entity architecture, while the right brain models emotional and persona features with short- and long-term attribution.
- Architecture: A four-stage streaming memory query completes retrieval within the time taken by a standard VAD.The query decomposes each retrieval step and runs them in real time.
- Pipeline and deployment: VoiceMem adds SLM-verified blackbox OPD, CHATMEM-400K, CHATMEM-BENCH, and a decoupled graph-on-graph memory engine.The benchmark covers Information, Persona, Affective Attribution, and Paralinguist&Environment across 14 fine-grained categories, while the lower memory layer remains interchangeable.
2 Preliminary
Prior memory retrieval methods emphasize semantic or affective relevance but remain limited in emotional association and long-term affective accumulation. VoiceMem extends this line with parallel factual and affective structures that are incrementally updated and jointly queried.
- Prior retrieval: Standard RAG retrieves external records by semantic similarity, while general memory engines also support memory writing and updating.The memory store M_t is queried with the current query q_t.
- Prior retrieval: Emotion-aware retrieval adds affective compatibility, and related systems use emotional context in memory construction and updating.These approaches incorporate affective representations of the current query and memory items.
- Limitations: Existing emotion-aware methods remain information-centric, with limited emotional association and long-term affective accumulation.This is identified as the limitation motivating the dual-brain extension.
- Streaming dual-brain memory: VoiceMem maintains parallel factual and affective cognitive structures connected by cross-brain associations.The left brain organizes factual cells into evolving fact clusters, while the right brain maintains affective-attribution cells.
- Streaming dual-brain memory: Both brains activate new cells and update incrementally for every incoming utterance, then support joint retrieval.The framework separates maintenance of information at scale from accumulation and attribution of affect.
3 VoiceMem: Streaming Dual-brain Architecture
VoiceMem instantiates a streaming dual-brain architecture that separately organizes factual information and emotional persona evidence, while using indexed candidate pools and staged retrieval to meet real-time latency constraints.
- VoiceMem targets three requirements: dense information under limited memory, long-term personality and emotion understanding, and no perceptible spoken-dialogue latency.
- Left Brain: Efficient Memory Access: The left brain uses a cluster–entity–memory-item index to narrow search to compact, semantically coherent candidates before backend retrieval.
- Left Brain: Efficient Memory Access: A two-level schema–entity hierarchy routes queries from coarse semantic categories to concrete people, events, or concepts.
- Left Brain: Efficient Memory Access: Streaming matching identifies schemas and entities from partial transcripts, expands one-hop neighbors, and searches only the resulting candidate pool for accurate top-5 retrieval.
- Left Brain: Efficient Memory Access: Cluster emergence promotes repeatedly co-retrieved, coherent subgraphs after relevance, importance, and completeness checks, while asynchronous updates reconcile new facts off the critical path.
- Right Brain: Knowing the Person: The right brain maintains independent persona nodes for persistent traits and cross-entity nodes for context-dependent affect, preserving both stability and emotional targets.
- Right Brain: Knowing the Person: Short-horizon attribution adapts persona evidence during an interaction, while long-horizon attribution consolidates recurrent affective and behavioral patterns across sessions.
- Streaming Dual-Brain Retrieval: Four-stage streaming retrieval supports near-zero added latency, with dense dual-brain retrieval costing 134 ms against a 500 ms VAD threshold.
4 Infrastructure: Model Training, Validation and Deployment
VoiceMem includes a training, validation, and deployment infrastructure that adapts speech models through online distillation, constructs memory-dependent datasets, and keeps the backend engine interchangeable.
- The infrastructure converts speech-input/text-output model families into memory-augmented speech language models through online black-box on-policy distillation.
- Training and Validation: CHATMEM-400K is built through memory-world construction, SLM-verified online distillation, and human curation over dynamic histories, acoustic inputs, and memory-dependent conversations.
- Training and Validation: CHATMEM-BENCH is a human-curated subset evaluating information recall, persona understanding, affective attribution, and paralinguistic or environmental reasoning.
- Training and Validation: Proprietary teacher models and online distillation are used to mitigate catastrophic forgetting during adaptation.
- Decoupled Deployment: The upper-level routing and dual-brain organization are decoupled from the interchangeable MemSearch backend, currently instantiated with MEM0.
5 Experiments
The experiments evaluate VoiceMem across information, persona, and voice-grounded memory benchmarks, alongside latency, cost, ablation, structural, and backend-transfer analyses. VoiceMem achieves strong retrieval with a small budget, contributes across components, produces structured dual-brain stores, and transfers across compatible backends.
- Main Results: VoiceMem averages 76.39 on information memory and 74.16 on persona memory, beating Mem0 by +24.12 and MemOS by +1.89, respectively.Its persona score reaches 76.56 with the fine-tuned response model.
- ChatMem-Bench: VoiceMem leads 11 of 14 ChatMem-Bench categories, with the largest advantage in Paralinguistics & Environment.It reaches 45.16 to 53.84 on three acoustic categories, while text systems score between 3.23 and 26.92.
- Efficiency: 91.2 accuracy uses 430 memory tokens and 134 ms retrieval, exceeding EverMemOS’s 83.13 with 4.4× fewer tokens.Retrieval remains near 134 ms from K=3 to K=100 because schema routing bounds the candidate pool.
- Efficiency: At K=5, VoiceMem preserves nearly all accuracy while using a small retrieval budget; K=10 adds 1.3 points and K=100 adds 2.3 for 8× the tokens.At K=1, its margins are +11.8 over EverMemOS and +26.5 over Mem0.
- Ablation: Removing the upper-layer index causes the largest ablation loss across datasets, ranging from −4.4 to −9.9 points.Removing the right brain is the next-largest loss, ranging from −4.3 to −6.3 points.
- Memory Structure: The left store contains 510 preset-slot items plus 254 items in two emergent slots, while the right store contains 251 items linked by 280 edges.Pets & Outdoor and Engineering Project span preset-slot boundaries rather than refining within one slot.
- Backend Transfer: The index improves three compatible backend stores by 15.8–29.5 points without threshold retuning, although backend quality still bounds recovery.The index requires identifier-addressable items, similarity search, and UPDATE writes.
6 Case Study
The case study presents VoiceMem as augmenting real-time voice models with persistent, personalized memory. Across representative cases, it supports emotion sensing, preference-aware responses, long-term recall, and multimodal audio memory unavailable to a memoryless baseline.
- Case Study: VoiceMem jointly models semantic, preference, affective, and audio-related memories to condition responses on interaction history.Its streaming dual-brain architecture separates factual memory from affective attribution while supporting long-horizon and multimodal recall.
- Case Study: Across four representative cases, VoiceMem enables user-specific emotion sensing, preference-aware responses, long-term recall, and multimodal audio memory unavailable to a memoryless baseline.The figure summarizes these capabilities in real-time voice interaction.
7 Conclusion
The conclusion presents VoiceMem as a streaming dual-brain memory framework for informational and emotional memory in real-time conversational systems. It combines dense top-5 retrieval, affective persona modeling, streaming retrieval, memory-aware training, long-horizon evaluation, and interchangeable backends.
- Conclusion: VoiceMem equips real-time conversational systems with informational and emotional memory without breaking their latency budget.Its left brain organizes factual knowledge, while its right brain models persona through short- and long-horizon affective attribution.
- Conclusion: VoiceMem reaches 91.2 on LoCoMo with 430 memory tokens and 134 ms retrieval, while the same index lifts three backends by 15.8–29.5 points.The four-stage streaming query hides retrieval inside the silence a standard VAD already waits out.
A Related Work
Related work spans agent memory, speech language models, and memory-interaction systems. These lines respectively develop factual and affective stores, increasingly capable spoken interaction, and architectures connecting internal state with interactive behavior.
- Agent Memory: Agent memory has progressed from scalable factual stores to self-organizing and graph-structured systems that revise links or consolidate memories over time.A parallel line incorporates affect into retrieval and updating.
- Speech Language Models: Speech language models have advanced from conversational overlap toward full-duplex interaction and reasoning during speech.Representative systems progressively support simultaneous listening, speaking, and low-latency interaction.
- Memory-Interaction Systems: Memory-interaction systems study how internal state shapes interactive behavior through parallel pathways, emotionally salient memory, dual-system models, and proactive agents.These approaches connect fast interaction or inferred intent with deeper reasoning or persistent state.
B.1 One-hop neighbours: compress or expand
One-hop neighbour information can be compressed or expanded into competing memory items, with compression selected because it bounds the candidate pool rather than improves accuracy.
- Compression and macro-expansion perform similarly on LoCoMo, scoring 91.2 and 91.10 respectively.
- Compression outperforms macro-expansion on ES-MemEval by +2.76 points.
- Ignoring neighbours reduces performance by 3.4 points on LoCoMo and 9.1 points on ES-MemEval.
- Compression keeps the ES-MemEval candidate pool smaller, at 144.1 versus 165.7 items, while adding 71.2 prompt tokens on LoCoMo.
B.2 Cluster maintenance over a growing history
The emergence strategy maintains clusters over growing histories by selecting coherent repeated-retrieval subclusters and promoting them only after relevance, importance, and completeness checks.
- The evaluation covers all 252 questions across 32 sessions at K=5.
- Emergence reaches 74.40 on ES-MemEval P1, ahead of static by 1.80 points, random_split by 2.00, and size_threshold by 2.79.The comparison with random_split controls for the number of splits, isolating split placement as the relevant difference.
- Random_split performs the same number of splits as emergence, but remains 2.00 points worse.
- Size_threshold falls 0.99 points below static, showing that splitting in the wrong place can underperform no splitting.