Source-linked AI summary
Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models
Heeseung Kim, Che Hyun Lee, Sangkwon Park, Jiheum Yeom, Nohil Park, Sangwon Yu, Sungroh Yoon
TL;DR
Whether open-source voice interaction models retain and use long-range dialog history remains unclear, with no benchmark explicitly requiring history use. The paper introduces ContextDialog to evaluate recall and retrieved-context use, finding speech models struggle more than text-based systems and remain sensitive to retrieval errors.
Problem
It remains unclear whether open-source voice interaction models can retain and use long-range dialog histories, and no benchmark explicitly requires dialog-history use to generate responses.
Source: §1 · p. 1Method
The authors introduce ContextDialog, a speech-to-speech benchmark based on spoken recall QA pairs, and use it to evaluate past-utterance recall and externally retrieved context.
Source: §1 · pp. 1–2Results
Open-source speech models generally perform worse than text-based systems, struggle more with speech-based information, and remain sensitive to retrieval errors despite retrieval augmentation.
Source: §1 · p. 2Takeaways & Limitations
The findings highlight a gap in open-source models’ conversational memory and point to stronger memory retention and retrieval robustness as improvement needs.
Source: §5 · p. 9The retrieval analysis uses text-based RAG, does not account for RAG latency, and leaves speech retrievers for future development.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Recent advancements in multi-turn voice interaction models have improved user-model communication. However, while closed-source models effectively retain and recall past utterances, whether open-source models share this ability remains unexplored. To fill this gap, we systematically evaluate how well open-source interaction models utilize past utterances using ContextDialog, a benchmark we proposed for this purpose. Our findings show that speech-based models have more difficulty than text-based ones, especially when recalling information conveyed in speech, and even with retrieval-augmented generation, models still struggle with questions about past utterances. These insights highlight key limitations in open-source models and suggest ways to improve memory retention and retrieval robustness.
1 Introduction
Open-source voice models’ ability to retain and use long-range dialog history remains unclear, and existing benchmarks do not explicitly require history use. This work evaluates recall with ContextDialog and finds speech models struggle with past speech, while retrieval augmentation remains vulnerable to errors.
1 Introduction
- Whether open-source multi-turn systems can retain and use long-range histories remains unclear, and no benchmark explicitly requires dialog-history use to answer questions.The introduction contrasts this open question with examples of closed-source systems referencing past interactions.Source: §1 · p. 1
- The study evaluates whether open-source voice models can recall prior dialog in spoken responses and incorporate externally retrieved utterances.ContextDialog is introduced as a speech-to-speech benchmark built from spoken question-answer pairs derived from existing dialogs.Source: §1 · pp. 1–2
- Speech models generally underperform text-based systems on past-utterance recall, especially for information conveyed in speech, and retrieval augmentation does not reliably overcome the gap.Models are highly susceptible to retrieval mistakes, which can leave performance unchanged or degrade it.Source: §1 · p. 2
- The authors identify memory retention and retrieval robustness as areas for future improvement in open-source models.They describe these as limitations that restrict model applicability.Source: §1 · p. 2
2 Related Works
Prior work has developed multi-turn voice models, but explicit long-context recall methods and evaluations remain limited. ContextDialog differs from existing benchmarks by requiring relevant past utterances to answer questions, while this paper studies whether retrieved context can be used in voice generation.
Voice Interaction Models
- Most multi-turn voice-model work does not propose explicit methods or evaluations targeting long-context recall, leaving real-world history handling largely unexplored.SLAM-Omni stores dialog history as transcribed text prepended at inference, while Lyra explores long-audio-history techniques and extends context windows within its architecture.Source: §2 · p. 2
- The evaluation includes SLAM-Omni and Lyra as long-context baselines, alongside Freeze-Omni, GLM-4-Voice, MiniCPM-o, and Moshi as open-source multi-turn candidates.SLAM-Omni and several additional candidates are covered in Appendix A.1.6.Source: §2 · p. 2
Benchmarks
- Unlike benchmarks that assess multi-turn semantics without requiring past information, ContextDialog explicitly requires retrieval and use of relevant prior utterances.Existing benchmark examples cover entity and dialog-state recognition, response coherence, and non-verbal properties such as emotion and background noise.Source: §2 · p. 3
Retrieval in Voice Interaction Model
- Prior spoken-dialog retrieval work emphasizes entity extraction, spoken question answering, or long-speech-document retrieval, whereas this study tests retrieved information in recent open-source multi-turn voice models.The paper examines whether an external module’s retrieved data can be used during generation.Source: §2 · p. 3
3 ContextDialog
ContextDialog is built from spoken dialogs to evaluate whether models can retain and use relevant information across turns. Its construction generates and validates recall questions and answers, then renders them as speech with speaker-adaptive synthesis and pronunciation checks.
3 ContextDialog
- ContextDialog uses MultiDialog, a roughly 340-hour two-speaker corpus with conversations of at least 10 turns, and draws from its test_freq and test_rare splits.The source splits contain 450 and 381 spoken dialogs, respectively; some data are filtered during generation and validation.Source: §3 · p. 3
3.1 Text Question-Answer Generation
- Recall QA pairs are generated from information mentioned only once, covering user and system utterances from either half of a conversation, with four pairs per dialog.Each pair includes a supporting utterance and is designed to require a detailed answer rather than a yes/no response.Source: §3.1 · pp. 3–4
- Three context checks validate that answers require the supporting utterance and remain consistent with it and the full conversation; failed pairs are filtered out.The first check uses dialog context before the supporting utterance and expects the answer not to be inferable; the next two expect consistency.Source: §3.1 · p. 4
3.2 Spoken Question-Answer Generation
- Speaker-adaptive TTS renders each QA pair in the target speaker’s voice, selecting the lowest-WER sample from five generations and manually reviewing nonzero-WER outputs.The longest original speech segment for each speaker is used as the voice reference, and mispronounced samples are filtered out.Source: §3.2 · p. 4
4 Experiments
Across two ContextDialog experiments, open-source voice interaction models struggle to recall past information and make little use of retrieved utterances. Their recall is weaker than text-based counterparts, particularly for user information conveyed only in speech.
4 Experiments
- The experiments evaluate four open-source voice models on recall and retrieved-utterance use, including comparisons of spoken responses and intermediate text.The evaluation uses ContextDialog and examines both past user and model utterances.Source: §4 · pp. 4–5, §4.1 · p. 5
4.1 Does Your Model Truly Recall Past Information?
- Voice interaction models underperform text-based counterparts on multi-turn recall, with especially pronounced degradation in Freeze-Omni.The comparison covers both intermediate text and transcribed spoken responses.Source: Table 2 · p. 5, §4.1 · p. 6
- Models perform better on questions about their own past utterances than on user utterances, and attend less to supporting utterances when answering about user speech.The authors attribute this gap to text accompanying model-generated speech, while user utterances must be recalled from speech alone.Source: §4.1 · p. 6, Figure 3 · p. 6
4.2 Does Your Model Reliably Augment Retrieved Information into Generation?
- With RAG, all models perform similarly or worse, showing little to no improvement as the number of retrieved utterances increases.The authors identify retrieval noise and longer prompts that conflict with models’ tendency to avoid lengthy spoken responses as contributing factors.Source: §4.2 · p. 7, Figure 4 · p. 7
4.2.1 Analyses
- Providing the correct supporting utterance improves performance for models other than MiniCPM-o, while an incorrect utterance degrades it.This suggests that retrieval errors, rather than augmentation itself, are a major obstacle for most models.Source: Table 3 · p. 7, §4.2.1 · p. 8
- ASR has minimal impact on RAG performance with text-based retrievers, but the speech retriever leads to a performance drop.The authors link the speech-retriever decline to its retrieval limitations and training challenges, including longer audio and limited data.Source: §4.2.1 · p. 8, Table 4 · p. 8, Table 5 · p. 8
5 Conclusion
The paper introduces ContextDialog to study how open-source voice interaction models maintain and use past utterances. The benchmark experiments find weak recall and sensitivity to retrieval errors, motivating stronger conversational memory methods.
5 Conclusion
- ContextDialog is a speech-to-speech benchmark for systematically evaluating recall of utterances from previous turns.It addresses the lack of benchmarks explicitly requiring accurate reference to past dialog.Source: §5 · p. 9
- Experiments find that models struggle to recall past utterances and remain highly sensitive to retrieval errors, limiting gains even with a dedicated retriever.Source: §5 · p. 9
- The findings point to stronger conversational memory methods, including improved long-context modeling, robust RAG, or dedicated memory modules.Source: §5 · p. 9
Limitations
The analysis is limited to selected open-source models, text-based retrieval, synthetic data, and basic direct-recall questions. The authors identify broader model coverage, speech retrieval, real-world data, and deeper reasoning as areas for extension.
Limitations
- The analysis covers selected open-source models; models that generate speech without intermediate text remain outside its scope.The authors also note that other open-source models may not be included.Source: Limitations · p. 9
- Retrieval-based analysis uses text prompts and does not account for RAG latency; speech retrieval integrating linguistic and non-verbal cues remains a need for real-time applications.The authors report no well-established open-source speech retrievers and difficulty with speech-based RAG prompting.Source: Limitations · p. 9
- ContextDialog is synthetic: GPT-4o generates question-answer pairs and a separate TTS model synthesizes the audio.The authors identify real-world, human-collected multi-turn voice-assistant data as a next step.Source: Limitations · p. 9
- ContextDialog tests the simplest past-information questions—direct retrieval and use of prior context—not deeper reasoning or more complex recall such as fragmented information.The authors suggest benchmarks covering deeper reasoning and fragmented information, such as a phone number given in segments.Source: Limitations · pp. 9–10
Ethical Considerations
The paper highlights a privacy trade-off for personalized voice assistants: useful services rely on past interactions, while stored conversational data may be vulnerable to unauthorized access. It calls for privacy protection alongside memory advances.
Ethical Considerations
- Personalized voice assistants rely on past interactions to provide customized services.The paper connects this reliance to their recall capabilities.Source: Ethical Considerations · p. 10
- Stored conversational data may be vulnerable to unauthorized access, raising security and privacy concerns.Source: Ethical Considerations · p. 10
- The paper calls for robust protection of stored history alongside advances in memory retention and utilization.Source: Ethical Considerations · p. 10
A.1 Additional Details and Analysis
The appendix expands the paper’s model and evaluation analysis, covering dataset statistics, model-specific text-based RAG, additional comparisons, datasets, prompts, and failure cases.
A.1 Additional Details and Analysis
- Appendix A.1.1 provides ContextDialog statistics, and A.1.2 describes four evaluated models and how text-based RAG is applied to each.The models are GLM-4-Voice, Freeze-Omni, Lyra, and MiniCPM-o.Source: §A.1 · p. 14
- Appendix A.1.3 reports intermediate-text results, while A.1.4–A.1.5 add GPT Score comparisons with closed-source models and human evaluations.Source: §A.1 · p. 14
- Appendix A.1.6–A.1.9 cover model inclusion rationale, additional datasets and retrieval prompts, and categorized failure cases.Source: §A.1 · p. 14
A.1.1 Dataset Details
The dataset details describe how ContextDialog statistics are organized and identify two approaches for generating text alongside spoken responses to support semantic coherence.
A.1.1 Dataset Details
- Table 6 breaks down ContextDialog statistics by data split, recalled-information source, and utterance type.Source: §A.1.1 · p. 14
- Figure 5 presents two representative approaches for generating text alongside spoken responses to enhance semantic coherence in voice interaction models.Source: Figure 5 · p. 14
A.1.2 Model Details
The section describes how four voice interaction models generate speech and text, and how retrieved utterances are incorporated into their response pipelines.
A.1.2 Model Details
- GLM-4-Voice interleaves 13 text tokens with 26 speech tokens per step, while retrieved utterances fill text-token slots before prompt removal for evaluation.Its waveform tokenizer combines pooling and vector quantization with a causal Whisper encoder; speech tokens are reconstructed by a CosyVoice-based module.Source: §A.1.2 · pp. 14–15
- Freeze-Omni keeps the backbone LLM frozen and routes speech through a plug-in encoder and decoder, with retrieved utterances prefixed to text generation.The decoder uses only the subsequently generated response, excluding the retrieval prefix and its hidden states.Source: §A.1.2 · p. 15
- Lyra uses Whisper-large-v3 speech features and generates text alongside discrete speech units that are converted into waveforms.The LLM’s hidden states are upsampled using the average text-to-unit length ratio to produce speech units.Source: §A.1.2 · p. 15
- MiniCPM-o supplies 25Hz speech features to the LLM and generates speech in chunks using both its hidden features and text response.Speech synthesis begins in parallel once the text output reaches a specified chunk size.Source: §A.1.2 · p. 15
A.1.3 Results on Intermediate Text Responses
Text-based RAG has minimal impact on overall performance, and the same pattern appears in intermediate text responses. This indicates that speech-synthesis errors do not account for the observed pattern.
A.1.3 Results on Intermediate Text Responses
- Text-based RAG has minimal impact on overall performance, a pattern that persists in intermediate text responses and is not influenced by speech-synthesis errors.Figure 6 shows results with RAG for each model against the no-RAG baseline, using intermediate text responses for evaluation.Source: §A.1.3 · pp. 15–16
A.1.4 Results on Closed-Source Models
This section reports appendix evaluations of closed-source model recall, speech-recall gaps, RAG gains, and related failure modes, alongside the prompts and examples used to generate and assess ContextDialog question-answer samples.
A.1.4 Results on Closed-Source Models
- Despite receiving only concatenated audio or textual history, gpt-4o-mini-audio-preview outperformed the open-source models by a large margin on recall.The API could not take both speakers’ past utterances as separate audio inputs; the evaluation instead supplied history as text or concatenated audio.Source: §A.1.4 · p. 16
A.1.5 Human Evaluation Results
- Human ratings showed better recall for system than user utterances and for text-based than speech-based models, while RAG ranged from slightly better to slightly worse than baseline.The evaluation used 180 participants and the same 5-point scale as the GPT Score; the authors note some measurement noise but say overall trends remained consistent with the main text.Source: §A.1.5 · p. 16, Table 8 · p. 17, Figure 7 · p. 17
A.1.6 Analysis on Additional Models
- Additional-model results reaffirmed that SLAM-Omni underperformed text-based counterparts and Moshi recalled user speech significantly worse than model speech (p < 0.01).Both models scored below those analyzed in the main paper; SLAM-Omni’s 0.5B backbone is identified as the likeliest cause of its weak recall, while Moshi’s interaction and fixed-window generation may impair performance.Source: Table 9 · p. 18, §A.1.6 · p. 18
A.1.7 Analysis on Additional Dataset
- On SpokenWOZ, the main trends persisted—speech models particularly struggled with past user utterances and RAG had minimal impact—but MiniCPM-o and GLM-4-Voice showed opposite trends from earlier findings.The authors treat these results as reference-only because ASR transcripts diverged from the audio, the source audio was low quality, and Table 10 showed high WER.Source: §A.1.7 · pp. 18–19, Table 10 · p. 19
A.1.8 Analysis on Retrieval Prompts
- The main RAG prompt minimized performance degradation relative to alternatives, although outcomes varied significantly by prompt and other prompts showed similar declines.Together with results showing gains from providing supporting utterances, the analysis supports the chosen template’s effectiveness while leaving further prompt and augmentation design as an open direction.Source: §A.1.8 · p. 20, Table 11 · p. 20
A.1.9 Failure Cases
- Common failures included saying information was not recalled, retrieving the wrong utterance, answering from intrinsic knowledge, and combining multiple error types.Examples include an answer based on unrelated Lion King knowledge and a hallucinated “Oklahoma!” reference unrelated to the conversation.Source: §A.1.9 · p. 20
A.3.1 Prompt for ContextDialog
- ContextDialog prompts generate questions, answers, and supporting utterances that target a single relevant past utterance and favor detailed answers over yes/no responses.The benchmark also requires questions to use information mentioned only once and provides the supporting utterance as a reference for evaluating augmented generation.Source: §A.3.1 · pp. 21–22
A.4 Evaluation
- On 10,448 samples, gpt-4o-mini and gpt-4o scores had a Pearson correlation of 0.8787 (p < 0.01), supporting gpt-4o-mini’s use for reported GPT Scores.The comparison covered spoken-response performance for the four voice interaction models.Source: §A.4 · p. 22
Failure Cases
- Text QA generation requires a unique fact stated once by the system, a natural confirmation question, and an exact supporting line from the conversation’s first half.The output is specified in English and must include the question, answer, and evidence line.Source: Failure Cases · pp. 24–25
- The validation prompt determines whether the entire answer is fully deducible from the conversation, while the evaluation prompt assesses a voice assistant’s performance in a multi-turn speech interaction scenario.Figure 11 presents the validation prompt template, and Figure 15 presents the template for evaluating a voice-interaction model in a multi-turn scenario.Source: Failure Cases · pp. 26, 31, Figure 11 · p. 26, Figure 15 · p. 31
- Spoken QA refinement standardizes punctuation and spacing, converts numbers and symbols into speech-friendly forms, and preserves interjections, fillers, and short repetitions.The prompt asks for context-sensitive spoken forms for items such as currency, phone numbers, addresses, and postal codes.Source: Failure Cases · pp. 27–28