Source-linked AI summary
MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
He Zhang, Wenqian Cui, Haoning Xu, Xiaohui Li, Lei Zhu, Haoli Bai, Shaohua Ma, Irwin King
TL;DR
Existing FD-SLM benchmarks largely focus on single-round interactions and omit the turn-boundary, context-consistency, and broader evaluation challenges of multi-round communication. MTR-DuplexBench addresses these gaps through turn segmentation and comprehensive assessment across four dimensions, while experiments show that current FD-SLMs struggle to maintain consistent performance across rounds and dimensions.
Problem
Existing benchmarks primarily evaluate single-round FD-SLM interactions and often focus on conversational features, leaving multi-round turn boundaries, context consistency, instruction following, and safety insufficiently covered.
Method
MTR-DuplexBench segments continuous full-duplex dialogues into discrete turns and evaluates conversational features, dialogue quality, instruction following, and safety.
Results
Current FD-SLMs struggle to maintain performance consistently across multiple rounds and evaluation dimensions.
Takeaways & Limitations
The benchmark provides a comprehensive framework for assessing FD-SLMs in multi-round conversational scenarios.
Takeaways & Limitations
The benchmark relies on natural and synthetic datasets that may not capture the diversity and unpredictability of real-world conversations, and focuses primarily on English speech.
Abstract
from arXiv · showhide
Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench
1 Introduction
Existing FD-SLM benchmarks mainly evaluate single-round conversational behavior, leaving multi-round turn boundaries, context consistency, and broader capabilities insufficiently assessed. MTR-DuplexBench addresses these gaps with turn-by-turn segmentation and evaluation across conversational features, dialogue quality, instruction following, and safety.
- FD-SLMs support simultaneous listening and speaking, enabling interruptions and backchannels during real-time conversations.
- Existing benchmarks primarily assess single-round interactions, although real-world conversations commonly unfold across multiple rounds.
- Full-duplex dialogues lack clear speaking-turn boundaries, making response quality difficult to evaluate for a specific round.
- Model-generated responses can diverge from ground-truth dialogue history, producing context mismatches that reduce evaluation reliability.
- Existing benchmarks also overlook instruction following and safety alongside conversational features.
- MTR-DuplexBench segments continuous full-duplex dialogues into discrete turns and evaluates conversational features, dialogue quality, instruction following, and safety.
2 Related Works
SLMs are categorized by whether listening and speaking occur sequentially or simultaneously, while existing full-duplex benchmarks provide incomplete multi-round coverage. MTR-DuplexBench extends evaluation toward diverse, turn-level multi-round assessment.
- HD-SLMs listen to a complete user query before generating the corresponding spoken response in turn-by-turn interaction.
- FD-SLMs listen and speak simultaneously to support interruptions and backchannels, and include cascaded architectures that condition behavior on dialogue states.
- Full-Duplex-Bench and its v1.5 extension mainly evaluate one round, including pause handling, backchanneling, turn taking, interruptions, and overlapping speech.
- FD-Bench uses up to five rounds but primarily targets interruptions with synthetic speech and emphasizes overall response quality over individual-round performance.
3 MTR-DuplexBench
MTR-DuplexBench combines turn segmentation with a benchmark spanning conversational features, dialogue quality, instruction following, and safety. Its pipeline supports turn-level analysis of both synthetic feature scenarios and natural spoken dialogues.
- Turn Segmentation: The benchmark segments continuous full-duplex dialogues into discrete turns to enable turn-by-turn evaluation.
- Turn Segmentation: The segmentation pipeline extracts speech transcripts and timestamps, uses GPT-4o to identify user-turn boundaries, groups candidate turns, and resolves overlaps.
- Turn Segmentation: For each user turn, the assistant responds from the current turn’s start until the next user turn, while previous assistant responses use ground-truth speech to prevent context mismatch.
- Evaluation Framework: MTR-DuplexBench evaluates conversational features, dialogue quality, instruction following, and safety across separate evaluation dimensions.
- Conversational Features: Conversational-feature evaluation adapts smooth turn-taking, interruption, pause handling, background speech, and backchanneling to multi-round interactions.
- Conversational Features: Feature tests use synthetic speech from 200 ten-round conversations and measure per-round success, response latency, and backchannel frequency.
- Dialogue Quality: Dialogue-quality evaluation uses 200 natural 120-second Candor dialogues, segmented into rounds and scored for meaningfulness and coherence.
4 Experiments
The experiments evaluate FD-SLMs across multi-round conversational features, dialogue quality, instruction following, and safety. Results show degradation with more rounds or combined features, while latency and model design strongly affect interaction quality.
- Experimental Setup: The evaluation covers conversational features, dialogue quality, instruction following, and safety across multi-round interactions.The experiments include baseline FD-SLMs, an HD-SLM, and a cascaded approach.
- Conversational Features: Ten-round single-feature tests show that success rates decrease and latency increases as interaction rounds accumulate.The findings hold across all four conversational features, making first-package latency insufficient for fair latency assessment.
- Conversational Features: Half-duplex and cascaded systems handle most features effectively but fail under background speech, while cascading three modules produces extremely high latency.The half-duplex systems stop speaking when user input is detected, whereas the cascaded pipeline may require custom optimization to reduce latency.
- Conversational Features: Performance consistently degrades as conversational features are combined, with increasingly complex scenarios posing greater challenges.The combinations progress from smooth turn-taking to interruption, pause handling, and background speech across ten rounds.
- Dialogue Quality: Dialogue meaningfulness is lower for stronger duplex capability: Moshi scores 3.13, Freeze-Omni 3.48, and VocalNet 3.96.The cascaded approach receives the lowest GPT score because high latency limits response completion, with roughly 10 versus 20+ words per response.
- Instruction Following & Safety: Moshi performs worst on instruction following and uniquely deteriorates across rounds, whereas all baselines maintain safety performance with refusal rates at or above 90%.Interruption-based evaluations show no significant performance difference from smooth turn-taking scenarios.
5 Conclusion
MTR-DuplexBench evaluates FD-SLMs in multi-round conversations using turn segmentation and four evaluation dimensions. Experiments show that existing models struggle to maintain performance consistently, motivating more robust FD-SLM development.
- Conclusion: MTR-DuplexBench comprehensively evaluates FD-SLMs in multi-round conversational scenarios.The benchmark addresses multi-round assessment rather than only single-round interaction.
- Conclusion: Its turn segmentation methodology addresses blurred turn boundaries and context inconsistency for dialogue-quality evaluation.Continuous full-duplex dialogues are converted into discrete turns for assessment.
- Conclusion: The benchmark covers dialogue quality, conversational features, instruction following, and safety.This range supports broader assessment of FD-SLM capabilities across multiple dimensions.
- Conclusion: Existing FD-SLMs struggle to maintain performance across evaluation dimensions in multi-round interactions.The conclusion identifies this pattern as evidence for developing more robust models.
Limitations
MTR-DuplexBench has methodological and scope limitations involving evaluation-design variance, dataset realism, language coverage, and omitted interaction dynamics.
- The benchmark combines natural and synthetic datasets, which may not fully capture the diversity and unpredictability of real-world conversations.
- The benchmark focuses primarily on English speech interactions, limiting applicability to multilingual or cross-lingual FD-SLMs.
- Its conversational-feature design follows prior full-duplex benchmarks for comparability, so certain dynamics such as user backchannels are not explicitly evaluated.
- Teacher-forced dialogue-quality evaluation prevents context mismatch but creates a gap from the model’s true unguided inference trajectory.
- Conversational-feature, instruction-following, and safety evaluations avoid teacher forcing and instead let models autoregressively build on past outputs.
A Formal Definitions of Evaluation Pipelines and Metrics for Conversational Feature Assessment
The section formalizes the evaluation pipelines and metrics for conversational features by defining turn-level notation and feature-specific success conditions. It covers smooth turn-taking, interruption, pause handling, background speech, and turn-boundary segmentation evaluation.
- A Formal Definitions of Evaluation Pipelines and Metrics for Conversational Feature Assessment: Each conversational round combines one user turn and one assistant turn, with timestamps, utterance content, and transcription lengths defined for both speakers.These notations support subsequent metric definitions and calculations.
- A Formal Definitions of Evaluation Pipelines and Metrics for Conversational Feature Assessment: Moshi receives 15 seconds of manually inserted user-channel silence to prompt response termination, whereas other models detect response endings autonomously.This implementation difference applies specifically to determining when the model has finished speaking.
- A.1 Smooth Turn-Taking: Smooth turn-taking evaluates whether the model takes over after the user stops speaking, using Takeover to measure whether it responds rather than producing silence or backchannel.Backchannel is defined as speech shorter than 1 second and containing fewer than two words.
- A.1 Smooth Turn-Taking: The improved smooth-turn-taking success criterion additionally requires the model to remain silent while the user is speaking, preventing false positives from responses delivered entirely during the user’s turn.The original metric does not account for whether model speech begins before or after user speech.
- A.2 Interruption: Interruption evaluation tests both stopping during user interruption and resuming afterward by truncating a previous response and inserting the current user input.Truncation is randomly placed between one-quarter and one-half of the previous response length.
- A.3 Pause Handling: Pause handling inserts a brief pause between two consecutive user words and assigns success when the model remains silent throughout that pause.Ellipses during CosyVoice2 synthesis create examples such as “Today is ... a sunny day”.
- A.4 Background Speech: Background-speech evaluation assumes the model’s state remains unchanged when generated background speech is inserted into the user channel during the model’s response.The response is randomly truncated between one-quarter and one-half of its length before the background-speech scenario is simulated.
- A.4 Background Speech: Table 8 presents human evaluation results for GPT-4o turn-boundary segmentation.The table is associated with the background-speech subsection in the supplied section.
B Human Evaluation of GPT-4o Turn Boundary Segmentation
A human evaluation assessed GPT-4o’s turn segmentation using boundary correctness and omitted-turn counts. The results indicate fairly accurate segmentation, while also showing that omitted-turn judgments are subjective.
- Five annotators evaluated 20 GPT-4o-segmented turns using boundary correctness and omitted-turn criteria.Boundary correctness measures whether a turn boundary is accurate; omitted turns count valid smaller turns that were missed.
- Boundary correctness scores whether each assigned boundary accurately marks the current turn.
- Omitted-turn counts measure how many valid smaller turns GPT-4o failed to segment, including their time boundaries.
- The evaluation metrics collectively indicate that GPT-4o turn segmentation is fairly accurate.
- All human-identified omitted turns existed in the raw GPT-4o segmentation before majority voting, clustering, and filtering.
- Each omitted turn was reported by roughly two of five annotators, indicating limited human consensus about conversational boundaries.
C Robustness to Accent Variations
The benchmark includes a preliminary accent-robustness investigation using matched American and British speech. Moshi shows more statistically significant feature variations than Freeze-Omni across the tested accents.
- The study preliminarily investigates whether accent changes produce statistically significant performance differences in multi-round FD-SLM evaluation.The broader motivation includes robustness to accents, environmental noise, and speech rates, though exhaustive acoustic testing requires dedicated benchmarks.
- Identical multi-round dialogues were synthesized with American and British male voices and evaluated on four conversational features.The features were smooth turn-taking, interruption, pause handling, and background speech handling.
- Wilcoxon Signed-Rank Tests were used to assess the statistical significance of performance differences between the two accents.
- Moshi exhibited more statistically significant metric variations across features than Freeze-Omni.
- The results suggest Moshi is currently less robust to input-accent shifts than Freeze-Omni’s cascaded full-duplex approach.
D Detailed Computing Infrastructure and Experiment Specification
Experiments used PyTorch 2.5.0 and Transformers 4.44.2, with the complete four-task evaluation requiring roughly 11 hours on one GPU and each experiment run once.
- The experiments used PyTorch 2.5.0 and Transformers 4.44.2.
- The complete evaluation across all four tasks required roughly 11 hours on one GPU.
- Each experiment was run once.
E Potential Risks
The benchmark acknowledges risks from GPT-4o-generated dialogue content, including personally identifying information, biased stereotypes, and offensive language. Generated scripts and audio were manually inspected to exclude sensitive or inappropriate samples.
- Potential Risks: Synthetic multi-round dialogues generated by GPT-4o may contain personally identifying information, biased stereotypes, or offensive language.
- Potential Risks: The authors manually inspected all generated dialogue scripts and associated audio before using them.
- Potential Risks: The inspection ensured that generated dialogues contained no sensitive personal details or inappropriate content.