Source-linked AI summary
Full-Duplex Interaction in Spoken Dialogue Systems: A Comprehensive Study from the ICASSP 2026 HumDial Challenge
Chengyou Wang, Hongfei Xue, Guojian Li, Zhixian Zhao, Shuiyuan Wang, Shuai Wang, Xin Xu, Hui Bu, Lei Xie
TL;DR
Many speech dialogue systems rely on rigid turn-taking, while full-duplex research lacks sufficient datasets and evaluation methods for realistic interaction. The paper releases a dual-channel human-recorded dataset, HumDial-FDBench, and a public leaderboard. In reported results, Gemini leads interruption handling, while Freeze-Omni leads rejection handling, and Gemini has the lowest latency.
Problem
Many speech dialogue systems use rigid turn-taking, and full-duplex research remains constrained by limitations in available datasets and evaluation methodologies.
Source: §1 · p. 1Method
Results
Gemini scored 79.8 in interruption handling, Freeze-Omni led rejection handling with 50.2, and Gemini had the lowest latency at 1.301 seconds.
Source: §5.2 · pp. 3–4Takeaways & Limitations
The dataset, benchmark, and leaderboard provide shared resources for transparent, reproducible study of full-duplex spoken dialogue systems.
Source: §1 · p. 1, §7 · p. 4Real-time fluency remains challenged by latency under uncertain acoustics, overlapping speakers, and background noise, which can impair speaker discrimination and trigger false activations or missed detections.
Source: §5.2 · p. 4
Abstract
from arXiv · showhide
Full-duplex interaction, where speakers and listeners converse simultaneously, is a key element of human communication often missing from traditional spoken dialogue systems. These systems, based on rigid turn-taking paradigms, struggle to respond naturally in dynamic conversations. The Full-Duplex Interaction Track of ICASSP 2026 Human-like Spoken Dialogue Systems Challenge (HumDial Challenge) aims to advance the evaluation of full-duplex systems by offering a framework for handling real-time interruptions, speech overlap, and dynamic turn negotiation. We introduce a comprehensive benchmark for full-duplex spoken dialogue systems, built from the HumDial Challenge. We release a high-quality dual-channel dataset of real human-recorded conversations, capturing interruptions, overlapping speech, and feedback mechanisms. This dataset forms the basis for the HumDial-FDBench benchmark, which assesses a system's ability to handle interruptions while maintaining conversational flow. Additionally, we create a public leaderboard to compare the performance of open-source and proprietary models, promoting transparent, reproducible evaluation. These resources support the development of more responsive, adaptive, and human-like dialogue systems.
1. Introduction
Human dialogue uses simultaneous speaking and listening, while many speech dialogue systems wait for detected speech completion or turn boundaries. The HumDial Challenge introduces a dual-channel dataset, benchmark, and leaderboard for evaluating full-duplex interaction.
1. Introduction
- Many existing speech dialogue systems use rigid turn-taking, disrupting continuity and limiting responsiveness to unexpected inputs or contextual changes.Human dialogue instead involves simultaneous speaking and listening, with conversational control regulated dynamically through cues and feedback.Source: §1 · p. 1
- Available datasets and evaluation methods often omit realistic overlap and interruption behavior, while common metrics overlook interactional dimensions such as response timing and resuming after disruption.The resulting misalignment limits fair system comparison and slows progress toward deployable full-duplex dialogue models.Source: §1 · p. 1
- The challenge introduces a real human-recorded dual-channel dataset, HumDial-FDBench for interruption handling and conversational continuity, and a public leaderboard comparing open-source, proprietary, and challenge systems.The resources are intended to support research toward more responsive and human-like conversational systems.Source: §1 · pp. 1–2
2. Related Work
Prior work includes cascaded and end-to-end spoken dialogue architectures, but a unified, stable full-duplex modeling paradigm remains absent. Dialogue benchmarks have expanded toward overlap and longer interactions, while fine-grained assessment of complex behaviors remains limited.
Full-Duplex Spoken Interaction
- Cascaded systems chain ASR, LLM, and TTS modules, while end-to-end systems offer a more streamlined alternative; both appear in mainstream spoken dialogue architectures.Cascaded designs can incur high end-to-end latency and progressive error accumulation, and end-to-end systems often still use interruption-related turn-decision strategies.Source: §2 · p. 2
- Most end-to-end systems still rely on additional interruption-related modules or strategies, and a unified, stable paradigm for full-duplex interaction remains absent.Examples named include voice activity detection, TEN-Turn, and Smart-Turn.Source: §2 · p. 2
Benchmarks for Dialogue Systems
- Earlier benchmarks often focus on single-turn or half-duplex settings; Full-duplex bench centers overlap handling, and FD-bench extends evaluation to complex, longer multi-turn scenarios.Benchmarks also provide limited fine-grained measures for subtle rejection behavior and feedback signals.Source: §2 · p. 2
3. Released Dataset
The released dataset covers interruption and rejection scenarios, and is constructed by combining generated scripts with human performances. It contains more than 100 hours of bilingual interactive speech with annotations and train–validation–test splits.
3.1. Scenarios
- The dataset comprises two major categories, Interruption and Rejection, spanning eight scenarios.Table 1 reports instance counts per split.Source: §3.1 · p. 2, Table 1 · p. 2
- Interruption scenarios test adapting an ongoing response to follow-up questions, disagreement, repetition requests, topic switches, or requests to stop speaking.After a stop request, the model should cease output and remain ready to resume upon further prompting.Source: §3.1 · p. 2
- Rejection scenarios test withholding responses to non-actionable speech, including backchannels, pauses, third-party speech, and utterances directed to others.The model should wait for intent after pauses and respond only to the target user amid third-party speech.Source: §3.1 · p. 2
3.2. Data construction
- Scripts generated by DeepSeek embed interaction cues, then professional actors record them to preserve natural overlap timing, prosody, and interaction rhythm.The dataset contains more than 100 hours of Chinese and English speech, with detailed annotations and train–validation–test splits.Source: §3.2 · p. 2
4. HumDial-FDBench
HumDial-FDBench extends an existing evaluation protocol to assess full-duplex systems across behavior and latency. Its metrics distinguish interruption from rejection behavior and combine behavioral scores with a latency score.
4. HumDial-FDBench
- The benchmark extends Full-Duplex-Bench v1.5 to support more complex interaction scenarios and a broader assessment of full-duplex dialogue systems.For behavior evaluation, time-aligned ASR transcripts are categorized into Respond, Resume, Uncertain, and Unknown.Source: §4 · p. 2, §4.1 · pp. 2–3
4.2. Evaluation metrics
- Interruption handling counts only Respond as correct, while rejection uses Resume for specified cases and timestamp-based criteria for pauses and third-party speech.Rejection outcomes are binary; behavior evaluation categorizes responses as Respond, Resume, Uncertain, or Unknown.Source: §4.2 · p. 3, §4.1 · p. 3
- First response latency measures the interval from the end of the first user question to the start of the first model response, alongside stop and response latency.Latency metrics are reported in seconds; rejection latency is measured for successful rejections from the end of the true user question to the first valid model response.Source: §4.2 · p. 3
4.3. Overall Score
- The total score weights interruption and rejection scores at 0.4 each and latency score at 0.2.The latency score averages stop, response, and first response latency across their sub-scenarios, then applies logarithmic normalization using the minimum observed latency and a baseline latency of 60.Source: §4.3 · p. 3
5. Leaderboard
The evaluation compares open-source and closed-source dialogue models using an Easy-Turn and Osum-EChat baseline. Results differ across interruption handling, rejection handling, and latency, while the discussion identifies remaining challenges in difficult acoustic settings.
5.1. Model Evaluation
- The evaluation includes open-source and closed-source models and uses Easy-Turn integrated with Osum-EChat as its baseline.Examples include Freeze-Omni, Moshi, Osum-EChat, and Gemini 2.5.Source: §5.1 · p. 3
5.2. Experimental Results
- 79.8: Gemini's interruption-handling score exceeds Moshi's 35.4 and Freeze-Omni's 29.6.Table 2 reports results for models on HumDial-FDBench.Source: §5.2 · p. 3, Table 2 · p. 3
- 50.2: Freeze-Omni leads rejection handling, ahead of Gemini at 36.5 and Moshi at 22.8.Source: §5.2 · pp. 3–4
- 1.301 seconds: Gemini has the lowest latency, attributed to its VAD-based interruption strategy.Source: §5.2 · p. 4
- Latency control, multi-speaker speaker identification, and robustness to background noise remain challenges under complex conditions.Overlapping third-party speech can complicate speaker identification, while transient noise can cause false activations or missed detections.Source: §5.2 · p. 4
6. Analysis
Challenge submissions vary in architecture, turn-taking strategy, and training approach. The analysis describes trade-offs among modular control, end-to-end capabilities, specialized decision components, and training resources.
6.1. System Architecture
- Most submissions use cascaded or semi-cascaded pipelines, while some use end-to-end multimodal models.Cascaded pipelines typically separate turn decisions from response generation; semi-cascaded systems combine independent perception with unified decision-making, and end-to-end systems unify perception and generation.Source: §6.1 · p. 4
6.2. Turn-Taking Strategy
- Turn-taking decisions must distinguish interruptions, backchannels, and rejection cases such as noise, side speech, or third-party utterances.Source: §6.2 · p. 4
- Heuristic systems use handcrafted lexical, semantic, or timing rules, which need little training but can struggle in complex acoustic conditions.RhythmSense uses staged arbitration, moving from lexical cues to semantic matching and then conservative rules when evidence is weak.Source: §6.2 · p. 4
- Specialized lightweight classifiers filter irrelevant noise before LLM use, reducing the LLM's decision burden.Examples include a Whisper-tiny-based four-class classifier, TinyBERT text classification with rules, and a classifier built on discrete audio representations.Source: §6.2 · p. 4
- Delegating turn decisions to a multimodal LLM can help with ambiguous pauses but may be less sensitive to purely acoustic cues than dedicated acoustic classifiers.Examples include Qwen2.5 producing text and classification labels and Qwen3-Omni emitting decision tokens.Source: §6.2 · p. 4
6.3. Training Strategy
- Training approaches range from extensive model or adapter fine-tuning and classifier training to frozen pipelines using prompts, modular design, or external decision logic.Extensive training can improve rejection accuracy with diverse backchannel and third-party patterns; frozen pipelines can use open-source multimodal backbones with minimal training when perception and turn control are well designed.Source: §6.3 · p. 4
7. Conclusion
The study releases a dual-channel dataset and introduces HumDial-FDBench to evaluate interruption handling and conversational continuity during concurrent listening and generation. It also establishes a public leaderboard and aims to support more responsive, human-like dialogue systems.
7. Conclusion
- The study introduces HumDial-FDBench to assess interruption handling and conversational continuity during concurrent listening and generation.Source: §7 · p. 4
- A public leaderboard enables transparent, reproducible comparisons of open-source and proprietary models and challenge submissions.Source: §7 · p. 4
- The authors state that these resources will contribute to developing more responsive and human-like dialogue systems.Source: §7 · p. 5
8. Generative AI Use Disclosure
Generative AI tools were used only for language refinement and editorial support. The authors state that they independently conducted the intellectual and scientific work and take responsibility for the manuscript.
8. Generative AI Use Disclosure
- Generative AI tools were used solely for language refinement and editorial support.Source: §8 · p. 5
- The tools were not involved in methodology development, experiments, results, or conclusions.Source: §8 · p. 5
- The authors independently made the intellectual contributions, take responsibility for the manuscript, and report that all authors approved its final version.Source: §8 · p. 5