Source-linked AI summary
Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, Hung-yi Lee
TL;DR
Full-duplex spoken dialogue models require evaluation of real-time behaviors that turn-based benchmarks largely overlook. Full-Duplex-Bench introduces an automated benchmark covering pause handling, backchanneling, turn-taking, and interruption management, and experiments reveal distinct model behaviors and improvement areas. The released data and behavior-specific metrics provide a practical evaluation foundation, while current analysis remains limited to English and does not connect behaviors to human preferences.
Problem
Existing spoken dialogue benchmarks largely assume turn-based interactions, leaving real-time behaviors of full-duplex models under-explored and difficult to compare consistently.
Method
Full-Duplex-Bench uses scenario-driven tests and automatic metrics to evaluate pause handling, backchanneling, turn-taking, and user interruption management.
Results
Experiments reveal distinct model features and areas for improvement across the benchmark’s interaction dimensions.
Takeaways & Limitations
The released benchmark, data, and behavior-specific metrics provide a practical foundation for systematic and reproducible full-duplex dialogue evaluation.
Takeaways & Limitations
The framework does not yet link behaviors to human preferences, and the present analysis is limited to English.
Abstract
from arXiv · showhide
Spoken dialogue modeling poses challenges beyond text-based language modeling, requiring real-time interaction, turn-taking, and backchanneling. While most Spoken Dialogue Models (SDMs) operate in half-duplex mode-processing one turn at a time - emerging full-duplex SDMs can listen and speak simultaneously, enabling more natural conversations. However, current evaluations remain limited, focusing mainly on turn-based metrics or coarse corpus-level analyses. To address this, we introduce Full-Duplex-Bench, a benchmark that systematically evaluates key interactive behaviors: pause handling, backchanneling, turn-taking, and interruption management. Our framework uses automatic metrics for consistent, reproducible assessment and provides a fair, fast evaluation setup. By releasing our benchmark and code, we aim to advance spoken dialogue modeling and foster the development of more natural and engaging SDMs.
I. INTRODUCTION
Full-duplex SDMs listen and speak simultaneously, but existing evaluations largely miss the real-time interaction behaviors that distinguish them. Full-Duplex-Bench addresses this gap with an automated, reproducible framework for evaluating four turn-taking dimensions.
- Full-duplex SDMs process continuous audio while listening and speaking simultaneously, modeling overlap, pauses, background noise, and backchanneling.
- As full-duplex systems proliferate, fair and open evaluation is needed to guide spoken dialogue research.
- Existing benchmarks emphasize content, instruction following, or paralinguistic abilities while mostly assuming turn-based interactions.
- Recent timing evaluations face interpretability, dataset-specific ground truth, generalizability, and reproducibility limitations.
- Full-Duplex-Bench evaluates pause handling, backchanneling, turn-taking, and user interruption management with objective, rapid, automated metrics.
- The benchmark provides an open toolkit for consistent cross-model comparisons and large-scale evaluation, while leaving behavioral desirability to developers’ application requirements.
A. Full-duplex Spoken Dialogue Models
Full-duplex spoken dialogue models include transparent systems with reproducible implementations or descriptions and closed-source commercial systems available mainly through demonstrations. Their diversity strengthens the need for standardized evaluation across architectures and access models.
- Transparent Models: Transparent models provide open implementations or detailed technical descriptions, enabling reproducibility and deeper analysis.
- Closed-Source Commercial Systems: Closed-source commercial systems expose full-duplex capabilities through public demonstrations but lack open-source implementations and architectural details.
- Standardized benchmarks are needed to compare interaction quality across transparent and closed-source full-duplex systems.
III. FULL-DUPLEX-BENCH FRAMEWORK
The framework defines operational behaviors and metrics for analyzing full-duplex interaction. Its backchanneling definition uses short, sparse utterances, while Takeover Rate averages a binary takeover variable across dialogue turns.
- Backchanneling consists of short listener utterances that signal engagement while another person is speaking.
- The benchmark classifies speech as backchanneling when it lasts less than 1 second and contains fewer than two words.
- The backchanneling criteria aim to preserve speaking pace and avoid interrupting the current speaker’s turn.
- Takeover Rate (TOR) is the average binary takeover value across the dataset’s dialogue turns.
A. Overview
Full-Duplex-Bench simulates real-time user interactions, records model speech, aligns it with word-level transcripts, and applies dimension-specific metrics for automated evaluation.
- The framework feeds a unified input.wav stimulus to an SDM and collects its output.wav speech response.
- An ASR model produces word-level, time-aligned output.json transcriptions for evaluating the model response.
- Dedicated metrics assess each interaction dimension after targeted test samples are processed.
B. Evaluation Dimensions
Full-Duplex-Bench evaluates four real-time interaction behaviors: pause handling, backchanneling, smooth turn-taking, and user interruption management. It uses takeover, frequency, divergence, response-quality, and latency measures to assess these behaviors.
- Pause Handling: Pause handling measures whether models avoid taking over while the speaker is still holding the turn.A lower Takeover Rate indicates better pause management.
- Smooth Turn-taking: Smooth turn-taking assesses whether the model takes the turn at an appropriate time during conversation.The figure identifies smooth turn-taking as one of the benchmark’s four evaluation dimensions.
- Backchanneling: Backchanneling evaluates whether models provide timely, appropriately frequent acknowledgments without interrupting the speaker.The benchmark uses TOR, backchannel frequency, and Jensen–Shannon Divergence against human timing.
- Backchanneling: Jensen–Shannon Divergence measures similarity between model-predicted and human backchannel timing distributions, ranging from 0 for perfect alignment to 1 for complete divergence.The metric is calculated only when the model does not take over the turn.
- User Interruption: User interruption management measures whether the model responds to sudden user input with appropriate timing, coherence, relevance, and adaptability.Metrics include TOR, GPT-4o response-quality scores from 0 to 5, and latency after interruption.
C. Data Curation
The benchmark combines real conversational data with controlled synthetic events to curate task-specific evaluation samples. Candor supports pause and turn-taking evaluation, while ICC supplies human backchannel timing distributions.
- Candor: Candor provides 850 hours of two-channel spontaneous conversations for pause handling and smooth turn-taking.The dataset contains open-ended conversations with separate audio channels.
- Candor: Pause-handling samples select single-speaker turns containing internal pauses between 0.4 and 1.0 seconds.Automatic filtering is followed by manual review using dual-channel evidence.
- ICC: ICC contributes 28.33 minutes of informal American English dialogue with responses from 118 native speakers for backchannel evaluation.Responses are segmented into 200-ms windows and normalized to produce human backchannel distribution Q.
- Synthetic Data: Synthetic data uses ChatTTS’s [uv_break] tag to insert controlled pauses and includes 200 user-interruption samples and 137 pause-handling samples.The controlled events support robustness evaluation for interruptions and intra-turn pauses.
IV. MODELS UNDER EVALUATION
The evaluation includes publicly available speech-to-speech systems and Gemini Live as a representative commercial model. The selected models span textless, real-time, cascaded, and API-based full-duplex architectures.
- Model Selection: The benchmark evaluates dGSLM, Moshi, Freeze-Omni, and Gemini Live using available implementations or the official commercial API.Complete speech-to-speech checkpoints are not publicly released for many models.
- Transparent Models: dGSLM is a textless speech-to-speech model that generates conversations directly from audio using HuBERT+k-means, a dual-tower Transformer, and HiFi-GAN.The authors adapt its official implementation for live interaction.
- Transparent Models: Moshi combines a 7B LLM with a neural codec and supports overlapping speech and interruptions through a multi-stream architecture.Its Inner Monologue step is intended to improve fluency.
- Transparent Models: Freeze-Omni uses a cascaded frozen-LLM pipeline with VAD-triggered encoding and a classification head that predicts dialogue states for turn-taking control.Parallel modules support streaming, speaking, and monitoring.
- Commercial Model: Gemini Live streams 16 kHz PCM-16 audio in 30-ms chunks to the Gemini Live API, where server-side VAD triggers responses.A new session begins after each model reply, and outputs preserve silent regions of the input duration.
V. RESULTS
Results across the four dimensions show distinct trade-offs among interruption avoidance, response timing, backchannel alignment, and interruption handling. Commercial Gemini Live generally performs strongly, while model architecture influences pause and interruption behavior.
- Pause Handling: All three open-source SDMs show high Takeover Rates during speaker pauses, with dGSLM and Moshi interrupting more often than Freeze-Omni.Freeze-Omni’s dedicated speaking/listening-state prediction module is associated with lower TOR.
- Backchanneling: Gemini Live achieves the lowest TOR among evaluated models and has the best JSD for backchannel timing.On Candor, it is more likely to take over than on synthetic data.
- Turn-taking Latency: dGSLM and Moshi respond in around 0.3 seconds while exhibiting high takeover rates, whereas Freeze-Omni has higher latency and lower TOR.The cascaded architecture may miss turn-end opportunities when detecting when to take over.
- User Interruption: Freeze-Omni handles user interruptions with significantly higher contextual relevance while maintaining acceptable latency.Its model-as-a-server strategy uses a pool of models to manage user barge-ins.
- User Interruption: Moshi responds promptly but obtains a GPT-4o score of 0.765, while end-to-end systems struggle with interruption coherence.Gemini Live shows comparable interruption performance with relatively better TOR and latency but a slightly lower GPT-4o score than Freeze-Omni.
VI. CONCLUSION
Full-Duplex-Bench evaluates pause handling, backchanneling, smooth turn-taking, and user interruption management in full-duplex spoken dialogue models. Its automatic, behavior-specific metrics support systematic and reproducible evaluation while revealing model differences and improvement areas.
- The benchmark targets four interaction dimensions: pause handling, backchanneling, smooth turn-taking, and user interruption management.
- Automatic metrics are tailored to real-time interaction for systematic and reproducible evaluation.
- Experiments reveal distinct model features and highlight areas for improvement.
- Released data and behavior-specific metrics provide a practical foundation for evaluating full-duplex spoken dialogue systems.
VII. LIMITATION AND FUTURE WORK
The framework does not yet connect described behaviors with human preferences, and its present analysis is limited to English. Future work should incorporate human judgment and extend evaluation across languages.
- The framework does not link described behaviors to human preferences, so users must define desirable behavior for their goals.
- The present analysis is limited to English, leaving cross-linguistic generality for future work.