Source-linked AI summary
Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models
Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, Shinji Watanabe, Hung-yi Lee
TL;DR
Overlapping speech is central to natural full-duplex dialogue but remains under-evaluated by conventional benchmarks. Full-Duplex-Bench v1.5 provides automated, scenario-controlled evaluation of five systems across behavior, timing, and prosodic adaptation, revealing a trade-off between rapid responsiveness and floor holding. The open-source framework supports reproducible comparison of overlap handling.
Problem
Existing speech benchmarks underrepresent overlapping dialogue, despite overlap being common and important for natural full-duplex interaction.
Method
Full-Duplex-Bench v1.5 streams audio to open-source and commercial systems, simulates four overlap scenarios, and measures behavior, timing, and prosodic adaptation.
Results
Across five systems, the benchmark reveals a stable trade-off between rapid responsiveness to true interruptions and floor holding that filters non-addressed overlapping speech.
Takeaways & Limitations
The benchmark provides scenario-specific metrics and reproducible comparisons for developing and evaluating robust full-duplex dialogue systems.
Takeaways & Limitations
The desired stop-latency value depends on the overlap scenario, because systems may need either rapid yielding or continued floor holding.
Abstract
from arXiv · showhide
Full-duplex spoken dialogue systems promise to transform human-machine interaction from a rigid, turn-based protocol into a fluid, natural conversation. However, the central challenge to realizing this vision, managing overlapping speech, remains critically under-evaluated. We introduce Full-Duplex-Bench v1.5, the first fully automated benchmark designed to systematically probe how models behave during speech overlap. The benchmark simulates four representative overlap scenarios: user interruption, user backchannel, talking to others, and background speech. Our framework, compatible with open-source and commercial API-based models, provides a comprehensive suite of metrics analyzing categorical dialogue behaviors, stop and response latency, and prosodic adaptation. Benchmarking five state-of-the-art agents reveals two divergent strategies: a responsive approach prioritizing rapid response to user input, and a floor-holding approach that preserves conversational flow by filtering overlapping events. Our open-source framework enables practitioners to accelerate the development of robust full-duplex systems by providing the tools for reproducible evaluation.
1. INTRODUCTION
Full-duplex dialogue aims to support natural simultaneous speaking and listening, but existing evaluation largely misses the diverse, consequential dynamics of overlapping speech. Full-Duplex-Bench v1.5 addresses this gap with automated, controlled scenarios and multidimensional analysis across five systems.
- Overlapping speech is common in natural dialogue and includes interruptions, backchannels, side conversations, and ambient speech, making graceful handling central to full-duplex interaction quality.Poor overlap handling can produce truncated responses, awkward silences, and degraded interaction quality.
- The framework evaluates categorical behavior, interaction timing, and prosodic adaptation rather than only what a model says.
- Existing benchmarks emphasize single-turn, half-duplex settings, while human evaluations are costly and corpus-level or classifier-based methods miss scenario-specific semantic appropriateness.
- Full-Duplex-Bench v1.5 is an automated benchmark that streams audio to open-weight and API-based systems across Interruption, Backchannel, Talking to Others, and Background Speech scenarios.
- Benchmarking five state-of-the-art systems reveals contrasting overlap strategies: rapid responsiveness versus floor holding that filters overlapping events.
2.1. Interaction Framework
The framework simulates live conversation by streaming prerecorded user audio while capturing model output, enabling controlled overlap evaluation across model architectures.
- The framework streams prerecorded user audio while simultaneously capturing model output to simulate a live conversation.
- Each trial introduces a controlled overlap event while the model is speaking.
- The model-agnostic design supports both open-source models and closed commercial APIs without assuming internal architecture.
2.2. Controlled Overlap Scenarios
Four overlap scenarios probe reactive turn-yielding, backchannel filtering, addressee detection, and robustness to ambient speech through controlled synthetic utterances.
- The benchmark uses four scenarios to probe distinct and essential conversational capabilities.
- User Interruption: User interruptions test rapid floor yielding and semantic repair using 200 synthesized utterances.The interruptions use the same speaker voice as the initial query without acoustic channel differences.
- User Backchannel: Backchannels test whether models filter non-floor-taking affirmations using 99 synthesized utterances.Examples include “yeah,” “right,” and “mm-hmm,” generated from a curated list with the same speaker voice.
- Talking to Others: Talking-to-others scenarios test addressee detection and graceful resumption with 100 semantically directed utterances.The utterances are acoustically processed to simulate speech directed away from the device.
- Background Speech: Background-speech scenarios test robustness to unrelated ambient speech using 100 utterances from a different speaker voice.The speech is simulated as distant environmental interference through volume reduction, filtering, and echo.
2.3. Evaluation Metrics
The evaluation combines categorical post-overlap behavior with stop and response latency measures, whose desired profiles depend on the overlap scenario.
- Post-overlap responses are classified as Respond, Resume, Uncertain, or Unknown to characterize the model’s semantic strategy.The categories distinguish addressing the overlap, continuing the prior utterance, expressing confusion, and producing an irrelevant response or silence.
- Stop latency measures the interval from overlapping user-speech onset to when the model stops speaking.
- Response latency measures the interval from the end of overlapping speech to the model’s next utterance.
- Table 1 summarizes the desired categorical behavior and latency profile for each overlap scenario.
- Stop latency indicates overlap awareness, with shorter values favoring rapid yielding and longer values sometimes appropriate for holding the floor.Response latency instead represents the post-overlap gap and is ideally minimized across scenarios.
3. EVALUATION SETUP
The evaluation benchmarks five full-duplex systems through streaming interfaces and reports behavioral distributions plus stop and response latencies across overlap scenarios.
- Systems: Five systems are evaluated through their streaming interfaces, including open-source Freeze-Omni and Moshi and commercial Gemini and Nova Sonic.The supplied setup also identifies GPT-4o as an evaluated system through the benchmark results.
- Evaluation protocol: The benchmark compares model behavior using streaming audio interactions rather than a single-turn evaluation setup.The framework supports both open-source systems and commercial APIs.
- Metrics and scenarios: Table 2 reports response-category distributions alongside average stop and response latencies for four overlap scenarios.Shaded rows identify the desired outcome for each scenario, while boldface marks best performance.
4. RESULTS
Results expose a stable trade-off between rapid responsiveness and floor holding: models that yield quickly handle interruptions better, while conservative models filter incidental speech more reliably.
- User Interruption: GPT-4o responds most strongly to interruptions (RESPOND=0.78, tstop=0.23 s), while Freeze-Omni achieves the shortest response latency (tresp=1.35 s).Gemini and Sonic often resume their prior turns and take more than two seconds to yield; Moshi responds to 0.50 of interruptions.
- User Backchannel: Sonic and Gemini best preserve the floor during backchannels (RESUME=0.98 and 0.93), whereas Moshi produces UNKNOWN=0.92.GPT-4o halts rapidly even for brief acknowledgments, while Freeze-Omni maintains continuity but resumes more slowly.
- Talking to Others: Gemini nearly perfectly ignores speech directed to others (RESUME=0.99), while GPT-4o responds to most such speech (RESPOND=0.91).Sonic also performs strongly with RESUME=0.90; Freeze-Omni and Moshi show mixed or uncertain control.
- Background Speech: Sonic best filters background speech (RESUME=0.98, RESPOND=0.01), whereas GPT-4o nearly always yields (RESPOND=0.93).Gemini and Freeze-Omni often respond inappropriately, and Moshi remains unstable with UNKNOWN=0.71.
- Cross-Model Patterns: Across scenarios, GPT-4o favors fast yielding, Sonic and Gemini favor floor holding, Freeze-Omni is more balanced, and Moshi is unstable outside floor-taking settings.Effective repair typically requires tresp ≤ 1.5 s, but fast responses can still reflect false RESPOND decisions.
5. PROSODIC AND QUALITY SHIFTS
The benchmark examines prosodic and quality changes when models respond to interruptions, revealing distinct re-entry strategies while largely stable predicted speech quality.
- Measurement: The analysis compares pre-overlap and post-overlap RESPOND segments using speaking rate, pitch, intensity, and predicted MOS with paired t-tests.Expected adaptation includes modest rate and pitch or energy increases without large intensity swings or MOS degradation.
- Adaptation regimes: GPT-4o and Gemini accelerate and raise pitch, while Freeze-Omni and Sonic reduce mean intensity but increase intensity variability during re-entry.These form tempo/pitch-lift and soft-but-dynamic intensity regimes, respectively.
- Adaptation regimes: Moshi combines an extreme speed-up with lower pitch, reduced pitch variability, and lower mean intensity, indicating brittle re-entry control.Its intensity variability increases despite the reduction in mean intensity.
- Quality: Predicted UTMOSv2 remains largely stable from pre- to post-overlap, with only a small decline for Sonic.The results distinguish controlled prosodic adaptation from rushed or uneven changes.
6. CONCLUSION
Full-Duplex-Bench v1.5 makes overlap handling measurable through controlled scenarios, behavior categories, and timing metrics, exposing a trade-off between responsiveness and floor holding.
- Contribution: The benchmark formalizes expected behavior and tstop and tresp across four overlap cases, enabling reproducible comparison beyond turn-based evaluation.It is fully automated and scenario-controlled.
- Main finding: Across five systems, fast yielders excel on true interruptions but over-accommodate incidental speech, whereas floor holders resist non-addressed input but delay repairs.This trade-off captures the central cross-model result.
- Implication: The open-sourced tasks, metrics, and code provide a practical yardstick for diagnosing weaknesses and tracking progress in full-duplex dialogue systems.The stated scope is overlap handling in real-time conversational AI.