Source-linked AI summary
Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts
Shuofeng Zhao, Hongwei Cai, Wenke Fan, Qingxiang Guo, Zhou Wang, Dawei Yang, Zhiyang Zhou, Yingxin Shang, Weixu Wang, Lin Yang, Shuran Zhou, Yang Song
TL;DR
Real-time spoken dialogue systems must decide whether overlapping user speech calls for yielding or continued speech, and the insertion transcript alone may not determine the right action. ECHO tests this with shared transcripts in rewritten contexts and paired action scoring. Three of four evaluated speech systems keep the floor on fewer than 13% of backchannels and achieve at most 4% PASR on interruption–backchannel pairs.
Problem
Overlapping user speech may claim the floor or be feedback or off-talk, so systems must distinguish cases that require YIELD from those that permit KEEP.
Source: §1 · p. 1Method
ECHO pairs Chinese dialogue examples that share an insertion transcript but use rewritten contexts, and PASR requires correct actions on both members of an action-flip pair.
Source: §1 · p. 2, §3.2 · p. 4Results
Three of four speech systems keep the floor on fewer than 13% of backchannels and achieve at most 4% PASR on interruption–backchannel pairs, although two exceed 90% interruption accuracy.
Source: §4 · p. 4Takeaways & Limitations
Interruption accuracy alone is insufficient to characterize balanced overlap handling; class-conditioned KEEP rates and pair-level success should also be reported.
Source: §4 · p. 4ECHO is a replayed, event-level diagnostic rather than an evaluation of closed-loop interaction quality, user adaptation, or response latency.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Real-time spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Existing benchmarks typically score events independently and may therefore assign high scores to systems with fixed action preferences rather than context-sensitive decision policies. We introduce ECHO, a paired diagnostic benchmark for Chinese turn-taking evaluation. ECHO pairs examples with the same overlap transcript but contrasting preceding multi-turn dialogue contexts, with one requiring Yield and the other Keep. It additionally includes off-talk examples for diagnosing unnecessary yielding. We introduce pair accuracy, which requires correct decisions on both members of a pair and assigns no credit to constant-action policies. Experiments on four speech systems show that three exhibit a severe over-yielding bias: they correctly keep the floor on fewer than 13% of backchannels, resulting in near-zero pairwise success rates equal or less than 4%. While the remaining system remains comparatively balanced across contexts, these findings broadly demonstrate that interruption-only evaluation can severely overestimate practical turn-taking reliability.
1. INTRODUCTION
Overlapping speech can be a floor-claiming interruption, non-floor-claiming feedback, or off-talk, so systems must choose whether to yield or continue using dialogue context. ECHO tests whether systems change that decision when the same insertion appears in rewritten contexts.
1. INTRODUCTION
- The evaluation concerns event-level YIELD/KEEP decisions under replayed trajectories, not closed-loop interaction quality, user adaptation, or response latency.The task applies across full-duplex models and modular systems with turn detectors or interruption mechanisms.Source: §1 · p. 1
- Existing evaluations often vary the overlapping utterance and preceding context together, so correct predictions do not establish that systems respond to context changes.Some non-interruptive subsets also permit lexical shortcuts, including frequent feedback characters or disjoint vocabularies.Source: §1 · p. 1
- The same utterance can be affiliative feedback, a floor-claiming addendum, or off-talk depending on the preceding dialogue.Its transcript alone does not determine whether the system should yield or continue.Source: §1 · pp. 1–2
- Across four speech systems evaluated through their supported interfaces, three yield correctly on many interruption instances but keep the floor on fewer than 13% of backchannels, with interruption–backchannel PASR at most 4%.Source: §1 · p. 2
- ECHO pairs Chinese dialogues with a shared insertion transcript and rewritten contexts that assign different interaction roles and target actions.The benchmark also includes off-talk examples; linked insertions are matched by text, not fully acoustically.Source: §1 · p. 2
2. ECHO DATASET CONSTRUCTION
ECHO constructs Chinese dialogue examples in which rewritten contexts change an insertion’s interactional role while its text stays fixed. The dataset distinguishes interruption, backchannel, and off-talk roles, renders them as speech, and organizes instances into linked groups and contrast pairs.
2. ECHO DATASET CONSTRUCTION
- ECHO uses insertion-text-matched contrasts, retaining candidate insertions only when a rewritten context can plausibly support an alternative interpretation.Dialogue skeletons are sampled from Duplex-Drama, and DeepSeek-V4-Pro selects overlap positions and generates initial insertions.Source: §2 · p. 2, §2.2 · p. 2
2.1. Interaction Roles and System Actions
- An interruption is assistant-directed and requires YIELD, while assistant-directed backchannels and self- or third-party-directed off-talk map to KEEP.The dataset retains three-way role labels because backchannels and off-talk differ in whether they should enter the dialogue state.Source: §2.1 · p. 2
2.2. Context-Rewritten Contrast Generation
- Claude-3.5-Sonnet rewrites dialogue history while preserving the insertion text, and linked groups may contain two or all three role variants.The equality constraint applies to insertion text, not synthesized waveforms; human annotators judged 97.45% of reviewed candidates consistent with target labels.Source: §2.2 · pp. 2–3
2.3. Speech Synthesis and Overlap Rendering
- Turns are independently synthesized into speaker-separated dual-channel audio, with role-dependent overlap rendering and no waveform identity guarantee across paired examples.Backchannel and off-talk speech overlays the assistant track, whereas interruptions receive onset emphasis and the assistant track fades after a short reaction interval.Source: §2.3 · p. 3
2.4. Dataset Organization and Scope
- ECHO contains 266 groups, 549 unique audio instances, and 300 pair relations, with 183 instances per role and 100 pairs for each role contrast.All 100 interruption–backchannel pairs share identical insertion text, removing within-pair lexical discrimination.Source: §2.4 · p. 3
3. EXPERIMENTS
The experiments evaluate speech systems through their native interfaces and use class-conditioned and pair-level measures to assess context-sensitive overlap handling. Most speech systems perform poorly on interruption–backchannel action flips, while MiniCPM-o 4.5 is more balanced.
3.1. Evaluated Systems and Native Interfaces
- Four speech systems were tested through their supported interfaces, so results are behavioral evaluations rather than controlled comparisons of model architectures or context representations.The study also evaluates a text-only Gemini reference, which is not a modality-matched baseline or an upper bound.Source: §3.1 · p. 3
3.2. Action Alignment and Metrics
- Class-conditioned accuracy reports correct YIELD on interruptions and correct KEEP on backchannels and off-talk; Gemini’s role predictions are mapped to binary actions for these metrics.Macro accuracy averages the three label-specific accuracies, counting each unique sample once.Source: §3.2 · p. 3
- PASR counts an interruption–backchannel or interruption–off-talk pair only when both target actions are predicted correctly, giving constant-YIELD and constant-KEEP policies zero.Source: §3.2 · p. 4
- PKC measures KEEP on both members of backchannel–off-talk pairs, while PRSR requires correct original-role predictions on both members of each pair type.PKC and PASR should be interpreted jointly because constant-KEEP maximizes PKC but yields zero PASR.Source: §3.2 · p. 4
3.3. Results and Analysis
- Interruption–backchannel PASR is 0.00% for Easy Turn, 4.00% for SoulX-Duplug and Lychee-FD, 54.00% for MiniCPM-o 4.5, and 79.00% for text-only Gemini.Correctly handling one member of a matched contrast generally did not imply success on its action-flipped counterpart.Source: §3.3 · p. 4
- Easy Turn and SoulX-Duplug correctly yield on 98.91% and 91.26% of interruptions, but keep the floor on only 1.09% and 6.56% of backchannels.Interruption-only accuracy would portray these systems favorably despite their near-zero success on matched interruption–backchannel contrasts.Source: §3.3 · p. 4
- Lychee-FD keeps the floor on only 12.02% of backchannels despite 55.19% interruption accuracy, while MiniCPM-o 4.5 scores 63.39% and 65.57% on those classes, respectively.Source: §3.3 · p. 4
4. CONCLUSION
ECHO evaluates overlap handling in Chinese spoken dialogue using matched insertion transcripts across rewritten dialogue contexts. Results expose over-yielding and show why interruption accuracy alone is insufficient, while the synthetic event-level design limits what the benchmark can assess.
- ECHO is a paired diagnostic benchmark for Chinese spoken dialogue that holds insertion transcripts fixed across rewritten contexts and evaluates joint action correctness on matched pairs.Source: §4 · p. 4
- Three of four speech systems keep the floor on fewer than 13% of backchannels and achieve at most 4% PASR on interruption–backchannel pairs, although two exceed 90% interruption accuracy.The findings motivate reporting class-conditioned KEEP rates and pair-level success alongside interruption accuracy.Source: §4 · p. 4
- ECHO is synthetic and event-level, and its transcript-matched pairs differ acoustically, so PASR cannot isolate dialogue-context effects from correlated acoustic cues.Fixed-trajectory replay also cannot evaluate latency, mutual adaptation, post-yield recovery, or subsequent response quality.Source: §4 · p. 4