Source-linked AI summary
Less can be More: What Aspects of Speech Drive End-of-Turn Detection
Rini Sharon, Manickavela A, Kadri Hacioglu, Andreas Stolcke
TL;DR
Streaming end-of-turn detection must balance premature interruption against delayed responses, while the relative contribution of modalities remains incompletely understood. The paper systematically ablates acoustic, prosodic, and text signals and analyzes their feature spaces. Acoustic–prosodic and trimodal systems meet the stated latency and false-alarm constraints, while text integration adds inference cost without recovering it in the studied domain.
Problem
The relative value of combining acoustic, prosodic, and semantic signals for end-of-turn detection remains only partially understood, although premature or delayed detections affect conversational flow.
Source: §1 · p. 1Method
Results
Acoustic–prosodic modeling has the highest F1 and lowest false-alarm rate overall; only acoustic–prosodic and trimodal systems meet real-time latency and false-alarm constraints at their optimal operating points.
Source: §4.1 · p. 4, §5 · p. 4Takeaways & Limitations
Acoustic–prosodic modeling appears sufficient for deployment-grade EOT detection in the conversational domain studied, while text inference adds cost not recovered by its integration.
Source: §5 · p. 4Results are limited to in-domain two-speaker English telephony, and text may contribute more where acoustic EOT cues are weaker; cross-lingual and cross-domain evaluation remains future work.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
In conversational AI, detecting when a speaker has finished talking is crucial for natural turn taking. While recent work incorporates semantics, the relative contribution of different modalities remains unclear. We present a controlled ablation of acoustic, prosodic, and semantic signals for streaming end of turn detection using a lightweight trimodal classifier. Under identical training conditions, the acoustic prosodic combination achieves the best balance of accuracy and latency, achieving utterance F1 of 0.93 with 7.8% false alarms at 400ms median latency. Adding text increases premature detections without improving performance. Feature space analysis confirms that prosodic features have the strongest class separability, while text representations overlap substantially. These findings suggest that turn-taking is primarily conveyed through intonation and silence patterns rather than semantic completeness, enabling faster and more reliable systems without expensive text inference.
1. Introduction
The paper asks when additional modalities improve streaming end-of-turn detection, a question left open by incomplete comparisons of multimodal fusion. It introduces a controlled architecture, systematic modality-subset evaluation, and feature-space analysis.
1. Introduction
- EOT detection trades premature interruption against delayed responses, while fixed-silence systems can confuse brief hesitations with genuine turn completions.Silence thresholds must exceed typical conversational pauses to avoid premature interruptions, creating response delays.Source: §1 · p. 1
- Whether multimodal fusion improves EOT detection remains only partially understood because added signals can yield diminishing or negative returns when they lack independent information.Source: §1 · p. 1
- APT enables controlled modality ablations by zeroing disabled streams while preserving input dimensionality, parameter count, training data, and optimization.The authors state that this isolates signal presence from changes in model capacity or training dynamics.Source: §1 · p. 2
- The study evaluates all seven non-empty acoustic, prosodic, and text subsets under matched conditions, spanning unimodal baselines through all multimodal combinations.This tests when added modalities improve on the best single-stream model.Source: §1 · p. 2
- The paper adds feature-space analysis with frozen pretrained encoders to quantify each modality’s class separability independently of the classifier.Earlier prosodic studies did not analyze how these signals separate in contemporary representation spaces.Source: §1 · p. 2
2. Methodology
APT is a streaming, frame-level detector that combines acoustic, prosodic, and text representations. Its fusion head supports controlled modality ablations while keeping model capacity and training conditions fixed.
2. Methodology
- APT fuses acoustic representations from a frozen ASR encoder, handcrafted waveform prosody, and semantic embeddings from a running ASR transcript into per-frame EOT probabilities.The pretrained Zipformer and MiniLM components remain frozen; only a lightweight classification head of about 261K parameters is optimized.Source: §2 · p. 2
2.1. Input Streams
- Prosodic features are extracted at 100 Hz and downsampled to 25 Hz; cached text embeddings update when a new word is decoded while positional encoding advances each frame.The resulting divergence within the causal window implicitly encodes elapsed time since the last decoded word as a learned silence cue.Source: §2.1 · pp. 2–3, Table 1 · p. 2
2.2. Fusion and Classification
- The three stream outputs are concatenated and passed through linear projections to produce a per-frame EOT probability; disabled streams are zeroed and trained independently under identical hyperparameters.The fixed input dimensionality and unchanged parameter count support comparisons across modality configurations.Source: §2.2 · p. 3
3. Experimental Setup
The experiments use single-turn segments from English telephony calls, with recording-level splits and a shared 5,000-utterance test set. Streaming performance is evaluated using a sustained-threshold rule and measures that capture detection accuracy, premature triggers, misses, and latency.
3. Experimental Setup
- The proprietary English telephony corpus covers banking, insurance, retail, and telecommunications, with 10K training, 2K validation, and 5K test segments split by recording.Single-turn segments retain up to 1.3 seconds of post-EOT silence, and frame-level labels switch to positive at EOT.Source: §3 · p. 3
- All configurations use the same 5,000 test utterances, enabling paired McNemar tests for binary outcomes and Wilcoxon signed-rank tests for latency and false-alarm counts.The tests use α = 0.05.Source: §3 · p. 3
3.1. Evaluation Metrics
- A detection requires the sigmoid output to exceed threshold τ for 8 consecutive frames, and detections more than 50 ms before the boundary count as false alarms.The asymmetric tolerance treats late responses as true positives with positive latency and reflects the stated preference for late responses over premature interruptions.Source: §3.1 · p. 3
- Evaluation reports utterance-level F1, false-alarm percentage, miss percentage, premature triggers per utterance, and median latency over correct detections.FA% measures whether the first detection was premature, whereas FA/utt counts premature triggers throughout an utterance; the FA%-latency curve is also reported across thresholds.Source: §3.1 · p. 3
4. Results and Discussion
Across the tested modality combinations, acoustic–prosodic modeling offers the strongest overall detection results, while text increases premature detections. Feature-space analysis finds the strongest class separation in prosodic features, and the authors limit their conclusions to in-domain English telephony.
4.1. Modality Ablation
- Adding prosody increases F1 in every reported combination, but prosodic-only detection remains in a high-FA region across the threshold sweep.Reported gains are +0.003 for A+P over A, +0.034 for APT over A+T, and +0.455 for P+T over T.Source: §4.1 · pp. 3–4
- A+P achieves the highest F1 and lowest false alarm rate overall, and both A+P and APT meet the stated latency and false-alarm constraints.A and A+T meet both constraints marginally; P, P+T, and T do not.Source: Figure 2 · p. 3, §4.1 · p. 4
4.2. Feature Space Analysis
- Prosodic raw features have the strongest reported silhouette (0.301), acoustic projection gains +0.158 over raw input, and text features show substantial class overlap at 0.108.The text overlap is consistent with the high false alarm rates observed when text is included in fusion.Source: Figure 3 · p. 4, §4.2 · p. 4
4.1. Modality Ablation
- Every text-containing configuration has higher FA/utt than its text-free counterpart; APT has lower median latency than A+P but lower F1 because of more false alarms.The FA% and FA/utt increases from A+P to APT are each reported as significant at p < 10^-9.Source: §4.1 · p. 4
4.2. Feature Space Analysis
- Text embeddings update only at non-blank decoder emissions, leaving representations unchanged across most frames apart from positional encoding.The authors connect the resulting 0.108 silhouette to substantial EOT/non-EOT overlap and high false alarms when text is fused.Source: §4.2 · p. 4
4.3. Limitations and Future Work
- Results are limited to in-domain two-speaker English telephony; cross-lingual and cross-domain evaluation would test whether A+P generalizes, while boundary-aware language-model suppression is proposed for future work.Text may contribute more where acoustic EOT cues are weaker, such as low-resource languages or noisy conditions.Source: §4.3 · p. 4
5. Conclusion
The conclusion reports a systematic comparison of modality subsets and finds acoustic–prosodic modeling sufficient for deployment-grade detection in the conversational domain studied. It cautions that adding text brings inference cost without recovering accuracy when acoustic features are present.
- All seven non-empty acoustic, prosodic, and text modality subsets were evaluated under identical training and evaluation conditions.Source: §5 · p. 4
- Acoustic features provide the primary detection signal, prosodic features add complementary precision, and text increases false alarms in every configuration containing it.Source: §5 · p. 4
- Acoustic–prosodic modeling appears sufficient for deployment-grade detection in the studied conversational domain, while text adds inference cost without improving accuracy when acoustic features are present.Source: §5 · p. 4