PaperlayerView exact revision on arXiv

Source-linked AI summary

The Importance of Speech Genre for End-of-Turn Detector Training

Rini Sharon, Kadri Hacioglu, Jeena Prakash, Andreas Stolcke

arXiv:2610.39742v1eess.AScs.AI

TL;DR

End-of-turn detection has focused less on training-data genre, despite genres differing in the within-turn events they contain. The paper compares matched conversational and single-turn training data using the same detector and labeling approach. Conversational training performs better across the four test sets, with the effect also found at utterance level on conversational test speech.

  • Problem

    Training-data genre has drawn less attention than features, models, and modalities, and genre comparisons are often entangled with architecture, features, and evaluation.

    Source: §I · pp. 1–2

  • Method

    The study trains the same detector on equally sized conversational and single-turn datasets labeled with the same rule, then compares them across four test sets.

    Source: §I · p. 2, B. Datasets · p. 3

  • Results

    Conversational training yields higher F1 on all four test sets, and its utterance-level advantage appears only on conversational test data.

    Source: B. Conversational Training Generalises… · p. 4, §IV · p. 6

  • Takeaways & Limitations

    Evaluations limited to single-turn speech leave the benefit of conversational training untested.

    Source: §IV · p. 6

    The training sets differ in genre and source domain, so part of the observed gap could reflect domain rather than genre.

    Source: E. Limitations · p. 6

Abstract

from arXiv · show

End-of-turn (EoT) detection has been studied mostly through features, models, and modalities, while the speech genre of the training data has drawn far less attention. We isolate this factor. Using one reproducible annotation pipeline, we build a matched pair of corpora, multi-turn human-human conversational and single-turn human-machine, and train the same streaming ConvLSTM EoT detector on each. Across four test sets the conversational model reaches up to +0.45 higher F1, driven by fewer false alarms on the mid-turn pauses and hesitations that are rare in single-turn corpora. This conversational-training effect is not specific to frame-level decoding. Under a matched complete-or-not test it also appears at the utterance level, across two encoders, but only on conversational test data. Testing only on single-turn speech, as most prior work does, therefore hides the benefit of conversational training data. Against open-source baselines, our 1.8M-parameter model attains the highest F1 on every test set.

I. INTRODUCTION AND RELATED WORK

End-of-turn detection research has emphasized features, models, and modalities, while training-data genre has received less attention. This study compares conversational and single-turn speech under matched labeling and examines whether genre effects extend from frame-level to utterance-level decisions.

I. INTRODUCTION AND RELATED WORK

  • Turn-taking systems must avoid both interrupting users with early responses and making exchanges unnatural by waiting too long.Fixed silence thresholds treat pauses alike and cannot distinguish a mid-turn hesitation from a true turn end.Source: §I · p. 1
  • Conversational speech contains hesitations, backchannels, and mid-turn pauses that should not trigger an end-of-turn decision, unlike genres with fewer within-turn events.These events provide cues a model can learn from; human-to-system, command-style, and scripted speech carry fewer such events.Source: §I · pp. 1–2
  • Training-data genre remains underexplored and often entangled with architecture, features, and evaluation, limiting attribution of reported gains to the data itself.Prior work often trains and evaluates on a single corpus without cross-data comparison.Source: §I · pp. 1–2
  • Frame-level detectors score every frame, whereas utterance-level classifiers decide at a pause; whether conversational training helps the latter remains untested.The study controls training genre using one labeling pipeline and compares both decision granularities across test genres and encoders.Source: §I · p. 2, Figure 1 · p. 2

II. METHODOLOGY

The methodology section is organized around the data, the model and training, and the evaluation comparison.

II. METHODOLOGY

  • The section describes how the data are built and the model that consumes them.It also covers the training setup.Source: §II · p. 2
  • The section describes the datasets used for comparison and the baselines.Source: §II · p. 2
  • The section also describes the metrics used to evaluate the systems.Source: §II · p. 2

A. Data Curation Pipeline

The curation pipeline extracts speaker turns, appends validated trailing silence, and labels the resulting audio at frame level.

A. Data Curation Pipeline

  • The conversational corpus uses 8 kHz stereo call-center recordings, with one channel per speaker.The passage states that relevant pitch, energy, and pause cues are preserved at this native telephony rate.Source: A. Data Curation Pipeline · p. 2
  • Turn extraction splits calls into single-speaker turns and marks each end where the other speaker takes over or the call ends.The pipeline appends up to 1.3 seconds of trailing silence, stopping if either speaker resumes and retaining turns with at least 0.3 seconds of clean silence.Source: A. Data Curation Pipeline · pp. 2–3
  • Frame-level labels use 10 ms resolution, assigning 0 before the end-of-turn and 1 from that boundary through appended silence.Silence extension is VAD-validated to contain at most 10% speech and is capped at the file boundary.Source: A. Data Curation Pipeline · p. 3

B. Datasets

The dataset design pairs equally sized conversational and single-turn training sets with four test sets spanning conversation, human-machine dialogue, and command speech.

B. Datasets

  • Conv-10K and ST-10K each contain 10,000 utterances and use the same labeling rule, model architecture, and hyperparameters.Conv-10K draws from multi-turn human-human calls; ST-10K samples complete real and synthetic human-to-system turns disjoint from evaluation utterances.Source: B. Datasets · p. 3
  • The conversational tests include 5,000 human-verified internal call segments and 2,500 independent CASPER conversation turns with hesitations and mid-turn pauses.CASPER is processed through the same full-call pipeline to test transfer beyond the source domain.Source: B. Datasets · p. 3
  • The SmartTurn pool subset keeps only real complete turns, while SLURP comprises 2,500 short voice commands with no multi-turn structure, testing whether an EoT detector transfers to command-and-control speech.SmartTurn frame-level labels come from VAD-based speech detection.Source: B. Datasets · p. 3

C. Model Architecture

The detector processes 10 ms frames with acoustic and prosodic features, using convolution and recurrent layers for streaming EoT prediction. Training uses chunked processing with carried-forward LSTM states and a weighted loss.

C. Model Architecture

  • Each 10 ms frame combines 80-dimensional log-Mel features with five prosodic features, and the model produces a per-frame EoT probability.The architecture uses a Conv1D, a two-layer unidirectional LSTM, and a three-layer MLP; its 1.8M parameters support streaming with unbounded left context.Source: C. Model Architecture · p. 3

D. Training Methodology

  • Training processes samples chunk by chunk, carries LSTM states forward, and masks loss to valid frames.This matches streaming inference; the loss is binary cross-entropy weighted by the 5.6 ratio of non-EoT to EoT frames.Source: D. Training Methodology · p. 3
  • Optimization uses two epochs of linear warmup, cosine decay, gradient clipping, and early stopping on validation F1.Training runs for up to 30 epochs with patience 7.Source: D. Training Methodology · p. 3

E. Baselines

The baselines include two silence-threshold VAD systems and learned utterance-level classifiers. Additional frozen-encoder classifiers compare training genres while holding the encoder fixed.

E. Baselines

  • WebRTC VAD and Silero VAD trigger EoT from silence duration, with thresholds swept from 0 to 1000 ms in 50 ms steps.The reported result uses each baseline’s best F1 per dataset; WebRTC uses aggressiveness 3 and Silero VAD v5 uses its default 0.5 threshold.Source: E. Baselines · p. 3
  • SmartTurn v3 uses a Whisper-Tiny encoder and linear classifier to classify speech up to a pause as a complete or incomplete turn.SmartTurn FT is the same model fine-tuned on conversational data; Silero VAD supplies the pauses.Source: E. Baselines · p. 3
  • Two additional utterance-level systems train small classifiers on frozen Whisper-Tiny or wav2vec2-base encoders, keeping training genre as the variable.The encoders were pretrained differently; unlike the frozen setup, SmartTurn updates its Whisper-Tiny encoder during training.Source: E. Baselines · pp. 3–4

F. Evaluation Metrics

Frame-level endpoints use a sustained threshold crossing and a one-sided tolerance window that treats premature triggers as false alarms. Utterance-level classifiers are evaluated on silence-matched complete and incomplete segments using AUROC.

F. Evaluation Metrics

  • An endpoint is detected when sigmoid output exceeds threshold τ for eight consecutive frames, and the predicted endpoint is the last frame of that crossing.At 100 Hz, eight frames correspond to 80 ms.Source: F. Evaluation Metrics · p. 4
  • A detection is correct if it is no more than 50 ms early; there is no late cutoff, and latency is reported separately.F1 is interpreted alongside latency because the one-sided window could reward a system that simply waits.Source: F. Evaluation Metrics · p. 4
  • Frame-level systems are scored against the same ground-truth boundary, while the utterance-level test matches complete and incomplete examples on 150 ms of trailing silence.The utterance-level decision must therefore rely on speech before the pause, not silence duration alone.Source: F. Evaluation Metrics · p. 4
  • Reported frame-level metrics include F1, false-alarm and miss rates, false alarms per utterance, and median latency over correct detections.The utterance-level setting uses AUROC, which is threshold-free and imbalance-robust.Source: F. Evaluation Metrics · p. 4

III. RESULTS AND DISCUSSION

The training and evaluation genres differ mainly in how many pauses occur within turns. Conversational training data contain more within-turn pause structure than single-turn data.

III. RESULTS AND DISCUSSION

  • The comparison scores each system at its own best-F1 threshold on each dataset, making the reported numbers an upper bound rather than a deployable result.The same tuning is applied to the model and baselines, so the comparison addresses their ranking.Source: §III · p. 4

A. The Two Genres Differ Mainly in Within-Turn Events

  • Conv-10K has roughly 60% more internal pauses per utterance, nearly double the silence ratio, and over 70% more long pauses than ST-10K.The evaluation sets vary along the same axis: Int-5K matches Conv-10K, CASPER retains a high silence ratio despite shorter utterances, and ST-Complete falls between the training distributions.Source: A. The Two Genres Differ Mainly in… · p. 4

B. Conversational Training Generalises Across Domains

Conversational training improves frame-level end-of-turn detection across all four test sets, with fewer false alarms on mid-turn pauses; at the utterance level, its advantage appears on conversational speech, not single-turn speech.

B. Conversational Training Generalises Across Domains

C. At the Utterance Level, the Benefit Holds Only on Conversational Speech

  • On conversational test speech, conversational training raises utterance-level AUROC for both encoders: +0.21 and +0.38 on Int-5K and CASPER for Whisper-Tiny, and +0.37 and +0.29 for wav2vec2-base.The two encoders share no pretraining, and CASPER is out of domain for both.Source: Figure 2 · p. 5, C. At the Utterance Level, the Benefit… · p. 5
  • On single-turn speech, utterance-level AUROC is tied on out-of-domain SLURP (ΔAUROC +0.01 and −0.03), while single-turn training wins on source-matched ST-Complete.The ST-Complete reversals are significant; the conversational head remains above chance there, with AUROC 0.78 and 0.73.Source: C. At the Utterance Level, the Benefit… · p. 5

D. Outperforming Open-Source Baselines with Fewer Parameters

The ConvLSTM outperforms open-source baselines on all three conversational or multi-turn test sets with substantially fewer parameters than SmartTurn.

D. Outperforming Open-Source Baselines with Fewer Parameters

  • The ConvLSTM leads in F1 on all three test sets with 1.8M parameters, about 4.4× fewer than SmartTurn’s 8M.It is the only system to keep false alarms, misses, and latency in check at once.Source: D. Outperforming Open-Source Baselines… · p. 6

E. Limitations

The conversational-training advantage is observed across the reported evaluations, but conclusions are bounded by differences between corpora and by the two genres studied. The paper identifies multilingual evaluation and mixed-genre training as next steps.

E. Limitations

  • The training sets differ in genre and source domain, so some of the observed gap may reflect domain rather than genre.The out-of-domain CASPER result argues against domain as the full explanation, but does not eliminate it.Source: E. Limitations · p. 6
  • The comparison covers two genres with abundant reliably labelled real speech; extending it to other genres awaits suitable corpora.The paper names multilingual settings and mixtures of conversational and single-turn data as next steps.Source: E. Limitations · p. 6, §IV · p. 6

IV. CONCLUSION

  • Across four test sets, the conversationally trained ConvLSTM beats the single-turn-trained model, with fewer false alarms on mid-turn pauses and hesitations.The effect appears at frame level and utterance level, but the utterance-level result is limited to conversational test speech.Source: §IV · p. 6

Related papers

Loading 2610.39742v1…