PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model

Yang Xiao

arXiv:2610.21150v1eess.AScs.SD

TL;DR

Open turn-taking detectors face a mismatch between segmented-utterance training and decisions made throughout real pauses, while related decisions often require additional calls. VoxTurn trains on real pauses and answers multiple-choice turn-taking questions in one pass; it reaches 11.5% English false cut-offs at 300 ms and the lowest 14-language mean among open models.

  • Problem

    Open detectors make one turn-taking decision per call, while deployed agents need several decisions during pauses and face a training-data mismatch between segmented utterances and real pauses.

    Source: §1 · pp. 1, 3

  • Method

    VoxTurn trains on real conversational pauses with timing-derived labels and uses a speech LLM to answer multiple-choice turn-taking questions in one pass.

    Source: §1 · p. 3

  • Results

    11.5% English false cut-offs at 300 ms were comparable to two commercial detectors, and VoxTurn’s 19.9% 14-language mean was the lowest among open models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    In Pipecat and LiveKit replays, VoxTurn cut off 17% and 9% of turns, respectively, and answered sooner.

    Source: §1 · p. 3

Abstract

from arXiv · show

A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.

1 Introduction

Open turn-taking detectors struggle with real pauses and often require separate calls for each decision. VoxTurn combines pause-timed training and one-pass multi-question inference, achieving low cut-off rates across eot-bench and strong results in framework replays and zero-shot TurnBench.

1 Introduction

  • Open detectors take the turn in 28% or more of mid-turn pauses on eot-bench’s 14-language mean at a 300 ms budget, with much of the gap traced to training on segmented utterances rather than real pauses.A deployed agent also needs decisions about whether a pause will end soon, whether speech is an interruption or backchannel, and whether a sound is speech; open detectors answer one per call.Source: §1 · p. 3
  • Training on real conversational pauses with decision times drawn throughout them and labels derived from pause timing reduced eot-bench false cut-offs from 26.2% to 16.5% in English and from 33.0% to 20.0% on the 14-language mean.These training choices brought more of the gain than a speech-aligned backbone, multi-question training, or four times more training.Source: §1 · p. 3
  • VoxTurn answers five turn-taking decisions in one pass, with 8.3 ms per decision and four questions in 9.0 ms instead of 33.2 ms.The block-isolating attention mask makes the multi-question result equivalent to asking questions one by one.Source: §1 · p. 3
  • 11.5% English false cut-offs at 300 ms were comparable to two commercial detectors, while the 19.9% 14-language mean was the lowest among open models.The 14-language mean was 19.9% among open models evaluated in all 14 languages.Source: §1 · p. 3
  • With repeated queries during silence, VoxTurn cut off 17% of Pipecat turns and 9% of LiveKit turns, versus 45% and 18% with default detectors, and answered sooner.The framework results come from re-querying during each silence.Source: §1 · p. 3
  • Without TurnBench training, VoxTurn ranked fifth of 24 in end-of-turn detection and sixth of 23 in interruption detection.TurnBench evaluates two-party conversations.Source: §1 · p. 3

2 Related work

Prior turn-taking systems use different models or calls for different decisions, while VoxTurn covers these decisions with one model and one pass. It also uses a shared audio encoding to score multiple-choice answers.

Turn-taking models

  • VoxTurn covers turn-taking decisions that prior systems handle with dedicated detectors, dialogue-state models, or implicit full-duplex turn-taking.The decisions include end of turn, interruption, and backchannel.Source: §2 · p. 3

LLMs as classifiers

  • VoxTurn scores multiple-choice answer options with a learned linear head and lets several questions share one audio encoding.This applies option scoring to speech rather than generating free text.Source: §2 · p. 4, Figure 2 · p. 4

Benchmarks

  • The evaluation suite includes repeated end-of-turn decisions during silences, interruption events, and tests of dialogue-state and nonverbal-vocalisation systems.It reports eot-bench, TurnBench, Easy Turn, TEN, and a nonverbal-vocalisation set.Source: §2 · p. 4

3 VoxTurn

VoxTurn frames turn-taking decisions as multiple-choice questions over audio and optional text, scores answer options, and isolates question blocks to answer several questions in one pass. Its training objective adds exact auxiliary labels and a consistency term; deployment re-queries during silences.

3.1 Turn-taking decisions as multiple-choice questions

  • VoxTurn answers five fixed-option questions using the last 8 seconds of audio, optional transcript or partner words, and a speaker marker.Each question has 12 English and Chinese phrasings; training samples one and inference uses a fixed phrasing.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 Model and readout

  • A frozen audio encoder feeds Qwen3-0.6B, while a linear head scores each option’s final hidden state instead of generating text.Only the projector, LoRA weights, and head are trained, totaling 11.8M parameters.Source: §3.2 · pp. 4–5

3.3 One-pass multi-question inference

  • A block-isolating attention mask and single-question position ids make each question’s probabilities match separate passes, independent of question order.The maximum measured probability difference is 1.5×10^-5 in fp32.Source: §3.3 · p. 5

3.4 Training objective

  • Training adds auxiliary questions with exact labels and a second phrasing of the main question, with both auxiliary loss weights set to 0.5.Pause timing supplies resume labels; the second phrasing contributes a consistency term.Source: §3.4 · p. 5

3.5 Deployment: re-querying during a silence

  • During deployment, VoxTurn is queried every 0.25 seconds while silence lasts, and the turn ends once its end-of-turn probability reaches the threshold.The default threshold is 0.8; a framework timeout is the fallback.Source: §3.5 · pp. 5–6

4 Training data and recipe

VoxTurn training combines segmented utterances with real conversational pauses sampled at varying decision times. The recipe uses 300,000 samples and takes 78 minutes for one run on an NVIDIA L40S.

Data

  • Real pauses come from phone calls, face-to-face dialogue, and time-aligned transcripts, with hold and end-of-turn labels derived from speaker timing.Pauses are sampled between speaker segments; overlaps and backchannels are skipped.Source: §4 · p. 6
  • For each pause, the decision time is drawn uniformly from the first min(d, 1.5 s), teaching the model how the answer changes as silence unfolds.Seamless audio is band-limited to 8 kHz half the time so bandwidth does not reveal the source.Source: §4 · p. 6

Training

  • The model is trained on 300,000 samples; one run takes 78 minutes on one NVIDIA L40S, with 0.1–0.5-point English false-cut-off variation across hardware reruns.Training uses AdamW and cosine learning-rate decay.Source: §4 · p. 6

5 Evaluation suite

The evaluation suite covers turn-taking benchmarks and reports detector performance under explicit cut-off and latency trade-offs. Evaluation audio is not used for training.

eot-bench protocol

  • eot-bench scores decisions every 100 ms during silences in 14 languages, sweeping policies to measure cut-offs, latency, and hold-versus-end AUC.Reported cut-off and latency values are points on a trade-off curve; lower is better for both.Source: §5 · p. 6

Evaluation suite

  • The suite includes eot-bench, TurnBench, Easy Turn, TEN, and a nonverbal-vocalisation set, covering single-speaker turns and two-party events.Table 3 summarizes what each benchmark measures.Source: Table 3 · p. 7

Baselines

  • Baseline results come from official eot-bench results, the turn-1 model card, and the TurnBench test leaderboard; eot-bench audio is not used for training.The cited TurnBench leaderboard is dated October 2026.Source: §5 · p. 7

6 Results

VoxTurn achieves low English false cut-offs at tight latency budgets and the lowest open-model mean across 14 languages. Results also show gains from conversational-pause data and a speech-aligned backbone, support several additional decisions, and transfer to TurnBench and voice-agent frameworks.

6.1 eot-bench English: the latency–accuracy trade-off

  • 11.5% false cut-offs in English at a 300 ms latency budget, fewer than Baton (12.3%) and Deepgram Flux (12.9%).VoxTurn joins LiveKit v1 and Baton in the lowest band across latency budgets; other open models need 1.6–2.2× its latency for the same 5% or 10% cut-off rate.Source: §6.1 · p. 7, Figure 3 · p. 8

6.2 Multilingual results

  • Conversational-pause data lowered the 14-language mean from 33.0% to 20.0%, with pause cues transferring to languages without such training data.English pauses from task-oriented calls gave the largest single English gain, reducing false cut-offs from 26.2% to 16.3%; four times more updates gained only 0.7 points.Source: §6.2 · pp. 8–9, §6.3 · p. 9

6.3 What drives the gains

  • Replacing Whisper-small and a lightly trained adapter with Qwen3-ASR-0.6B lowered English cut-offs from 16.6% to 12.8%.Chinese calibration error of p(eot) on eot-bench fell from 0.30 to 0.11; multi-question training improved English by 0.4–1.8 points between 250 and 500 ms, while the 14-language mean was nearly unchanged (19.7 vs. 19.9%).Source: §6.3 · p. 9

6.4 Other decisions from the same pass

  • From audio alone, VoxTurn reaches 88.3% accuracy across four Easy Turn dialogue states and flags 99.6% of OpenSLR 99 nonverbal vocalisations as invalid.It rejects 0.06% of 1,786 valid speech clips; on TEN, finished-turn recognition is 83.5% in English and 89.7% in Chinese, while unfinished-turn recognition is 99.0% and 97.9%.Source: §6.4 · p. 9, Figure 6 · p. 10

6.5 Two-party conversations: TurnBench

  • On the TurnBench test set, VoxTurn ranks fifth of 24 in end-of-turn detection (0.924) and sixth of 23 in interruption detection (0.951).On development data, partner speech raises own-channel EOT recall from 0.700 to 0.810, and the full policy reaches 0.930; replacing VoxTurn with VAD preserves recall (0.924) but increases latency from 667 to 821 ms.Source: Figure 7 · p. 11, §6.5 · p. 12, Figure 8 · p. 12

6.6 Efficiency and use in voice-agent frameworks

  • Four questions in one pass take 9.0 ms instead of 33.2 ms, a 3.7× saving, while serving mode preserves the eot-bench result (11.6% vs. 11.5% at 300 ms).In framework replay, VoxTurn reduced cut-offs and answered sooner than each default detector: Pipecat 16.7% vs. 45.3% and 0.36 s vs. 1.32 s; LiveKit 9.3% vs. 18.0% and 0.47 s vs. 0.73 s. Re-checking every 0.25 s matters: one-shot Pipecat averaged 1.26 s.Source: §6.6 · pp. 12–13, Table 7 · p. 12, Figure 9 · p. 13

7 Conclusion

VoxTurn answers several turn-taking questions with a speech LLM in one forward pass. Its reported results span eot-bench, decision latency, and TurnBench end-of-turn detection.

7 Conclusion

  • Real conversational pauses, a speech-aligned backbone, and supervision of timing-derived labels bring an open 0.8B model to 11.5% English false cut-offs at 300 ms and the best 14-language mean among open models.VoxTurn runs at 8.3 ms per decision and ranks fifth in TurnBench end-of-turn detection.Source: §7 · p. 13

A Per-language eot-bench results

The appendix table presents per-language eot-bench results for false cut-offs at a fixed latency budget and latency at a fixed false cut-off rate.

A Per-language eot-bench results

  • Table 8 reports per-language false cut-offs at a 300 ms mean-latency budget and mean latency at 5% false cut-offs; lower values are better.Boldface marks the best open model in each column.Source: Table 8 · p. 16

B Training hyperparameters

The appendix provides a table of VoxTurn v1 training configuration.

B Training hyperparameters

  • Table 9 lists the VoxTurn v1 training configuration.The supplied passage identifies the table but does not specify its hyperparameter values.Source: Table 9 · p. 16

C TurnBench decision policy

The TurnBench policy opens a candidate for each silence and commits it using duration, EOT-probability, partner-speech, or deadline criteria. It also specifies event timing and tunes ten parameters for development recall under an FPR constraint.

C TurnBench decision policy

  • Each silence opens a candidate, committed at the first frame meeting the policy’s silence-duration, EOT-probability, partner-speech, or deadline criteria.The supplied passage does not specify the thresholds or classification labels for these criteria.Source: C TurnBench decision policy · pp. 16–17
  • Events wait 0.5 s after speech offset; a confirming event follows at silence tconf, and the final silence ends with an event at audio end.A pause costs at most one false positive.Source: C TurnBench decision policy · p. 17
  • Ten parameters are random-searched for best development recall subject to FPR ≤0.1.Source: C TurnBench decision policy · p. 17
Loading 2610.21150v1…