PaperlayerView exact revision on arXiv

Source-linked AI summary

XTurnix: Large-Scale Self-Supervised Turn Control through Two-State Binary Decisions

Zhanxun Liu, Yifan Duan, Hengtao Wu, Chen Yang, Qinyuan Cheng, Kun Wang, Xingyu Zeng, Xipeng Qiu, Chaochao Lu, Xie Chen

arXiv:2610.04400v1cs.CLcs.AIcs.HC

TL;DR

Turn control must decide when to start speaking, keep listening, continue speaking, or stop, while existing detectors vary in modality and output categories. XTurnix makes state-conditioned decisions from full dialogue history using timestamp-derived supervision and synthetic fine-tuning. It is leading or competitive across third-party benchmarks and achieves the strongest, most balanced four-class performance on the self-curated benchmark.

  • Problem

    Turn control requires deciding when to start speaking, keep listening, continue speaking, or stop, while existing detectors vary in input modality and output categories.

    Source: §1 · p. 1

  • Method

    XTurnix predicts one state-legal control token from complete dialogue history, using timestamp-derived self-supervision and synthetic multi-turn fine-tuning.

    Source: §1 · p. 1, §4.3 · p. 5

  • Results

    XTurnix is leading or competitive across third-party benchmarks and has the strongest, most balanced four-class performance on the self-curated benchmark.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A single model can decide when to start responding, keep listening, continue speaking, or stop speaking, rather than only detect user-turn endpoints.

    Source: §6 · p. 9

    Easy-Turn evaluation excludes backchannel samples because isolated acknowledgments may be backchannels or affirmative answers depending on context.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

General turn-taking behavior in real-time dialogue systems requires deciding whether to keep listening or start responding while listening, and whether to continue or stop while speaking. Existing turn detectors use heterogeneous, task-specific label spaces and are often trained on limited annotations or evaluated on isolated utterances, making them difficult to use as a unified causal controller with comprehensive context. We propose XTurnix, a compact text-based model that formulates turn control as two binary decisions conditioned on the AI's current listening or speaking state and predicts a single control token from the complete dialogue history. XTurnix is pretrained on 5.5 million causal action examples automatically derived from timestamped two-speaker transcripts, then fine-tuned on synthetic multi-turn examples with a flatter distribution across the four state-action labels. We evaluate XTurnix on four public benchmarks and a balanced self-curated benchmark. Across the public benchmarks, XTurnix achieves the best results on all SemanticVAD and LiveKit splits, ties the native Smart-Turn model on Smart-Turn Bench, and achieves the highest incomplete-turn accuracy on Easy-Turn. On the self-curated benchmark, it reaches 89.06% accuracy, more than 20 percentage points above the strongest third-party baseline at 68.75%, while maintaining F1 scores between 84.21% and 90.63% across all four categories. These results demonstrate unified listening- and speaking-state turn control in a single compact model. Code is available at https://github.com/xcc-zach/xturnix, with an interactive demo at https://huggingface.co/spaces/xcczach/xturnix-demo.

1 Introduction

Real-time turn control requires decisions conditioned on whether the AI is listening or speaking, while existing detectors vary in input modality and output categories. XTurnix uses full dialogue history, state-conditioned actions, and timestamp-derived self-supervision; it performs competitively on public benchmarks and leads on the self-curated benchmark.

1 Introduction

  • Existing audio- and text-based detectors address utterance completion, backchannels, and multilingual endpointing, but their modalities and output categories vary.This variation makes it difficult to use them through a common turn-control interface.Source: §1 · p. 1
  • A general controller must use scalable supervision and full dialogue history, because earlier questions, instructions, and commitments can change how the same fragment should be handled.The required action space is conditioned on whether the AI is listening or speaking.Source: §1 · p. 1
  • The controller predicts a single token from complete dialogue history, choosing between keep listening and start speaking while listening, or keep speaking and stop while speaking.Text input supports incremental decisions from ASR output while retaining multi-turn context.Source: §1 · pp. 1–2
  • The self-supervised procedure interleaves timestamped two-speaker transcript tokens and derives token-level action labels, enabling pretraining without manual turn annotations.The paper frames this as one of three contributions.Source: §1 · pp. 1–2
  • XTurnix achieves competitive performance on public benchmarks and the highest overall accuracy among evaluated models on the multi-turn benchmark, with strong F1 across all four state-action categories.The model is a compact text-based turn controller evaluated on four public Chinese benchmark subsets and a balanced multi-turn benchmark.Source: §1 · p. 2

2 Related Work

Related work covers turn-taking and endpoint detectors built with varied modalities and output categories, from speech-activity prediction to transcript classification and interaction control.

2 Related Work

  • TurnGPT (Ekstedt and Skantze, 2020) predicts takeover readiness from dialogue, while Voice Activity Projection (Ekstedt and Skantze, 2022) predicts near-future participant speech activity.VAP uses recorded conversation speech activity and can indicate continuation, takeover, or a brief backchannel.Source: §2 · p. 2
  • Text-based systems classify accumulated ASR transcripts or multilingual completion, while Smart-Turn and Easy-Turn use audio or speech representations for completion and turn-state decisions.Easy-Turn distinguishes complete, incomplete, backchannel, and wait; Smart-Turn estimates completion with a Whisper-based classifier.Source: §2 · p. 2
  • Other systems combine streaming audio and ASR-derived text for endpointing or interruption handling, while Anyreach predicts turn-taking actions from dialogue context.The passage also describes systems with separate ASR and turn-state heads and models that classify audio into multiple states.Source: §2 · p. 2

3 Task Formulation

Turn control is formulated as a policy that maps dialogue history and the AI’s current state to a legal action from a state-dependent binary action space.

3 Task Formulation

  • Four state-conditioned labels pair each of the two dialogue states with its two legal control actions.The AI state is listening or speaking, and the final message in the history is always from the user.Source: §3 · p. 3
  • Each updated ASR version of the current user message defines a decision point and triggers an inference; a detected speech-segment end inserts a pause token and triggers another inference.The policy receives the available message history at each decision point.Source: §3 · p. 3
  • The legal action space depends on state, and the state transition is deterministic.The passage identifies the transition rule using the start, stop, and keep actions.Source: §3 · p. 3

4 Method

XTurnix combines timestamp-derived causal action pretraining with synthetic multi-turn fine-tuning across the four state-action labels. The training procedure turns interleaved dialogue into causal action examples and uses synthetic examples to flatten label distribution.

4.1 Self-Supervised Training Data Construction

  • Timestamped two-speaker transcripts are tokenized and interleaved, then scanned from an initial listening state to create causal examples for user tokens, with actions determined by temporal order.Each user token, including an inserted pause, creates an example; the listening-state action depends on whether the next timeline token is also from the user.Source: §4.1 · pp. 3–4

4.2 Synthetic Training Data Construction

  • Because self-supervised labels are highly imbalanced, the authors choose a target label before generating an example and prioritize labels furthest behind their target proportions.The target proportions are maintained throughout generation rather than corrected only after the dataset is complete.Source: §4.2 · pp. 4–5
  • Synthetic contexts contain at least two user and assistant messages and end with an assistant message; they are reused for distinct labels, with transformations creating truncated turns, backchannels, or interruptions.Most paired examples share the same dialogue context and differ only in turn-relevant content.Source: §4.2 · p. 5

4.3 Training

  • 5,504,000 self-supervised examples are used for task-specific pretraining, followed by fine-tuning on 10,026 synthetic examples; both stages train the model to emit one legal action token.The final model requires neither free-form decoding nor natural-language-output postprocessing.Source: §4.3 · p. 5

5 Experiments

The experiments compare XTurnix with third-party detectors across four public benchmarks and a balanced multi-turn benchmark, using each benchmark’s AI state and mapped action labels. XTurnix leads on all SemanticVAD and LiveKit splits, ties Smart-Turn on its pooled and Chinese results, and performs strongly across all four decisions on the self-curated benchmark.

5.1 Evaluation Benchmarks

  • The public evaluation uses Easy-Turn, Smart-Turn, SemanticVAD, and LiveKit, while the self-curated Chinese benchmark contains 128 balanced examples across four state-action combinations.Its examples provide dialogue history, the AI’s current state, and an incremental final user message, which may require interpreting that fragment against earlier dialogue.Source: §5.1 · pp. 5, 7

5.3 Results on Third-Party Benchmarks

  • XTurnix-ZH-Base reaches 92.14% on SemanticVAD (All), ahead of the strongest third-party model’s 80.66%, while XTurnix-PT reaches 73.70% on LiveKit (All), ahead of 70.91%.The XTurnix checkpoints jointly hold the best result on all SemanticVAD and LiveKit splits; XTurnix-PT ties native Smart-Turn on pooled and Chinese results at 98.81% and 98.52%, respectively, while Easy-Turn leads outright on Easy-Turn Bench at 97.33% versus XTurnix’s 95.33%.Source: Table 3 · p. 6, §5.3 · pp. 7–8
  • 100.00% incomplete-turn accuracy makes XTurnix-PT the best on Easy-Turn’s incomplete class, while XTurnix-ZH-Base reaches 99.33%.On Easy-Turn, both XTurnix checkpoints achieve 99.28% ROC-AUC for separating completed from continuing turns; backchannels are excluded because isolated acknowledgments may be either backchannels or affirmative answers.Source: Table 5 · p. 6, Table 6 · p. 6, §5.3 · p. 8
  • On Smart-Turn, XTurnix-ZH-Base reaches 96.75% with endfiller annotated but 58.88% without it, when turn status is more ambiguous.Smart-Turn reaches 98.52% and 85.96%, respectively; either decision can be plausible for utterances without an explicit endfiller cue because discourse context and action preference matter.Source: Table 4 · p. 6, §5.3 · p. 8

5.4 Results on the Self-Curated Benchmark

  • 89.06% accuracy for XTurnix-ZH-Base exceeds the strongest third-party model’s 68.75%, with XTurnix class F1 scores between 84.21% and 90.63%.The reported results characterize XTurnix as strongest overall and most balanced across the four decisions.Source: Table 7 · p. 7, §5.4 · p. 8
  • Within XTurnix, fine-tuning raises overall accuracy from 86.72% to 89.06%, while listening-start F1 increases from 85.71% to 90.63% and stop F1 from 84.21% to 88.57%.XTurnix checkpoints occupy the top two positions on both speaking decisions; endpoint detectors project speaking predictions from endpoint scores without multi-turn history or AI-state input.Source: §5.4 · pp. 8–9

5.2 Models and Evaluation Protocol

  • Third-party models receive benchmark-appropriate audio or text inputs, while XTurnix receives original multi-turn history and explicit AI state on the self-curated benchmark.Benchmark labels and model probabilities are mapped to the paper’s actions, and AI state is assigned before inference; XTurnix also removes terminal periods for Smart-Turn.Source: §5.2 · p. 7

5.3 Results on Third-Party Benchmarks

  • Fine-tuning shifts training signal toward transition actions, increasing their share after class weighting from 23.31% to 41.76%.XTurnix-ZH-Base gains Chinese accuracy over XTurnix-PT by 9.33% on Easy-Turn and 8.39% on SemanticVAD, but loses 1.77% on Smart-Turn and 4.10% on LiveKit; its <|start|> recall rises from 69.00% to 91.33% on Easy-Turn and from 15.22% to 51.37% on Smart-Turn, alongside lower <|keep|> recall.Source: §5.3 · pp. 7–8

6 Conclusion

The conclusion frames turn-taking as two state-conditioned decisions and presents XTurnix as a single model for controlling both listening and speaking, with leading or competitive benchmark performance.

6 Conclusion

  • The paper formulates general turn-taking behavior as two state-conditioned binary decisions for listening and speaking.Source: §6 · p. 9
  • XTurnix achieves the strongest, most balanced four-class performance on the self-curated benchmark and leading or competitive performance across third-party benchmarks.The results support using one model to decide when to start responding, keep listening, continue speaking, or stop speaking.Source: §6 · p. 9
  • XTurnix is trained through self-supervision on large-scale timestamped conversational data and augmented with synthetic multi-turn examples.Source: §6 · p. 9

A Complementary Pooled and English Turn-Detection Results

The section presents pooled and English turn-detection results, including analyses grouped by endfiller status.

A Complementary Pooled and English Turn-Detection Results

  • Table 8 reports pooled and English accuracy grouped by endfiller.Its caption says the table extends prior results, but supplies no outcome values here.Source: Table 8 · p. 11
  • The table notation marks best results in each column in bold and second-best results with underlining.Source: A Complementary Pooled and English… · p. 11
  • Table 9 reports pooled and English turn-completion metrics for Smart-Turn samples annotated endfiller=false.The caption identifies it as an extension of Table 6.Source: Table 9 · p. 11

Related papers

Loading 2610.04400v1…