PaperlayerView exact revision on arXiv

Source-linked AI summary

TurnTurk-GPT: Text-Based Turn-Taking Prediction for Turkish

Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Nur Yağmur Tuncer, Mustafa Sertaç Türkel

arXiv:2610.42010v1cs.CL

TL;DR

Turn-taking systems need to estimate plausible turn endings as speech unfolds, but real Turkish performance had not been reported under a public evaluation protocol. TurnTurk-GPT fine-tunes a Turkish GPT-2 model with a TurnGPT-style objective and evaluates it on real Turkish conversation. It beats a multilingual end-of-turn detector on both splits, while its advantage over a zero-shot Turkish language model depends on whether test speakers appeared in training.

  • Problem

    Real Turkish conversation had not been evaluated under a public protocol, despite deployed multilingual end-of-turn detectors listing Turkish among their supported languages.

    Source: §I · p. 1

  • Method

    TurnTurk-GPT fine-tunes a Turkish GPT-2 backbone with a TurnGPT-style turn-shift objective and evaluates it on real Turkish conversation against zero-shot and multilingual baselines.

    Source: §I · p. 1, A. Model and objective · p. 2

  • Results

    TurnTurk-GPT outperforms the multilingual detector on both splits, but surpasses the zero-shot Turkish model only for familiar speakers; on unseen speakers they are statistically indistinguishable.

    Source: §VIII · p. 5

  • Takeaways & Limitations

    Turn-final suffixes carry the Turkish end-of-turn cue, while text-only baselines remain close to chance when a speaker pauses.

    Source: §VIII · p. 5

    Evaluation is offline on final transcripts; evaluation on streaming-recognizer partial hypotheses and latency remains a natural extension.

    Source: D. Scope of the evaluation · p. 5

Abstract

from arXiv · show

This paper presents TurnTurk-GPT, a text-based turn-taking predictor for Turkish obtained by fine-tuning a Turkish GPT-2 model with the TurnGPT objective on synthetic and real dialogue, and evaluates it on real Turkish conversation against rule baselines, a zero-shot Turkish language model and a widely used multilingual end-of-turn detector. Evaluation uses word-level decision points on unpunctuated, lower-cased text approximating the output of streaming speech recognisers, recording-disjoint and speaker-disjoint folds, nested threshold selection and paired cluster bootstrap tests. TurnTurk-GPT outperforms the multilingual detector under both recording-disjoint and speaker-disjoint evaluation. It outperforms the zero-shot language model only when the test speakers occur in training; on unseen speakers the two are statistically indistinguishable. Removing the suffixes of the turn-final word reduces AP by up to 44%, against 2% for a sentence-internal word, locating the Turkish end-of-turn cue in verb-final morphology. Synthetic dialogue alone yields the weakest trained model, and when a speaker pauses the zero-shot and multilingual baselines remain close to chance. The model and the evaluation protocol are publicly released.

I. INTRODUCTION

Turn-taking prediction needs to detect plausible turn endings incrementally, but performance on real Turkish conversation has not been reported under a public evaluation protocol. TurnTurk-GPT addresses this gap with a Turkish-backbone predictor and Turkish-specific evaluation and analyses.

I. INTRODUCTION

  • Performance of deployed multilingual end-of-turn detectors on real Turkish conversation had not been reported under a public evaluation protocol.Turkish is verb-final and agglutinative, so its turn-final cues may differ substantially from English.Source: §I · p. 1
  • TurnTurk-GPT is evaluated on real Turkish conversation against a zero-shot Turkish language model and a multilingual detector using recording-disjoint and speaker-disjoint folds.The protocol uses word-level decision points, average precision, nested threshold selection and paired cluster bootstrap tests.Source: §I · p. 1
  • The study analyzes Turkish-specific cues, tokeniser choice, synthetic-data training and baseline behavior when speakers pause.The model is publicly available on Hugging Face as a Turkish turn-taking baseline.Source: §I · p. 1

II. RELATED WORK

Turn-taking prediction has moved from silence thresholds toward continuous estimates, while evaluation remains fragmented. This work adopts a text-based prediction formulation, situates its Turkish setting, and evaluates it against text and deployed-system baselines.

A. Turn-taking prediction

  • Silence-threshold systems can interrupt speakers who pause or respond late to speakers who finish quickly, while continuous prediction uses future speech activity and acoustic and lexical features.Voice Activity Projection learns discretised future voice-activity patterns, and multilingual extensions cover English, Mandarin and Japanese; benchmark efforts address fragmented evaluation for English.Source: A. Turn-taking prediction · p. 2

B. Text-based models

  • TurnGPT uses a turn-shift token and causal language-modelling loss to predict turn shifts incrementally from preceding context.A deployed text-based detector and a zero-shot language-model reading serve as baselines for task-specific fine-tuning.Source: B. Text-based models · p. 2

C. Turkish

  • Turkish is agglutinative and predominantly verb-final, concentrating tense, person, mood and question marking in suffixes on the clause-final verb.The paper uses a recently available monolingual Turkish transformer as both its backbone and zero-shot baseline.Source: C. Turkish · p. 2

III. DATA

The study uses synthetic dialogues and real dyadic conversations, but their turn structures differ: the synthetic turns are short and complete, while real turns are longer and can pause before resuming. Real-TurnTurk turn boundaries are derived from voice activity and aligned transcript segments.

III. DATA

  • Syn-TurnTurk contains topic-conditioned two-party dialogues generated by a large language model, while Real-TurnTurk contains six dyadic video-call recordings with six distinct speakers.The real corpus includes separate audio channels, per-channel voice activity detection, backchannel annotation and platform live-caption transcripts; only text and turn structure are used.Source: §III · p. 2
  • Among transcript words, 4.7% in segments straddling a turn boundary and 0.5% conflicting with voice-activity annotation are discarded [11].Real turns merge consecutive segments from one speaker into a floor-holding turn. Transcript segments are assigned to turns by temporal overlap, and backchannels do not split floor-holding turns.Source: §III · p. 2
  • Of 930 turn-internal pauses, 633 could be attributed to a word, with a median duration of 0.50 s; 297 lasted longer than 0.5 s.These pause positions are treated as hard negatives; unlike short, complete synthetic turns, real turns are long and frequently resume after backchannels.Source: §III · p. 2

IV. METHOD

TurnTurk-GPT applies a TurnGPT-style causal language-model objective to serialized Turkish dialogue and compares four training configurations. Its input is normalized to approximate streaming-recognizer text, although evaluation uses final transcripts offline.

A. Model and objective

  • TurnTurk-GPT adds a turn-shift token and two speaker tokens to turkish-gpt2, serializes turns with a leading and turn-closing <ts>, and scores each word by P(<ts>) at its final subword token.Speaker identity is supplied through token_type_ids, and training uses the causal language-modelling loss.Source: A. Model and objective · p. 2

B. Training configurations and input

  • S, R, SR and S2R denote synthetic-only, real-only, one-stage mixed-data and synthetic-pretraining-then-real-finetuning configurations, respectively.A fixed step budget accommodates corpora differing in size by two orders of magnitude; each configuration uses three seeds averaged per position.Source: B. Training configurations and input · p. 2
  • The input is lower-cased text with punctuation removed to approximate streaming-recognizer output, but evaluation uses final transcripts and is offline.For early stopping, the final 12% of windows from each training dialogue form the development set, with an adjacent window discarded to prevent overlap.Source: B. Training configurations and input · pp. 2–3

V. EVALUATION

The evaluation scores every word in real Turkish conversation as a turn-ending decision point, using average precision and two disjoint data splits. It compares simple, zero-shot, and multilingual baselines under a nested threshold and recording-level bootstrap protocol.

A. Protocol

  • The protocol labels each of 15,522 real-corpus word positions as a decision point, with 578 positive turn ends; average precision is primary, and random AP is 0.037.Thresholds are selected on other folds; false alarms are also measured on 633 words preceding turn-internal pauses.Source: A. Protocol · p. 3
  • Recording-disjoint evaluation leaves one recording out, while speaker-disjoint evaluation excludes test speakers from training, leaving two or three training recordings.Recording-level cluster bootstrap intervals and paired comparisons resample six recordings; a difference is significant when its 95% interval excludes zero.Source: A. Protocol · p. 3

B. Baselines

  • B0 is random, while B1 is a turn-length prior estimated from the training folds.Source: B. Baselines · p. 3
  • The zero-shot baseline scores each word using four untuned-backbone probabilities, including newline probability as a direct estimate of whether the turn ends.The four readings are punctuation, end-of-text, newline, and their sum.Source: B. Baselines · p. 3
  • The multilingual baseline uses LiveKit’s released detector and its own lower-casing, punctuation-removal, and context processing.Its input includes the current turn through the scored word.Source: B. Baselines · p. 3

VI. RESULTS

TurnTurk-GPT beats the multilingual detector in both evaluation splits, while its advantage over the zero-shot model is limited to recording-disjoint evaluation. Synthetic pre-training brings no significant general gain, though it improves calibration.

A. Main comparison

  • S2R reaches AP 0.120 on the speaker-disjoint split.Source: A. Main comparison · p. 3, Table II · p. 3
  • TurnTurk-GPT outperforms the multilingual detector on both splits, but exceeds the zero-shot model only on recording-disjoint evaluation; unseen-speaker differences are not significant.The main results and paired-comparison tables present these evaluations.Source: Table II · p. 3, Table III · p. 3, A. Main comparison · p. 3
  • Synthetic pre-training yields no significant general gain over real data alone, but improves calibration: the threshold for 70% recall is 0.016 for S2R versus 0.00000 for R.The paper identifies calibration, rather than AP gain, as the synthetic corpus’s clearest contribution.Source: A. Main comparison · p. 3

B. Operating points

At fixed recall, the fine-tuned model and zero-shot model perform similarly, while both compare favorably with the multilingual detector. False alarms remain substantial, limiting the usefulness of text-only systems as stand-alone turn-end triggers.

B. Operating points

  • At 70% recall, the multilingual detector produces 16.0 false alarms per correct detection versus 9.6 for S2R, and predicts turn endings during pauses in 65.4% versus 51.7% of cases.Source: Table IV · p. 4, B. Operating points · p. 4
  • At matched recall, the fine-tuned and zero-shot models perform almost identically; at 90% recall, zero-shot is slightly ahead, so S2R’s AP edge comes from high-confidence, low-recall decisions.Source: B. Operating points · p. 4
  • At 70% recall, the fine-tuned model produces approximately ten false alarms per correct detection, and none of the text-only systems is suitable as a stand-alone trigger.Source: B. Operating points · pp. 3–4

C. When the speaker pauses

Text-only baselines struggle to distinguish completed turns from within-turn pauses. Turkish-specific analyses show that turn-final suffixes carry important end-of-turn information, while an English tokenizer fragments that signal.

C. When the speaker pauses

D. Turkish-specific analyses

  • Stemming the final word cuts AP by 32% (R) and 44% (S2R), versus 0.2% and 2.2% for internal-word stemming.The end-of-turn cue is located in suffixes of the final, typically verbal word.Source: D. Turkish-specific analyses · p. 4, Table VI · p. 4
  • The English GPT-2 tokenizer segments Turkish into 2.94 tokens per word versus 1.17 with the Turkish tokenizer, reducing R AP from 0.116 to 0.042, near random.Source: D. Turkish-specific analyses · p. 4

E. Synthetic data

The untuned Turkish model, read through newline probability, is a strong baseline against the multilingual detector. Synthetic-only training is weak, while fine-tuning gains over zero-shot are limited to familiar speakers.

E. Synthetic data

  • Synthetic-only training reaches AP 0.082, below the zero-shot model, and has the highest pre-pause false-alarm rate at 33.5%.Source: E. Synthetic data · p. 4
  • Synthetic pre-training helps relative to mixing synthetic and real data in one stage, but not significantly relative to real data alone; its clearest benefit is calibration.Source: E. Synthetic data · p. 4

A. Why the zero-shot model is a strong baseline

B. Implications for deployment

Deployment comparisons should use matched recall and in-domain thresholds: practical false-alarm differences do not always track statistical significance. The discussion also identifies limits in synthetic dialogue and in the evaluation setting.

B. Implications for deployment

  • At 70% recall, TurnTurk-GPT has approximately six fewer false alarms per correct detection than the multilingual detector; its +0.019 AP over zero-shot is significant but does not reduce false alarms at practical recall.Source: B. Implications for deployment · p. 5
  • At the multilingual detector's published Turkish threshold, 85.5% of pre-pause words trigger false alarms, supporting in-domain threshold selection.Source: B. Implications for deployment · p. 5
  • Compare systems at matched recall: false-alarm rates at each model's own optimal threshold can differ twofold despite equivalent performance.Source: B. Implications for deployment · p. 5

C. Implications for synthetic dialogue

D. Scope of the evaluation

  • The six-conversation corpus has six distinct speakers, and the speaker-disjoint split measures generalization to speakers absent from training.Source: D. Scope of the evaluation · p. 5
  • Evaluation uses offline final transcripts, not streaming recognizer hypotheses; testing partial hypotheses and latency is a natural extension.Source: D. Scope of the evaluation · p. 5
  • Labels mark floor transfer, which is related to but not identical with whether a response is appropriate; systems are compared on the same decision points.Source: D. Scope of the evaluation · p. 5

VIII. CONCLUSION

TurnTurk-GPT is a Turkish text-based turn-taking predictor evaluated against rule-based, zero-shot, and deployed systems. It beats the multilingual detector on both splits, while its zero-shot advantage is limited to familiar speakers.

VIII. CONCLUSION

  • TurnTurk-GPT outperforms the multilingual detector on both splits, but its advantage over zero-shot is limited to familiar speakers; at deployment-relevant recall, the two text models are equivalent.Source: §VIII · p. 5
  • Turn-final suffixes carry the end-of-turn signal, and an English tokenizer that fragments them reduces performance to near chance; text-only baselines also approach chance during pauses.Source: §VIII · p. 5
  • Future work calls for more speakers, synthetic dialogue with pauses and resumptions, and prosodic or visual cues; the model and evaluation protocol are released.Source: §VIII · p. 5
Loading 2610.42010v1…