PaperlayerView exact revision on arXiv

Source-linked AI summary

TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog

Erik Ekstedt, Gabriel Skantze

arXiv:2010.10874v1cs.CLcs.AI

TL;DR

Turn-taking models have used linguistic information in limited ways despite the importance of syntactic and pragmatic completeness. TurnGPT models turn-shifts with a transformer language model, and it outperforms prior linguistic baselines while analyses indicate it uses dialog context. The paper also explores generating possible futures to project turn-completions.

  • Problem

    Prior turn-taking models used linguistic information in limited ways, although syntactic and pragmatic completeness are important for predicting turn-shifts.

    Source: §1 · p. 1

  • Method

    TurnGPT fine-tunes a GPT-2-based transformer on dialog and estimates turn-shift probability from speaker-token outputs.

    Source: §1 · p. 1, §4 · p. 3

  • Results

    TurnGPT achieved the best results on all evaluated datasets, and context ablations and model inspections indicate that it uses dialog context and pragmatic completeness.

    Source: §6.2 · p. 5, §6.3 · p. 6, §6.4 · p. 7

  • Takeaways & Limitations

    The model’s context representation supports identifying pragmatic rather than merely syntactic completion, and language-model generation offers a way to inspect possible futures for upcoming turn-shifts.

    Source: §6.5 · p. 7, §7 · p. 8

    The study models linguistic cues without evaluating how prosody or other cues such as gaze and gestures contribute or combine with the model.

    Source: §1 · p. 1, §7 · p. 8

Abstract

from arXiv · show

Syntactic and pragmatic completeness is known to be important for turn-taking prediction, but so far machine learning models of turn-taking have used such linguistic information in a limited way. In this paper, we introduce TurnGPT, a transformer-based language model for predicting turn-shifts in spoken dialog. The model has been trained and evaluated on a variety of written and spoken dialog datasets. We show that the model outperforms two baselines used in prior work. We also report on an ablation study, as well as attention and gradient analyses, which show that the model is able to utilize the dialog context and pragmatic completeness for turn-taking prediction. Finally, we explore the model's potential in not only detecting, but also projecting, turn-completions.

1 Introduction

Turn-taking depends on coordinating speaker transitions, yet simple silence thresholds can cause interruptions or sluggish responses. TurnGPT uses a transformer language model to predict turn-completion points from words, addressing the limited use of complex linguistic information in prior models.

1 Introduction

  • Silence-threshold turn-taking systems can produce interruptions or sluggish responses, while human speakers use prosodic and linguistic cues as well as gaze and gestures.People can detect and project turn completions using these cues.Source: §1 · p. 1
  • Prior models have used linguistic representations simplistically or relied solely on prosody, despite linguistic cues depending on complex semantic and pragmatic functions and preceding context.The paper identifies capturing these functions as a challenge for machine learning models.Source: §1 · p. 1
  • TurnGPT applies a transformer-based language model to predict turn-completion points from words and dialog datasets.The model is evaluated on written and spoken dialog and analyzed through ablations and model inspection.Source: §1 · p. 1

2 Background

Turn-taking accounts describe possible transition points and distinguish syntactic from pragmatic completeness. Prior prediction models have represented linguistic cues simply, motivating more powerful linguistic models.

2 Background

  • Turn-taking is organized into units, after which a transition-relevant place is available for a possible, but not necessary, turn-shift.Whether a shift occurs depends on speaker selection or another speaker self-selecting.Source: §2 · p. 2
  • Syntactic completeness means an utterance can be interpreted as a complete clause in context, whereas pragmatic completeness means a complete conversational action in its sequential context.A syntactically complete question may still be pragmatically incomplete when the preceding context makes an answer relevant.Source: §2 · p. 2
  • The example contrasts an incomplete turn-initial “yesterday” with a syntactically complete “tomorrow,” illustrating that syntactic completeness depends on discourse context.The example marks syntactic completeness with slashes.Source: §2 · p. 2, Figure 1
  • Prosody can signal pragmatic completion, but split-second turn-shifts suggest listeners must project completions ahead of time rather than rely only on turn-final prosody.The cited response-time estimate is around 600–1500 ms, compared with turn-shifts around 200 ms.Source: §2 · p. 2
  • Earlier models often used simple part-of-speech cues or LSTMs, and linguistic features yielded less performance gain than expected, motivating more powerful linguistic models.The reviewed LSTM approaches encoded part-of-speech tags, words, or senones.Source: §2 · pp. 2–3

3 Approach

The approach treats turn-shifts as probabilistic transition-relevance points learned from conversational language, rather than requiring explicit annotations. It studies linguistic cues in both written and spoken dialog without relying on punctuation or capitalization.

3 Approach

  • Because transition-relevance places are not overtly marked and manual annotation is labor-intensive, the approach infers their probabilistic strength from observed turn-shifts.This treats transition relevance as varying in appropriateness rather than as strictly binary.Source: §3 · p. 3
  • The approach learns turn-shift probabilities as a language-model distribution over special speaker-specific turn-shift tokens inserted at the beginning of each turn.The probabilities associated with these tokens are interpreted as probabilities of transition-relevance points, while the model continues predicting all vocabulary tokens.Source: §3 · p. 3
  • The study uses written and spoken dialog to learn linguistic transition-relevance distributions, recognizing that pragmatic points may overlap across the two data types.The stated goal is to avoid relying on punctuation or capitalization, which spoken language lacks.Source: §3 · p. 3

4 Model

TurnGPT is a GPT-2-based transformer that incorporates speaker identity into language modeling and uses speaker-token outputs to estimate transition-relevance probabilities. The study compares it with POS-bigram and LSTM linguistic baselines.

4 Model

  • The model uses a unidirectional GPT-2 language model, adapting the TransferTransfo setup by omitting its task-specific token and next-sentence prediction loss.The paper chose GPT-2 for its performance on language generation.Source: §4 · p. 3
  • TurnGPT uses word, position, and speaker-id embeddings, with transition-relevance probability defined as the maximum output probability assigned to speaker tokens.The speaker tokens are included in the language-modeling task.Source: §4 · p. 3
  • The experiments fine-tune two pretrained models, GPT-2 trained on WebText and DialoGPT, using the smallest models available from the Transformers library: GPT-2-base and DialoGPT-small.Source: §4 · p. 3
  • Table 1 summarizes dataset statistics, including dialog counts, turn counts, words per turn, and unique words.The table covers the datasets used in the experiments.Source: Table 1 · p. 4, §4 · p. 4
  • TurnGPT is compared with a POS-bigram model and an LSTM classifier trained on preceding words.The LSTM baseline can use up to three layers with hidden size 768.Source: §4 · p. 4

5 Data

The study draws on eight dialog datasets spanning assistant, written social, and spontaneous spoken conversations. It standardizes the data and defines a custom turn-extraction policy for overlapping spontaneous speech.

5 Data

  • Assistant datasets primarily represent user queries to automated systems, while the collection also includes written social and spontaneous spoken human-human dialogs.Assistant data were collected through Wizard-of-Oz and self-written dialogs; written social dialogs are more open and social.Source: §5 · p. 4
  • Written datasets include DailyDialog's varied everyday topics and Persona conversations in which crowdworkers get to know each other.Spontaneous spoken examples include Maptask route-giving and topic-constrained Switchboard telephone conversations.Source: §5 · p. 4
  • The eight datasets were combined into a Full dataset, then split into training, validation, and test sets using predefined or 90/5/5 random splits.Table 1 reports basic statistics for each dataset.Source: §5 · p. 4

5.1 Data Extraction

  • For spontaneous speech, isolated backchannels were removed, IPUs were formed using a 500 ms threshold, and consecutive speaker IPUs were merged into turns.Overlapping IPUs fully contained within another speaker's IPU were omitted, and the resulting turns were ordered by time.Source: §5.1 · p. 5

6 Experiment

The experiments evaluate TurnGPT with balanced accuracy, vary the dialogue context available to it, inspect attention and integrated gradients, and probe generation-based turn-completion projection.

6.1 Training

  • TurnGPT models are trained on Assistant and Full datasets with cross-entropy loss; the LSTM baseline is trained as a binary turn-shift classifier.Models with the lowest validation loss are selected for testing.Source: §6.1 · p. 5

6.2 Evaluation

  • Balanced accuracy (bAcc) evaluates true and false turn-shifts to account for the much greater number of word tokens than turn-shifts.The models’ TRP probabilities are thresholded to classify turn-shifts.Source: §6.2 · p. 5
  • In the example, TurnGPT distinguishes “yesterday” from context-dependent “tomorrow,” while LSTM assigns high turn-shift probability to a syntactic but not pragmatic completion.The LSTM also gives a fairly high probability after “when will you meet,” unlike TurnGPT.Source: §6.2 · pp. 5–6

6.3 Context Ablation

  • TurnGPT performance increases with more context, with the largest drop when moving from some context to none; LSTM follows this trend less strongly.On Written data, LSTM slightly improves when moving from one context turn to none.Source: §6.3 · p. 6

6.4 Model Inspection

  • Attention averages about 70% on the current turn, while the remaining 30% comes from the four preceding turns.Attention is normalized across the five most recent turns, which sum to 1 for each sample.Source: §6.4 · p. 7, Figure 5 · p. 7
  • Integrated gradients show positive and negative contributions: the immediate turn contributes positively on average, while preceding turns contribute more negatively.The authors suggest, but leave for future investigation, that prior context may distinguish syntactic from pragmatic completion.Source: §6.4 · p. 7, Figure 6 · p. 7

6.5 Future Prediction

  • Autoregressive sampling estimates tokens remaining until a generated speaker token, offering a way to project rather than only detect turn-completions.In the constructed example, predicted completions become shorter and more stable nearer the end of the turn.Source: §6.5 · pp. 7–8

7 Conclusions and Discussion

TurnGPT formulates turn-shift prediction as language modeling and outperforms prior-work baselines, with model inspection indicating a role for context and pragmatic completion. The discussion frames the work as a step toward richer turn-taking models and further analysis of how words shape predictions.

7 Conclusions and Discussion

  • TurnGPT, a fine-tuned GPT-2 with special turn-shift tokens, outperformed prior-work baselines; ablations and inspection link performance partly to context and pragmatic completion.The paper attributes this partly to the model's ability to identify pragmatic rather than merely syntactic completion.Source: §7 · p. 8
  • Autoregressive generation lets TurnGPT inspect possible futures as a way to predict upcoming turn-shifts; Figure 7 pairs predicted turn-length histograms with generated sample text.Turns in the figure are separated by dashed lines.Source: §7 · p. 8, Figure 7 · p. 8
  • For spoken dialog, integrating prosody with TurnGPT is future work; prosodic information can help disambiguate pragmatic completion.The discussion also identifies gaze and gestures as cues for a more powerful turn-taking model.Source: §7 · p. 8
  • Further inspection and ablation could pinpoint how words shape turn-completion predictions and potentially inform understanding of human turn coordination.Source: §7 · p. 8

Related papers

Loading 2010.10874v1…