Source-linked AI summary
Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness
Mark Dourado, Karim Haddad, Henrik G. Hassager, Stefania Serafin
TL;DR
Turn-taking in noisy, natural four-person dialogue requires predicting whether floor transfers become gaps or overlaps from interacting verbal and non-verbal cues. This study trains interpretable logistic-regression models on pre-turn gaze, loudness, and interpersonal-closeness features, finding that multimodal prediction is robust and that gaze complements speech intensity.
Problem
Polyadic conversation creates multiple competing next-speaker possibilities, motivating models of how multimodal cues predict gap or overlap outcomes.
Method
The study trains logistic-regression models on interpretable pre-turn gaze, speech-intensity, and directional interpersonal-closeness features from natural four-person dialogue.
Results
ROC AUC was 0.76 ± 0.04 for the multimodal model versus 0.58 ± 0.05 for gaze-only, with positive gains from adding gaze or phon features across folds.
Takeaways & Limitations
Gaze carries predictive structure, complementing loudness by indexing recipient-oriented and anticipatory behaviour across speakers and acoustic conditions.
Takeaways & Limitations
The feature-engineered pre-turn windows may underrepresent dynamics outside the interval, and the analysis does not examine how interpersonal closeness covaries with gaze or speech.
Abstract
from arXiv · showhide
Smooth speaker transitions are fundamental to effective conversation and rely on an interlocutor's ability to predict when to enter the conversation. This ability depends on accurately interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-party settings, with multiple interlocutors available. This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue. Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event to classify floor-transfer outcomes as gaps or overlaps. Predictors included gaze features such as transition motifs and behavioural contrasts, entropy, gaze-based addressee identity, and mutual gaze, alongside speech features derived from speaker loudness, as well as perceived interpersonal closeness (IOS) between speakers. Results show that gaze features capture predictive structure, and that combining them with loudness improves performance (ROC AUC = 0.76 +- 0.04). Loudness reflected speaker control, while gaze dispersion and addressing indexed listener readiness and competitive entry. Performance remained robust across noise conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking dynamics.
1 Introduction
Turn-taking in natural four-person conversation requires integrating verbal and non-verbal cues because multiple interlocutors compete for the floor, especially under noise.
- Turn transitions typically occur within a few hundred milliseconds, requiring listeners to anticipate upcoming speaker changes.
- Gaze regulates turns by signaling completion, selecting an intended recipient, or holding the floor through withdrawal.
- Gaze shifts and head orientations can predict the next speaker and onset timing several hundred milliseconds before speech begins.
- The GaMMA corpus supports analysis of spontaneous four-person coordination using synchronised audio, gaze, and motion data from quiet and noisy conditions.
- The study models whether floor transfers become conversational gaps or overlaps using gaze, speech, and interpersonal closeness.
2 Floor-Transfer Offset and Conversational Event Detection
The study detects floor-transfer events from voice activity and characterizes their timing with signed floor-transfer offset, distinguishing gaps from overlaps.
- Voice-activity detection produces binary speech streams that define temporal regions where conversational events can occur.
- Turns are continuous speech periods by one speaker, bounded by transitions in floor ownership and derived from VAD-based activity streams.
- Floor-transfer offset measures the interval between an outgoing speaker’s offset and an incoming speaker’s onset when the floor changes.
- FTO_i < 0 denotes an overlap transfer, whereas FTO_i ≥0 denotes a gap transfer with silence between turns.
3 Gaze Data Representation and CSC Alignment
Gaze is represented as continuous relative direction and discrete target states, then aligned with speech through the CSC framework for multimodal feature extraction.
- Each participant’s gaze is recorded continuously as a 3D vector and converted into relative angles to other participants’ centres of mass.
- Discrete gaze states identify one of the other three co-participants or an Outside category, with active targets defined by a ±15° angle threshold.
- CSC consolidates speech activity, IPUs, FTOs, floor changes, and synchronised per-speaker loudness into machine-readable sequences.
- The integrated CSC representation combines speech-state and discretised gaze streams as the basis for subsequent multimodal feature extraction.
Model Summary
The models use interpretable multimodal features computed before each floor-transfer event, with matched multimodal and gaze-only variants for comparison.
- A 3 s pre-turn window aligned to each event captures speech and gaze activity immediately preceding the next speaker’s onset.
- The feature set includes loudness, gaze contrasts, gaze motifs, static and transition entropy, interpersonal closeness, addressee identity, and mutual gaze.
- Interpersonal closeness is represented by directional dyadic IOS scores between the active speaker and candidate target.
- Addressing variables encode addressee identity and mutual gaze at the end of an interpausal unit.
- Two variants use identical folds and event sets: a multimodal model and a gaze-only model excluding Phons.
4 Methods
The study extracts interpretable gaze, loudness, and interpersonal-closeness features from pre-turn windows in four-person dialogue. Gaze dynamics are represented through symbolic states, transition structure, entropy, and addressee-related measures.
- Loudness Features: Loudness was computed in phons from calibrated close-microphone signals, using whole-window and 200 ms pre-transition averages.The ISO 532-1 Zwicker method was applied to denoised, cross-talk-attenuated recordings.
- Symbolic Gaze Bigrams: Gaze streams were encoded in a seven-state space representing interlocutors and Outside, with mutual gaze included as a state.Successive identical states were collapsed before transition extraction.
- Symbolic Gaze Bigrams: Bigrams were counted in normalized 7 × 7 transition matrices and flattened into 49-dimensional feature vectors.These vectors were subsequently reduced using PCA or NMF.
- Dimensionality Reduction: PCA retained five components explaining more than 90% of variance, while NMF produced additive nonnegative motif activations.Both decompositions were fit separately within each cross-validation fold to prevent information leakage.
- Gaze Entropy: Static Gaze Entropy measured gaze-target dispersion, whereas Gaze Transition Entropy measured transition unpredictability.Lower entropy indicates concentrated or repetitive gaze behavior; higher entropy indicates dispersed or variable behavior.
- Addressing and Interpersonal Closeness: Addressee features used the final 600 ms of gaze, a 15° target cone, mutual-gaze coding, categorical identity, and fold-wise addressed weights.Interpersonal closeness was represented by directional, static dyadic IOS ratings that were standardized within each rater.
5 Model Representation and Training
The models classify genuine floor-transfer events as gaps or overlaps using multimodal predictors and regularised logistic regression. Evaluation uses leakage-controlled feature construction and held-out group and condition folds.
- Outcome Variable: Floor-transfer labels classify FTO_i ≥0 as Gap and FTO_i <0 as Overlap, excluding backchannels and within-speaker overlaps.Labels are assigned only when the conversational floor changes.
- Feature Representation: Each event vector combines loudness, gaze entropy, PCA, NMF, IOS, addressee identity, mutual gaze, and addressed-weight predictors.All features were computed independently within folds.
- Model Training: Elastic-net logistic regression used α = 0.5, with λ tuned by inner five-fold cross-validation.The outer evaluation used leave-one-group-out and leave-one-condition-out validation across 15 folds.
- Evaluation: Held-out performance was evaluated with ROC AUC, PR AUC, F1-score, Precision, Recall, and Accuracy.These metrics were computed under both group-based and condition-based outer validation.
6 Results
Multimodal models combining gaze and speech intensity more reliably distinguished gaps from overlaps than gaze-only models. Gaze contributed complementary, noise-robust information, while loudness remained the dominant cue associated with speaker control.
- 6.1 Overall Predictive Performance: 0.76 ± 0.04 ROC AUC and 0.77 ± 0.05 PR AUC were achieved by the multimodal model, versus 0.58 ± 0.05 and 0.60 ± 0.06 for gaze-only.Average paired improvements were ΔAUC = +0.18 and ΔPR = +0.17, consistent across folds (Wilcoxon p < 0.001).
- 6.2 Added Value of Gaze and Phonetic Features: +0.17 in both PR-AUC and ROC-AUC resulted from adding phonetic features to the gaze-only baseline, whereas adding gaze to phon-only improved PR-AUC by +0.08 ± 0.02 and ROC-AUC by +0.03.All folds showed positive gains when gaze was added to the phon-only baseline (Wilcoxon p < 0.001).
- 6.3 Performance Across Acoustic Conditions: 0.74–0.77 multimodal AUCs exceeded gaze-only AUCs of 0.56–0.59 across quiet, 55 dB, 65 dB, and varying acoustic conditions.The multimodal advantage persisted across noise environments.
- 6.3 Performance Across Acoustic Conditions: At the best-F1 threshold, multimodal performance was Precision = 0.62, Recall = 0.80, F1 = 0.69, and Accuracy = 0.71.The corresponding gaze-only values were 0.50, 1.00, 0.66, and 0.58.
- 6.4 Feature Contributions and Stability: Phon intensity windows of 200–600 ms were the dominant feature block, while gaze entropy and addressing cues were frequently selected and stable across folds.NMF gaze-transition motifs showed elevated odds ratios toward overlap, whereas PCA axes had mixed associations.
- 6.5 Directionality of Strong Predictors: High phon intensity decreased overlap probability, while gaze patterns retained directional tendencies toward recipient readiness and competitive entry, albeit with smaller relative magnitude.Together, the modalities provided complementary signals: vocal intensity indexed speaker hold strength, while gaze captured recipient design and readiness.
7 Discussion
The study finds that gaze contributes stable, interpretable information for predicting gap and overlap outcomes, complementing loudness across speakers and acoustic conditions. Its conclusions are bounded by the static treatment of interpersonal closeness, data-specific interpretations, sampling constraints, and pre-turn feature windows.
- Gaze-derived features alone forecast floor-transfer outcomes, while multimodal integration improves prediction across speakers and acoustic conditions.The predictors include entropy, addressing, and component-level gaze structure.
- Phon intensity primarily captures speaker control, whereas gaze captures anticipatory, recipient-oriented behaviour during turn competition.Together, the modalities represent complementary aspects of speaker continuation and listener approach to the floor.
- Interpersonal closeness was included as a static, directional dyadic feature, but its interaction with gaze and speech behaviours was not examined.The authors identify relational expression through specific behavioural patterns as a direction for subsequent analysis.
- PCA and NMF provide interpretable representations of recurrent gaze transitions, with NMF components strongly associated with overlap.PCA represents broad variance contrasts, while NMF captures additive motif-like structures.
- Addressing showed a positive association with gap, plausibly reflecting preparation time for the selected recipient before uptake.The authors present this as a data-specific explanation whose expression may depend on condition and local interactional context.
- Sparse, current-speaker-only loudness features reduce leakage risk but necessarily limit raw AUC values for speech-based models.Features were aligned to the outgoing speaker’s offset and excluded candidate or future-speaker information.
- Purposive sampling of native speakers in controlled four-person conversations limits the setting’s participant and recording-condition scope.The study used unscripted free conversation without imposed tasks or turn-taking constraints.
- Feature-engineered pre-turn windows may underrepresent dynamics outside the interval, motivating temporally continuous, context-aware modelling.The authors also propose adding head orientation, gesture, conversational structure, and linguistic context.
8 Conclusion
The study presents a multimodal model of gap-versus-overlap outcomes in natural four-person dialogue. It finds that gaze carries predictive structure complementary to speech intensity and remains observable when acoustic information is degraded.
- Gaze entropy, addressing, and component-level transitions provide consistent information beyond speech intensity for predicting gaps and overlaps.
- The multimodal model generalises across speaker groups and acoustic conditions while avoiding leakage from diarisation or candidate speakers.
- Gaze offers a continuous behavioural stream that complements speech cues, which depend on signal-to-noise ratio and occur in discrete units.
Data Availability
The analysed data are publicly available through the GaMMA corpus on Zenodo.
- The GaMMA corpus is publicly available via Zenodo.The associated data descriptor documents collection procedures, processing pipelines, ethics approval, and usage notes.
Code Availability
The full analysis code is not publicly available because it is subject to intellectual property restrictions, although the procedures are described for independent replication.
- The full analysis codebase cannot currently be released because an industrial research collaboration imposes intellectual property restrictions.The Methods section describes the analysis procedures and modelling steps in sufficient detail for independent replication.