Source-linked AI summary
Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Prediction
Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Bekir Berker Türker, Mustafa Sertaç Türkel, Alper Kaplan
TL;DR
Naturalistic Turkish turn-taking data remain limited, despite the difficulty of predicting turn completion in synchronous dialogue. The paper introduces a multimodal corpus and evolves interpretable hybrid AND–OR rules over visual, acoustic, and linguistic features. The resulting rule achieves F1 = 57.0%, modestly above the always-positive baseline of F1 = 50.0, while speaker imbalance limits interpretation for unseen speakers.
Problem
Naturalistic Turkish conversational corpora with primary turn-taking annotations are lacking, while silence-based systems struggle with variable human pausing and overlapping speech.
Method
The study builds a multimodal unscripted dyadic corpus and formulates turn-transition prediction as binary classification using interpretable hybrid AND–OR rules optimized by a genetic algorithm.
Results
F1 = 57.0%, above the always-positive baseline of F1 = 50.0, with the final rule using five of 28 features and one visual condition.
Takeaways & Limitations
The rule combines alternative prosodic, visual-acoustic, and linguistic cue pathways and can be inspected directly rather than treated as a black box.
Takeaways & Limitations
Recurring speakers appear in every conversation, so the reported figures should be read as an upper bound on generalization to unseen speakers.
Abstract
from arXiv · showhide
Turn-taking is a basic organizational feature of human conversation and remains difficult to model in natural, synchronous dialog systems. While existing research has explored multimodal approaches and large language models for turn-ending prediction, there is a lack of naturalistic conversational corpora specifically addressing turn-taking dynamics in Turkish. This study introduces a multimodal Turkish conversational dataset of unscripted dyadic interactions, comprising synchronized front-facing video, per-speaker audio channels that allow overlapping speech to be attributed to individual speakers, and time-aligned transcriptions. Turn-taking prediction is formulated as a binary classification problem, and a Genetic Algorithm (GA) is employed to optimize interpretable decision rules derived from visual, acoustic, and linguistic features. A hybrid AND-OR rule representation is adopted in the proposed framework to represent the alternative cue combinations that precede a turn transition.
I. INTRODUCTION
Turn-taking systems often rely on silence thresholds that mishandle variable human pausing and cause premature transitions or overlap. Real-TurnTurk addresses the lack of naturalistic Turkish turn-taking corpora with multimodal unscripted conversations and interpretable rule optimization.
- Motivation: Silence-based completion rules can misjudge rapid or deliberate pauses, causing premature transitions and degraded dialogue synchronization.Human speakers vary in pause duration, so a predetermined silence interval cannot reliably identify turn completion.
- Motivation: Acoustic thresholds alone cannot capture hold-versus-shift distinctions, while conversational context and task type affect turn-taking dynamics.Metrics from structured elicitation tasks may not generalize to naturalistic dialogue.
- Motivation: Naturalistic Turkish resources remain limited for primary turn-taking annotation despite growth in sentiment and synthetic turn-taking datasets.The study targets this corpus gap directly.
- Contribution: Real-TurnTurk provides unscripted dyadic interactions with synchronized video, mostly per-speaker audio, and time-aligned transcriptions.The dataset supports speaker-attributed overlapping speech and multimodal analysis.
II. DATA
The dataset contains unscripted dyadic conversations with synchronized audiovisual and transcript data, and turn changes are labelled using speaker, duration, content, and timing criteria. The corpus includes substantial overlap and a speaker imbalance that constrains generalization evaluation.
- Dataset: The corpus contains 11 unscripted dyadic interactions totaling 4.23 hours of synchronized text, audio, and video.Audio was recorded separately per speaker, while video used 1920×1080 resolution at 16 fps.
- Dataset: 1,750 turn-taking instances were annotated against 3,500 uniformly sampled negatives with a fixed 1:2 class ratio and a ±1.0 s exclusion buffer.Negative sampling was stratified by conversation duration and used seed 42.
- Dataset: Two recurring interlocutors appear in every conversation and account for 60.8% of words and 60.1% of speaking time.This imbalance constrains speaker-level cross-validation and generalization to unseen speakers.
- Turn Labelling: Turn-change labelling merges consecutive same-speaker transcript cues, detects speaker transitions, and filters backchannels using four concurrent criteria.The criteria concern speaker identity, duration, textual content, and incoming-segment onset time.
- Turn Labelling: A 0.5 s duration threshold excludes brief Turkish feedback signals, while filler tokens and segments of three characters or fewer are excluded as acknowledgments.These rules distinguish candidate turn initiations from minimal feedback signals.
- Dataset: 81.4% of annotated transitions contain overlap, measured per speaker in the nine interactions with separated channels.Overlapping speech was retained rather than excluded from analysis.
III. FEATURE GENERATION
Each candidate turn-change point is represented by 28 features extracted from the current speaker during the preceding two-second analysis window.
- Feature Generation: 28 features are extracted per 2-second analysis window preceding each candidate turn-change point.All features describe the current speaker holding the turn during that window.
A. Visual Features
The visual feature set captures facial activity, gaze, head motion, and expressiveness from the current speaker. These features target anticipatory behaviors such as looking away, nodding, or reduced facial activity before yielding.
- A. Visual Features: 9 visual features capture facial activity, gaze behavior, head motion, and expressiveness.The set includes landmark motion, action-unit change, blinking, gaze direction, mouth aperture, head rotation and translation, eyebrow displacement, and lip-corner pull.
- A. Visual Features: Speakers may display anticipatory visual signals shortly before yielding, including looking away, nodding, or reduced facial activity.These behaviors motivate including visual cues in turn-taking prediction.
B. Acoustic Features
The acoustic feature set represents prosodic and spectral properties of the current speaker, including pitch, energy, spectral measures, voicing, and onset activity.
- B. Acoustic Features: 12 acoustic features capture prosodic and spectral properties of the current speaker.They include F0, RMS energy, spectral centroid, rolloff, bandwidth, contrast, voiced-frame ratio, and onset rate.
- B. Acoustic Features: Energy drops and flattened pitch contours are commonly associated with completed turns.
C. Linguistic Features
The linguistic feature set captures lexical, syntactic, and discourse cues in the current speaker’s preceding two-second window, while the extraction pipeline aligns multimodal signals to that window.
- C. Linguistic Features: Seven linguistic features represent lexical, syntactic, and discourse properties of the current speaker.They include word duration, syllables per word, filler, interrogative, affirmative, repetition, and syntactic completeness indicators.
- C. Linguistic Features: Fillers and incomplete structures typically accompany continuation, whereas interrogatives and completed units mark potential transitions.
- D. Extraction Tools and Windowing: The pipeline uses each speaker’s audio channel and forced-aligned word timings while restricting features to the 2.0 seconds before the event.Visual features use only the current speaker’s crop, and no information at or after the event enters the feature vector.
A. Problem Formulation
Turn-transition prediction is framed as interpretable binary classification over multimodal features, with genetic search evolving hybrid AND–OR rules and evaluating them by F1.
- A. Problem Formulation: The classifier uses the preceding two-second feature window to predict whether the current speaker will yield or continue.A label of 1 denotes yielding the turn, while 0 denotes continuation.
- A. Problem Formulation: The framework seeks interpretable, computationally inexpensive rules suitable for real-time systems rather than black-box benchmark comparisons.
- A. Problem Formulation: Turn transitions may arise from alternative cue combinations, so the representation combines conjunctions and disjunctions rather than relying on one cue.
- B. Genetic Algorithm Based Rule Optimization: Each genetic-algorithm individual encodes feature conditions, numeric thresholds, comparison operators, and logical operators into a complete rule.Chromosomes contain 3 to 7 conditions, each linked to the next by AND or OR.
- B. Genetic Algorithm Based Rule Optimization: The genetic algorithm searches the combinatorial space of feature–threshold–operator combinations using population-based evolutionary optimization.
- B. Genetic Algorithm Based Rule Optimization: The procedure initializes random rules, evolves them across generations, and returns the rule with the highest F1-score.The supplied algorithm describes elitism, tournament selection, crossover, mutation, and an early-stopping condition.
- B. Genetic Algorithm Based Rule Optimization: Tournament selection, crossover, mutation, and elitism respectively regulate selection, recombination, perturbation, and preservation of strong rules.
- B. Genetic Algorithm Based Rule Optimization: Performance is tracked across generations and summarized in the 5-fold-average evolution table.
C. Cross-Validation Strategy
Cross-validation partitions data by conversation, trains the genetic algorithm only on training conversations, and evaluates the discovered rule on unseen test conversations.
- C. Cross-Validation Strategy: Five-fold cross-validation assigns whole conversations to folds, with four folds for training and one for testing in each run.Because there are 11 conversations, fold sizes are three, two, two, two, and two conversations.
- C. Cross-Validation Strategy: Speaker-specific leakage remains because recurring speakers occur in both training and test folds.The reported figures should therefore be read as an upper bound on generalization to unseen speakers.
V. RESULTS
The final GA rule improves turn-change prediction over the always-positive baseline and uses a small, interpretable subset of multimodal cues. Its selected conditions combine prosodic, visual-acoustic, and linguistic signals, while the evaluation cautions against interpreting the margin as high predictive accuracy.
- Performance: F1 = 57.0% exceeds the always-positive baseline by 7.0 points, with precision 46.1% and recall 74.6%.The baseline has precision 33.3%, recall 100.0%, and F1 = 50.0%; all silence thresholds perform below it.
- Evaluation context: The 1:2 class ratio makes the always-positive baseline a primary reference, while overlap and scarce non-speech weaken pause duration as an indicator.The corpus contains 1,750 positives and 3,500 negatives; 81.4% of transitions involve overlap and non-speech accounts for 12.4%.
- Evaluation context: The 7.0-point improvement is not treated as evidence of high predictive accuracy, and no statistical-significance claim is made because the corpus has 11 conversations.This limitation applies specifically to the margin over the always-positive baseline.
- Rule analysis: The GA rule is retrained on the entire dataset after evaluation to produce one interpretable rule for qualitative discussion.Its thresholds use the units listed in Table III.
- Rule analysis: Only 5 of 28 features are selected, with one visual condition and unequal contributions from the three modalities.The rule combines word duration with energy change, gaze shifts with mean F0, and prolonged fillers.
VI. CONCLUSION
The study contributes a multimodal Turkish corpus and an interpretable GA-based approach to turn-transition prediction. The resulting rule achieves a modest improvement over baseline performance while remaining directly inspectable.
- Contribution: The corpus contains 4.23 hours of synchronized video, per-speaker audio, time-stamped transcripts, and 1,750 filtered turn-change events.The prediction procedure uses an interpretable hybrid AND–OR rule representation over multimodal signals.
- Findings: The resulting rule reaches F1 = 57.0%, above the silence thresholds and the always-positive baseline of F1 = 50.0.The margin over the baseline is described as modest.
- Future work: Future work will test balanced non-hub recordings, speaker-normalized acoustic features, additional classifiers, and integration into real-time LLM-based agents.The proposed comparisons include Random Forest, XGBoost, and transformer-based classifiers.