Source-linked AI summary
Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios
Long-Vu Hoang, Naomi Harte
TL;DR
The generalisation of predictive turn-taking models to conversations with overlapping speech and background noise remains underexplored. This paper evaluates audio-visual models trained on clean data on AVCocktail and studies fine-tuning for adaptation. Performance degrades across modalities under domain shift, while fine-tuning can improve robustness in the new domain.
Problem
How clean-trained predictive turn-taking models generalise to cocktail-party conversations, where overlapping speech and noise challenge both acoustic and visual cues, remains underexplored.
Source: §I · p. 1Method
The study evaluates audio-visual turn-taking models trained on clean Candor data on AVCocktail, deriving shift/hold labels from sentence-level transcriptions and examining fine-tuning.
Source: §I · p. 1, C. The AVCocktail Dataset · p. 2, A. Model Architectures · p. 3Results
Across modalities, clean-pretrained VAP models lose 2% to 38% relatively in weighted F1 under domain shift, while fine-tuning can improve robustness by up to 30%.
Source: §V · p. 6Takeaways & Limitations
Fine-tuning can improve robustness in the new domain when pre-training captures basic turn-taking dynamics, especially with contextual information from audio and visual cues.
Source: §V · p. 6
Abstract
from arXiv · showhide
Current predictive turn-taking models (PTTMs) achieve strong performance on benchmarks with controlled acoustic conditions and clean audio signals. Their generalisation to conversations with overlapping speech and background interference remains underexplored. In this research, we evaluate audio-visual PTTMs trained with clean data on a challenging cocktail-party testbed derived from the AVCocktail dataset, and analyse their adaptation behaviour to this new domain. Experimental results show consistent performance degradation across audio and visual modalities under noisy conditions, with up to 38% relative drop in weighted F1. Fine-tuning on the new domain improves robustness, but gains vary across modalities and depend on the size of the available pre-training data. These findings provide insights into the different generalisation and adaptation capabilities of the audio and visual modalities, and indicate the need for robust modelling strategies to adapt to the complexities of human interactions in noise. All code and turn labels are made publicly available to facilitate further research.
I. INTRODUCTION
Predictive turn-taking models perform well in controlled settings, but their generalisation to overlapping speech and background interference remains underexplored. This study evaluates clean-trained audio-visual models in cocktail-party conversations and examines their adaptation to that domain.
I. INTRODUCTION
- The paper makes its code and turn labels publicly available.The repository is identified in the paper.Source: §I · p. 1
- PTTMs have performed strongly in quiet benchmarks, but their generalisation to cocktail-party conversations with overlap and background noise has not been analysed.Prior noisy-setting studies examined noise augmentation and visual cues, but not this combination of generalisation and adaptation to more complex interactions.Source: §I · p. 1
- Cocktail-party conversations shift both acoustic signals and visual expressions compared with quiet settings, creating domain differences in both modalities.Noise can alter speech articulation and vocal intensity, while speakers may also change lip movements and other visual cues.Source: §I · p. 1
- The study benchmarks audio-visual models trained on clean Candor data on AVCocktail and investigates fine-tuning, modality-specific adaptation, and pre-training dataset size.It also proposes an automated procedure for deriving turn-taking labels from AVCocktail transcriptions.Source: §I · p. 1
II. THE TURN-TAKING PREDICTION PROBLEM
The paper frames turn-taking as predicting who speaks when, using shifts and holds between speakers as events. It describes the Candor dataset and the event-label convention used to train and evaluate models.
A. Problem Formulation
- In speech technology, a turn is a sequence of one speaker’s inter-pausal units (IPUs) uninterrupted by another speaker’s IPUs; an IPU contains no silence exceeding a threshold.After an IPU, a speaker may yield the floor (shift) or continue holding it (hold).Source: A. Problem Formulation · p. 2
B. The Candor Dataset
- Turn-taking events are defined around silences longer than 250 ms, with shifts or holds determined by whether the speaker before and after the silence changes.The Candor training and development subsets use the first fold described in prior work.Source: B. The Candor Dataset · p. 2
- Candor comprises 850 hours across 1,657 unscripted English dyadic videoconferencing conversations, recorded as 30-FPS video and 16-kHz stereo audio.The dataset is used for training and evaluation of the predictive turn-taking models.Source: B. The Candor Dataset · p. 2
C. The AVCocktail Dataset
The paper adapts AVCocktail, a multi-party in-person conversation dataset, for audio-visual turn-taking prediction. It selects dyads and derives shift/hold labels automatically from sentence-level timestamps.
C. The AVCocktail Dataset
- AVCocktail provides multi-party cocktail-party conversations, with up to eight active speakers and four parallel conversations, and speech overlap reaching 100% at some moments.Sessions last approximately 5–7 minutes and vary in room layout and speakers.Source: C. The AVCocktail Dataset · p. 2
- AVCocktail’s manual sentence-level transcriptions enable automatic label derivation, making the dataset suitable for turn-taking evaluation in a challenging in-person setting.The paper describes AVCocktail as the only dataset with sufficient features for automatic evaluation of audio-visual turn-taking prediction in a cocktail-party scenario.Source: C. The AVCocktail Dataset · p. 2
- The dataset is adapted for turn-taking by selecting dyads and extracting their video crops and single-channel audio for model inputs.The single-channel audio is duplicated into stereo before the audio and face streams are passed to the models.Source: C. The AVCocktail Dataset · p. 2
- Shift/hold labels are derived from sentence starts: overlap or proximity to another speaker indicates a shift, while proximity to the same speaker indicates a hold.Sentences shorter than one second are excluded, and speech more than τ seconds away is treated as a new turn.Source: C. The AVCocktail Dataset · p. 2, Figure 2 · p. 2
D. Domain Shift Between Candor and AVCocktail
Compared with Candor, AVCocktail differs in interaction dynamics, audio quality, and visual cues. These differences span both the acoustic and visual domains relevant to turn-taking prediction.
D. Domain Shift Between Candor and AVCocktail
- AVCocktail has more complex interaction dynamics than Candor, with approximately 1.5 times as many shifts per minute as holds in AVCocktail-train.Candor-train shows the contrasting distribution.Source: D. Domain Shift Between Candor and… · p. 3
- Median SNR is around 15 dB in AVCocktail, compared with 90–100 dB in Candor, indicating substantially noisier AVCocktail audio.Candor also has a secondary SNR mode at 40 dB.Source: D. Domain Shift Between Candor and… · p. 3
- AVCocktail speakers show consistently high cheek-raiser and lid-tightener intensities across conditions, including intervals when neither dyad participant is speaking.The paper suggests this may reflect a Lombard effect in visual signals under noise.Source: D. Domain Shift Between Candor and… · p. 3
III. TURN-TAKING PREDICTION EXPERIMENTS
The experiments compare audio-only, video-only, and multimodal VAP models, all predicting future speaker activity while also estimating current activity.
A. Model Architectures
- VAP predicts both speakers’ activity in a two-second future window divided into eight bins, while the auxiliary VAD task predicts activity in the current context window.The future-activity label has 2^8 possible outputs.Source: A. Model Architectures · p. 3
- Audio-VAP and Video-VAP use self-attention for within-channel representations and cross-attention to fuse information across channels.Audio-VAP takes stereo audio as input; Video-VAP substitutes video signals.Source: A. Model Architectures · p. 3
- MM-VAP combines audio and video representations, then uses cross-attention to model dependencies between interlocutors before VAD and VAP classification.Video features are linearly projected to match the audio embedding dimension.Source: A. Model Architectures · p. 3
1) Feature Extraction:
The setup extracts audio and video features, uses Candor and AVCocktail data splits, and specifies the training configuration for the VAP variants.
1) Feature Extraction:
- Audio features are extracted at 50 Hz, while video is standardised to 50 FPS and represented by 60 OpenFace features.MM-VAP upsamples video features to 256-dimensional embeddings to match audio for cross-attention.Source: 1) Feature Extraction · pp. 3–4
2) Dataset Splits:
- The study uses the first Candor fold’s training and development splits and AVCocktail’s public splits, excluding its evaluation split because it lacks transcription labels.Source: 2) Dataset Splits · p. 4
3) Training Configurations:
- Full-Candor checkpoints come from Russell and Harte’s open-source repository; other experiments use AdamW, summed VAD and VAP losses, and the specified attention configurations.Experiments run for 10 epochs with batch size 4; the best development-set epoch is reported. AVCocktail training uses a constant learning rate of 3 × 10^-5.Source: 3) Training Configurations · p. 4
4) Evaluation Configurations:
The evaluation predicts shifts or holds from activity probabilities in the next 200 ms and compares models trained on Candor or AVCocktail across both datasets.
4) Evaluation Configurations:
- At each shift or hold, models predict a shift when cumulative shift probability in the next 200 ms exceeds the threshold; shift, hold, and weighted F1 are reported.The threshold is 0.5 × W × f, where W is the prediction window and f is the 50 Hz prediction rate.Source: 4) Evaluation Configurations · p. 4
A. Generalisation of Turn-taking Models
- 56.61% weighted F1 (down from 82.57%) was MM-VAP’s AVCocktail result after Candor training; its hold F1 fell from 88.87% to 57.05%.Audio-VAP fell to only above-random performance on AVCocktail, while Video-VAP remained comparatively stable at 70.68% on Candor and 69.23% on AVCocktail.Source: A. Generalisation of Turn-taking Models · p. 4, Table II · p. 5
- 54.14% weighted F1 was Audio-VAP’s AVCocktail result after AVCocktail training; MM-VAP achieved 69.84% on AVCocktail and 64.56% on Candor.AVCocktail is about 100 times smaller than Candor; the passage attributes Audio-VAP’s weak result to insufficiently robust representations amid data scale and task complexity.Source: A. Generalisation of Turn-taking Models · pp. 4–5, Table II · p. 5
- AVCocktail-trained Video-VAP achieved 70.63% weighted F1 on AVCocktail and 63.78% on Candor, while MM-VAP offered the best balance across the two datasets.Video-VAP’s reported weighted F1 on Candor was 63.78%.Source: A. Generalisation of Turn-taking Models · pp. 4–5, Table II · p. 5
B. Audio-Visual Adaptation of Turn-taking
Fine-tuning on AVCocktail improved multimodal and video-model results there, but outcomes differed by modality; the experiments also examine forgetting, representations, and synthetic-noise training.
B. Audio-Visual Adaptation of Turn-taking
- Fine-tuning raised MM-VAP’s AVCocktail weighted F1 to 73.65%, above the AVCocktail-trained model’s 69.84%, while Video-VAP improved from 69.23% to 73.01%.Audio-VAP did not adapt effectively: its weighted F1 fell by 4.70% relative to the Candor-trained AVCocktail result, to only above-random.Source: B. Audio-Visual Adaptation of… · p. 5
- AVCocktail results for AVCocktail-trained and fine-tuned MM-VAP and Video-VAP were not significantly different (p > 0.05); the authors suggest noisy overlap may limit audio’s contribution and current fusion.The authors identify more robust fusion strategies as a future need.Source: B. Audio-Visual Adaptation of… · p. 5
- After fine-tuning, Candor performance fell 3–5% relative to pre-finetuning models, while MM-VAP and Video-VAP had AVCocktail gains of up to 30.10% and 5.46%, respectively.The reported Candor losses for the visual-supported models were under 6%.Source: B. Audio-Visual Adaptation of… · p. 5
- In t-SNE visualisations, Candor-only MM-VAP embeddings for shifts and holds remained intertwined, while fine-tuned embeddings separated the events more clearly and brought same-event before-and-after samples closer.Separation remained imperfect in the cocktail-party setting.Source: B. Audio-Visual Adaptation of… · p. 5, Figure 5 · p. 6
- Training on Candor augmented with synthetic noise improved AVCocktail results without harming the original domain, but remained below training with AVCocktail data.MM-VAP reached 60.55% versus 69.84%, and Audio-VAP reached 49.89% versus 54.14%; fine-tuning from noise-augmented pretraining brought only marginal changes.Source: B. Audio-Visual Adaptation of… · p. 6
C. Effect of Pre-training Dataset Size
Increasing Candor pre-training data improves some downstream AVCocktail results, but adaptation varies by modality. Fine-tuning also causes Candor performance loss, which diminishes with more pre-training data for multimodal models.
C. Effect of Pre-training Dataset Size
- After fine-tuning, Video-VAP rises from 67% weighted F1 with 6h to 73% with 600h, while MM-VAP rises from nearly 70% to roughly 74%; Audio-VAP does not improve.Video-VAP’s steepest change occurs at 50h of pre-training; MM-VAP improves more steadily.Source: Figure 6 · p. 6, C. Effect of Pre-training Dataset Size · p. 6
- From 6h to 600h of Candor pre-training, MM-VAP and Audio-VAP gain around 10% absolute weighted F1 on Candor, while Video-VAP peaks at 70% with 50h and then plateaus.Video-VAP consistently trails Audio-VAP and MM-VAP.Source: Figure 6 · p. 6, C. Effect of Pre-training Dataset Size · p. 6
- Fine-tuning on AVCocktail causes catastrophic forgetting on Candor, but the effect diminishes with more pre-training data for MM-VAP and is smaller for Audio-VAP.Audio-VAP remains close to random on AVCocktail with or without fine-tuning.Source: C. Effect of Pre-training Dataset Size · p. 6
V. CONCLUSIONS
The paper evaluates turn-taking prediction in cocktail-party conditions and finds that clean-domain VAP models are sensitive to domain shift. Fine-tuning can improve noisy-domain robustness, but forgetting and modality-specific limitations remain.
V. CONCLUSIONS
- Across modalities, pre-trained VAP models show 2% to 38% relative weighted F1 degradation under background noise and complex interaction dynamics.Fine-tuning can yield up to 30% relative improvement in the new domain, while causing 1.6% to 5.6% relative drops on the original domain.Source: §V · p. 6
- Multimodal models indicate that audio-visual cues are essential for mitigating forgetting and exploiting turn-taking signals across pre-training data sizes.Source: §V · p. 6
- The authors identify high-level OpenFace features and the current audio-visual fusion architecture as limitations for high-quality turn-taking systems.Source: §V · p. 6
APPENDIX MANUAL TURN LABELS VERIFICATION OF AVCOCKTAIL-DEV
Manual verification of AVCocktail-dev labels preserves the model ranking and increases the gap between MM-VAP and Video-VAP. The checks also identify timestamp errors and rule-based misclassification of backchannels.
APPENDIX MANUAL TURN LABELS VERIFICATION OF AVCOCKTAIL-DEV
- Automatic labels can mistake backchannels for shifts and assign false shifts when speech-end timestamps are late; long semantically backchannel-like IPUs also challenge rule-based decisions.Source: Figure 7 · p. 8, APPENDIX MANUAL TURN LABELS… · p. 8
- Manual verification checks shift/hold labels, corrects timestamps, and annotates failed interruptions and backchannels using the Figure 8 decision diagram.Two sessions were removed after inspection because they were not dyadic conversations.Source: APPENDIX MANUAL TURN LABELS… · p. 8, Figure 8 · p. 8
- On manually verified AVCocktail-dev labels, the relative model ranking remains unchanged and the MM-VAP–Video-VAP performance gap is larger.Table IV reports VAP performance and absolute changes relative to automatic labels.Source: APPENDIX MANUAL TURN LABELS… · p. 8, Table IV · p. 9