Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

Siddhant Arora; Zhiyun Lu; Chung-Cheng Chiu; Ruoming Pang; Shinji Watanabe

Source-linked AI summary

Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, Shinji Watanabe

arXiv:2503.01174v1Mar 3, 2025cs.CLcs.SDeess.AS

TL;DR

Audio FMs have been insufficiently evaluated for natural, interactive turn-taking beyond standard speech and dialogue benchmarks. The paper proposes a supervised human-conversation judge and timing-focused benchmarks, finding diverse system weaknesses and substantial room for improvement while noting protocol limitations.

Problem
Audio FMs lack comprehensive evaluation of natural conversational turn-taking, including fluent succession of turns, backchannels, interruptions, and appropriate silence.
Method
The paper trains a supervised model on human-human conversations to judge turn-taking events and evaluates dialogue systems and audio FMs with automated metrics and curated benchmarks.
Results
Existing systems sometimes fail to speak up, Moshi interrupts too aggressively, and both systems rarely backchannel; audio FMs remain near random on backchannel understanding.
Takeaways & Limitations
The evaluation protocol and planned public platform provide a basis for diagnosing and improving audio FMs’ turn-taking capabilities.
Takeaways & Limitations
The protocol requires supervised conversational data, uses heuristic backchannel identification, and has judge-human agreement below 80% for many metrics.

Abstract

from arXiv · show

The recent wave of audio foundation models (FMs) could provide new capabilities for conversational modeling. However, there have been limited efforts to evaluate these audio FMs comprehensively on their ability to have natural and interactive conversations. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Inspired by this, we ask whether the recently proposed audio FMs can understand, predict, and perform turn-taking events? To answer this, we propose a novel evaluation protocol that can assess spoken dialog system's turn-taking capabilities using a supervised model as a judge that has been trained to predict turn-taking events in human-human conversations. Using this protocol, we present the first comprehensive user study that evaluates existing spoken dialogue systems on their ability to perform turn-taking events and reveal many interesting insights, such as they sometimes do not understand when to speak up, can interrupt too aggressively and rarely backchannel. We further evaluate multiple open-source and proprietary audio FMs accessible through APIs on carefully curated test benchmarks from Switchboard to measure their ability to understand and predict turn-taking events and identify significant room for improvement. We will open source our evaluation platform to promote the development of advanced conversational AI systems.

1 Introduction

Audio foundation models need conversational abilities beyond speech recognition, text-to-speech, and text dialogue, including timely, fluent turn-taking. The paper introduces an evaluation protocol and applies it to dialogue systems and audio FMs to expose their turn-management strengths and limitations.

Motivation: Natural interaction requires systems to manage turns, backchannels, interruptions, and cues about retaining or yielding the conversation floor.These behaviors support two-way communication while users and systems listen and speak.
Motivation: Existing evaluations have made limited progress on comprehensively measuring audio FMs’ conversational capabilities.Standard ASR, TTS, and text-based dialogue benchmarks do not cover the full requirements of interactive conversation.
Evaluation protocol: The proposed protocol trains a predictor on human-human conversations to judge the local timing of turn changes, backchannels, and interruptions.It complements corpus-level statistics, which do not evaluate when an event occurs within its local context.
Findings: A user study finds that existing dialogue systems sometimes fail to speak when users yield, Moshi interrupts too aggressively, and both systems rarely backchannel.The study also reports insufficient floor-retention cues and different responses to user interruptions.
Benchmark: The paper curates a human-human conversation benchmark to compare audio FMs’ understanding and prediction of turn-taking events.The benchmark examines relative performance across event types and quantifies room for improvement.

2 Related Study

Prior benchmarks evaluate standard audio processing or response generation, but they do not specifically assess the timing and conversational management of turn-taking events. This work targets that gap with metrics for understanding, predicting, and performing turn-taking events.

Turn-taking prediction: Prior turn-taking research predicts turn changes, backchannels, and interruptions from audio, text, or multimodal information.Voice Activity Projection models instead predict future speech activity from spoken dialogue data in an unsupervised manner.
Audio FM benchmarks: Existing audio FM benchmarks assess standard speech-processing tasks but do not specifically evaluate conversational capabilities.Spoken-dialogue evaluation has also focused on response-generation quality using automated or subjective metrics.
This work: The paper introduces timing-sensitive metrics for turn-taking events alongside previously proposed corpus-level statistics.These metrics are intended to reveal limitations in engaging in interactive and natural conversations.

3 Turn-Taking Analyses based on Conventional Corpus-level Statistics

The paper first evaluates dialogue systems with corpus-level turn-taking statistics and defines the conversational events underlying those analyses. These statistics show different latency and overlap patterns, but they cannot determine whether events occur at appropriate local moments.

Experimental setup: Participants converse with Moshi and a cascaded system, while the study computes corpus-level statistics from collected human-AI conversations.The cascaded system combines VAD, Whisper, a language model, and TTS components.
Turn-taking representation: A conversation is divided into non-overlapping chunks with speaker voice-activity sequences used to represent turn-taking events.The analysis derives event structure from chunk-level speech activity for two speakers.
Turn-taking events: An Inter-Pausal Unit is continuous speech in one channel bounded by silences exceeding 200ms, while pauses separate successive units from the same speaker.Successive IPUs separated by pauses are grouped into a turn; gaps separate different speakers’ turns.
Turn-taking events: Overlap occurs when both speakers try to speak, and backchannels are short listener utterances that acknowledge the current speaker without taking the turn.Backchannels typically occur during the speaker’s turn.
Turn-taking events: An interruption is overlapping speech in which one speaker tries to take the turn, classified as successful floor-taking or unsuccessful butting-in.The distinction depends on whether the interrupting speaker continues beyond the first speaker’s IPU or yields at a natural break.
Results: 11.8% of cumulative duration is the gap for Moshi, whereas the cascaded system has a 32.4% gap and minimal overlap.Moshi’s overlap is lower than natural human dialogue, while the cascaded system’s higher latency makes conversation feel less natural.
Limitation: Corpus-level statistics capture global event distributions but cannot evaluate the exact timing or appropriateness of individual turn-taking events.For example, they cannot distinguish supportive overlap from unexpected, disruptive overlap.

4 Proposed Turn-Taking analyses with Timing-Centric Metrics

The paper evaluates turn-taking at precise conversational moments by training a causal supervised model to predict events in 40 ms chunks, then using it as a judge of AI dialogue systems. The resulting analyses expose timing-specific weaknesses in speaking up, backchanneling, interruptions, and conveying turn willingness.

4.1 Judge turn-taking Model: The causal judge predicts the next turn-taking event every 40 ms from a prior context window encoded by Whisper representations.Its output likelihoods support timing-sensitive evaluation of turn-taking decisions.
4.1 Judge turn-taking Model: The label sequence distinguishes no speech, backchannel, interruption, turn change, and continuation events.Turn changes include interruptions that take the floor, while continuation is the default remaining case.
4.1 Judge turn-taking Model: The supervised judge performs strongly on interruptions and generalizes similarly to out-of-domain spoken-dialogue corpora.The reported evaluation uses ROC-AUC on Switchboard, Columbia Games Corpus, and Fisher Corpus; direct comparison with word-boundary predictions is limited.
4 Proposed Turn-Taking analyses with Timing-Centric Metrics: The timing-centric metrics cover when systems should speak, backchannel, interrupt, convey willingness to yield or retain the floor, and handle user interruptions.These capabilities cannot be evaluated adequately with corpus-level event distributions alone because timing determines whether overlap is supportive or disruptive.
4.4 Metric (a) When user speaks: when system should speak up?: Moshi sometimes fails to speak when users appear ready to yield, while the cascaded system speaks up in 37.1% of pauses.The paper links Moshi’s behavior to its EPAD/PAD generation mechanism and suggests increasing EPAD bias during pauses.
4.5-4.9 Metrics: Both systems rarely backchannel and show low agreement with judge labels, while Moshi also interrupts too aggressively and often fails to convey its willingness to retain the floor.Moshi’s interruption false-positive rate is 2.09%, its turn-change errors are mainly false negatives at 2.21%, and users speak up 46.5% of the time when it pauses.

5 Additional Evaluation on Understanding and Predicting turn-taking events

The paper evaluates audio foundation models on understanding and predicting turn-taking events, finding strong performance in limited settings but substantial weaknesses, especially for backchannels and several future-event predictions.

The study tests whether audio foundation models can understand turn-taking events from initial audio chunks, focusing on turn changes, backchannels, and interruptions.
Qwen-Audio-Chat performs best among open-source models, while a Whisper–GPT-4o cascade generally outperforms open-source systems.
All models perform close to a random-guess baseline when understanding backchannels.
The prediction task covers turn change, backchannel, interruption, and floor-taking interruption as future events.
The Whisper–GPT-4o cascade predicts turn changes well but cannot reliably predict backchannels, interruptions, or whether interruptions lead to turn changes.
Open-source audio foundation models perform close to a random-guess baseline across all future-event prediction tasks, leaving substantial room for improvement.

6 Discussions and Conclusions

The paper introduces a broad evaluation framework for turn management and reports limitations concerning supervision, backchannel modeling, judge accuracy, and user-study scope.

The work identifies core turn-management capabilities, proposes automated metrics, and offers insights into strengths and limitations of spoken dialogue systems.
The evaluation protocol requires a supervised dataset to train its judge model, limiting scalability and applicability.
The current backchannel-identification approach relies on heuristics and may miss certain backchannels.
Agreement between judge labels and human judgments is below 80% for many metrics.
The user study tests only a few audio foundation models and may contain participant-selection bias.

A.1 Prior works on predicting turn-taking events

Prior work predicts turn-taking from acoustic, linguistic, multimodal, and voice-activity signals, while newer audio and multimodal foundation models target real-time conversational interaction.

Prior studies identify transition-relevant places where turn shifts tend to occur and use acoustic and prosodic properties to predict them.
End-of-turn models help voice assistants decide when to speak by distinguishing gaps from pauses, but most are not designed for overlapping speech.
Speaker-change and speaker-diarization research models changes in active speakers, often assuming minimal or no overlap.
Backchannel prediction research has emphasized acoustic and prosodic cues, with early work relying primarily on acoustic information.
Multi-task models jointly predict turn changes with backchannels, fillers, or turn-management willingness.
Voice Activity Projection models predict each speaker’s future speech activity and have become multilingual without requiring voice-activity-detection features.
Moshi is a full-duplex end-to-end spoken dialogue framework that models the user’s and system’s speech in parallel streams without explicitly modeling speaker turns.

A.1.1 Prior Turn taking evaluation Metrics

The paper extends prior turn-taking evaluation by assessing interactive human–AI decisions with separate metrics for listener and speaker behavior and new interruption labels.

Prior metrics: Prior turn-taking metrics forecast whether events such as turn changes or backchannels will occur soon in human-human conversations.
Evaluation scope: This work evaluates whether AI systems actively take the floor, yield, backchannel, or interrupt during spontaneous human interactions.
Evaluation challenges: Human–AI conversations lack inherent ground truth, so human relevance judgments are needed to assess AI turn-taking decisions.
Evaluation design: The protocol separately models turn-taking decisions when the AI listens and when the AI speaks.
Evaluation design: The protocol uses distinct metrics for AI interruptions during human turns and AI responses to user interruptions during system turns.
Metric design: The authors survey prior work to identify turn-taking abilities and adapt or create metrics for each ability.
New metrics: A new labeling sequence distinguishes successful floor-taking interruptions from unsuccessful butting-in interruptions.
Conclusion: The resulting protocol combines adapted and new metrics to evaluate all identified turn-taking abilities in human–AI interaction.

A.2 User Study

The user study collected real-time conversations with Moshi and a cascaded dialogue system, using participant sessions to evaluate spoken dialogue interactions. The study materials included system interfaces, participant instructions, and supervised-model validation documentation.

Study procedure: Participants conversed with Moshi through its public demo and with a cascaded dialogue system while their conversation recordings were collected.The study used the publicly available Moshi demo and a separately configured cascaded system.
Study materials: The cascaded-system interface was shown to participants through a study screenshot, while the judge-model architecture was documented separately.The supplied materials identify the cascaded demo screenshot and the causal turn-taking model architecture as study figures.
Study procedure: The study instructed participants to hold short, approximately five-minute conversations about topics such as pronunciation feedback and places to eat.Participants were asked to follow conversation guidelines during the sessions.
Evaluation materials: The supervised turn-taking model’s validation performance was reported in Table 5 as part of the evaluation materials.The table is identified as reporting validation performance for the supervised turn-taking prediction model.

A.3 Architecture of Judge Turn-taking Model

The judge model predicts turn-taking labels causally from short windows of preceding mixed speech. It uses pretrained speech representations followed by a linear classifier and softmax to estimate the next chunk’s event likelihood.

Causal formulation: The sequence model estimates the label sequence by maximizing P(L|X), then factorizes that probability into conditional probabilities for successive labels.The formulation begins with MAP estimation and applies the product rule to the label sequence.
Causal formulation: The model partitions mixed speech into non-overlapping chunks and predicts the next turn-taking event using only preceding chunks.This causal setup conditions predictions on U1:i−1 rather than future audio.
Causal formulation: The model can further restrict prediction to a prior context window of size W instead of the entire preceding conversation.The context-window formulation conditions on ˆUi, a bounded history before chunk i.
Architecture: Whisper encodes the context window, a weighted sum produces acoustic representations, and a linear layer with softmax predicts the turn-taking event.The representation uses the encoder output of the last audio frame before classification.
Architecture: Using single-channel mixed speech, the supervised model estimates each event likelihood and is trained with cross-entropy loss.This design models both speakers’ turn-taking behavior without requiring separate speaker channels.

A.4 Experiment Setup of Supervised Turn-Taking Model

The experiment evaluates turn-taking models and audio foundation models across validation, out-of-domain, user-study, and benchmark settings. Results indicate robust judge-label agreement in some settings, while conversational systems and open-source audio FMs show important turn-taking weaknesses.

Model setup: The judge model uses 40-ms chunks, a 30-second context window, and Whisper-medium acoustic representations.Backchannel annotations use common isolated one- and two-word phrases, with label classes balanced by downsampling.
Evaluation protocol: The evaluation reports judge-label alignment with human decisions and confidence intervals for dialogue-system consistency across proposed metrics.Tables 6 and 7 document uncertainty in judge agreement and system consistency.
Datasets: The model is evaluated on Switchboard and two out-of-domain corpora: Columbia Games Corpus and Fisher Corpus, using ROC-AUC for performance reporting.The Fisher transcriptions contain inaccuracies and substantial untranscribed portions, requiring heuristics for evaluation preparation.
Audio-FM benchmarks: The benchmark tests whether audio FMs understand or predict turn change, backchannel, interruption, and floor-taking interruption using balanced yes/no samples.Understanding benchmarks contain 1,500 Switchboard samples with equal positive and negative examples; prediction samples use up to 30 seconds of audio.
Audio-FM benchmarks: The floor-taking-interruption benchmark contains only 250 samples because interruptions are rare.Other prediction benchmarks contain 1,500 samples with equal positive and negative instances.
User-study results: Both dialogue systems speak faster than average humans and produce fewer backchannels than humans.These findings are reported from natural dialogue event statistics covering speaking rate and backchannel rate.
Threshold analysis: Turn-taking events are highly imbalanced, with continuation and silence comprising more than 95% of instances, motivating threshold tuning.The evaluation varies thresholds systematically and examines judge agreement and margin of error.
Threshold analysis: Judge-label agreement with human decisions does not fluctuate greatly under threshold changes, and the model also shows good agreement on the Columbia Games Corpus without dataset-specific tuning.The reported analyses use margin-of-error estimates and an out-of-domain consistency check.