Source-linked AI summary
Distributional Metrics for Evaluating Spoken Conversational Systems
Shree Harsha Bokkahalli Satish, Erica Cooper, Patrícia Schmidtová, Maike Züfle, Éva Székely, Nicholas Sanders, Ondřej Klejch
TL;DR
Evaluating conversational systems raises the question of what distributions of their behavior reveal relative to human conversations, beyond tests of specific interactional capabilities. The paper introduces CDS to compare behavior distributions against conversational references and evaluates it with listener judgments. The reported findings support CDS as an interpretable complement, while short excerpts of one minute or less do not reproduce rankings reliably.
Problem
Specific interactional evaluations test selected capabilities, while the paper asks what distributions of conversational behavior reveal about a system relative to human conversations.
Source: §1 · p. 1Method
CDS compares pooled distributions of conversational features with human-reference and conversational-distractor distributions, using Wasserstein–2 distances.
Source: §2 · p. 2Results
Agreement with listener judgments depends on the reference scale and whether evaluation concerns system ordering or preferences between conversations.
Source: §6 · p. 4Takeaways & Limitations
CDS provides an interpretable complement for evaluating conversational models, with individual features describing specific behavioral differences.
Source: §6 · p. 4Short excerpts of one minute or less fail to reproduce rankings reliably.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.
1. INTRODUCTION AND BACKGROUND
Conversational systems are often evaluated on specific interactional capabilities, but conversational behavior is distributional, motivating comparisons against distributions of human conversations. The section frames questions about distinguishing systems and conversation types, listener agreement, feature alignment with preferences, and the data needed for stable rankings.
1. INTRODUCTION AND BACKGROUND
- Conversational benchmarks test specific capabilities such as turn-taking and interruption handling, while distributional comparisons target the broader range of conversational behavior.The paper argues that conversational behavior is naturally distributional and asks what its distributions reveal about a system relative to human conversations.Source: §1 · p. 1
- CDS addresses a common-scale gap: earlier distributional statistics were not placed on a scale that readily supports combining behaviors or comparing systems.The paper brings distributional comparison into spoken interaction using distances to human reference distributions.Source: §1 · p. 1
- The first research question asks whether distributional metrics distinguish human–human from human–AI conversations and whether system rankings agree with third-party judgments.It also asks whether the metrics distinguish among conversational AI systems.Source: §1 · p. 1
- The remaining questions concern which conversational features align with listener preferences and how many callers or minutes are needed for rankings to stabilize.The supplied research-question text identifies these as separate evaluation questions.Source: §1 · p. 1
2. CONVERSATIONAL DISTRIBUTION SCORE
CDS compares distributions of conversational features from system conversations with reference distributions using Wasserstein–2 distance. Its features cover speech pace, rhythm, and turn interaction, with lexical measures kept as a separate semantic baseline.
2. CONVERSATIONAL DISTRIBUTION SCORE
- For each behavior, CDS pools scalar feature values across turns into a distribution, averages conversation distributions for a system, and compares that distribution with a reference using Wasserstein–2 distance.The scalar-distance calculation uses inverse cumulative distribution functions.Source: §2 · p. 2
- With a human reference and a conversational distractor, CDS places a target on a 0–100 scale: 100 is reference-like, 0 distractor-like, and 50 equidistant.The two anchors can both represent conversations, allowing comparison of conversational behaviors rather than speech against noise alone.Source: §2 · p. 2
2.1. CDS Features
- The feature set covers pace through phone and Sylber segment rates and duration distributions, and rhythm through ordered Sylber durations.Rhythm features include nPVI and the log ratio of a turn-final duration to the median of earlier durations.Source: §2.1 · p. 2
- Turn interaction features represent turn duration, signed response gaps, and each speaker’s share of combined floor time.Silence is positive and overlap is negative in the signed response-gap feature.Source: §2.1 · p. 2
- Lexical content is a separate two-feature semantic baseline based on reply entailment and GPT–2 mean response surprisal.The entailment measure uses a RoBERTa model fine-tuned on InferConvAI, and both measures can use up to two preceding turns as context.Source: §2.1 · p. 2
3. DATASETS
The evaluation combines human reference conversations with goal-oriented conversations involving human and AI partners, including third-party humanness judgments. Figure 2 presents conversation-level listener ratings alongside phone-rate and Sylber-duration CDS scores.
3. DATASETS
- CANDOR supplies high-, low-, and medium-success human conversation references, with 88 high-success and 34 low-success conversations retained and 35 medium-success conversations held separately.The success groups are based on participants’ post-conversation surveys.Source: §3 · p. 2
- VoiceArena Goals includes 374 conversations across five scenarios and five partner types, yielding 296 human–AI and 76 human–human conversations.The four AI partners are Gemini, OpenAI, Inworld, and Grok; the fifth partner is human.Source: §3 · p. 2
- Third-party listeners contributed 1,026 pairwise humanness votes for four AI systems across 295 unique conversation pairs; Behavior–SD interactions serve as AI–AI distractor anchors.Figure 2 plots listener humanness ratings against phone-rate and Sylber unit-duration CDS for pooled turns, with each point representing a conversation.Source: §3 · p. 2, Figure 2 · p. 3
4. METHODOLOGY AND EXPERIMENTS
The experiments compare reference-scaled behavior distributions, system rankings, feature-level alignment with listener preferences, and ranking stability under reduced caller counts or recording durations. The methodology uses two estimation orders and evaluates recovery against the full-recording, all-caller ordering.
4. METHODOLOGY AND EXPERIMENTS
- Turn extraction uses Silero VAD, with 700 ms of silence defining a new turn; pace and rhythm use segments from 3 to 30 seconds, while interaction features use the full speech timeline.The authors choose simpler VAD for better generalization across languages and datasets.Source: §4 · p. 3
- The analysis averages scores over 50 HSC splits and equally combines the pace, rhythm, and interaction families into the final non-semantic CDS.MSC and Behavior–SD serve as alternative distractor anchors against the HSC reference.Source: §4 · p. 3
- The RQ1 analyses compare human–human and human–AI feature distributions under two reference scales, then test whether CDS system orderings match listener rankings.System comparisons account for differing caller and scenario mixtures by assigning each caller/scenario/system block its own baseline; listener preference rates come from a Bradley–Terry model of pairwise votes.Source: §4 · p. 3
- For RQ2, Spearman correlations relate CDS differences for matched conversation pairs to differences in listener vote shares, using both reference scales and TTSDS2.A positive correlation means a larger CDS advantage tends to accompany stronger listener preference for the same conversation.Source: §4 · p. 3
- For RQ3, 2,000 caller samples are evaluated from the first 30 seconds through the full conversation, tracking the share that retains at least five of six full-ordering pair comparisons.Figure 3 shows this recovery fraction for caller count and recording duration under HSC–MSC, alongside median correlation between conversation CDS differences and listener preferences.Source: §4 · p. 3, Figure 3 · p. 4
- The score-first estimator computes conversation CDS before averaging within caller/scenario/system blocks, across scenarios, and then across callers; pool-first combines system turns before computing CDS.Pool-first gives each conversation the same total weight.Source: §4 · p. 4
5. RESULTS AND DISCUSSION
Full recordings stabilized listener rankings more reliably than short excerpts, while individual features aligned with preferences and reference choice shifted how AI distributions compared. Pooled phone-rate CDS also distinguished human–human from human–AI conversations.
5. RESULTS AND DISCUSSION
- Pooled phone-rate CDS averaged 59.2 for human–human and 46.6 for AI partners under HSC–MSC, and 72.9 and 39.2, respectively, with Behavior–SD.Scores above 50 indicate a distribution is more HSC-like than the distraction anchor.Source: §5 · p. 4
- Many AI phone-rate and Sylber-duration distributions were nearly equidistant on HSC–MSC, while HSC–Behavior–SD shifted and spread scores toward the distractor.The shift indicates greater similarity to Behavior–SD.Source: §5 · p. 4
- 99.0% of subsets were recovered at 30 callers with pooling, versus 95.0% at 40 when scoring conversations first; neither method recovered five of six comparisons from 30-second or 1-minute excerpts.The short-excerpt result used all 56 callers.Source: §5 · p. 4
- Sylber duration had higher preference correlation than overall CDS under both anchors; Wav2Vec2 activations reached ρ=0.555, while overall TTSDS2 had ρ=−0.062.The reported correlations concern listener preferences between conversation pairs.Source: §5 · p. 4, Table 1 · p. 4
6. CONCLUSION
The conclusion presents CDS as an interpretable complement to conversational-model evaluation. Its agreement with listeners depends on the reference scale and whether the goal is system ordering or conversation preferences, while excerpts of one minute or less do not reproduce rankings reliably.
6. CONCLUSION
- CDS provides interpretable distributional comparisons as a complement to evaluating conversational models, with listener agreement varying by reference scale and evaluation target.Short excerpts of one minute or less fail to reproduce rankings reliably.Source: §6 · p. 4