Source-linked AI summary
Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models
Kian Shamsaie, Iman Modarressi
TL;DR
Fixed-window benchmarks classify silence after a turn and overlap before its end without accounting for the speaker’s intent. TACT evaluates timing with intent-conditioned reference laws and speaker memory; across eleven systems, its scores align more closely with human judgments than binary metrics.
Problem
Fixed-window turn-taking metrics judge responses by pause or turn-end timing, although whether silence or overlap is appropriate depends on the speaker’s latent intent.
Source: §I · p. 1Method
TACT combines dialogue history, per-speaker memory, and an intent posterior to score timing with a strictly proper continuous rule weighted by intent-conditioned human timing kernels.
Source: §I · p. 1, §VII · p. 6Results
TACT correlated with human judgments at Spearman ρ = 0.81 versus 0.46 for binary metrics; the best system scored 0.47 against a human topline of 0.86.
Source: §I · pp. 1–2Takeaways & Limitations
TACT’s evaluation treats speaking time as conditional on speaker intent rather than as a latency problem.
Source: §VII · p. 6The six-class taxonomy discretizes a continuum and has label noise (α = 0.73); kernels are fit on English corpora, though the recipe enables refitting where norms differ.
Source: F. Limitations · p. 6
Abstract
from arXiv · showhide
Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by completeness of the prior turn. We argue that the appropriateness of a response offset, whether delayed silence or anticipatory overlap, is conditional on the speaker's latent intent, identifiable only from that speaker's behavior. We introduce TACT, a benchmark of 9,728 episodes and 73.2 hours from five dyadic corpora; each episode carries dialogue history, a per-speaker memory profile, and an annotator-derived posterior over six intent classes. Scoring replaces binary windows with a strictly proper threshold-weighted continuous ranked probability score whose weights are intent-conditioned timing kernels fitted to human floor-transfer-offset distributions, proving boundedness, consistency, and binary reduction. Across eleven systems the best model reaches 0.47 against a human topline of 0.86, is nearly invariant to speaker profiles, and TACT agrees with human judgments at Spearman 0.81 versus 0.46 for binary metrics.
I. INTRODUCTION
Fixed-window metrics treat silence after a completed turn as failure and overlap before completion as intrusion, although appropriate timing depends on the speaker’s latent intent. TACT conditions timing evaluation on intent and speaker memory, and its scores align more closely with human judgments than binary metrics.
I. INTRODUCTION
- Silence after a completed turn and overlap before completion are not intrinsically failures; their appropriateness depends on the speaker’s intent and behavior across interactions.The same silence or incoming speech can invite a response or make it inappropriate.Source: §I · p. 1
- TACT replaces fixed-window evaluation with continuous, strictly proper scoring based on intent-conditioned timing kernels fitted to human floor-transfer-offset distributions.The benchmark includes per-speaker memory and a posterior over six latent intent classes.Source: §I · p. 1
- The best system attained a composite of 0.47 versus a human topline of 0.86; TACT correlated with held-out human judgments at Spearman ρ = 0.81, versus 0.46 for binary metrics.Behavior was essentially invariant to speaker profile, while held-out human behavior was not.Source: §I · pp. 1–2
II. RELATED WORK
- Conversation-analysis work describes floor-transfer offsets as intent-dependent, with reflective responses delayed, rhetorical or floor-holding turns permitting long silence, and backchannels allowing overlap.Turn-shift predictors such as TurnGPT and VAP supply machinery repurposed for scoring.Source: §II · p. 2
- Prior benchmarks increasingly cover overlap, multi-turn structure, and real disfluent audio, but define appropriate behavior from observed context rather than latent, speaker-specific intent.TACT’s stated distinction is intent-conditioned continuous scoring, not being the first multi-turn or overlap-aware benchmark.Source: §II · p. 2
III. THE TACT BENCHMARK
TACT episodes combine audio, dialogue history, speaker memory, and intent annotations, with episodes drawn from five public corpora and synthetic probes. Speaker profiles summarize behavior from other episodes of the same speaker.
A. Episode Construction
- Each episode combines dual-channel audio, time-aligned history averaging 3.2 minutes and 11.4 turns, a focal-speaker profile, and annotations conditioning the scoring.Systems receive the focal-speaker channel with a rendered profile; output is recorded over a 2-second anticipatory pre-horizon and 5-second horizon, with three runs and the median-scoring run retained.Source: A. Episode Construction · p. 2
- 9,728 episodes spanning 73.2 hours come from five public dyadic corpora, supplemented by synthetic probes for sparse conditions.Episodes cover pause handling, backchanneling, smooth turn-taking, and user interruption.Source: A. Episode Construction · p. 2
B. Speaker Memory Profiles
- Speaker profiles are built from other episodes of the same speaker, averaging 14.6 minutes of evidence, and summarize timing, pause tolerance, backchanneling, articulation, and floor-holding behavior.Profiles are the only channel for speaker-specific expectations to reach systems.Source: B. Speaker Memory Profiles · p. 2
C. Intent Taxonomy and Annotation Protocol
TACT annotates six intent classes from full context, speaker profile, and human continuation, then fuses human labels with a calibrated LLM judge into a posterior. The taxonomy distinguishes timing expectations on both sides of the turn end.
C. Intent Taxonomy and Annotation Protocol
- The six intents are immediate-answer-seeking, reflective, rhetorical, floor-holding, cooperative-incoming-inviting, and turn-abandonment; BCK covers continuers, collaborative completions, and urgent clarifications.Three to five annotators label each episode and rate silence acceptability and acceptable onset windows on both sides of the turn.Source: C. Intent Taxonomy and Annotation… · p. 3
- Intent-label agreement was Krippendorff α = 0.73; onset-window agreement was 0.84 and overlap-acceptability agreement was 0.79.The empirical intent distribution ranges from 9.2% RHE to 31.6% ANS.Source: C. Intent Taxonomy and Annotation… · p. 3
- TACT stratifies by gender, age band, first-language region, signal-to-noise ratio, and articulation rate, fitting kernels per slice where data permit.The stated aim is to avoid encoding one population’s timing norm as universal.Source: C. Intent Taxonomy and Annotation… · p. 3
A. Notation and Intent Posterior
- The intent posterior blends majority human annotations with a calibrated, temperature-scaled LLM judge; λ = 0.3 was chosen by held-out annotator log-likelihood, and intent is ambiguous in 38% of episodes from history alone.Conditioning on Ms(e) resolves over half of these ambiguities.Source: A. Notation and Intent Posterior · p. 3
B. Reference Laws and Weights for All Six Intents
TACT models each intent with a human reference law that combines a no-response probability and a response-conditional timing density. Intent-specific laws and asymmetric weights distinguish silence, delayed responses, and anticipatory overlap.
B. Reference Laws and Weights for All Six Intents
- Each intent’s reference law combines an empirical no-response probability with a conditional continuous floor-transfer-offset density fitted on responsive turns disjoint from test.RHE and HOLD place most reference mass on no response; BCK’s fitted onset distribution places 72% of onset mass inside the ongoing turn.Source: B. Reference Laws and Weights for All… · p. 3
- Fitted no-response masses range from βANS = 0.05 to βHOLD = 0.90, with βRHE = 0.85 and βBCK = 0.45.The masses equal mean annotator acceptability of silence.Source: B. Reference Laws and Weights for All… · p. 3
- Intent-conditioned weights preserve asymmetric timing priorities: ANS weights late offsets above premature overlap, while BCK favors a well-placed overlap over any post-completion onset.The weights remain positive across the offset domain through a floor ϵ = 0.02, supporting strict propriety.Source: B. Reference Laws and Weights for All… · p. 3
C. Threshold-Weighted Continuous Ranked Probability Scoring
The intent-conditioned twCRPS scores a model’s predictive timing law against a realized continuation, weighting timing errors by their communicative severity. It also represents no response within the same scoring framework.
C. Threshold-Weighted Continuous Ranked Probability Scoring
- The predictive law includes a no-response atom, so withheld floor and post-completion onset are scored by the same object.For a no-response outcome, the event indicator is zero at every finite threshold.Source: C. Threshold-Weighted Continuous… · p. 3
- twCRPS integrates the weighted squared discrepancy between a predictive CDF and the realized-event step function, scoring timing errors according to intent-conditioned weights.The weights derive from asymmetric timing kernels fitted to human floor-transfer-offset distributions.Source: C. Threshold-Weighted Continuous… · p. 3
- The episode score compares the posterior-mixed reference law with the realized continuation and normalizes by the posterior-weighted intent-specific weight totals.The episode mean estimates the population timing score, subject to irreducible single-continuation variance.Source: C. Threshold-Weighted Continuous… · p. 4
D. Length, Overlap, and Composite
Length and overlap receive distinct scoring treatments, then combine with timing into category scores and an overall composite. The overlap rule distinguishes prosodically licensed competitive entry from mid-clause interruption.
D. Length, Overlap, and Composite
- Length uses the same proper scoring construction against the human duration density, with uniform weights over the duration support.The bounded aggregate reaches 1 uniquely when the model’s duration CDF matches the human duration density.Source: D. Length, Overlap, and Composite · p. 4
- Competitive overlap is weighted by prosodic completion: high-completion entry is barely penalized, while mid-clause interruption is penalized in proportion to duration.Backchannels are exempt; pairing this rule with the BCK reference law makes cooperative overlap a high-scoring action.Source: D. Length, Overlap, and Composite · p. 4
- The composite is the unweighted mean of four category scores, with BCa bootstrap intervals computed over 104 resamples.The category combination gives timing and length shared weight in backchanneling, while timing dominates pause handling and overlap dominates interruption.Source: D. Length, Overlap, and Composite · p. 4
E. Formal Guarantees
The scoring construction is bounded and strictly proper under the stated conditions, and its composite retains those guarantees. Binary fixed-window metrics arise as a limiting special case, while a separate result identifies speaker-specific intent mixtures under distributional assumptions.
E. Formal Guarantees
- For positive, integrable weights that are strictly positive on the interior of the timing domain, the timing score lies in [0, 1] and is uniquely optimized by the human conditional law.The composite is bounded and reaches 1 exactly when the model reproduces each combined human conditional timing, duration, and overlap law.Source: E. Formal Guarantees · pp. 4–5
- Binary fixed-window takeover and overlap metrics are recovered as indicator-weight limits of the twCRPS score.A degenerate intent posterior and deterministic forecast yield the takeover statistic; a two-sided indicator weight recovers windowed overlap statistics.Source: E. Formal Guarantees · p. 5
- Speaker priors and intent-specific response laws are identifiable up to label permutation when the laws are pairwise distinct and the response law otherwise does not depend on speaker.The proof separates the no-response atom from continuous components and establishes their linear independence under the stated distribution families.Source: E. Formal Guarantees · p. 5
F. Reconciling the Population Optimum with the Empirical Topline
The empirical human topline is 0.86 although the population optimum is 1, because each episode is scored against a single held-out continuation. Single-draw variance and held-out speaker variation account for the shortfall.
F. Reconciling the Population Optimum with the Empirical Topline
- The empirical human topline is 0.86, below the population optimum of 1 because the per-episode estimator scores a single held-out continuation.The paper describes the topline as a principled estimate of the population optimum despite this finite-sample gap.Source: F. Reconciling the Population Optimum… · p. 5
- Each episode contributes one realized continuation, creating an irreducible single-draw term even for a forecaster equal to the human reference law.The passage identifies this term as R_z/W_z > 0.Source: F. Reconciling the Population Optimum… · p. 5
- Held-out speakers’ timing can deviate from pooled human timing laws, and the gap shrinks only with evidence from that speaker.Together with single-draw variance, this speaker variation accounts for the 0.14 shortfall.Source: F. Reconciling the Population Optimum… · p. 5
V. EXPERIMENTAL SETUP
The evaluation covers eleven frontier systems, reference baselines, and a human topline, using local streaming for open models and realtime APIs for proprietary systems.
V. EXPERIMENTAL SETUP
- Eleven systems were evaluated: eight open models and three proprietary systems, with Gemini tested at minimal and high thinking.Open models ran locally in streaming mode; proprietary systems used realtime APIs.Source: §V · p. 5
- Reference baselines included a VAP-gated cascade, a never-respond policy, and the human topline.The VAP cascade used a reference baseline; reproduced binary v1 metrics matched published Moshi pause-takeover and latency figures.Source: §V · p. 5
- The validity study used 1,280 held-out episodes judged by five raters across 88 system-condition cells.Source: §V · p. 5
A. Main Comparison
TACT separates the best system from the human topline and substantially reorders system rankings relative to binary metrics. Intent-conditioned results reveal over-eagerness and limited speaker adaptation, while TACT tracks human judgments more closely than binary scoring.
A. Main Comparison
- 0.47 for gpt-realtime-2 versus 0.86 for the human topline; TACT and reproduced binary v1 rankings correlate at 0.55.Moshi falls from second under binary scoring to tenth under TACT; Gemini high overtakes Gemini minimal on TACT despite ranking seventh under v1.Source: A. Main Comparison · p. 5, Table I · p. 6
B. The Over-Eagerness Pathology
- Intent-conditioned Fig. 2 scores show systems approach humans only on ANS, degrading two- to ten-fold on REF, RHE, and ABN.Across systems, ANS scores are 0.38–0.78, versus 0.09–0.36 on REF and 0.06–0.30 on RHE; humans overlap on 63% of invited backchannels, versus 14% for PersonaPlex and under 6% elsewhere.Source: B. The Over-Eagerness Pathology · p. 5, Figure 2 · p. 6
C. Memory Ablation: No Per-Speaker Adaptation
- Swapping speaker profiles leaves all eleven systems essentially invariant (∆S ≤0.01; JSD2 ≤0.024), while held-out human behavior shifts substantially (JSD2 = 0.23).The largest system JSD2 is 0.024 for gpt-realtime-2; the smallest is 0.003 for dGSLM.Source: C. Memory Ablation: No Per-Speaker… · p. 6
D. Metric Validity
- TACT correlates with held-out human judgments at Spearman ρ = 0.81, versus 0.46 for binary composite and 0.39 for latency.An intent-conditioned binary baseline reaches ρ = 0.62; ablating memory, calibration, or prosody yields ρ = 0.71, 0.78, and 0.74.Source: D. Metric Validity · p. 6
E. Fairness and Robustness Slices
Human topline composites remain flat across the reported demographic and signal slices, while system scores vary. The paper notes scope limits in its intent taxonomy, kernel data, and scoring components, and concludes that turn timing is an inference over latent speaker intent rather than simply a latency problem.
E. Fairness and Robustness Slices
- The human topline stays at 0.84–0.87 across gender, age, first-language region, SNR, and articulation rate; the best system loses 0.07 across North American and Indian-English speakers and 0.06 across high- and mid-SNR conditions.The system gaps are concentrated in timing; gender and age gaps are smaller at 0.02 and 0.05.Source: E. Fairness and Robustness Slices · p. 6
F. Limitations
- The six-class taxonomy discretizes a continuum, and its timing kernels are fitted on English corpora, though the recipe enables refitting where norms differ.The paper also notes label noise (α = 0.73), possible shared judge failure modes, and reliance on VAP completion estimates for anticipatory scoring.Source: F. Limitations · p. 6
VII. CONCLUSION
- The conclusion frames turn-taking as inference over a speaker’s latent intent, not a latency problem.TACT replaces fixed-window scoring with an intent-conditioned twCRPS over human floor-transfer-offset laws, per-speaker memory, and a calibrated posterior.Source: §VII · p. 6