PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Semantic Uncertainty to Estimate Transition Relevance in Turn-taking

Muhammad Umair, Jan P. de Ruiter

arXiv:2609.10934v1cs.CL

TL;DR

The paper asks how to predict within-turn response opportunities that are poorly represented by observable speaker changes. It models evolving semantic constraint through sampled continuations and semantic uncertainty, then uses uncertainty changes to identify TRPs. Semantic-uncertainty prediction substantially outperforms prompt-based and fine-tuned text-only baselines, while the evidence remains limited by dataset and modality scope.

  • Problem

    Within-turn TRPs are difficult to predict because many perceived response opportunities do not produce observable speaker changes, leaving systems without direct supervision for them.

  • Method

    The paper samples continuations for each turn prefix, measures their semantic dispersion as semantic uncertainty, and predicts TRPs from changes in that uncertainty.

  • Results

    Semantic-uncertainty-based prediction substantially outperforms prompt-based inference and supervised fine-tuning, with performance stable from two sampled continuations onward.

  • Takeaways & Limitations

    Evolving semantic structure provides a useful text-only signal for predicting perceived within-turn response opportunities, though semantic uncertainty is only a model-mediated proxy for listener expectations.

  • Takeaways & Limitations

    Evaluation uses a small English-only dataset of single-speaker turns and excludes acoustic, prosodic, and other multimodal cues, limiting generalization to broader interaction settings.

Abstract

from arXiv · show

Turn-taking is a fundamental mechanism that governs when interlocutors speak and listen. Although Spoken Dialogue Systems (SDS) exploit a range of linguistic, acoustic, and non-verbal cues, they produce ill-timed responses in unscripted interaction. A central challenge is anticipating Transition Relevance Places (TRPs), or opportunities, not obligations, for a listener to take the floor. Human listeners do not wait for turn endings; as an utterance unfolds, they use expectations about its developing meaning to anticipate TRPs and decide whether to take the floor. We examine whether these evolving expectations can be modeled through semantic uncertainty -- an LLM-derived measure of how strongly a turn so far constrains what may plausibly come next. To do so, we sample possible continuations of an ongoing turn and use changes in semantic dispersion to identify TRPs within turns. We evaluate this account on a dataset with TRP labels derived from real-time listener responses, rather than retrospective annotation. Our approach substantially outperforms prompt-based and fine-tuned text-only baselines, providing empirical support for the view that evolving semantic constraints inform perceived turn-taking opportunities in unscripted interaction.

1 Introduction

Within-turn TRPs are response opportunities that often leave no observable trace, making them difficult to supervise and predict. The paper tests whether listeners’ evolving expectations about turn continuation can be modeled through semantic uncertainty.

  • 1 Introduction: Within-turn TRPs can occur without speaker change, so recorded interaction provides limited evidence about these perceived response opportunities.A listener may respond but not take the floor, leaving the opportunity difficult to observe retrospectively.
  • 1 Introduction: Spoken Dialogue Systems still produce ill-timed responses because observable turn-taking behavior supervises acted-on responses, not all perceived opportunities.This motivates asking what information listeners use to anticipate TRPs and whether systems can exploit it.
  • 1 Introduction: Listeners project possible turn completions incrementally, revising expectations about continuation as each new word arrives.Projection describes prospective recognition rather than waiting for the turn to end.
  • 1 Introduction: The paper models these evolving expectations with semantic uncertainty derived from sampled continuations and tests whether its changes predict TRPs better than direct baselines.Semantic uncertainty measures how strongly an unfolding turn constrains plausible next meanings.

2 Related Work

Prior work shows that TRP projection depends on developing linguistic structure alongside prosodic and nonverbal cues, while computational systems largely target observable speaker changes. Semantic uncertainty addresses the underexplored evolution of plausible meanings within turns.

  • 2 Related Work: Developing lexico-syntactic structure supports completion projection, with experiments finding stronger prediction when words remain recognizable than when only intonation is preserved.This places semantic and lexical information alongside prosodic and nonverbal cues in projection research.
  • 2 Related Work: Listeners incrementally revise expectations as each word changes the lexico-syntactic and semantic continuations they consider plausible.Some words narrow possibilities, whereas extensions, repairs, qualifications, and redirections can introduce or redirect them.
  • 2 Related Work: Most computational turn-taking models predict realized speaker changes or future speech activity, rather than within-turn opportunities that need not produce floor transfer.Text models use speaker-change targets, while VAP forecasts joint speech activity from acoustic signals.
  • 2 Related Work: Within-turn TRPs are underrepresented because dialogue corpora primarily record where speakers responded, not where listeners could have responded but did not.This limits direct study of perceived opportunities in standard interaction datasets.
  • 2 Related Work: Semantic uncertainty methods compare variation in meaning rather than only lexical form, but their use for tracking changing semantic constraint during turns remains relatively unexplored.Existing applications have focused mainly on question answering, summarization, and translation.

3 Approach

The approach predicts whether each prefix affords a possible within-turn response by tracking semantic uncertainty over sampled continuations. A causal decision rule then identifies local changes in uncertainty as candidate TRPs.

  • 3.1 Within-Turn TRP Prediction Task: The task predicts a binary opportunity label after each word using only the preceding prefix, regardless of whether a listener actually takes the floor.The reference sequence marks positions that afford possible response within a single speaker’s unfolding turn.
  • 3.2 Predicting TRPs via Semantic Uncertainty: Semantic uncertainty is a scalar measure of how strongly each prefix constrains plausible continuations, producing a causal trajectory as the turn unfolds.Low uncertainty indicates semantically similar continuations, while high uncertainty indicates greater variation.
  • 3.3 Estimating Semantic Uncertainty: For each prefix, the method samples continuations from a language model, embeds them, and computes pairwise semantic similarity for SNNE-based dispersion.SNNE uses sampled continuations and semantic similarity rather than token-level probabilities or entailment clusters.
  • 3.3 Estimating Semantic Uncertainty: More negative SNNE values indicate more semantically constrained continuations, whereas less negative values indicate greater indeterminacy.Applying the estimator across prefixes yields a trajectory of evolving semantic constraints.
  • 3.4 Rule over Uncertainty Dynamics: The decision function uses local shifts in semantic constraint, considering both convergent changes and divergent unexpected extensions as candidate TRPs.The rule is deterministic to preserve interpretability while leaving learned alternatives for future work.
  • 3.4 Rule over Uncertainty Dynamics: Uncertainty values are converted into changes between successive prefixes, smoothed to reduce sampling noise, and compared with a trailing window for adaptive decisions.The causal rule operates on the prefix-indexed trajectory and preserves local change structure.

4 Experimental Setup

The evaluation uses real-time participant responses to construct within-turn TRP labels from a single-speaker dataset. Labels are derived from word-adjacent response proportions, producing a sparse and imbalanced prediction problem evaluated with precision-sensitive metrics.

  • 4.1 Data: Empirical Within-Turn TRPs: The dataset contains 55 single-speaker stimuli with responses from 118 participants, using reversed presentation lists to counterbalance stimulus-order effects.After pooling presentations, each stimulus had a mean of 59 participants and a median of 58.
  • 4.1 Data: Empirical Within-Turn TRPs: Participants verbalized brief backchannels in real time when they perceived an opportunity for speech, rather than providing retrospective annotations.This paradigm targets perceived within-turn opportunities whether or not the listener took the floor.
  • 4.1 Data: Empirical Within-Turn TRPs: Reference labels align response onsets with word-adjacent intervals following each prefix, using each interval’s empirical response proportion.The authors reconstructed transcripts and word-level and response-onset timings manually from the released recordings.
  • 4.1 Data: Empirical Within-Turn TRPs: 16.2% of 5,195 candidate positions are labeled positive, and 72% of the 842 positives are local maxima as a diagnostic rather than a labeling definition.Zero-response intervals are excluded from threshold estimation, and the turn-final position is always positive.
  • 4.2 Evaluation: False-positive predictions may interrupt an ongoing turn, so evaluation reports F0.5 and True Negative Rate alongside balanced accuracy rather than relying on accuracy.F0.5 weights precision more heavily, while TNR measures suppression of spurious predictions.
  • 4.2 Evaluation: Metrics use a two-level macro-average that gives equal weight to each stimulus and each presentation list.This prevents TRP-dense stimuli from dominating pooled results.

5 Experiments and Results

Prompting and supervised fine-tuning do not reliably predict within-turn TRPs, whereas semantic-uncertainty trajectories substantially improve prediction. The semantic-uncertainty pipeline remains stable across embedding and sampling choices, with performance driven chiefly by the decision rule.

  • 5.1 Prompt-Based TRP Prediction: Prompt-based prediction remains weak across models and conditions, with balanced accuracy near chance and low precision and F0.5.Neither imagined future context nor oracle access to upcoming same-speaker words consistently improves performance.
  • 5.2 Supervised Fine-Tuning (SFT) Based TRP Prediction: Supervised fine-tuning provides modest but limited and non-monotonic gains, so performance does not consistently scale with additional training supervision.The comparison uses nested Low, Medium, and Full supervision regimes on a fixed held-out split.
  • 5.3 Semantic-Uncertainty-Based TRP Prediction: Decision-rule choices account for 75.4% of F0.5 variance, while smoothing alone explains 52.6%.Decoding setting, embedding model, and τ explain only 5.5%, 0.9%, and 0.88%, respectively.
  • 5.3 Semantic-Uncertainty-Based TRP Prediction: Semantic dispersion contributes information beyond token-probability uncertainty: NTE and NPE mean F0.5 scores are 0.319 and 0.328, respectively.Their maxima, 0.356 and 0.376, remain below the semantic-uncertainty minimum of 0.385.
  • 5.3 Semantic-Uncertainty-Based TRP Prediction: Mean F0.5 reaches approximately 0.54 at K = 2 and remains near that level for larger continuation samples.The shallow optimum occurs around four to five continuations, and SNNE at K = 2 outperforms NPE at K = 25.

6 Discussion

Within-turn TRP prediction benefits from tracking how semantic expectations evolve over an utterance. Semantic-uncertainty-based prediction outperforms direct text-to-label approaches, though deployment remains computationally costly.

  • Because sampling and embedding are computationally costly, real-time deployment in dialogue systems remains future work.
  • Semantic-uncertainty-based prediction substantially outperforms prompt-based inference and supervised fine-tuning for within-turn TRPs.Prompting yields low F0.5, while SFT shows limited, non-monotonic gains.
  • Predictive-entropy control reaches the SNNE performance range under the same decision rule, suggesting SNNE captures uncertainty beyond token-probability variation.
  • Performance remains stable from two sampled continuations onward, indicating that informative local uncertainty changes can be recovered from small samples.
  • Local changes in semantic uncertainty provide a signal for identifying response opportunities that listeners perceive but do not necessarily act on.The labels come from real-time listener responses rather than retrospective annotation.

7 Conclusion

The findings show that semantic uncertainty provides a more reliable basis for identifying within-turn TRPs than direct text-to-label prediction. More broadly, changes in plausible semantic continuations carry information about perceived response opportunities.

  • Semantic-uncertainty-based prediction provides a more reliable basis for identifying within-turn TRPs than direct text-to-label prediction.The improvement is robust across the tested settings.
  • Changes in the space of plausible continuations carry information about perceived turn-taking opportunities.

8 Limitations

The evaluation is constrained by its small English-only dataset, text-only design, model-mediated uncertainty measure, computational cost, simple decision rule, and interactional cost assumptions. These limitations restrict generalization, deployment claims, and interpretation of reported performance.

  • The evaluation uses a small, English-only dataset of single-speaker turns, limiting generalization beyond the tested setting.The findings have not yet been validated on widely used dialogue datasets such as Switchboard or SpokenWOZ.
  • The text-only evaluation does not establish that semantic uncertainty alone is sufficient for deployed response timing.Acoustic, prosodic, and multimodal cues may interact with semantic uncertainty.
  • Semantic uncertainty remains model-mediated, and alternative language models, embedding spaces, or similarity functions could produce different trajectories.The diagnostic analysis reduces but does not eliminate concern about surface-form effects.
  • Sampling and embedding remain more expensive than single-pass prediction even when smaller continuation samples reduce cost.Mean F0.5 reaches 0.54 at K = 2 and remains near its best values for small K.
  • The fixed causal decision rule prioritizes interpretability over expressive capacity, so reported performance is not an upper bound on uncertainty-based projection.The decision rule accounts for 75.4% of F0.5 variance, with smoothing alone explaining 52.6%.
  • The precision-weighted metrics assume false positives are more disruptive than missed opportunities, an interactional cost structure that may not hold universally.The metrics assess local prediction quality rather than downstream conversational outcomes.

9 Ethical Considerations

The paper warns that TRP signals should remain advisory because deployment can produce inappropriate interruptions or privilege particular interactional norms. Participant data were collected under ethical oversight with informed consent and anonymity protections.

  • Treating turn-prediction signals as entitlements to speak could cause inappropriate interruptions or missed response opportunities.
  • Systems operationalizing TRPs may privilege particular cultural or interactional norms if deployed without care.
  • Participant-judgment data were collected with informed consent, anonymity protections, and institutional review board approval.

A Operationalizing TRPs: Stimulus-Specific Thresholding and Local Maximality

The labeling procedure uses stimulus-specific response thresholds to accommodate dispersed responses and differing stimulus baselines, then diagnoses whether selected intervals are locally prominent.

  • A Operationalizing TRPs: Stimulus-Specific Thresholding and Local Maximality: Stimulus-specific thresholds identify response proportions elevated relative to each stimulus, accommodating temporal dispersion and differing baseline responsiveness.A fixed global threshold would be poorly suited to these response characteristics.
  • A Operationalizing TRPs: Stimulus-Specific Thresholding and Local Maximality: Local peak margin compares an interval’s response proportion with the larger proportion of its immediate neighbors, with positive values indicating strict local maxima.Figure 4 visualizes the resulting distinction between detected TRPs and non-TRP intervals.
  • A Operationalizing TRPs: Stimulus-Specific Thresholding and Local Maximality: 72% of 842 positive labels occur at strict local maxima, showing that stimulus-specific thresholding generally selects locally prominent response positions.Local maximality is diagnostic rather than part of the labeling rule.
  • A Operationalizing TRPs: Stimulus-Specific Thresholding and Local Maximality: The study varies model scale and architecture, including dense and mixture-of-experts instruction-tuned open-weight LLMs, to test whether behavior depends on model family or capacity.Parameter counts and quantization conventions are summarized in Table 5.

B.2 Experimental Infrastructure

The experiments use controlled, resource-efficient infrastructure to evaluate prompting, fine-tuning, and semantic-uncertainty configurations across models, decoding settings, and embedding models.

  • B.2 Experimental Infrastructure: Efficient batched inference, 4-bit weight quantization, FP16 computation, and LoRA adapters reduce memory or training requirements for large-scale evaluation.The implementation uses custom MLC-LLM and HuggingFace adaptations on NVIDIA A100 hardware.
  • B.2 Experimental Infrastructure: Prompting uses temperature 0.4 and top-p 1.0 with one prediction per interval, controlling stochasticity while limiting computational cost.The same fixed decoding configuration is applied across all evaluated LLMs and prompting conditions.
  • B.2 Experimental Infrastructure: Prompt outputs contain binary TRP decisions, confidence scores, and justifications, but evaluation uses only the binary decision; deterministic parsing succeeds for 99.5% of 124,680 generations.Confidence and justifications are recorded but excluded from reported metrics.
  • B.2 Experimental Infrastructure: Fine-tuning adapts LLaMA-3.1-8B-Instruct with LoRA and completion-only causal-language-modeling loss rather than optimizing F0.5 directly.The choice tests supervised adaptation while leaving larger-model fine-tuning for future work.
  • B.2 Experimental Infrastructure: The semantic-uncertainty evaluation sweeps continuation sampling, uncertainty, and decision-rule parameters across a structured configuration space.The method uses an 8B continuation model and varies continuation diversity across six temperature and top-p settings.
  • B.2 Experimental Infrastructure: Semantic uncertainty is computed with three embedding models and pairwise cosine similarity because short continuations can differ lexically while preserving meaning.The embedding set tests whether the signal varies with embedding-model scale.

E.2 Semantic Uncertainty Robustness

Robustness analyses show that decoding stochasticity broadens and diversifies sampled continuations while preserving prefix-level semantic regions, and that embedding-based semantic similarity is supported in-domain.

  • E.2 Semantic Uncertainty Robustness: As decoding becomes more stochastic, mean SNNE changes from −11.74 to −10.61 while prefix-level standard deviation decreases from 0.56 to 0.21.These values indicate broader continuation sets but compressed contrast across prefixes.
  • E.2 Semantic Uncertainty Robustness: Continuation similarity decreases from 0.72 to 0.54 with greater decoding stochasticity, indicating more semantically diverse continuations for each prefix.The measure averages cosine similarity among distinct sampled continuations.
  • E.2 Semantic Uncertainty Robustness: The robustness sweep evaluates 2,304 scored configurations by crossing 72 uncertainty-signal settings with 32 decision-detector settings.The two-phase design separates uncertainty estimation from trajectory-to-TRP decision conversion.
  • E.2 Semantic Uncertainty Robustness: Mean cosine similarity exceeds 0.94 across decoding settings, indicating that greater continuation diversity does not substantially move a prefix to a different semantic region.The diagnostics therefore support prefix-tied uncertainty rather than dependence on one sampling configuration.
  • E.2 Semantic Uncertainty Robustness: Increasing τ compresses SNNE scale, but the main sweep indicates limited downstream effect because TRP prediction depends on interpreting local trajectory changes.At τ = 100, the calibration contrast no longer captures the full empirical range of the signal.
  • E.2 Semantic Uncertainty Robustness: In 77 of 78 triplets, the original continuation is closer to its paraphrase than to an unrelated continuation, with a mean paired difference of 0.207.This in-domain diagnostic supports using embedding dispersion as a model-mediated semantic-variation proxy for short continuations.
Loading 2609.10934v1…