PaperlayerView exact revision on arXiv

Source-linked AI summary

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He, Jiarui Hai, Mingrui Liang, Kaavya Chaparala, Thomas Thebaud, Laureano Moro-Velazquez, Najim Dehak, Jesus Villalba

arXiv:2608.27176v1cs.CLcs.AIcs.LGeess.AS

TL;DR

The paper addresses whether spoken-dialogue models genuinely use acoustic evidence when transcripts provide misleading shortcuts. It constructs ContraTalk with conflict and consistent cases and introduces Audio Twin as an explicit acoustic-evidence interface; experiments show improved conflict-case behavior but selective grounding remains necessary.

  • Problem

    Existing spoken-dialogue evaluations can be solved through transcript shortcuts or single-modality reasoning, obscuring genuine acoustic grounding.

  • Method

    The paper constructs ContraTalk from cross-modal disagreement and consistent cases, and represents localized acoustic cues through the text-readable Audio Twin framework.

  • Results

    Audio Twin improves conflict-case behavior and reduces transcript-biased errors, while direct Audio-LLMs provide only partial grounding.

  • Takeaways & Limitations

    Robust spoken-dialogue reasoning requires deciding when transcript evidence is reliable and when acoustic cues should revise a surface interpretation.

  • Takeaways & Limitations

    The work focuses on controlled cross-modal disagreement and does not exhaust the full range of acoustic, pragmatic, social, and cultural cues in real-world conversations.

Abstract

from arXiv · show

Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.

1 Introduction

Spoken dialogue understanding must reconcile lexical transcript content with paralinguistic acoustic evidence, because existing evaluations can reward transcript shortcuts. The paper introduces ContraTalk and Audio Twin to evaluate and support this reconciliation.

  • 1 Introduction: Cross-modal disagreement arises when transcript wording and acoustic delivery support different interpretations of spoken dialogue.Relevant cues include hesitation, suppressed frustration, timing, prosody, and speaking behavior.
  • 1 Introduction: Existing benchmarks can let models rely on transcript shortcuts while underusing acoustically informative paralinguistic signals.This makes single-modality solutions appear successful without demonstrating genuine speech grounding.
  • 1 Introduction: ContraTalk converts disagreement regions between transcript-based and speech-grounded interpretations into controlled QA examples, while also retaining consistent cases.The benchmark spans interaction behavior, emotion state, social stance, dialogue act, and conversational intent.
  • 1 Introduction: Strong text-only LLMs perform well on consistent cases but degrade substantially under cross-modal disagreement, while Audio Twin improves conflict-case behavior.Direct Audio-LLMs reduce some transcript-biased errors but still frequently select text-biased traps.
  • 1 Introduction: Audio Twin converts localized speech-derived cues into text-readable evidence that reasoning models can inspect against transcript interpretations.The framework is designed to expose acoustic evidence for revising or confirming a transcript-based reading.

2 Related Work

The paper positions ContraTalk as a disagreement-aware benchmark and formulates spoken dialogue understanding as inference over paired textual and acoustic evidence. Its construction derives controlled QA cases from modality relationships and verifies them through automated checks and human review.

  • 2 Related Work: Multimodal systems can favor predictive textual shortcuts over complex acoustic evidence, producing modality bias and shortcut learning.The related work connects this issue to broader multimodal learning and spoken emotion-recognition findings.
  • 2 Related Work: Agentic multimodal approaches decompose complex inputs and route queries to specialized perceptual modules or retrieval loops.ContraTalk’s framing explicitly addresses disagreement between transcript-derived interpretations and acoustic signals.
  • 3.1 Problem Formulation: The task represents each dialogue as D = (T, A), distinguishing transcript-only reasoning from full spoken-dialogue reasoning with acoustic evidence.Cross-modal disagreement requires detecting, interpreting, and resolving conflicting evidence rather than assuming modality alignment.
  • 3.1 Problem Formulation: Conflict instances use acoustics to determine the gold answer when transcript reasoning yields a plausible incorrect surface answer, whereas consistent instances test calibrated agreement.Questions target interaction behavior, emotion state, social stance, dialogue act, or conversational intent.
  • 3.2 ContraTalk Benchmark Construction: ContraTalk is built from spoken dialogues, transcripts, and construction-time speaker prompts, which are not provided to models during evaluation.The construction process derives transcript and audio-grounded interpretations, then creates conflict or consistent examples.

4 Agentic-style Inference with Audio Twin

The Audio Twin framework makes localized speech cues explicit and retrieves question-relevant evidence alongside transcripts, enabling controlled comparison of lexical and acoustic interpretations. ContraTalk evaluates this behavior across conflict and consistent cases, showing improved conflict-case grounding but backbone-dependent consistency.

  • Audio Twin representation: Audio Twin converts speech-derived cues into transcript-aligned, text-readable evidence that can be retrieved before answer prediction.The representation links acoustic evidence to dialogue turns and speakers while preserving the full lexical context.
  • Audio Twin representation: Audio Twin exposes prosodic, affective, fluency, timing, and overlap cues so reasoning can compare how dialogue was spoken with what was said.The design targets modality collapse by making speech delivery explicit rather than leaving audio opaque.
  • Agentic-style evidence retrieval: The retrieval module selects relevant transcript anchors and returns question-relevant Audio Twin entries for answer prediction.An evidence plan specifies needed speech-grounded evidence types without choosing an answer.
  • Conflict-case results: 33.0–47.7% accuracy and 34.5–45.0% mislead rates show that text-only LLMs are vulnerable to transcript-biased traps on conflict cases.Direct Audio-LLMs reach 33.6–46.8% accuracy with 29.7–39.9% mislead rates, while Audio Twin variants reach 43.2–50.5% accuracy with 29.4–34.8% mislead rates.
  • Consistent-case results: 98.2% overall accuracy is reached by Opus 4.7 on consistent cases, while direct Audio-LLM performance varies from 89.9% to 69.6%.Consistent cases test whether models preserve transcript-supported answers when transcript and speech agree.
  • Cross-split comparison: Paired results reveal a modality-collapse pattern in which speech can improve conflict behavior yet destabilize cases where transcripts already support the correct answer.Other Audio-LLMs show limited conflict improvement and continued reliance on transcript-based cues.

6 Conclusion

ContraTalk evaluates spoken dialogue understanding under both cross-modal conflict and agreement, exposing transcript shortcuts and testing whether models remain calibrated when audio need not revise the text. Experiments show that explicit acoustic evidence aggregation improves conflict-case behavior, while selective use of audio remains necessary.

  • ContraTalk separates conflict cases, where acoustic evidence contradicts plausible transcript interpretations, from consistent cases, where both modalities support the same answer.
  • Text-only LLMs perform well on consistent cases but degrade sharply on conflict cases, revealing transcript-centric failures in speech-grounded understanding.
  • Direct AudioLLMs provide partial grounding, improving some conflict cases but retaining substantial transcript-biased errors and sometimes reducing consistent-case accuracy.
  • Audio Twin improves conflict-case behavior by providing an explicit interface to localized acoustic evidence for reasoning.
  • Explicit acoustic grounding must be applied selectively so that speech evidence revises unreliable surface readings without disrupting correct text-supported reasoning.

A Limitations

The study addresses controlled cross-modal disagreement rather than complete spoken dialogue understanding, and Audio Twin is only one instantiation of explicit audio-grounded reasoning. Its interface also depends on the fidelity of extracted speech cues.

  • The benchmark focuses on controlled transcript shortcuts under cross-modal disagreement rather than the full range of real-world spoken-dialogue cues.
  • ContraTalk spans several discourse dimensions but does not exhaust acoustic, pragmatic, social, and cultural cues shaping real-world conversations.
  • Audio Twin is one instantiation of explicit audio-grounded reasoning, leaving room for more complex agents, encoders, and evidence-selection strategies.
  • Audio Twin’s utility depends on extracted-cue fidelity, which may be limited by noise in transcription, alignment, prosodic analysis, or affect estimation.

B ContraTalk Benchmark Construction Details

ContraTalk is constructed by deriving transcript-based surface interpretations, locating disagreement or agreement regions, converting selected regions into QA instances, and applying automatic and human quality assurance.

  • Overview: The pipeline first captures text-only surface interpretations, identifies disagreement regions, generates question-answer pairs, and performs multiple quality-assurance stages.
  • Step 1: Text-only surface interpretation: A language model annotates transcript-only interpretations across interaction behavior, emotion state, social stance, dialogue act, and conversational intent.
  • Step 2: Speaker-conditioned label comparison: Speaker-conditioned candidate grounded labels are compared with text-only interpretations to identify audio-dependent conflict regions and text-solvable consistent regions.
  • Step 3: Conflict and consistent question generation: Selected regions become multiple-choice QA instances whose conflict questions pair the intended spoken interpretation with a plausible transcript-biased distractor.
  • Step 4: Automatic quality control: Automatic quality control tests whether questions and answer choices reveal answers without transcript or audio evidence.
  • Step 5: Human verification: Human verification checks answerability, evidence quality, and distractor validity, prioritizing conflict cases with validator disagreement and sampling consistent cases.

C Statistical Uncertainty

Uncertainty is quantified with dialogue-level nonparametric bootstrap intervals, preserving dependence among questions derived from the same dialogue while excluding uncertainty from alternative construction or labeling decisions.

  • Table 5 reports aggregate confidence intervals for the main metrics in Tables 2–4.
  • The analysis uses a nonparametric dialogue-level bootstrap, sampling dialogues with replacement and including all questions derived from each sampled dialogue.
  • 95% percentile confidence intervals use 10,000 bootstrap samples and the 2.5th and 97.5th percentiles of each metric distribution.
  • The intervals quantify score variability under dialogue-level resampling but do not capture uncertainty from alternative benchmark construction or labeling decisions.

D Human Verification Protocol

ContraTalk’s human verification prioritizes audio evidence, checks answerability and option clarity, and revises or removes examples that fail these criteria. The protocol covers both conflict and consistent cases through targeted review and sampling-based quality control.

  • Quality control: Conflict candidates receive targeted review, while consistent candidates undergo random sampling-based quality control.Disagreement among a text-only LLM, a direct Audio-LLM, and Audio Twin system identifies higher-risk conflict cases for manual review.
  • Evaluator workflow: Seven reviewers verify answer correctness, distinguishability of choices, and answerability without construction-time information.Reviewers are blind to the speaker prompts used during benchmark construction.
  • Quality control: Examples are revised or removed when audio evidence is unclear, answers are insufficiently grounded, options are ambiguous, or distractors are too close.When the correct answer is present, reviewers update its index; otherwise, they revise a choice and provide the updated text.
  • Coverage: 350 of 501 retained examples were manually reviewed, covering 230 conflict cases and 120 consistent cases.Conflict cases were checked for answerability, speech-grounded evidence, and distractor quality; consistent cases were checked for aligned interpretations.
  • Evaluator workflow: Evaluators listen to the audio first, use the transcript only as support, and trust audio when the modalities suggest different interpretations.
  • Mislead options: Conflict questions retain a mislead option even when it is text-biased, and evaluators select it if the audio actually supports it.

F Audio Twin Representation Details

Audio Twin converts aligned speech-derived cues into structured, text-readable evidence cards for reasoning. Its representation combines turn-level, speaker-baseline, dialogue-dynamics, contextual, and limitation-aware evidence.

  • Representation: Audio Twin stores speech-derived evidence as compact JSON-like cards with stable identifiers and fixed fields.The representation is built from transcript and audio-derived features without gold answers, prompts, labels, or verification decisions.
  • Alignment and textualization: The system aligns transcript turns to acoustic segments using timestamp overlap when available and transcript-text similarity as fallback.Timestamp alignment is high reliability when overlap covers at least half the turn duration; fallback similarity requires at least 0.50.
  • Evidence families: Turn cards link transcript turns to loudness, pitch, timing, overlap, affect, emotion, confidence, fluency, alignment method, and reliability.
  • Evidence families: Speaker baseline cards summarize usual delivery and affective profiles, enabling speaker-relative interpretation of individual turns.Baseline reliability is high with at least four same-speaker observations and low otherwise.
  • Evidence families: Dialogue-dynamics cards summarize turn counts, speaking time, overlap, overlap duration, and response-delay patterns for each speaker.
  • Alignment and textualization: Acoustic and timing features are textualized relative to speaker distributions, while affective scores are discretized into fixed qualitative labels.Valence uses thresholds 0.30, 0.45, 0.55, and 0.70; arousal, dominance, and emotion confidence use 0.40 and 0.70.
  • Evidence notes: Evidence notes state what cues support, remain compatible with, or cannot establish without discourse context.Sincerity, social intent, sarcasm, hostility, and confidence cannot be inferred from low-level acoustic cues alone.

G Agentic-style Evidence Retrieval Details

The agentic retrieval pipeline plans and validates localized evidence before diagnostic answer selection. It separates transcript localization, evidence retrieval, grounding, and answering so the process remains focused and auditable.

  • Evidence planning: The system assigns evidence plans by discourse dimension and retrieves the speech evidence needed for each question type.Plans cover emotion state, interaction behavior, speaker-level style, local prosodic delivery, and speaker comparison.
  • Transcript localization: A locator selects transcript anchors without predicting answers, then normalization repairs identifiers, removes invalid lines, and merges duplicates.Repair is triggered when required targets or quoted lines are missing or too few valid anchors remain.
  • Evidence retrieval: Retrieval uses selected transcript turns, aligned Audio Twin cards, speaker baselines, and local chronological context rather than the full transcript.
  • Evidence validation: Validation checks required anchors, target lines, context, speakers, and matching Audio Twin entries before answer prediction.Speaker-comparison plans require evidence from both speakers and comparison or representative-turn cards.
  • Evidence validation: Missing required evidence marks the bundle incomplete and records the missing type, although the pipeline may still produce a flagged forced-choice answer.Selecting transcript lines alone is not treated as successful retrieval.
  • Pipeline control: Stage boundaries and output-format checks prevent later stages from adding transcript regions or making answer choices prematurely.Execution traces retain intermediate plans, retrieval identifiers, validation results, grounding outputs, and final answers for audit.
  • Diagnostic grounding: Diagnostic grounding compares the target utterance, dialogue context, retrieved acoustic evidence, and support for each answer candidate.The final answer and citations are selected from the validated retrieval bundle.

H Model Details

The paper reports model checkpoints, evaluation settings, parameter counts, and computational resources for self-hosted and hosted runs.

  • Model and evaluation details: Table 7 lists model names, corresponding checkpoints, evaluation settings, and publicly available parameter-count information.
  • Computational resources: Self-hosted runs used at most 2 A100 GPUs, 8 CPUs, 64 GB RAM, and 24 hours per run.
  • Computational resources: Hosted API runs used at most 2 local CPUs, 8 GB RAM, and 4 hours per run.

I Example Agentic Reasoning Trace

The trace separates transcript localization from answer selection, then retrieves localized acoustic and contextual evidence before resolving a misleading backchannel interpretation. The final answer rejects the neutral transcript reading in favor of a stalling/bracing function before a difficult topic shift.

  • Inference procedure: The agentic procedure localizes relevant transcript lines, plans evidence retrieval, validates the evidence contract, grounds the diagnosis, and then outputs an answer.Answer selection is blocked during planning and permitted only after retrieval validation confirms the required evidence is present.
  • Evidence planning: The plan compares the target backchannel’s prosody with the same speaker’s routine backchannels and examines pauses, overlap, disfluencies, and the following topic shift.The comparison is designed to distinguish stalling or steeling from neutral listening, encouragement, or a smooth transition.
  • Diagnostic grounding: In isolation, “Mm-hmm” suggests a neutral continuer, but the local sequence places it before disfluent restarts and a face-threatening noise complaint.The diagnostic reading therefore treats the token as a stalling/bracing device at a topic-shift hinge point.
  • Diagnostic grounding: Acoustic marking relative to the speaker baseline, together with the following disfluencies and complaint, overturns the transcript-biased neutral choice.The trace records the isolated transcript reading as choice 3 and the audio-grounded final candidate as choice 0.
  • Final answer: The final output selects “Buying time and steeling herself for the topic shift” rather than neutral continued listening.The answer is based on speaker-relative prosody and local sequential context retrieved through the trace.
Loading 2608.27176v1…