PaperlayerView exact revision on arXiv

Source-linked AI summary

When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection

Yongjian Chen, Pengfei Wei, Yiqun Sun, Zhu Li, Lawrence B. Hsieh

arXiv:2608.30204v1cs.CL

TL;DR

The paper asks whether MLLMs use prosody for sarcasm inference or rely on surface acoustic stereotypes. It evaluates two Omni models across Mandarin and English with modality decomposition, acoustic diagnosis, and targeted manipulation, finding that elevated pitch and irregular pausing drive false positives and can induce rates up to 60%.

  • Problem

    Whether MLLMs exploit prosodic cues for pragmatic sarcasm inference, rather than surface acoustic patterns, remains systematically underexplored across languages and modality conditions.

  • Method

    The study evaluates two MLLMs on Mandarin and English sarcasm corpora using five modality conditions, 66 acoustic features, error diagnosis, and targeted prosodic manipulation.

  • Results

    Adding audio systematically inflates false positives without improving true-positive detection; manipulating only pitch and pause structure induces false positive rates of up to 60%.

  • Takeaways & Limitations

    Model errors track an expressive-prosody stereotype that diverges from corpus-derived sarcasm signatures in both languages and also appears in Gemini 3 Flash Preview.

  • Takeaways & Limitations

    Both corpora consist of performative television speech, so corpus-derived acoustic profiles may not fully generalize to spontaneous everyday interaction.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) process speech and text jointly, yet whether they exploit prosodic cues for pragmatic inference or rely on surface acoustic patterns has received little systematic investigation. We address this through sarcasm detection, evaluating Qwen2.5-Omni and Qwen3-Omni on Mandarin Chinese and English under five modality conditions that decompose the contributions of lexical content, vocal semantics, and prosodic structure. Adding audio systematically inflates false positives without improving true positive detection. Acoustic error diagnosis reveals that model errors cluster on a shared stereotype of expressive prosody, namely elevated pitch and irregular pausing, that diverges from the actual cues marking sarcasm in both languages. Targeted manipulation of only these two dimensions causally confirms the heuristic, inducing false positive rates of up to 60%. Applying the same manipulation template to Gemini~3 Flash Preview without modification replicates the effect, suggesting that the stereotype extends beyond the Qwen Omni family rather than arising from a single model architecture.

1 Introduction

Sarcasm depends on the relationship between lexical content and acoustic delivery, but prosodic cues vary across languages and MLLMs often underuse audio when text is present. The paper addresses this gap with a cross-lingual diagnostic framework that decomposes modalities, diagnoses acoustic errors, and validates heuristics causally.

  • Motivation: Prosody signals the gap between literal and intended meaning, but its acoustic realization varies across and within languages.Slower speech is broadly consistent, whereas pitch direction is unstable; Mandarin also shows creaky voice and lower F0.
  • Research gap: When text is available, prior MLLM studies report that models often underuse the acoustic channel on deeper paralinguistic tasks.This motivates testing whether spoken sarcasm models exploit prosodic cues or rely on surface patterns.
  • Approach: The study evaluates two MLLMs on Mandarin MSCD and English MUStARD++ using five conditions that separate lexical, vocal-semantic, and prosodic contributions.The framework is applied across both languages and models.
  • Approach: The framework combines modality decomposition, acoustic error profiling, and targeted manipulation to test whether identified prosodic heuristics causally produce false positives.The same diagnostic pattern is also examined beyond the Qwen Omni family.

2 Experimental Setup

The experiments use two balanced, televised-speech sarcasm corpora and two zero-shot Qwen Omni models. Five input conditions isolate text, speech, prosody, and their multimodal combinations, with enhancement, filtering, feature extraction, and repeated evaluation used to characterize performance and acoustic behavior.

  • Datasets: The study evaluates 2,705 Mandarin MSCD samples and 1,202 English MUStARD++ audio-transcript pairs, both approximately balanced for sarcasm.Both datasets contain performative television speech with studio audience reactions.
  • Evaluation: Performance is measured with binary sarcasm prompts across repeated train-fold, validation, and test evaluations.The training split is partitioned into five repeated evaluation folds rather than used for model fitting.
  • Models: Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B-Thinking are evaluated zero-shot, differing in audio encoder capacity and language-model backbone.Omni 2.5 uses a Whisper-based encoder, whereas Omni 3.0 uses a purpose-built Audio Transformer and 30B mixture-of-experts backbone.
  • Modality conditions: Five modality conditions isolate text, full speech, prosody-only audio, full-audio transcript pairs, and prosody-filtered transcript pairs.BIMODAL and BIPROSODY hold the transcript constant, allowing their gap to reflect vocal-semantic contribution rather than filtering-induced distribution shift.
  • Audio preprocessing: The analysis extracts 66 acoustic features across fundamental frequency, intensity and energy, rhythm and timing, and voice quality.Enhanced audio is used throughout subsequent processing, with MossFormerGAN selected over FRCRN across the evaluated metrics.
  • Audio preprocessing: A 300 Hz low-pass filter removes most lexical information while retaining F0 contours, intensity patterns, and rhythmic structure.ASR CER rises from 9.01% to 91.11% in English and from 13.61% to 153.74% in Chinese after filtering.

3 What Audio Changes: Performance and Error Shifts

Adding audio generally improves or preserves F1 while shifting errors toward false positives rather than improving true-positive detection. The strongest false-positive pattern appears when transcripts co-occur with expressive prosodic contours, which models interpret as ironic mismatch.

  • Error shifts: Adding audio trades false negatives for false positives rather than uniformly improving detection.The analysis compares audio additions against TEXT-ONLY and transcript additions against SPEECH-ONLY.
  • Overall performance: PROSODY-ONLY shows no reliable discriminative ability: Omni 3.0 reaches 22.2% F1 in English and 14.3% in Chinese, while Omni 2.5 is near chance.The condition is excluded from subsequent error analysis.
  • Overall performance: BIMODAL is top-ranked or joint-top across all four model-language panels, gaining +1.0 to +6.8 pp over TEXT-ONLY.BIPROSODY trails BIMODAL by only 1.0–3.7 pp, indicating a small vocal-semantic contribution relative to prosodic contour.
  • Error shifts: BIMODAL adds +9.8% false positives in English and +8.6% in Chinese for Omni 3.0, with corresponding false-negative reductions of −7.5% and −9.7%.BIPROSODY produces nearly identical inflation, while Omni 2.5 shows the same direction mainly in Chinese.
  • False-positive signature: Adding transcripts to speech or prosody-only audio adds 1–12% of samples as false positives across both models and languages.The transcript anchors literal content as non-sarcastic while audio supplies a perceived tonal contrast.
  • False-positive signature: False positives arise from the co-presence of transcript and audio: both full speech and prosody-only traces attribute irony to expressive prosodic contour.Neither transcript alone nor audio alone triggers the same false alarm.

4 Corpus-Derived Acoustic Profile of Sarcasm

Corpus-derived sarcasm profiles differ sharply between Mandarin and English. Mandarin sarcasm is characterized mainly by pitch contour complexity and temporal structure, whereas English sarcasm emphasizes energy, lower F0, and longer duration.

  • Chinese (MSCD): Mandarin sarcasm is dominated by temporal and pitch-structural features, led by total pause duration (d = 0.80), utterance duration (d = 0.76), and pitch contour complexity (d = 0.74).Top features replicate across all splits, and total pause duration reaches d = 0.90 in the test partition.
  • Chinese (MSCD): Mandarin sarcasm shows more pitch peaks and valleys (d ≈ 0.74), while mean F0 is not among the top discriminating features.This indicates contour complexity rather than pitch level as the distinctive F0-related signal.
  • English (MUStARD++): English sarcasm is led by intensity and energy features, including INTENSITY_MAX (d = 0.47), RMS_RANGE (d = 0.43), and RMS_MAX (d = 0.43).Lower F0 and longer duration follow as additional signals.
  • English (MUStARD++): English sarcasm has lower F0 across mean, median, and minimum, with effects from d = −0.30 to d = −0.40, plus longer duration (d = 0.32).The lower-F0 pattern holds across all splits.
  • Cross-linguistic comparison: The languages diverge most on F0 direction: Mandarin emphasizes contour complexity, whereas English consistently exhibits lower F0 summary statistics.The findings support caution against treating F0 mean direction as a stable cross-linguistic sarcasm cue.

5 Error Diagnosis: What Do Models Actually Respond To?

Prosodic input inflates false positives, but those errors do not reflect acoustic similarity to genuine sarcasm. Instead, the audio encoder represents false positives as intermediate between control and true-sarcasm speech, while the model responds to elevated pitch and irregular pausing.

  • Acoustic distance: False-positive utterances are acoustically closer to non-sarcastic controls than to true sarcasm in both languages.The FP↔TP centroid distance exceeds FP↔Control by 2.35× in English and 3.23× in Chinese under Euclidean distance, and by 6.68× and 20.11× under cosine distance.
  • Acoustic distance: Every false-positive sample assigns to the Control centroid in handcrafted acoustic feature space.
  • Encoder representation: FP representations lie significantly between Control and TP on the audio-encoder LDA axis across all four model–language conditions.They hover near the Control↔TP midpoint and substantially overlap both distributions; Mann–Whitney U, p < .001.
  • Encoder representation: The encoder therefore preserves ambiguity that is absent from handcrafted acoustic distances, though fusion may further amplify it.
  • Feature mismatch: Across languages, errors cluster on elevated pitch and irregular pausing rather than the corpus-derived sarcasm signatures.Chinese errors target pitch level and pause irregularity instead of contour complexity and total duration; English false positives show elevated F0 where sarcasm has suppressed F0.

6 Causal Verification

Targeted manipulation of control samples tests whether the diagnosed prosodic profile causes false positives. Manipulating pitch and pausing substantially increases misclassification across languages and models, transfers to Gemini, and preserves intelligibility within reported bounds.

  • Causal test: The manipulation is an out-of-sample test derived from FP–Control profiles and applied to separately classified Control samples.Language-specific targets include elevated F0 and pause-duration or pause-irregularity changes, with magnitudes bounded by the 75th percentile of FP prosody.
  • Causal test: Prosodic manipulation raises false-positive rates to 19.35–40.65% in Chinese and 33.55–60.53% in English.Only pitch and pausing were modified, and the manipulated samples were separate from those used to derive the heuristic.
  • Modality context: Omni 3.0 shows higher attack-induced rates under BIPROSODY than BIMODAL in both languages.The authors interpret this ordering as a modality-context effect in which full speech provides acoustic grounding that partially normalizes manipulated prosody.
  • Stimulus quality: Manipulated stimuli retain reported intelligibility, with negligible Chinese CER differences and a minimal English increase of 0.017.
  • Bidirectional evidence: Reverse manipulation toward control-like prosody partially recovers correct classification, with flip rates of 29.5–56.3%.
  • Model transfer: Gemini 3 Flash Preview reproduces the error pattern without model-family-specific tuning, and the Omni-derived template transfers across languages and audio conditions.Gemini flip rates range from 4.7% to 17.1%, while its false-positive increases over TEXT-ONLY reach +13.2 pp and +13.6 pp in English for BIMODAL and BIPROSODY.

7 Conclusion

Across Mandarin Chinese and English, prosodic audio increases false positives without improving true-positive detection. The shared elevated-pitch and irregular-pausing heuristic diverges from corpus-derived sarcasm cues and also transfers to Gemini, indicating a broader MLLM failure mode.

  • Conclusion: Prosodic audio systematically inflates false positives without improving true-positive detection across both languages and evaluated models.
  • Conclusion: Model errors track a language-independent stereotype of elevated pitch and irregular pausing rather than the actual sarcasm signatures.Targeted manipulation of these two acoustic dimensions induces false-positive rates of up to 60%.
  • Conclusion: The same shallow prosodic stereotype appears in Gemini 3 Flash Preview, not only the Qwen Omni family.

Limitations

The study identifies corpus scope and mechanism localisation as boundaries for interpreting its findings, while proposing prosody-aware training interventions as future work.

  • Corpus scope: Both benchmark corpora use performative television speech, so corpus-derived acoustic profiles may not fully generalise to spontaneous everyday interaction.The causal manipulation remains within the same recordings, holding lexical content and recording context fixed.
  • Training implications: Contrastive augmentation could pair identical transcripts with prosodically distinct audio to discourage reliance on surface acoustic patterns.The proposed intervention targets the identified elevated-pitch and irregular-pausing stereotype.
  • Training implications: Prosody-aware reinforcement learning is presented as a complementary direction for prioritising language-specific cues over surface correlations.Related work is cited as evidence that such an intervention can improve acoustic reasoning without sacrificing text-based performance in speech emotion recognition.
  • Mechanism localisation: The audio encoder’s ambiguous false-positive representations are characterised in isolation, leaving the fusion stage’s role unresolved.Probing where prosodic representations meet textual grounding could distinguish inherited encoder bias from bias constructed during fusion.
  • Experimental design: The five modality conditions use shared prompt templates within audio-only and audio-plus-transcript settings, reducing prompt variation as an explanation for condition differences.All templates retain the same binary classification instruction while varying the input modality.
  • Speech enhancement: Speech enhancement improves perceptual quality but generally decreases downstream sarcasm-detection F1 because increased false negatives outweigh reduced false positives.Both enhancement models improve over raw audio across perceptual metrics, while the downstream net effect is an F1 decrease driven by false-negative increases.

C Per-Split Results

Per-split evaluation and quality analyses support the stability of the main findings while showing that speech enhancement does not translate perceptual improvements into sarcasm-detection gains.

  • Per-split stability: The observed findings are consistent across training, validation, and test splits, rather than being artifacts of one data partition.The original training split is divided into five independently evaluated folds used as repeated evaluation runs.
  • Acoustic analysis: The acoustic feature inventory contains 66 Praat/Parselmouth-extracted features spanning four categories.The feature set supports the acoustic analyses in Sections 4–6.
  • Prosodic manipulation: The manipulation targets are derived from FP–Control acoustic profiles and compared with actual PSOLA outputs for target fidelity.The table distinguishes on-target, approximated, and redefined features.
  • Manipulation quality: Prosodic manipulation produces negligible Chinese CER differences and a minimal English CER increase while lowering DNSMOS OVRL.Reported CER values are 0.084 versus 0.079 for Chinese, with an English increase of 0.017; OVRL decreases by 0.30 in English and 0.17 in Chinese.

G Prosodic Feature Ranking (Sarcastic vs. Non-Sarcastic)

Prosodic feature rankings reveal language-specific sarcasm signatures: rhythm and timing dominate Chinese, whereas intensity, energy, and lower pitch characterise English.

  • Chinese: Chinese sarcasm is dominated by rhythm and timing features, with longer utterances, more pauses, and reduced rhythmic regularity.All top-10 Chinese features reach significance in all three splits.
  • Chinese: The largest negative Chinese effect is hnr_min (d = −0.69), consistent with lower harmonics-to-noise ratios in sarcastic speech.This pattern aligns with breathier or rougher voice quality.
  • English: English sarcasm is most strongly distinguished by intensity- and energy-related features, including greater dynamic range.Examples include intensity_max and rms_range.
  • English: English sarcastic speech has a lower overall pitch register, while pause_dur. std is unstable across splits despite ranking seventh by absolute effect size.Pause_dur. std reaches significance in 0/3 splits.
  • False-positive comparison: False-positive profiles are ranked against matched non-sarcastic controls using signed Cohen’s d, with significance assessed by Mann–Whitney U tests.The plots mark the small-effect threshold at |d| = 0.2.
  • Chinese false positives: Chinese false positives show elevated F0 statistics and pause irregularity, contrasting with genuine sarcasm’s dominance by total pause duration and other temporal features.All 10 Chinese FP–Control features are significant and positively directed.
  • English false positives: English false positives show elevated pitch extremes and increased pausing, opposite to genuine sarcasm’s lower-pitch profile.Only 4 of 10 English features are significant; rhythmic regularity and voiced fraction are negatively directed.
  • Causal correction: Reverse manipulation targets control-like prosody to measure the fraction of false positives whose predictions flip to non-sarcastic.Partial recovery is expected because false positives may contain additional acoustic or contextual features beyond the two targeted dimensions.

J Gemini 3 Flash Preview: Extended Results

Extended Gemini results show that the Omni-derived prosodic manipulation transfers across conditions, while Gemini’s own false-positive geometry indicates only partial overlap in the underlying acoustic dimensions.

  • Cross-model transfer: The Omni-derived template transfers to Gemini 3 Flash Preview across all audio conditions, with non-trivial flip rates.The template targets elevated F0 and irregular pausing.
  • Gemini acoustic geometry: For Gemini Chinese, false positives involve higher RMS energy, intensity variability, and F0 contour activity, which the Omni template does not target.This mismatch explains the lower Chinese transfer rates.
  • Gemini acoustic geometry: For Gemini English, false positives involve elevated central pitch, reduced F0 contour complexity, and slower temporal structure, partially overlapping with the Omni template.The overlap is consistent with the higher English transfer rates.
  • Per-split contrasts: Figure 10 compares false-positive and false-negative changes across four modality contrasts for train, validation, and test splits in English and Chinese.Marker shape encodes split, color encodes language, and size encodes model; the lower-right pattern replicates across splits.
  • Manipulation details: The manipulation targets and PSOLA outputs document how FP–Control differences motivate each prosodic adjustment.English F0_MAX and F0_RANGE share an averaged ×1.088 PSOLA factor, while pause insertion was avoided to preserve naturalness.
  • Cross-model transfer: Gemini’s cross-model transfer reflects a partial rather than complete instantiation of the same vulnerability.The transfer establishes that the prosodic heuristic is not confined to Qwen MLLMs.

K Chain-of-Thought Reasoning Traces

The reasoning traces show that false positives arise when models interpret expressive prosody as contradicting serious or factual text. This interpretation changes with modality context: identical manipulated prosody is normalised with full speech but read as mocking when vocal semantics are absent.

  • Acoustic error patterns: False positives show elevated pitch and irregular pausing in Chinese, while English false-positive comparisons identify only four significant separating features.The corresponding figures rank features by signed Cohen’s d and mark significance across splits.
  • Representative false positive: False-positive traces follow three steps: exaggerated or playful prosody, serious transcript content, then a sarcasm judgment based on perceived mismatch.The pattern appears under both BIMODAL and BIPROSODY for the same non-sarcastic utterance.
  • Causal and cross-model checks: Reverse manipulation measures how often false positives flip to correct non-sarcastic classifications, while Gemini evaluation compares an Omni-derived template with a Gemini-specific template.The supplied table descriptions define these comparisons but do not report their numerical outcomes.
  • Representative false positive: The false positive is attributed to text–prosody incongruence rather than information recoverable from vocal semantics.Both reasoning traces converge on the same mismatch-based conclusion.
  • Modality context effect: Identical manipulated prosody yields a correct non-sarcastic judgment in BIMODAL but a false positive in BIPROSODY.The manipulated Chinese sample is described as amused in BIMODAL and as strained, high-pitched, forced, and mocking in BIPROSODY.
  • Modality context effect: Full speech normalises manipulated prosody, whereas removing vocal semantics makes the same features appear mocking rather than expressive.The traces therefore treat prosodic features relative to the surrounding acoustic context rather than as having fixed perceptual weight.
Loading 2608.30204v1…