CLASH: Counterfactual Auditing of Lexical and Prosodic Reliance in Spoken Sarcasm Detection

Motivation

• A correct prediction alone does not reveal which cues a spoken sarcasm detector uses.Acoustic associations with sarcasm do not establish that prosody is a stable or sufficient decision signal; realisations vary across utterances and speakers, and human judgements depend on discourse context.Source: §1 · p. 1
• Aggregate gains from audio or learned speech embeddings do not isolate lexical, prosodic, or interaction effects.Learned embeddings can encode lexical and prosodic information together, so comparisons with and without them measure only an aggregate contribution.Source: §1 · p. 1

Approach

• CLASH pairs lexical-preserving, prosody-preserving, and approximately neutralised speech conditions with original speech to distinguish score shifts, label discrimination, and binary predictions.The framework evaluates handcrafted acoustic-feature systems, SSL probes, and speech LLMs on MUStARD and CMMA.Source: §1 · p. 1, Figure 1 · p. 2

Findings

• For target-only Qwen3-Omni, lexical-preserving speech retains a 0.135–0.148 AUROC advantage over prosody-preserving speech after duration balancing, and the advantage persists under alternative resynthesis.The paired design distinguishes score changes from their label-discriminative AUROC and binary predictions; duration balancing, within-identity comparisons, and alternative lexical resynthesis test robustness.Source: §1 · p. 1

Contributions

• CLASH’s utterance-paired factorial protocol audits lexical and prosodic reliance while holding detector parameters fixed.The framework is designed to examine acoustic score shifts that may coexist with weak label discrimination and unchanged binary predictions.Source: §1 · p. 1

2.1. Paired Speech Interventions

• Each utterance is evaluated in original (O), lexical-preserving (L), prosody-preserving (P), and approximately neutralised (F) conditions, with detector parameters fixed across conditions.Within-detector contrasts measure responses to input interventions; cross-detector comparisons also reflect representation and readout differences.Source: §2.1 · p. 2
• L attenuates pitch and intensity variation while preserving timing and pauses; P replaces the spectral envelope while retaining F0 and aperiodicity, and F combines both operations.In Mandarin, L reduces pitch variation while retaining lexical tone contours; retained timing and processing effects mean F is not an information-free baseline. An alternative L condition, LTTS, resynthesises reference transcripts with a fixed voice and neutral-delivery instructions, with intelligibility, acoustic preservation, and speaker changes assessed across generated conditions.Source: §2.1 · p. 2

2.2. Detector Evaluation

• The evaluation compares handcrafted acoustic-feature systems, SSL probes, and LALMs on CMMA and MUStARD, with corpus evaluations kept separate.The main estimates use 3,960 CMMA and 690 MUStARD utterances; the handcrafted and learned probe baselines use standardised features and L2-regularised logistic regression trained on original CMMA training speech, making MUStARD a cross-lingual, cross-corpus transfer evaluation. Context-aware inputs prepend preceding dialogue text, which remains available in F.Source: §2.2 · p. 2

2.3. Diagnostic Contrasts and Robustness

• The diagnostics distinguish score sensitivity from label discrimination: paired score differences measure sensitivity, their AUROC measures label discrimination, and Macro-F1 and positive-prediction rates describe binary decisions.Binary predictions use a fixed score threshold of zero; the principal discrimination contrasts compare L with P and LTTS with P. These contrasts describe implemented transformations, while L–P cancels dependence on F but leaves off-target differences uncontrolled; the interaction measures AUROC nonadditivity, not a specific incongruity mechanism.Source: §2.3 · p. 3
• Duration robustness is tested by weighting within five quantile bins of log(1 + original duration) to equalise class contributions, with 95% percentile intervals from 2,000 cluster-bootstrap replicates.The same weights apply across speech conditions; resampling uses CMMA conversation clusters and 21 MUStARD speaker clusters, and a ten-bin analysis tests sensitivity to binning.Source: §2.3 · p. 3

3.1. Cross-Model Intervention Profiles

• Qwen3-Omni has the highest original-speech AUROC in Table 1, but stronger original-speech performance does not consistently correspond to a larger prosody-preservation advantage.Across six systems on the common CMMA subset, AL−AP is positive, ranging from 0.025 to 0.058 for acoustic probes and 0.071 to 0.219 for LALMs; negative probe ∆P values show that relative advantage need not mean strong absolute discrimination.Source: §3.1 · p. 3, Table 1 · p. 4, Table 2 · p. 4

3.2. Score Sensitivity and Binary Predictions

• P increases log-odds over F for 74.8% of CMMA and 82.5% of MUStARD utterances, yet both conditions produce exclusively non-sarcastic predictions under target-only Qwen3-Omni.Difference-based AUROC is 0.566 and 0.524, respectively, demonstrating that score sensitivity, discrimination, and binary decisions can yield different conclusions.Source: §3.2 · p. 3

3.3. Duration and Lexical Resynthesis

• Duration alone yields 0.660 AUROC on MUStARD, where sarcastic utterances average 5.77 s versus 4.45 s for non-sarcastic utterances.Unintelligible-speech scores correlate negatively with duration (ρ = −0.72), and longer sarcastic utterances can receive lower scores after lexical removal, contributing to below-chance ordering in P and F.Source: §3.3 · p. 3
• 0.135 on MUStARD and 0.148 on CMMA are the duration-balanced AL−AP contrasts for target-only Qwen3-Omni, with both cluster-bootstrap intervals above zero.The contrasts remain positive under WORLD and TTS resynthesis, although binary decisions differ; on CMMA, TTS raises positive predictions from 14.4% to 30.8% while Macro-F1 falls from 0.594 to 0.557.Source: §3.3 · p. 4, Table 3 · p. 4
• WORLD-based L increases CMMA CER/MUStARD WER by 19.9/11.0 percentage points relative to O, while TTS reduces them by 30.4/15.4 points.Experts judged inspected generated samples acceptable in perceptual quality, but these checks do not establish perfect cue separation.Source: §3.3 · p. 4

3.4. Context Effects across Corpora

• Removing context yields larger target-only P−F score shifts, but within-identity estimates do not support a consistent compensatory benefit across corpora.The within-conversation CMMA estimate is 0.037 [−0.018, 0.089], and the within-speaker MUStARD estimate is −0.012 [−0.038, 0.034]; context and joint-condition effects vary across corpora, whose language and corpus differences are confounded.Source: §3.4 · p. 4

• Acoustic interventions shifted model scores without consistently improving sarcasm discrimination or changing binary predictions.Source: §4 · p. 4
• For target-only Qwen3-Omni, the lexical-preservation advantage persisted after duration balancing and under independent lexical resynthesis, while context and interaction effects varied across corpora.Source: §4 · p. 4
• The findings distinguish acoustic sensitivity from discrimination and show that audio-based performance gains alone provide insufficient evidence of prosodic cue use.Source: §4 · p. 4
• Imperfect cue separation, probe transfer, and corpus-specific structure limited the scope of attribution.Source: §4 · p. 4