PaperlayerView exact revision on arXiv

Source-linked AI summary

When Vocal Tone and Literal Meaning Diverge: An Acoustic-Semantic Incongruity Study for Large Audio-Language Models

Yu-Wen Chen, William Ho, Maxim Topaz, Zoran Kostic, Julia Hirschberg

arXiv:2608.28966v1eess.AS

TL;DR

The paper asks whether LALMs can disentangle acoustic emotion from semantic sentiment when the cues conflict. It introduces CREMA-ASIS, evaluates LALMs with multitask prediction and layer-wise probing, and finds semantic dominance and poor incongruity handling. Supervised fine-tuning improves acoustic information while preserving semantic processing, transcription accuracy, and joint emotion recognition.

  • Problem

    Evidence is limited on whether LALMs can disentangle acoustic and semantic affective cues in incongruent speech, despite conflicts occurring in real-world communication.

  • Method

    The paper creates CREMA-ASIS with controlled acoustic emotion and semantic sentiment pairings, then evaluates multitask LALM predictions and layer-wise modality decodability.

  • Results

    LALMs struggle with acoustic–semantic incongruence and underweight acoustic cues, while fine-tuning improves deeper-layer acoustic information and reduces modality disparity.

  • Takeaways & Limitations

    TTS-synthesized data may help LALMs better preserve and disentangle acoustic and semantic information without sacrificing generalization.

  • Takeaways & Limitations

    Most labels reflect intended rather than perceived emotion, and the acted, synthetic data may contain artifacts and transfer less directly to spontaneous speech.

Abstract

from arXiv · show

Affective cues across modalities may be incongruous (e.g., sarcasm or mocking praise), potentially leading to misinterpretation when relying on a single modality. Large Audio-Language Models (LALMs) have recently gained popularity and been applied to multimodal emotion recognition, but their ability to disentangle acoustic and semantic cues, especially in incongruent cases, remains underexplored. To address this gap, we introduce CREMA-ASIS, a dataset specifically created to investigate incongruence between acoustic emotion and semantic sentiment cues. It pairs acoustic emotion labels with semantic sentiment polarities. Using this dataset, we evaluate LALM biases within a multitask framework and conduct a layer-wise analysis to identify modality dominance across layers. Our findings reveal that LALMs struggle with semantic-acoustic incongruent cases, rarely predicting incongruity, and that LALMs are predominantly influenced by semantic information. However, supervised fine-tuning significantly improves LALM performance on our CREMA-ASIS test set while preserving transcription accuracy and joint emotion recognition. Results demonstrate potential for enhancing both acoustic and semantic understanding on out-of-domain data.

1 Introduction

The paper studies how LALMs handle conflicting acoustic and semantic affective cues, a setting common in sarcasm and mocking praise. It introduces a modality-separated framing and CREMA-ASIS to evaluate these biases and improve cue disentanglement.

  • Multimodal affect combines speech and text, but most studies assign one unified emotion label and overlook modality-specific information.
  • LALMs often prioritize textual information over acoustic evidence when the modalities conflict, potentially causing misinterpretation of speaker intent.
  • Prior datasets and single-task evaluations provide limited support for disentangling acoustic and semantic cues, motivating multitask predictions for each modality.
  • CREMA-ASIS pairs acoustic emotion categories with semantic sentiment labels to distinguish vocal expression from the general sentiment conveyed by words.
  • The paper introduces CREMA-ASIS, evaluates LALMs across acoustic–semantic pairings, and uses supervised fine-tuning and layer-wise probing to narrow the modality gap.

2 Method

The method constructs controlled speech with independently specified acoustic emotion and semantic sentiment, focusing on incongruent combinations. It uses emotion references, sentiment-grouped sentences, filtering, and synthetic audio generation to build CREMA-ASIS.

  • CREMA-ASIS targets cases where acoustic emotional valence contradicts the semantic sentiment expressed in an utterance.
  • The dataset pairs reference speech emotion with spoken content and instructs models to predict acoustic emotion and semantic sentiment simultaneously.
  • IndexTTS2 generates speech by cloning reference voices while controlling acoustic emotion through the reference sample and semantic content through the supplied text.
  • Incongruence scenarios include cheerful complaints, mocking praise, and positive messages delivered in a sad tone.
  • CREMA-D provides acoustic references, while GoEmotions supplies human-annotated sentences selected for opinion-expressing content.
  • An acoustic emotion recognizer filters clear mismatches between target and rendered emotion to improve synthetic-data consistency.

3 Experimental Settings

The experiments evaluate three LALMs on acoustic, semantic, and joint prediction tasks, then probe representations across model depth. They also test transfer to MELD and LISTEN and measure how fine-tuning changes modality information.

  • Three LALMs—Qwen2-Audio, Kimi-Audio, and Audio-Flamingo3—are evaluated on CREMA-ASIS, with MELD and CREMA-D data additionally used during fine-tuning.
  • Evaluation reports acoustic accuracy, semantic accuracy, and dual-condition accuracy requiring both predictions to be correct.
  • Generalizability is assessed on the LISTEN_full test split, which contains samples annotated with acoustic or semantic emotion categories.
  • Layer-wise probing measures hidden-state decodability for 5 acoustic and 3 semantic categories across encoders, projectors, selected LLM layers, and Kimi-Audio’s MIMO module.
  • Linear classifiers are trained after mean pooling so probe results reflect information encoded in hidden states rather than classifier capacity.

4 Results

CREMA-ASIS tests whether LALMs can jointly recognize acoustic emotion and semantic sentiment when cues align, conflict, or include neutral modalities. Results show strong semantic bias and poor incongruity recognition, while SFT improves acoustic processing and overall performance without harming transcription.

  • Dataset composition: CREMA-ASIS includes incongruous, congruous, and neutral-associated cases to evaluate separately aligned and partially misaligned acoustic-semantic cues.Incongruous pairs oppose acoustic valence and semantic sentiment; neutral-associated cases have one or both modalities labeled neutral.
  • Subjective validation: 64.6% majority-vote accuracy and Krippendorff’s α 0.54 support reliable reproduction of intended acoustic emotion in the TTS-generated samples.Perceived sentiment medians aligned with intended acoustic valence, although neutral and disgust remained difficult.
  • LALM performance: Existing LALMs rarely predict acoustic-semantic incongruity, yielding very low recall and F1 despite relatively high precision when both cues are strong.Among the four incongruous scenarios, angry–positive performed best.
  • LALM performance: Semantic-neutral cases are especially difficult: models commonly ignore emotional acoustics and predict neutral–neutral, whereas acoustic-neutral cases generally perform better.These patterns indicate a strong bias toward semantic information.
  • Layer-wise probing: In deeper base-model layers, semantic probing accuracy rises above 85% while acoustic accuracy falls to approximately 67% for Kimi-Audio and Audio-Flamingo3 and 59% for Qwen2-Audio.This pattern indicates a progressive shift toward semantic-oriented representations across model depth.
  • Layer-wise probing: After SFT, acoustic gains range from 5.3 pp to 17.4 pp, semantic accuracy changes remain within ±2.5 pp in nearly all layers, and the acoustic-semantic gap is reduced.The gains are concentrated in later layers; Kimi-Audio L28 is a notable semantic exception at −7.6 pp.
  • Test-set evaluation: After SFT, all open-source LALMs improve on CREMA-ASIS while transcription WER remains approximately 0.07, indicating no noticeable transcription degradation.Before SFT, Audio-Flamingo3 consistently outperformed the other models, while joint acoustic-semantic accuracy was low across models.
  • Out-of-domain evaluation: On out-of-domain LISTEN data, SFT improves acoustic performance for Qwen2-Audio and Audio-Flamingo3, but Kimi-Audio’s acoustic performance declines.Semantic performance improves for SFT Qwen2-Audio but decreases for Kimi-Audio and Audio-Flamingo3; LISTEN labels may not reflect transcript-level sentiment.

5 Conclusion

CREMA-ASIS exposes LALM difficulty with acoustic–semantic incongruence and semantic dominance, while fine-tuning improves acoustic information in deeper layers and reduces modality disparity.

  • CREMA-ASIS evaluates LALM bias toward acoustic and semantic affective cues in speech.The dataset is synthetic and designed for controlled evaluation.
  • LALMs struggle with acoustic–semantic incongruence and underweight acoustic cues when semantic content is neutral.
  • Deeper layers show reduced acoustic decodability and improved semantic decodability, widening the modality gap.
  • Fine-tuning improves acoustic emotion information in deeper layers while largely preserving semantic sentiment representations.

Limitations

The study’s limitations concern standardized adaptation settings, indirect probing, synthetic acted data, and restricted dataset coverage.

  • SFT uses shared hyperparameter settings that may not be optimal for every model.Alternative adaptation strategies are outside the study’s focus.
  • The dataset labels intended rather than perceived emotion except for 150 subjective-evaluation samples.Perception may differ because of listener subjectivity and semantic content.
  • Acted references may be exaggerated and easier to identify than spontaneous speech, limiting generalizability.CREMA-ASIS also covers only some GoEmotions sentences and basic emotion categories.

A Experiment Parameter Settings

The experiments use prompts for simultaneous acoustic-emotion and semantic-sentiment prediction, synthetic speech generation, filtering, SFT, and layer-wise probing.

  • Prompting: The LALM prompt instructs simultaneous prediction of acoustic emotion categories and semantic sentiment labels.The prompt includes acoustic categories beyond those in CREMA-ASIS.
  • Prompting: Sentence filtering replaces neutral with positive or negative when selecting sentences with corresponding sentiment.
  • Synthetic data: Synthetic speech uses IndexTTS2 with CREMA-D reference audio for speaker conditioning and filtered GoEmotions text input.Audio is enhanced with SEMamba.
  • Synthetic data: Acoustic filtering with AER removes samples whose conveyed emotion clearly differs from the target, while avoiding exact-match filtering that could discard valid samples.The authors note that sad audio is often classified as neutral by AER.
  • Training: SFT uses rsLoRA with rank r = 4, LoRAα = 32, learning rate 1 × 10−5, gradient accumulation of 4, and dropout 0.1.Checkpoints are selected by overall performance.
  • Probing: Linear probes are trained for 20 epochs with Adam and cosine annealing across four learning rates.Probe outputs target five acoustic-emotion and three semantic-sentiment classes.

A.7 LISTEN Dataset Setup

LISTEN samples are divided into acoustic and semantic subsets according to their questions, but source labels may reflect vocal performance rather than transcript semantics.

  • Subset construction: LISTEN is split into acoustic and semantic subsets based on whether questions target vocal prosody or literal textual content.
  • Dataset caveat: LISTEN’s single source-corpus emotion label is also attached to text-centric questions, even though it was annotated from audio or audiovisual delivery.Thus, the label encodes vocal performance rather than transcript semantics alone.
  • Label mapping: The semantic analysis maps labels to positive, negative, and neutral sentiment and excludes ambiguous mappings.

B Supplementary Analysis of LISTEN Dataset

The LISTEN evaluation can misrepresent semantic sentiment because its text-centric labels inherit delivered acoustic emotions, even when transcripts convey different sentiments. Reference choice therefore materially changes measured semantic agreement, and manual annotation remains necessary for verified semantic gold labels.

  • LISTEN’s text-centric labels can diverge from transcript sentiment because they reuse delivered acoustic-emotion labels under an assumption of cross-modal agreement.Examples include “Somebody please call 911,” labeled happiness despite textual distress or concern, and other transcripts whose text reads neutral-to-positive despite sadness or anger labels.
  • Qwen3-32B transcript predictions were used as a proxy for transcript sentiment, not as verified ground truth, when assessing LISTEN reference agreement.The comparison feeds transcripts alone to a text-only classifier and maps its predictions against LISTEN’s text-centric labels.
  • 0.594 to 0.731: Kimi-Audio’s agreement with transcript-level sentiment increased after SFT on CREMA-ASIS, while its score under LISTEN labels decreased despite identical predictions.The corresponding increases were smaller for Qwen2-Audio, from 0.622 to 0.662, and Audio-Flamingo3, from 0.711 to 0.720.
  • Kimi-Audio’s semantic score moves in opposite directions solely because the scoring reference changes, showing that the evaluation label source affects the apparent SFT outcome.Under transcript-level references the score rises, whereas under LISTEN’s labels it drops, although the model predictions are identical.
  • A true semantic sentiment gold label would require manual human annotation, so weaker LISTEN Accsem results should be interpreted cautiously.

C.1 Complete Probing Results

Complete probing results compare acoustic emotion, semantic sentiment, and their gap across layers for base and SFT models. SFT mainly improves acoustic accuracy, particularly in weaker layers, while semantic accuracy changes comparatively little and the gap narrows.

  • The probing results report acoustic emotion accuracy Accacou, semantic sentiment accuracy Accsem, and acoustic-semantic gap G for each layer in base and SFT models.Changes are represented as ∆ = SFT−base, with negative ∆G indicating a reduced gap.
  • Deeper layers widen the acoustic-semantic gap in base models as acoustic accuracy decreases while semantic accuracy increases.This pattern appears when probing last hidden states and is similar to the mean-pooling results.
  • SFT improves acoustic emotion accuracy most where base models perform worst, while changes in earlier layers are modest.
  • Gap reductions after SFT are driven largely by acoustic gains because semantic accuracy changes comparatively little.The resulting acoustic performance is more uniform across layers.

C.2 Kimi-Audio L28 Analysis

The Kimi-Audio L28 analysis extends probing across late LLM layers and MIMO layers to explain an acoustic-accuracy drop. Accuracy decreases through the LLM backbone, then rises sharply in MIMO, with SFT making the L28 drop sharper.

  • Kimi-Audio probing covers LLM backbone layers 22–28 and MIMO layers 1–5, using means from five random seeds.Reported standard deviations are omitted because all are below 0.005.
  • Base-model acoustic accuracy steadily decreases through the LLM backbone, then sharply increases in MIMO to levels comparable to early LLM layers.
  • SFT preserves this pattern but makes the Kimi-Audio L28 accuracy drop significantly sharper.
  • The authors hypothesize that a distribution mismatch between the text-pretrained LLM backbone and the 1B-audio-transformer MIMO module contributes to the Kimi-Audio pattern.They leave further investigation of this hypothesis for future work.
  • Other large accuracy dips occur at Qwen2-Audio L1, the Kimi-Audio Projector, and Kimi-Audio MIMO 2.SFT has no effect at Qwen2-Audio L1 and the Projector, while Kimi-Audio L28 and MIMO 2 show reduced or barely improved accuracy.
Loading 2608.28966v1…