Source-linked AI summary
From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
Hezhao Zhang, Thomas Hain
TL;DR
SER models that take audio directly do not indicate which information supports their predictions, while recognition remains low on many corpora. This paper uses an LLM concept bottleneck and concept removal to examine prediction dependence across three datasets. Results show that transcript influence differs by setting and corpus, and concept removal can change individual labels even when Macro-F1 changes little.
Problem
Direct-audio emotion recognition does not indicate which information informs predictions, making model decisions difficult to examine.
Source: §1 · p. 1Method
The study represents utterances with transcript, acoustic and speaker concepts, then uses an LLM to predict emotion and tests concept-group contributions and single-acoustic-concept removals.
Source: §1 · pp. 1–2, §2 · p. 2Results
Zero-shot CREMA-D Macro-F1 falls from 27.88 with acoustic concepts alone to 5.87 when the transcript is added, while fine-tuned CREMA-D rises from 41.87 to 45.10 with that addition.
Source: §4.1 · p. 3Takeaways & Limitations
Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D, while MELD predictions also change despite little Macro-F1 change.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.
1. INTRODUCTION
SER performance remains low on many corpora, and direct-audio models do not show what information informs their predictions. This work adapts concept bottlenecks to SER so fixed predictors can be examined through human-readable concepts and single-concept edits.
1. INTRODUCTION
- Direct-audio LALMs offer no indication of which information informs a prediction, leaving their decisions difficult to examine.Earlier SER systems combined acoustic representations with transcript features, while LALMs predict emotion directly from audio with text instructions.Source: §1 · p. 1
- Prior approaches either generate prosody descriptions or supply extracted acoustic descriptions, but do not establish which descriptions a fixed predictor depends on.The cited work also notes that generated reasoning may not faithfully reflect how a model reached its prediction.Source: §1 · p. 1
- Separate extractors represent utterances with transcript, pitch, intensity, speech rate, age and gender concepts, which an LLM uses to predict emotion.Removing one concept while holding the predictor fixed lets the study examine which class predictions change.Source: §1 · p. 1
- The study asks how concept groups contribute before and after fine-tuning, and how acoustic-concept dependence varies across three corpora.The analyses use CREMA-D, IEMOCAP and MELD with three LLMs.Source: §1 · pp. 1–2
2. SPEECH CONCEPT BOTTLENECK
A speech concept bottleneck represents an utterance through human-readable concepts and predicts emotion from those concepts alone. Editing a concept while keeping the predictor fixed allows the resulting prediction change to be examined.
2. SPEECH CONCEPT BOTTLENECK
- A concept bottleneck maps an input to human-readable concepts, then predicts its label from those concepts alone.In this SER adaptation, an utterance is represented by text-expressed speech properties and an LLM produces one emotion label from a fixed set.Source: §2 · p. 2
- A concept intervention changes one concept while holding the predictor fixed, then observes the change in the predicted label.The concepts are human-specified, interpretable properties whose predicted values form the LLM's input.Source: §2 · p. 2
- The concept groups cover transcript content, pitch, intensity, speech rate, and speaker age and gender.The transcript represents spoken words; pitch, intensity and speech rate are acoustic concepts, while age and gender are speaker concepts.Source: §2 · p. 2
3. IMPLEMENTATION AND EXPERIMENTS
The experiments extract transcript, acoustic and speaker concepts separately, compare concept-group combinations across zero-shot and fine-tuned predictors, and evaluate Macro-F1. The intervention analysis holds a fully fine-tuned predictor fixed while removing acoustic concepts.
3.1. Concept Extraction and Prompting
- Concept extraction uses Qwen3-ASR for transcripts, Praat for pitch and intensity, and Vox-Profile for speaker attributes; speech rate is transcript words divided by utterance duration.Acoustic measures are discretised into five levels using training-set quantiles; predicted gender matches metadata for 96.3% of CREMA-D and 95.2% of IEMOCAP utterances, while CREMA-D age bands are correct for 35.9%.Source: §3.1 · p. 2
3.2. Experimental Design
- The study measures each concept group's contribution by comparing four group combinations in zero-shot and separately fine-tuned settings.For acoustic-concept interventions, it holds the all-concepts fine-tuned predictor fixed and compares the same utterance with and without a selected concept.Source: §3.2 · p. 2
3.3. Experimental Setup
- Experiments use CREMA-D, IEMOCAP and MELD, three instruction-tuned LLMs, and Macro-F1; fine-tuning uses LoRA and three seeds per model and combination.CREMA-D has six classes and speaker-disjoint splits, IEMOCAP four classes and session-wise five-fold cross-validation, and MELD seven classes and official splits; unparseable outputs count as errors.Source: §3.3 · pp. 2–3, Table 2 · p. 3
4. RESULTS AND DISCUSSION
Concept-based SER performance varies with corpus and training: fine-tuning reverses transcript harm on CREMA-D, while concept removals expose prediction changes that Macro-F1 can miss.
4.1. Predictive Performance
- 27.88 Macro-F1 with acoustic concepts alone falls to 5.87 when the transcript is added in zero-shot Qwen2.5 on CREMA-D.The repeated script makes transcript-only models predict Neutral for every utterance, yielding 4.26 Macro-F1 for all three models.Source: §4.1 · p. 3
- 41.87 Macro-F1 with acoustic concepts alone rises to 45.10 when the transcript is added after fine-tuning on CREMA-D.The transcript alone gives 11.15; the reversal holds for all three models.Source: §4.1 · p. 3
- 74.37 versus 69.90 Macro-F1 on IEMOCAP favors adding acoustic concepts after fine-tuning, while Qwen2.5's MELD comparison is 37.03 versus 37.20.On conversational corpora, transcripts already carry emotion-related information; acoustic gains on MELD are small and inconsistent across models.Source: §4.1 · p. 3
- 45.57 Macro-F1 from concepts versus 78.67 from direct audio on CREMA-D; the gap shrinks to 7.69 on IEMOCAP and 2.59 on MELD.These are Qwen2.5-Omni test results; adding speaker concepts changes fine-tuned scores by less than two points per corpus, without consistent direction.Source: §4.1 · p. 3, Table 3 · p. 3
4.2. Concept Interventions
- 2.49–6.30 Macro-F1 points are lost by removing any acoustic concept for Qwen2.5 on CREMA-D, while no removal changes MELD by more than 0.3.On IEMOCAP, three concepts lose less than 0.6; intensity level and variation lose 2.12 and 1.93, respectively.Source: §4.2 · p. 4, Figure 1 · p. 4
- 48% of CREMA-D Neutral predictions become Disgust without speech rate, while MELD also shows class shifts despite little Macro-F1 change.Changes cluster at the highest or lowest levels of the removed concept, with few at middle levels; on MELD, 7% of Happy predictions become Neutral, and some Neutral predictions become Happy.Source: §4.2 · p. 4, Figure 2 · p. 4
5. CONCLUSION
The conclusion presents concept bottlenecks as a way to inspect how emotion predictions depend on interpretable inputs, with results varying by corpus and training setting.
5. CONCLUSION
- Concept bottlenecks enable examination of emotion-prediction dependence through controlled changes to predictor inputs, while aggregate scores can conceal individual prediction changes.The conclusion reports transcript bias in zero-shot scripted-corpus settings, corpus-dependent contributions after fine-tuning, and prediction changes after acoustic-concept removal.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study uses existing datasets and did not recruit participants or collect new data.
6. COMPLIANCE WITH ETHICAL STANDARDS
- The analysis used CREMA-D, IEMOCAP, and MELD data without new recruitment or data collection; ethical approval was not required for this secondary analysis.Source: §6 · p. 5