CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation

1. Introduction

• The system transforms audio into spectrograms for convolutional feature extraction followed by recurrent time aggregation, rather than using hand-crafted acoustic features.The authors describe this architecture as a way to reduce dimensionality and simplify training.Source: §1 · p. 1
• The authors argue that IEMOCAP results from five-fold cross-validation are not comparable when the evaluated data are unspecified, and propose 10-fold cross-validation instead.They report that performance depends strongly on which part of the data is used for scoring.Source: §1 · p. 1
• The best tested architecture used four convolutional layers and one Bi-LSTM layer.The tested range was one to six convolutional layers and one to four Bi-LSTM layers.Source: §1 · p. 1
• Vocal tract length perturbation improved performance, while per-layer optimizer adjustment did not yield a definite conclusion.The techniques address class imbalance and data scarcity; batch normalization of recurrent layers was also examined.Source: §1 · p. 1

1.1. Dataset description

• IEMOCAP contains twelve hours of recordings from ten actors; this study uses improvised utterances and four emotions, totaling 2,280 utterances.The authors focus on improvised speech because annotators reached consensus more often there than for scripted utterances: 83.1% versus 66.9%.Source: §1.1 · p. 1

2. Data augmentation

• IEMOCAP’s class imbalance and small size motivate VTLP augmentation and oversampling of happiness and anger.Figure 1 presents the class distribution of utterances in the improvised portion of the dataset.Source: §2 · p. 1, Figure 1 · p. 2
• VTLP rescaling in the approximate range (0.9, 1.1) models variation in vocal-tract length and is used to augment spectrograms without estimating each speaker’s scaling factor.The authors describe augmentation as an easier-to-implement alternative to speaker-specific normalization, with both approaches pursuing invariance to speaker-dependent features.Source: §2 · p. 2
• The frequency warp rescales frequencies below f0 by α, then rescales the remainder to keep the considered frequency range constant.The paper sets f0 above the highest significant formants and takes f0/fmax = 0.9.Source: §2 · p. 2
• The best reported augmentation strategy uses an independently generated α for each training and validation spectrogram, with test predictions averaged across eleven rescaled copies.The eleven test-set copies use α values from 0.9 to 1.1 in increments of 0.02.Source: §2 · p. 2

3. Model description and experiments

• Ten-fold cross-validation averages results across possible dataset splits because measured performance depends strongly on which speakers are held out.Weighted accuracy is computed over the whole test set; unweighted accuracy averages accuracy separately across emotions.Source: §3 · p. 2
• The best reported architecture uses four convolutional layers and one Bi-LSTM layer; dataset-wide spectrogram normalization speeds convergence but strongly overfits networks with at most two convolutional layers.The experiments considered 1–6 convolutional layers and 1–4 Bi-LSTM layers.Source: §3 · pp. 2–3
• VTLP increased unweighted accuracy by 1.1% and weighted accuracy by 0.7%; oversampling raised unweighted accuracy by 2.1% but lowered weighted accuracy by 2.9%.The table reports the best run for each experiment.Source: §3 · p. 3, Table 1 · p. 4
• Batch normalization results depend strongly on batch size: size 16 overfit faster and degraded performance, whereas size 64 nearly reached the best model's performance.GPU memory restrictions prevented testing whether still larger batches would improve results.Source: §3 · p. 3, Table 3 · p. 4

3.1. Difference in the gradient scaling of the convolutional and recurrent layers

• Convolutional-layer gradients were much larger than Bi-LSTM gradients, motivating tests of separate learning rates and momentum settings, but experiments found no definite benefit from layer-wise adjustment.The authors identify more thorough experiments as a possible next step.Source: §3.1 · p. 3

• The authors used 10-fold cross-validation as a more appropriate evaluation for IEMOCAP and applied VTLP augmentation plus minor-class oversampling to address scarcity and imbalance.Source: §4 · p. 4
• The mixed CNN-LSTM architecture extracts high-level representations with convolutional layers, while layer-wise optimizer adjustment yielded no definite conclusion in its favor.The authors note that convolutional and LSTM parameters train at very different paces.Source: §4 · p. 4
• Batch normalization did not improve performance over the baseline, possibly because GPU memory constraints required a small batch size.Source: §4 · p. 4