Source-linked AI summary
SETEAB: Multiscale approach with Squeeze-and-Excitation Temporal Enhanced Aware Block for Speech Emotion Recognition
Duy Vo, Kiet Anh Hoang, Hao Do
TL;DR
Speech emotion recognition must contend with variability in speaking styles, linguistic backgrounds, and environmental noise, while standard temporal-aware blocks have identified optimization and temporal-modeling limitations. SETEAB combines lightweight subsampling and emotion-aware temporal and channel modeling; the authors report strong benchmark accuracy-efficiency trade-offs and cross-corpus results.
Problem
Speech emotion recognition remains challenging under variation in speaking styles, linguistic backgrounds, and environmental noise, and standard temporal-aware blocks have identified optimization and temporal-modeling limitations.
Source: §1 · p. 1Method
SETEAB combines depthwise subsampling, Squeeze-and-Excitation channel refinement, a Temporal Enhanced Aware Block, and weighted bidirectional fusion for multiscale speech emotion recognition.
Source: §1 · p. 1Results
SETEAB achieved the best results in 7 of 12 cross-corpus evaluations and the best mean cross-corpus WA at 37.53%.
Source: §3.2.2 · p. 4Takeaways & Limitations
SETEAB offers a compact model with high recognition performance and stable cross-corpus generalization for robust SER.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper proposes a novel lightweight multiscale architecture for speech emotion recognition (SER) with three key innovations. First, a depthwise convolution-based subsampling module is introduced to reduce model size and computation while preserving salient emotional cues. Second, a Squeeze-and-Excitation block is integrated to enhance channel-wise recalibration and improve representation robustness. Third, a new Temporal Enhanced Aware Block is designed to strengthen temporal dependency modeling and produce more discriminative emotion-aware features. The proposed model is explicitly designed to jointly improve compactness, recognition performance, and generalizability. Experiments on benchmark SER datasets show that our method achieves higher accuracy with reduced computational complexity, while also delivering stronger cross-corpus performance than most recent advanced networks for SER.
1. Introduction
Speech emotion recognition remains difficult under variation in speaking styles, linguistic backgrounds, and noise, while standard temporal-aware blocks have specific modeling and optimization limitations. The paper proposes SETEAB to balance recognition accuracy, computational efficiency, and cross-corpus generalization.
1. Introduction
- CNNs capture local spectral patterns, RNNs model temporal dependencies, and self-supervised models achieve state-of-the-art performance but often incur high computational cost.Early systems used handcrafted acoustic features with conventional classifiers such as support vector machines.Source: §1 · p. 1
- Lightweight SER methods use depthwise separable convolutions and fully convolutional designs such as Light-SERNet, while TIM-Net uses dilated causal temporal blocks and MS-SENet adds multiscale fusion with Squeeze-and-Excitation.These approaches are presented as efforts to reduce model size or improve multiscale temporal and emotional representation learning.Source: §1 · p. 1
- Standard Temporal-Aware Blocks may face optimization instability, gradient attenuation, redundant features, limited temporal-resolution control, and inaccurate equal weighting of past and future cues.The authors identify absent explicit normalization, standard convolutions, and direct bidirectional summation as the respective design concerns.Source: §1 · p. 1
- The paper reports that its framework balances accuracy, computational efficiency, and cross-corpus generalization on benchmark SER datasets.The introduction characterizes the experiments as extensive but provides no numerical results in these passages.Source: §1 · p. 1
- SETEAB combines a novel TEAB, adaptive forward-backward fusion, an SE-Res2Block, and depthwise subsampling to model temporal dynamics and reduce computation while preserving affective cues.The stated aims also include more discriminative emotion-aware representations, channel-wise recalibration, robustness, and generalization.Source: §1 · p. 1
2. Method
SETEAB extends TIM-Net with a four-part multiscale pipeline: depthwise subsampling, SE-Res2 refinement, TEAB temporal modeling, and weighted bidirectional fusion.
2.1. Overall Framework
- The framework combines early temporal compression, local channel-aware refinement, bidirectional TEAB modeling, and adaptive fusion for emotion prediction.Its front end removes redundant short-term variation, while fusion aggregates directional and multi-level information.Source: §2 · pp. 1–2
2.2. Depthwise Convolution Subsampling
- Depthwise convolution subsampling compresses redundant short-term variation while preserving salient affective information for later temporal modeling.The front end uses a standard 2D convolution first, followed by depthwise separable convolutions.Source: §2.2 · p. 2
2.3. SE-Res2Block
- The SE-Res2Block refines local multi-scale patterns and recalibrates channels, preserving the original representation while emphasizing emotion-relevant channels.Its recalibration follows global average pooling of the block features.Source: §2.3 · p. 2
2.4. Temporal Enhanced Aware Block
- TEAB uses pre-normalization, channel expansion, depthwise temporal filtering, and gated residual learning to improve stability and feature reuse over TAB.It preserves a shortcut path and adaptively emphasizes temporally salient emotion cues while modeling both directions.Source: §2.4 · pp. 2–3
2.5. Weighted Bidirectional Fusion
- Weighted bidirectional fusion replaces direct summation with shared global weights for forward and reverse features, then aggregates fused features across levels.A separate learnable weight is assigned to each level.Source: §2.5 · p. 3
3. Experiments
Experiments use standardized intra- and cross-corpus evaluation, with SETEAB reporting strong benchmark performance, low computational cost, and improved cross-corpus results.
3.1.1. Datasets and evaluation metrics
- The EmoBox protocol evaluates intra-corpus performance on five datasets and cross-corpus WA on four datasets.Intra-corpus metrics are UA and macro-F1; cross-corpus evaluation uses WA.Source: §3.1.1 · p. 3
3.1.2. Implementation details
- Audio is resampled to 16 kHz and converted to 80-dimensional log-Mel features; training also uses stochastic augmentation and SpecAugment.Features use 25 ms windows and 10 ms shifts; training uses Adam, label smoothing, early stopping, and eight TEAB blocks per direction.Source: §3.1.2 · p. 3
3.2.1. Intra-corpus SER Results
- 47.69% average UA for SETEAB (R = 2) and 45.23% average F1 for SETEAB (R = 4) are the respective best means across five benchmarks.Each variant ranks second on the complementary metric.Source: §3.2.1 · p. 3, Table 1 · p. 4
- 0.4–0.5M parameters and 0.06–0.12 GFLOPs give SETEAB variants a substantially smaller footprint than wav2vec 2.0 base.The comparison baseline has 95M parameters and 33.53 GFLOPs; the figure plots FLOPs against UA.Source: §3.2.1 · p. 3, Figure 2 · p. 4
- Ablations report improved average UA and F1 after adding SE-Res2 and depthwise subsampling, with the full model performing best.The depthwise subsampling comparison includes a variant operating in both forward and backward directions.Source: §3.2.1 · p. 3, Table 2 · p. 4
3.2.2. Cross-corpus SER Results
- SETEAB wins 7 of 12 cross-corpus evaluations, with its strongest consistent advantage on RAVDESS test settings.The reported comparisons cover IEMOCAP, MELD, RAVDESS, and SAVEE.Source: §3.2.2 · p. 4, Table 3 · p. 4
- 37.53% mean cross-corpus WA is the best among SETEAB and the compared baselines.Table 4 reports overall cross-dataset WA as mean ± standard deviation.Source: §3.2.2 · p. 4, Table 4 · p. 4
4. Conclusion
SETEAB is a lightweight SER architecture combining compression, channel refinement, temporal modeling, and weighted bidirectional fusion. The paper reports a strong accuracy-efficiency balance and stable cross-corpus generalization.
4. Conclusion
- SETEAB combines depthwise subsampling, Squeeze-and-Excitation, TEAB, and weighted bidirectional fusion to target compactness, recognition performance, and generalization.The conclusion describes the approach as effective for robust SER.Source: §4 · p. 4