Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

1. Introduction

• Small annotated datasets make deep-learning emotion recognition prone to overfitting, while speech and text both carry relevant emotional information.The introduction describes speech as accessible and notes that its linguistic component can be transcribed in real-world applications.Source: §1 · p. 1
• Prior multimodal deep-representation fusion approaches include attention, graph neural networks, and transformers, and often produce architecturally complex models.The introduction also contrasts pretrained transfer learning and unsupervised embedding methods as approaches to representation learning.Source: §1 · p. 1
• SSL models can be fine-tuned for downstream tasks with limited labeled data, and BERT-like models offer sequence-level CLS representations.The introduction also notes the availability of pretrained models and their potential cost and time savings relative to training from scratch.Source: §1 · p. 1
• Prior work had not combined separate modality-specific SSL models for speech emotion recognition, while their high dimensionality can make fusion architecturally complex.The authors suggest similar architectural properties may permit simpler fusion mechanisms.Source: §1 · p. 1
• The paper jointly fine-tunes modality-specific BERT-like SSL models for speech and text, then compares simple fusion with Co-Attentional fusion.It also conducts ablations to examine factors affecting performance.Source: §1 · p. 1

2. Pre-Trained SSL models

• The framework uses pretrained SSL models available through Fairseq to process speech and text signals.Source: §2 · p. 2

2.1. VQ-Wav2Vec

• VQ-Wav2Vec applies vector quantization to Wav2Vec representations, and the paper uses a Librispeech-960-pretrained model to convert speech into a sequence of tokens.Source: §2.1 · p. 2

2.2. Speech-BERT

• Speech-BERT is a BERT-like Transformer trained on discretized speech tokens with masked-token prediction, and it is fine-tuned for multimodal emotion recognition.The model was trained on discretized Librispeech-960 and has 12 layers with a 768-dimensional embedding.Source: §2.2 · p. 2

2.3. RoBERTa model

• RoBERTa extends BERT without the next-sentence-prediction task and is fine-tuned for multimodal emotion recognition.Its architecture has 24 layers and a 1024-dimensional embedding.Source: §2.3 · p. 2

3. Methodology

• The methodology evaluates fusion mechanisms for outputs from Speech-BERT and RoBERTa, including both fine-tuned and frozen settings.The study compares the final model with published results on IEMOCAP, CMU-MOSEI, and CMU-MOSI and runs IEMOCAP ablations.Source: §3 · p. 2, §3.2 · p. 2

3.1. Model Pipeline

• Speech and text are processed in parallel by separate pipelines, fused before a single fully connected prediction layer, and the full framework is fine-tuned.Speech is discretized with VQ-Wav2Vec and text is tokenized with the GPT-2 tokenizer.Source: §3.1 · p. 2, Figure 1 · p. 2

3.2.1. Shallow Fusion

• Shallow-Fusion concatenates Speech-BERT and RoBERTa CLS tokens and sends the joint embedding to a classification head.The authors use it as the standard fusion mechanism in ablations because it performed better in their experiments.Source: §3.2.1 · p. 2

3.2.2. Co-Attentional Fusion

• Co-Attentional fusion lets each modality’s CLS-derived query attend to the other modality’s full embedding sequence, then concatenates the modified CLS tokens for prediction.The method exchanges the key-value vectors and query vector across modalities.Source: §3.2.2 · pp. 2–3, Figure 2 · p. 3

4. Experimental Setup

• Models were trained with distributed settings on two Tesla V100 32GB GPUs, using an effective batch size of 16 and Adam optimization.The initial learning rate was 1e-5 and dropout was 0.1.Source: §4 · p. 3

4.1.1. IEMOCAP Experiments

• IEMOCAP experiments use four emotion categories and compare models with prior work using binary accuracy, F1-score, and four-class unweighted accuracy.Source: §4.1.1 · p. 3

4.1.2. CMU-MOSEI and CMU-MOSI Experiments

• CMU-MOSEI has 22,000 examples and CMU-MOSI has 2,000; both are evaluated with 7-class accuracy, MAE, binary accuracy, and F1-score.Source: §4.1.2 · p. 3

5.1. Comparison between two fusion mechanisms in Fine-Tune state

• Shallow-Fusion slightly improves binary accuracy and per-emotion F1 over Co-Attentional fusion when Speech-BERT and RoBERTa are fine-tuned.The comparison uses a fusion mechanism followed by a classification head.Source: §5.1 · p. 3

5.2. Comparison between Unimodal inputs

• Text-only input performs better than speech-only, but the best multimodal model improves on text-only results.The authors suggest linguistic structure may provide high emotional clues.Source: §5.2 · p. 3

5.3. Comparison between fine-tuning and frozen SSL architectures

• Fine-tuning both SSL models with Shallow-Fusion leads to better performance than keeping them frozen.The ablation uses IEMOCAP, with the first three sessions for training, the fourth for validation, and the fifth for testing.Source: §5.3 · p. 3, §5 · p. 3

5.4. Comparison between the two fusion mechanisms when keeping SSL models frozen

• With frozen Speech-BERT and RoBERTa, Co-Attentional fusion performs better, while adding nearly 6 million trainable parameters versus Shallow-Fusion’s close to 14 thousand.The authors attribute its stronger adaptation to more interactions before the prediction layers.Source: §5.4 · pp. 3–4, Table 1 · p. 4

• Shallow-Fusion improves on more complex fusion mechanisms while keeping the multimodal framework straightforward.Source: §6 · p. 4
• The authors jointly fine-tune pretrained BERT-like SSL architectures for speech and text in multimodal emotion recognition.Source: §6 · p. 4
• Structurally similar BERT-like speech and text models allow straightforward fusion and adaptation of standard NLP practices.Source: §6 · p. 4
• Future work will visualize and further explore SSL behavior; BERT-like speech models could help bring NLP advances to speech.Source: §6 · p. 4