Source-linked AI summary
A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
Yingzhi Wang, Abdelmoumene Boumadane, Abdelwahab Heba
TL;DR
The paper examines whether wav2vec 2.0 and HuBERT provide strong representations beyond ASR, where evidence for non-ASR tasks remained limited. It compares partial and entire fine-tuning across emotion recognition, speaker verification, and spoken language understanding, achieving strong results on all three tasks. The models therefore demonstrate broad capacity for prosodic, voice-print, and semantic representations within the evaluated settings.
Problem
Evidence remained limited on whether wav2vec 2.0 and HuBERT outperform traditional supervised models for emotion, speaker, and semantic speech information beyond ASR.
Method
The study benchmarks partial and entire fine-tuning of wav2vec 2.0 and HuBERT with simple downstream frameworks across SER, SV, and SLU.
Results
Fine-tuned wav2vec 2.0 and HuBERT achieved state-of-the-art results across Speech Emotion Recognition, Speaker Verification, and Spoken Language Understanding.
Takeaways & Limitations
The results support the generalizability of wav2vec 2.0 and HuBERT for learning prosodic, voice-print, and semantic representations.
Abstract
from arXiv · showhide
Speech self-supervised models such as wav2vec 2.0 and HuBERT are making revolutionary progress in Automatic Speech Recognition (ASR). However, they have not been totally proven to produce better performance on tasks other than ASR. In this work, we explored partial fine-tuning and entire fine-tuning on wav2vec 2.0 and HuBERT pre-trained models for three non-ASR speech tasks: Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding. With simple proposed downstream frameworks, the best scores reached 79.58% weighted accuracy on speaker-dependent setting and 73.01% weighted accuracy on speaker-independent setting for Speech Emotion Recognition on IEMOCAP, 2.36% equal error rate for Speaker Verification on VoxCeleb1, 89.38% accuracy for Intent Classification and 78.92% F1 for Slot Filling on SLURP, showing the strength of fine-tuned wav2vec 2.0 and HuBERT on learning prosodic, voice-print and semantic representations.
1. INTRODUCTION
This work evaluates whether fine-tuned wav2vec 2.0 and HuBERT can learn prosodic, speaker, and semantic representations for three non-ASR speech tasks. It addresses limited evidence beyond frozen-feature benchmarks with partial and entire fine-tuning experiments.
- Research gap: Self-supervised speech models had surpassed existing ASR models, but their effectiveness for emotion, speaker, and semantic information remained unclear.The paper contrasts verbal representation gains with unresolved performance questions for Speech Emotion Recognition, Speaker Verification, and Spoken Language Understanding.
- Prior evidence: Frozen-encoder benchmarking had reported 67.62% ACC for SER and 5.11% EER for SV with HuBERT, while prior fine-tuning studies reported 65.4% WA for SER and 3.61% EER for SV.A prior SV result reached 1.91% EER after adding VoxCeleb2 to training.
- Scope and approach: The study explores partial and entire fine-tuning of wav2vec 2.0 and HuBERT across Speech Emotion Recognition on IEMOCAP, Speaker Verification on VoxCeleb1, and Spoken Language Understanding on SLURP.Partial fine-tuning freezes the CNN feature encoder while training the Transformer contextualized encoder; entire fine-tuning trains both parts.
- Contribution: Fine-tuned models achieved excellent results on all three tasks, supporting their capacity to construct problem-agnostic representations.The authors also open-sourced code and fine-tuned models for SER and SLU on SpeechBrain.
2. METHOD
The method compares wav2vec 2.0 and HuBERT pretraining variants under partial or entire fine-tuning, then attaches simple task-specific downstream adaptors. The framework uses contrastive or masked-prediction pretraining objectives and lightweight classifiers or decoders.
- Pretraining: wav2vec 2.0 masks CNN-derived speech representations and uses contrastive learning to match contextualized outputs with quantized targets.The objective includes 100 distractors, temperature κ = 0.1, L2 regularization, and a diversity loss.
- Model variants: The benchmark compares released base and large wav2vec 2.0 and HuBERT models, including ASR fine-tuned variants trained on LibriSpeech-derived data.HuBERT large is pretrained on 60k hours of Libri-Light data, while the compared wav2vec 2.0 models use 960h LibriSpeech pretraining.
- Pretraining: HuBERT masks CNN-encoded features and predicts cluster assignments using cross-entropy over masked timestamps.Initial labels come from 39-dimensional MFCC clustering; later targets use latent features from preceding HuBERT iterations and cluster ensembles.
- Fine-tuning: Partial fine-tuning freezes the CNN feature encoder and updates only Transformer blocks, whereas entire fine-tuning updates both CNN and Transformer modules.Partial fine-tuning is framed as top-level domain adaptation intended to prevent interference with lower-level features.
- Downstream models: SER and SID use average time pooling with a linear classifier, while SLU uses an attentional GRU-based decoder for intent and slot sequences.Speaker verification derives pairwise cosine-similarity scores from pretrained SID embeddings before the classification layer.
3. EXPERIMENTS
Experiments benchmark partial, entire, and frozen fine-tuning of wav2vec 2.0 and HuBERT on SER, SV, and SLU using IEMOCAP, VoxCeleb1, and SLURP. Fine-tuned models achieved strong task-specific results, with optimal fine-tuning strategy varying by task.
- Datasets: The experiments use IEMOCAP for SER, VoxCeleb1 for SV, and SLURP for SLU.SER is evaluated in speaker-dependent and speaker-independent settings; SV uses speaker verification; SLU includes intent classification and slot filling.
- Fine-tuning settings: The benchmark compares entirely fine-tuned, partially fine-tuned, and frozen wav2vec 2.0/HuBERT models, including models with ASR fine-tuning.Partial fine-tuning freezes the CNN feature encoder while fine-tuning the Transformer contextualized encoder.
- Speech Emotion Recognition: 79.58% WA for SER-SD and 73.01% WA for SER-SI were achieved by the partially fine-tuned HuBERT large model.These results improved on the state-of-the-art baselines by 3.40% and 1.26%, respectively.
- Speaker Verification: 2.36% EER for SV was achieved by the entirely fine-tuned HuBERT model with ASR fine-tuning.This surpassed the baseline by 0.78%, and entire fine-tuning outperformed partial fine-tuning for SV.
- Spoken Language Understanding: 89.38% ACC for intent classification and 78.92% F1 for slot filling were the best SLU results.Entire fine-tuning was best for intent classification, while partial fine-tuning was best for slot filling; frozen models performed substantially worse.
4. CONCLUSIONS
The paper benchmarks fine-tuning methods for wav2vec 2.0 and HuBERT across three non-ASR speech tasks. State-of-the-art results support their generalizability for learning prosodic, voice-print, and semantic representations.
- 4. CONCLUSIONS: The study benchmarks wav2vec 2.0 and HuBERT fine-tuning methods on SER, SV, and SLU.The three tasks probe prosodic, voice-print, and semantic representations.
- 4. CONCLUSIONS: State-of-the-art results were achieved across all three tasks.The authors present the findings as evidence of excellent generalizability for wav2vec 2.0/HuBERT representations.