The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech

1. INTRODUCTION

• Listeners tend to use the full rating scale regardless of sample quality, so MOS systems may underestimate high-quality speech or overestimate low-quality speech.This range-equalizing bias is especially relevant to zoomed-in tests that exclude worse-performing systems.Source: §1 · p. 1
• T05 placed first in 7 of 16 VMC 2024 Track 1 metrics and second in the remaining 9, significantly ahead of systems ranked third and below.The system targets zoomed-in MOS tests of high-quality synthetic speech.Source: §1 · p. 1
• T05 separately trains SSL-based and spectrogram-based MOS predictors, then fine-tunes them using fused features.The spectrogram features come from EfficientNetV2, a pretrained image feature extractor applied to speech spectrograms.Source: §1 · p. 1
• Ablations indicate that feature fusion improves correlation-based metrics, while training with neural-TTS-only MOS data or actual zoomed-in MOS data enhances prediction performance.The paper reports these as findings about essential system factors.Source: §1 · p. 1

2. THE VMC 2024 TRACK 1

• Track 1 assessed MOS prediction for comparisons of high-MOS speech synthesis systems using zoomed-in tests on BVCC speech.The three tests used 50%, 25%, and 12% zoom-in rates, representing the share of systems covered relative to the original BVCC dataset.Source: §2 · p. 1
• The organizers provided no training data for zoomed-in conditions, so participants had to build systems using publicly available MOS datasets.The validation set used 50% zoom-in; evaluation included 25% and 12% zoom-in tests.Source: §2 · p. 1
• Evaluation covered MSE, LCC, SRCC, and KTAU at both utterance and system levels.Source: §2 · p. 1

3.1. Basic Architecture

• UTMOSv2 combines SSL and spectrogram features with a dataset-domain embedding, then concatenates them for MOS prediction.The system uses a pretrained SSL speech feature extractor and a pretrained image feature extractor for spectrograms.Source: §3.1 · p. 1, Figure 1 · p. 2

3.1.1. Spectrogram Feature Extractor

• Multiple mel-spectrograms use different STFT window sizes on randomly extracted speech frames, and ImageNet-pretrained CNNs process them as fixed-size images.The spectrogram features are aggregated across window settings and pooled across time and frequency.Source: §3.1.1 · p. 2

3.1.2. SSL Feature Extractor

• The SSL extractor combines Transformer-layer hidden states with a weighted sum, then applies attention and max pooling along the sequence.The final output is denoted hSSL.Source: §3.1.2 · p. 2

3.1.3. Data-domain Encoding

• A lookup embedding maps each training dataset ID to a data-domain embedding, but unseen datasets lack IDs and may not work properly out of domain.The encoding is intended to address biases across MOS tests, possibly including range-equalizing bias.Source: §3.1.3 · p. 2

3.1.4. Fusion of Spectrogram Features and SSL Features

• A fully connected layer predicts MOS from the concatenated spectrogram feature, SSL feature, and data-domain embedding.Source: §3.1.4 · p. 2, Figure 1 · p. 2

3.2. Additional Data Collection

• No official organizer training sets were provided, so training data came from publicly available MOS test results.Source: §3.2 · p. 3
• The collected sources were BVCC, Blizzard Challenge 2008–2011, SOMOS, and publicly available zoomed-in BVCC data.Source: §3.2 · p. 3
• For Blizzard Challenge data, the authors kept English utterances, excluded BC2008 EUS listeners for using a different rating scale, and omitted BC2010 ES2 because it did not assess naturalness.Source: §3.2 · p. 3

• The training objective combines contrastive loss with mean squared error loss.Source: §3.3 · p. 3
• The margin α > 0 makes the model ignore errors smaller than the margin.Source: §3.3 · p. 3
• Hyperparameters λcon and λmse control the weights of the contrastive and MSE losses, respectively.Source: §3.3 · p. 3

3.4. Multi-Stage Learning

• Multi-stage learning is introduced to mitigate catastrophic forgetting that can worsen performance on learned domains.Source: §3.4 · p. 3
• The system trains the spectrogram and SSL extractors separately, trains the fusion layer with extractors frozen, then fine-tunes all parameters at a small learning rate.The staged strategy addresses the difficulty of training both feature extractors from scratch; Stage 2 also trains a new data-domain embedding layer.Source: §3.4 · p. 3
• SSL pretraining freezes the backbone before MOS-layer training and then fine-tunes all parameters, whereas spectrogram pretraining trains the CNNs and MOS layer together.The authors report that two-stage spectrogram pretraining did not significantly improve performance in a preliminary experiment.Source: §3.4 · p. 3
• Comparative experiments use data-domain embeddings on all datasets, while the submitted system omits that encoding and fine-tunes on sarulab-data after BVCC training.Source: §3.4 · p. 3

4. EXPERIMENTS

• The experiments ablate spectrogram/SSL fusion, multi-stage learning, and training datasets.Source: §4 · p. 3

4.1. Common Experimental Conditions

• The study uses EfficientNetV2, wav2vec2.0 base3 pretrained on LibriSpeech, and a one-dimensional data-domain embedding; it applies five-fold cross-validation and averages fold and five-frame-sampling predictions.Source: §4.1 · p. 3

4.4.1. Experimental Conditions

• Table 2 compares our systems against the B01 baseline and human-annotated BVCC MOS; all Table 1 datasets were used for training, with BVCC as the MOS-prediction data-domain ID in three systems.Source: Table 2 · p. 4, §4.4.1 · p. 4

4.2. Evaluation Metrics

• Evaluation uses system- and utterance-level MSE, LCC, SRCC, and KTAU on test sets with 25% and 12% zoom-in rates.Source: §4.2 · p. 4

4.3. VMC2024 Results of Our T05 System [7]

• T05 ranked first in 7 of 16 official metrics and second in the remaining 9, with a large margin over systems ranked third or lower.Source: §4.3 · p. 4

4.4. Ablation Study on Fusing Spectrogram/SSL Features

• The ablation compares prediction scores from the proposed fused model with spectrogram-only and SSL-only systems, alongside B01, trained on original BVCC samples and labels, and UTMOS.B01 was the VMC 2024 track 1 baseline; UTMOS is an open-source MOS prediction system.Source: §4.4 · p. 4, §4.4.1 · p. 4

4.4.2. Results and Discussion

• All three T05 variants outperform both baselines on correlation metrics, and fusion significantly improves those scores over either single-feature system; fusion also exceeds BVCC MOS on all metrics except system-level MSE.The single-feature variants show a trade-off: spectrogram-only has the best MSE in all cases but often the worst correlation metrics, while SSL-only has the highest MSE but often better correlations.Source: §4.4.2 · p. 4

4.5. Comparison of Multi-Stage Learning; 4.5.1. Experimental Conditions

• The comparison evaluates the full system against variants omitting Stage 2 or both Stages 1 and 2.Source: §4.5 · p. 4, §4.5.1 · p. 5

4.5.2. Results and Discussion

• Reducing the number of training stages moves models toward spectrogram-only behavior, with lower MSE but lower correlation metrics; the authors conclude multi-stage learning is essential for SSL features to capture sample differences.Source: §4.5.2 · p. 5, Table 3 · p. 5
• The authors suggest unequal learning speeds may cause the fusion layer to emphasize spectrogram features when both extractors are trained together before individual optimization.Source: §4.5.2 · p. 5

4.6.1. Experimental Conditions

• The ablation compares training on all datasets with training that excludes BVCC, BC, SOMOS, and sarulab-data, and varies the inference data-domain ID.For BC datasets, only mean values are reported despite domain IDs being available for each BC dataset.Source: §4.6.1 · p. 5

4.6.2. Results and Discussion

• Excluding SOMOS or sarulab-data tends to significantly degrade performance, while All datasets achieves the best scores overall; excluding BVCC or BC sometimes improves scores.The authors suggest excluding datasets likely to contain low-quality systems and using MOS datasets with as many state-of-the-art DNN-based synthesis results as possible.Source: §4.6.2 · p. 5
• MSE is lowest for sarulab-data and highest for BVCC, while this domain pattern does not appear in correlation-based metrics.The authors interpret this as range-equalizing bias affecting absolute MOS prediction more strongly.Source: §4.6.2 · p. 5
• Correlation-based metric scores are better for 25% than 12% zoomed-in datasets in almost all cases.The authors suggest training speech quality was closer to that of the 25% zoomed-in datasets.Source: §4.6.2 · p. 5

5. CONCLUSION

• T05 placed first in 7 of 16 VMC 2024 Track 1 metrics.The ablation found fusion of spectrogram and SSL features improved correlation-based metrics, while spectrogram features alone gave the best MSE.Source: §5 · p. 5
• The submitted system fused spectrogram features from a pretrained image extractor with speech features from pretrained SSL models, alongside multi-stage learning and multiple datasets.The authors report that wider dataset coverage and multi-stage learning enhanced MOS prediction performance.Source: §5 · p. 5
• Future work includes predicting speech aspects beyond naturalness, such as prosody.Source: §5 · p. 5