Source-linked AI summary
Generalization Ability of MOS Prediction Networks
Erica Cooper, Wen-Chin Huang, Tomoki Toda, Junichi Yamagishi
TL;DR
MOS predictors often perform well within one listening test but struggle when listeners, systems, speech characteristics, instructions, or scales change. The paper evaluates MOSNet and self-supervised speech models across listening-test contexts in zero-shot and fine-tuned settings, finding strong wav2vec2 generalization and especially difficult unseen-system cases.
Problem
MOS predictors typically fail to generalize from one listening-test context to another despite achieving high accuracy within the same test.
Method
The paper evaluates MOSNet and self-supervised speech models, including wav2vec2, on in-domain and out-of-domain listening-test data using zero-shot and fine-tuned settings.
Results
Wav2vec2 models generalize well to out-of-domain data, including challenging zero-shot utterance-level prediction, while fine-tuning improves performance.
Takeaways & Limitations
Fine-tuning large self-supervised speech models can adapt MOS prediction to new listening-test contexts using less labeled MOS data than training from scratch.
Takeaways & Limitations
Unseen systems remain the most challenging category for MOS prediction, despite being a likely real-world use case.
Abstract
from arXiv · showhide
Automatic methods to predict listener opinions of synthesized speech remain elusive since listeners, systems being evaluated, characteristics of the speech, and even the instructions given and the rating scale all vary from test to test. While automatic predictors for metrics such as mean opinion score (MOS) can achieve high prediction accuracy on samples from the same test, they typically fail to generalize well to new listening test contexts. In this paper, using a variety of networks for MOS prediction including MOSNet and self-supervised speech models such as wav2vec2, we investigate their performance on data from different listening tests in both zero-shot and fine-tuned settings. We find that wav2vec2 models fine-tuned for MOS prediction have good generalization capability to out-of-domain data even for the most challenging case of utterance-level predictions in the zero-shot setting, and that fine-tuning to in-domain data can improve predictions. We also observe that unseen systems are especially challenging for MOS prediction models.
1. INTRODUCTION
Human listening tests are accurate but costly, while MOS predictors often fail to generalize across listening-test contexts. The paper investigates models and evaluation splits designed to test and improve this generalization.
- Human-subject listening tests are the gold standard but become time-consuming and costly as more systems require evaluation.
- MOS prediction could support faster experimental iteration and larger-scale experiments, but current models typically generalize poorly across listening tests.
- The study creates challenging splits with unseen speakers, systems, listeners, and texts, then evaluates predictors on additional out-of-domain datasets.
2. RELATED WORK
Prior MOS-prediction research includes MOSNet, alternative speech representations, and models that explicitly account for listener dependencies.
- MOSNet predicts naturalness ratings for voice-conversion samples using a CNN-BLSTM architecture over magnitude spectrograms.
- Subsequent work investigated speech embeddings and other input feature representations for MOS prediction.
- MBNet explicitly models listener dependencies by using listener labels during training as input to a listener-bias component.
3. DATASETS
The study combines a main listening-test dataset with three out-of-domain archives and constructs balanced splits containing unseen categories. These datasets differ in language, task, ratings, and score distributions.
- Datasets: The main dataset combines samples from many systems and past listening tests, while three archived listening tests provide out-of-domain evaluation data.
- Dataset splits: The split-construction process matches MOS and rating-variability distributions to the overall data and evaluates candidates using earth-mover’s distance.
- Dataset splits: Development and test splits include unseen speakers, systems, texts, and listeners, with test categories unseen relative to both training and development.
- Out-of-domain datasets: ASV2019 uses machine-versus-human judgments on a 1–10 scale that were linearly adjusted to the standard 1–5 MOS scale.
- Out-of-domain datasets: BC2019 provides Chinese TTS samples rated by native Chinese speakers, creating a cross-language domain mismatch relative to the English main dataset.
- Out-of-domain datasets: COM2018 compares nine acoustic-model and vocoder combinations plus natural speech using Japanese data, while dataset score distributions differ across listening-test contexts.
4. EXPERIMENTS AND RESULTS
The experiments compare MOSNet and self-supervised speech models under in-domain, zero-shot, and fine-tuned conditions, including challenging unseen categories. Wav2vec2 models generalize well across listening tests, while fine-tuning improves performance and unseen systems remain particularly difficult.
- Experimental setup: The study evaluates MOSNet and Fairseq wav2vec2/HuBERT models using MSE and correlation metrics, including LCC, SRCC, and KTAU.The models are evaluated on BVCC and out-of-domain listening-test data.
- MOSNet: Training MOSNet from scratch on BVCC performed worse than using the pretrained model, whereas fine-tuning the pretrained model produced a large performance improvement.Augmenting the fine-tuning data yielded further improvements for MOSNet.
- Fairseq: The relatively small and large English-trained wav2vec2 models achieved the best results consistently among the evaluated self-supervised models.The multilingual model had the third-best development-set performance.
- Experimental setup: Training and evaluation include zero-shot prediction on out-of-domain datasets and fine-tuning on each dataset’s training portion before testing on its held-out portion.The fine-tuning setup represents availability of a small amount of data from the target listening-test context.
- Out-of-domain data experiments: Zero-shot prediction was more difficult than fine-tuning, but wav2vec2 models showed good generalizability and reached moderate utterance-level correlations across datasets.They consistently outperformed MOSNet in correlations, although MOSNet sometimes achieved the lowest MSE.
- Out-of-domain data experiments: Fine-tuning on small amounts of in-domain data reduced error rates and improved correlations at both utterance and system levels.Fine-tuning benefited wav2vec2 models more strongly and removed the multilingual xlsr advantage observed in zero-shot evaluation.
5. CONCLUSIONS AND FUTURE WORK
Fine-tuned self-supervised speech models support MOS prediction across new listening-test contexts with less labeled data than training from scratch. However, unseen systems remain the most challenging prediction case and a key direction for future work.
- Fine-tuning SSL models enables MOS prediction for new listening-test contexts with less human-labeled MOS data than training from scratch.The labeled MOS data is costly to obtain.
- Large SSL models demonstrate good performance for MOS prediction, especially when target listening-test data is available for fine-tuning.
- SSL models trained on multilingual data or mixed datasets show especially good generalization ability, including moderately strong zero-shot utterance-level prediction.
- Unseen systems remain the most challenging category for MOS prediction despite being a likely real-world use case.This difficulty indicates an important direction for future work.