Source-linked AI summary
UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, Hiroshi Saruwatari
TL;DR
MOS prediction is expensive to obtain subjectively, while general-purpose prediction across in-domain and OOD listening-test settings remains challenging. UTMOS combines fine-tuned SSL strong learners, feature-based weak learners, and several improvements, achieving the highest scores on several metrics in both challenge tracks. Ablations further examine the effectiveness of these methods and stacking.
Problem
Subjective MOS evaluation is costly, and developing a general-purpose automatic MOS prediction model remains challenging across different challenge settings.
Method
UTMOS ensembles SSL-fine-tuned strong learners with weak machine-learning learners using SSL features, adding contrastive learning, listener dependency, and phoneme encoding.
Results
UTMOS achieved the highest score on several metrics in both the main and OOD tracks of VoiceMOS Challenge 2022.
Takeaways & Limitations
Ablations report effectiveness for listener-dependent learning and stacking, while increasing the number of strong and weak learners tended to improve prediction accuracy.
Abstract
from arXiv · showhide
We present the UTokyo-SaruLab mean opinion score (MOS) prediction system submitted to VoiceMOS Challenge 2022. The challenge is to predict the MOS values of speech samples collected from previous Blizzard Challenges and Voice Conversion Challenges for two tracks: a main track for in-domain prediction and an out-of-domain (OOD) track for which there is less labeled data from different listening tests. Our system is based on ensemble learning of strong and weak learners. Strong learners incorporate several improvements to the previous fine-tuning models of self-supervised learning (SSL) models, while weak learners use basic machine-learning methods to predict scores from SSL features. In the Challenge, our system had the highest score on several metrics for both the main and OOD tracks. In addition, we conducted ablation studies to investigate the effectiveness of our proposed methods.
1. Introduction
UTMOS addresses the high cost of subjective speech evaluation and the challenge of building general-purpose MOS predictors. It combines strong and weak learners and introduces methods including contrastive learning and phoneme encoding for VoiceMOS Challenge 2022.
- Subjective speech evaluation is costly in time and money, motivating automatic performance measures, but general-purpose prediction remains challenging.
- UTMOS combines strong learners fine-tuned from SSL models with weak learners that predict MOS from SSL features using non-neural machine-learning methods.
- The proposed strong learner incorporates contrastive learning, listener dependency, and phoneme encoding.
- The system achieved the highest score on several metrics in both the main and OOD VoiceMOS Challenge 2022 tracks.
- Ablation studies examine listener-dependent learning and stacking with increasing numbers of strong learners.
2. VoiceMOS Challenge 2022
VoiceMOS Challenge 2022 evaluates automatic MOS prediction on synthetic speech across main and OOD tracks. The tracks differ in language, listening-test domain, and the amount of labeled and unlabeled data available.
- Main track: The main track uses English BVCC samples from 187 systems and tests unseen systems, speakers, and listeners from the same listening-test setting.
- OOD track: The OOD track uses Chinese synthetic speech collected with a different listening test and provides little labeled data alongside much unlabeled audio.
- Performance is measured with MSE, LCC, SRCC, and KTAU at both utterance and system levels.
3. UTMOS
UTMOS uses ensemble learning with strong SSL-based neural learners and weak traditional machine-learning learners. The system therefore combines waveform-based modeling with simpler predictors built from utterance-level SSL features.
- UTMOS uses multiple strong and weak learners in an ensemble.
- Datasets: Table 1 organizes challenge datasets by whether synthesizer systems are included in training and whether MOS labels are available.
- Strong learners: Strong learners are SSL-based neural networks that directly accept speech waveforms as input.
- Weak learners: Weak learners include ridge regression and support vector machines using utterance-level SSL features.
3.1. Fine-tuned SSL model
The fine-tuned SSL model predicts frame-level scores while incorporating contrastive, regression, listener/domain-dependent, phoneme, and augmentation components. These components target ranking accuracy, listener and domain variation, intelligibility, and limited-data overfitting.
- Architecture: The strong learner sends SSL frame-level features through BLSTM and linear layers to compute frame-level scores instead of averaging them.
- Contrastive loss: Contrastive loss compares score differences between utterance pairs and penalizes disagreement between predicted and ground-truth quality ordering.
- Contrastive loss: The margin α ignores small contrastive errors, while opposite signs between true and predicted differences receive a penalty suited to ranking metrics such as SRCC.
- Loss functions: The final objective combines clipped regression loss and contrastive loss as L = βL_reg + γL_con.
- Listener and domain dependence: Listener embeddings model listener-dependent scores, and a mean-listener embedding produces utterance-level MOS when listener identity is unavailable at inference.
- Listener and domain dependence: Listener and domain IDs account for variation across listeners and listening-test datasets during multi-domain training.
- Phoneme encoding: Phoneme encoding combines phoneme and ASR-derived reference sequences with SSL outputs to provide auxiliary information related to intelligibility.
- Data augmentation: Speaking-rate changing and pitch shifting augment utterances while maintaining MOS, helping address overfitting when challenge data are limited.
3.2. External data collection
Because the OOD track had only 136 labeled utterances, the authors collected MOS for 540 unlabeled utterances as external data.
- 136 labeled utterances were insufficient to train a robust OOD MOS prediction model.
- The authors selected BC2019-A, whose utterances were treated as natural speech after review by a Chinese native speaker.
- They conducted a standard 5-point-scale MOS test for the external utterances.
3.3. Ensemble learning with strong and weak learners
UTMOS combines fine-tuned SSL-based strong learners with feature-based weak learners through stacking to improve prediction robustness.
- UTMOS uses stacking to ensemble fine-tuned SSL models and simple regression models based on utterance-level SSL features.The paper calls the former strong learners and the latter weak learners.
- Weak learners extract pretrained-SSL mean embeddings by averaging frame representations before regression.The authors assume mean embeddings retain efficient information for MOS prediction despite potentially rough utterance-level characterization.
- The stacking pipeline trains strong and weak learners individually, predicts scores by cross-validation, and trains successive meta-learners from earlier-stage scores.
4. Experimental evaluations
UTMOS was evaluated across main and OOD tracks, where it achieved strong challenge results and ablations examined the contributions of its components and stacking ensemble.
- Experimental conditions: The experiments used multiple strong-learner configurations, selected stacking candidates through development-set system-level SRCC, and incorporated up to 17 main-track or 6 OOD strong learners.
- VoiceMOS2022 results: Utt. MSE = 0.165 (1), Utt. SRCC = 0.897 (1), Sys. MSE = 0.090 (1), and Sys. SRCC = 0.936 (3) were reported for the Main track.
- VoiceMOS2022 results: Utt. MSE = 0.162 (1), Utt. SRCC = 0.893 (2), Sys. MSE = 0.030 (1), and Sys. SRCC = 0.988 (1) were reported for the OOD track.
- Ablation study on SSL-based models: All proposed methods outperformed SSL-MOS in almost all indices, while UTMOS strong performed best on several OOD indices.
- Ablation study on SSL-based models: Removing listener ID significantly degraded performance in many main- and OOD-track cases.
- Evaluation on stacking: Increasing the numbers of strong and weak learners tended to improve prediction accuracy, reducing MSE while retaining high SRCCs.
5. Conclusion
The paper presents UTMOS, an ensemble of strong SSL-fine-tuned learners and weak SSL-feature machine-learning learners, and identifies broader data collection as future work.
- UTMOS combines strong learners obtained by fine-tuning SSL models with weak learners that predict MOS from SSL features.
- Future work includes constructing a larger-scale general-purpose MOS prediction model using a wider variety of collected data.