Source-linked AI summary
The VoiceMOS Challenge 2022
Wen-Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, Junichi Yamagishi
TL;DR
The VoiceMOS Challenge addresses the need for automatic MOS prediction when human listening tests are costly and their scores are context-dependent. It establishes common main-track and out-of-domain evaluations with multiple metrics and shared data, showing strong effectiveness of fine-tuned self-supervised models but persistent difficulty across unseen contexts.
Problem
Automatic MOS prediction is needed because listening tests are costly, cross-test MOS values are not meaningfully comparable, and models generally cannot combine data from different listening tests.
Method
The challenge evaluates MOS prediction on a main dataset and a separate out-of-domain listening test using shared data, multiple metrics, and common baselines.
Results
Fine-tuning self-supervised speech models was highly effective, while unseen speakers and listeners in the main track and unseen systems in the OOD track remained difficult.
Takeaways & Limitations
Generalization across listening-test contexts and precise score prediction with little labeled target-domain data remain challenging, making matched training and test distributions important.
Takeaways & Limitations
MOS values cannot be meaningfully compared across different listening tests because lexical content and participant instructions differ.
Abstract
from arXiv · showhide
We present the first edition of the VoiceMOS Challenge, a scientific event that aims to promote the study of automatic prediction of the mean opinion score (MOS) of synthetic speech. This challenge drew 22 participating teams from academia and industry who tried a variety of approaches to tackle the problem of predicting human ratings of synthesized speech. The listening test data for the main track of the challenge consisted of samples from 187 different text-to-speech and voice conversion systems spanning over a decade of research, and the out-of-domain track consisted of data from more recent systems rated in a separate listening test. Results of the challenge show the effectiveness of fine-tuning self-supervised speech models for the MOS prediction task, as well as the difficulty of predicting MOS ratings for unseen speakers and listeners, and for unseen systems in the out-of-domain setting.
1. Introduction
Human listening tests, especially MOS, are the standard for evaluating synthesized speech but are costly and context-dependent. The VoiceMOS Challenge addresses these limitations by organizing a shared task for automatic MOS prediction across common datasets and an out-of-domain setting.
- MOS listening tests collect human ratings of synthesized speech quality, commonly using a Likert scale.
- Listening tests are time-consuming and costly, despite becoming easier to conduct through crowdsourcing.
- MOS scores cannot be meaningfully compared across listening tests because sample content and participant instructions differ.
- Automatic non-intrusive MOS prediction models are needed because reference-based metrics are unsuitable and data from different listening tests cannot be combined.
- The VoiceMOS Challenge uses common datasets and an out-of-domain track to advance MOS prediction for speech synthesis research.
2. Challenge Description
The challenge provides shared datasets, evaluation procedures, metrics, and baseline systems for MOS prediction. Its main track covers diverse historical TTS and VC systems, while the out-of-domain track tests adaptation to a different listening test with limited labeled data.
- Datasets: The main track contains samples from 187 synthesis systems, with 38 samples per system rated by eight listeners.
- Datasets: The OOD track uses a separate Chinese TTS listening test with different systems and listeners, split into 10%/40%/10%/40% training, unlabeled, development, and test data.
- Evaluation: Evaluation uses system- and utterance-level MSE, LCC, SRCC, and KTAU to assess different aspects of prediction performance.
- Evaluation: System-level SRCC was the primary leaderboard ranking metric, while the distributed script was intended to reduce evaluation inconsistency.
- Baselines: Participants received three open-source baselines: SSL-MOS, MOSA-Net, and LDNet, representing self-supervised, cross-domain, and listener-aware approaches.
3. Participants and Submitted Systems
The challenge attracted 22 teams from academia, industry, joint collaborations, and an individual participant, with participation across Asia and Europe. The main and OOD tracks drew 21 and 15 teams, respectively.
- 22 teams submitted during the evaluation phase: 14 academic, five industry, two joint academia-industry, and one individual team.
- Teams came from Asian countries including Japan, Taiwan, and China, and European countries including the UK, Ireland, the Netherlands, Hungary, Romania, and Czechia.
4. Results, Discussion and Analysis
Results show strong baseline and top-system performance, with SSL fine-tuning especially effective in the main track. Prediction remains difficult for unseen listeners, some unseen systems in the OOD track, and systems whose training and test rating distributions differ.
- Baseline comparison: B01 was the strongest baseline for main-track system-level MSE and SRCC and OOD-track system-level SRCC, while B02 led OOD-track system-level MSE.B03 was consistently the weakest baseline, and both stronger baselines used SSL.
- Baseline comparison: 0.148 and 0.921 were B01’s main-track system-level MSE and SRCC, compared with top-system scores of 0.090 and 0.939.For the OOD track, B02 achieved 0.071 system-level MSE and B01 achieved 0.975 system-level SRCC, versus top scores of 0.030 and 0.979.
- Top prediction systems: Main-track top-ranking teams all fine-tuned SSL models, whereas this ordering did not hold in the OOD track.Ensembling, multi-task learning, and speech recognizers were also common approaches among top systems and teams overall.
- Top prediction systems: Seven teams, including three of the top five, used per-listener ratings, while no teams used the supplied listener demographics.One team used listener groups based on the utterances rated in the main-track listening test.
- Top prediction systems: Unlabeled-data use was uncommon in the OOD track, but T17’s added listening test labels supported a first-place system-level SRCC ranking.OOD system-level SRCCs were generally higher than in the main phase, while wider MSE variation indicated that precise scores across listening tests remained difficult.
- Seen vs. unseen categories: Unseen listeners were significantly harder for 16 teams and one baseline, while unseen speakers were harder for six teams and one baseline.Unseen synthesis systems were generally not significantly harder in the main track, but were harder for four teams and two baselines in the OOD track.
- Difficult synthesis systems: All four remaining difficult systems fell within the 15% with the highest training–test EMDs, linking distribution discrepancy with adverse prediction ability.The analysis computed earth-mover’s distance between each system’s training and test rating distributions.
- Metric analysis: LCC, SRCC, and KTAU correlated close to 1, whereas MSE correlated less strongly with them at approximately 0.87.The analysis suggests reporting SRCC while retaining MSE, or developing a metric combining both.
5. Conclusions
The challenge found strong effectiveness of SSL-based models for MOS prediction, while generalization across synthesis systems, speakers, listeners, and listening-test contexts remained difficult.
- SSL-based models were overwhelmingly effective for MOS prediction.
- Unseen synthesis systems were difficult in the out-of-domain setting, while unseen speakers and listeners were difficult in the main track.
- Generalizing to a different listening-test context and predicting scores precisely from small labeled datasets remained challenging.
- Well-matched training and test distributions were important for MOS prediction.
A. Challenge Rules
Challenge rules required participant submissions and system descriptions, permitted external data with disclosure, and restricted development-set use to model selection and tuning.
- Teams had to make at least one CodaLab leaderboard submission during the training phase.
- Participants were required to submit a system description for their team entry.
- External data was permitted, but teams had to specify its use and could not use samples from several named challenge sources.
- The development set could be used for model selection and parameter tuning, but not as additional evaluation-phase training data.
B. List of participants
Table 4 lists participant affiliations in random order.
- The table presents participant affiliations in random order.
C.1. Main track
The main-track results table reports evaluation metrics, while a separate table gives rankings for each metric.
- Main-track evaluation results are reported using MSE, LCC, SRCC, and KTAU, with lower MSE and higher correlation values preferred.
- Main-track rankings are listed separately for each evaluation metric.
C.2. OOD track
The OOD track reports performance using error, correlation, and rank-correlation metrics, with rankings provided separately for each metric.
- OOD performance is evaluated with MSE, LCC, SRCC, and KTAU.
- MSE is interpreted with smaller values indicating better performance.
- LCC, SRCC, and KTAU are interpreted with larger values indicating better performance, and rankings are reported separately for each metric.
D. Correlation analysis between metrics
The analysis relates evaluation metrics to prediction difficulty across systems. It also examines how score variability, train–test distribution differences, and MOS level relate to prediction errors.
- Linear correlations between different evaluation metrics are analyzed using main-track results.
- High standard deviation in a system’s scores does not explain why MOS predictors find that system difficult to predict.
- Large differences between training and test distributions contribute to prediction difficulty.
- All teams had low errors for low-scoring systems, while higher errors tended to occur for middle- and high-scoring systems.