Source-linked AI summary
How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?
Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara, Marc Delcroix, Naoyuki Kamo, Tetsuji Ogawa, Shoko Araki
TL;DR
Predicted MOS is widely used to compare speech enhancement systems, but correlation with human-rated MOS does not establish agreement on which system is better. The paper introduces SPA to measure that agreement and evaluates single models, ensembling, and domain adaptation. Results show reliability varies across settings, with domain adaptation substantially more effective in the closed than the practical open condition.
Problem
MOS prediction models are commonly assessed by correlation with human-rated MOS, which does not directly measure whether they reproduce human pairwise system preferences.
Source: §1 · p. 1Method
The paper introduces SPA, the rate of agreement between predicted- and human-MOS system preferences, and evaluates single models, ensembling, and domain adaptation.
Source: §1 · p. 1Results
Single-model SPA mainly ranged from 30.8% to 76.8%, while even the best ensemble left about 17% of judgments incorrect; domain adaptation was more reliable in the closed than open condition.
Source: §1 · p. 1, §6 · p. 4, §5.4 · p. 4Takeaways & Limitations
Predicted MOS alone can lead to unreliable conclusions in practical speech enhancement system comparisons, and neither ensembling nor domain adaptation guarantees reliable judgments.
Source: §6 · p. 4SPA considers both (A, B) and (B, A) to avoid dependence on which system is labeled first.
Source: §5.5 · p. 4
Abstract
from arXiv · showhide
We investigate whether predicted Mean Opinion Scores (MOS) can reliably support system-level comparisons of speech enhancement (SE) methods by introducing system-level preference accuracy (SPA). Although MOS prediction models are widely used to evaluate SE systems, their performance is typically assessed by correlation with human-rated MOS, which does not guarantee agreement on which system is better. SPA addresses this gap by directly evaluating whether predicted and human-rated MOS yield the same system preferences. Using SPA, we systematically evaluate three settings: single prediction models, ensembling, and domain adaptation. Through experiments, SPA varies substantially across single prediction models, from 9.4% to 76.8%. Even the best model disagrees with human judgments in approximately 23% of system comparisons. Ensembling yields only limited improvement, while domain adaptation tends to substantially improve SPA in the closed condition but brings only modest gains in the more practical open condition, where neither the target systems nor the speakers are known. These results suggest that SPA can reveal errors correlation-based evaluation alone does not expose, and that predicted MOS alone can lead to unreliable conclusions in practical SE system comparison.
1. INTRODUCTION
Human-rated MOS supports system comparisons but requires costly annotation, so speech enhancement research increasingly uses predicted MOS. This section introduces SPA to test whether those predictions reproduce human system preferences and reports substantial reliability concerns across evaluated approaches.
Motivation
- Human-rated MOS is commonly used to compare speech enhancement systems, but collecting human annotations is time-consuming and financially costly.Researchers therefore increasingly use MOS prediction models and compare systems using their averaged predicted scores.Source: §1 · p. 1
Evaluation gap
- Correlation metrics such as LCC and SRCC do not directly measure whether predicted MOS correctly identifies which system is superior.A high SRCC can coexist with reversed pairwise system orderings.Source: §1 · p. 1
Contribution
- SPA measures the rate at which predicted and human-rated MOS agree on pairwise system preferences, including wins, losses, and ties, with bootstrap sampling quantifying data variability.The evaluation addresses whether predictions reproduce human preferences, rather than relying only on correlation with human-rated MOS.Source: §1 · p. 1
Study scope
- The study evaluates single prediction models, ensembling, and domain adaptation as factors affecting SPA.The two approaches are examined because they have been reported to improve MOS prediction performance.Source: §1 · p. 1
Key findings
- Across six models and two speech enhancement tasks, SPA mainly ranged from 30.8% to 76.8%; even the best model was wrong in about 23% of comparisons.Ensembling did not consistently improve on the best single model, and domain adaptation was more reliable in the closed than the open condition.Source: §1 · p. 1
2. RELATED WORK
Related work includes MOS-prediction training losses that align utterance-level preferences, but utterance-level and system-level preferences differ. System-level accuracy better matches how speech enhancement systems are typically compared.
- Prior studies introduced training losses aligning predicted and human-rated utterance-level preferences and reported utterance-level preference accuracy.Those studies compare preferences between pairs of utterances, rather than system-level averaged MOS.Source: §2 · pp. 1–2
- Because speech enhancement systems are typically compared using system-level averaged predicted MOS, system-level preference accuracy is better aligned with that evaluation practice.Utterance-level and system-level preferences tend to exhibit different properties.Source: §2 · p. 2
3. SYSTEM-LEVEL PREFERENCE ACCURACY
SPA evaluates whether predicted and human-rated MOS agree on pairwise system preferences. It treats the comparison as a win, loss, or tie and uses bootstrap sampling to quantify uncertainty.
- The method classifies each pair as a win, loss, or tie and uses bootstrap sampling to account for evaluation-dataset variability.It compares the differences in average MOS for each pair under human ratings and predicted scores.Source: §3 · p. 2
- SPA is the proportion of system pairs whose predicted-MOS preference agrees with the human-rated-MOS preference, with values from 0 to 1.The agreement is averaged across system pairs; bootstrap resampling of utterances yields a mean SPA and 95% confidence interval.Source: §3 · p. 2
4. APPROACHES FOR IMPROVING MOS PREDICTION
The paper studies averaging and majority voting as ensemble strategies, and fine-tunes MOS prediction models with limited target-domain data under closed and open adaptation conditions.
4.1. Ensemble of Multiple MOS Prediction Models
- Ensembling independently developed MOS prediction models tests whether combining predictors improves the reliability of system comparisons.The study considers averaging predicted MOS scores and majority voting over predicted preferences.Source: §4.1 · p. 2
- Averaging combines models' predicted MOS scores before SPA is computed, whereas majority voting combines their system-pair preferences.Majority voting does not produce predicted MOS scores, so correlation-based metrics cannot be evaluated for that strategy.Source: §4.1 · p. 2
4.2. Domain Adaptation of MOS Prediction Model
- Domain adaptation fine-tunes a MOS prediction model on limited target-domain data to address training–evaluation domain mismatch.The closed condition assumes target speakers, systems, speech, and corresponding human MOS are available for adaptation; the open condition represents cases where target speakers and systems are unknown in advance.Source: §4.2 · p. 2
5. SPEECH QUALITY EVALUATION EXPERIMENTS
Experiments across two speech-enhancement datasets show that system-level preference accuracy (SPA) varies across individual models, improves only selectively with ensembling, and benefits most from domain adaptation in the closed condition. Errors are dominated by confusion around ties rather than win/lose reversals.
5.1. Experimental Setup
- The experiments evaluated six MOS prediction models on CHiME-7 UDASE and URGENT24 using sys-SRCC and bootstrap SPA with a tie threshold of ϵ = 0.125.Cross-validation subsets were defined by speaker or speaker group and system pair; open adaptation excluded subsets sharing the evaluation speaker or system.Source: §5.1 · pp. 2–3
5.2. Results of Individual MOS Prediction Models
- SPA ranged from 30.8% to 65.7% on CHiME-7 UDASE, while the best URGENT24 model reached 76.8%; SPA can differ from sys-SRCC.The authors present SPA as complementary to sys-SRCC because models’ differences on the two metrics do not always align.Source: §5.2 · p. 3
5.3. Results of Multiple MOS Prediction Models (Ensemble)
- Ave outperformed MV on both datasets; on CHiME-7 UDASE, Ave (All) scored 40.8% versus the best single model’s 65.7%, while oracle Ave reached 80.6%.On URGENT24, Ave (All) reached 79.8%, above the best single model’s 76.8%, and Ave (Best) reached 83.1%.Source: §5.3 · p. 3
- On URGENT24, the best single-model SPA was 76.8%, versus 83.1% for oracle Ave; adaptation helped most in closed conditions, with more limited open-condition gains.Even the oracle ensemble’s 80.6% SPA on CHiME-7 UDASE was described as potentially insufficient for reliable comparisons.Source: §5.3 · p. 3, §5.2 · p. 3, §5.4 · p. 4
5.4. Results of Domain Adaptation for MOS Prediction Models
- On CHiME-7 UDASE, closed adaptation raised SCOREQ and Distill-MOS SPA to 98.2% and 98.8%, but their open-condition scores diverged to 85.9% and 44.2%; URGENT24 gains were limited.On URGENT24, some models improved modestly while others worsened, and additional learning-rate experiments did not resolve the degradation.Source: §5.4 · p. 4
5.5. Analysis of SPA Error Types
- Win/lose reversals ranged from 0.0% to 7.6%, while tie-related confusions ranged from 20.8% to 40.4% and accounted for most SPA errors.For NISQA, false differentiation errors totaled 20.6%, compared with 4.0% for false equivalence errors.Source: §5.5 · p. 4, Figure 1 · p. 4
6. CONCLUSION
SPA directly tests whether predicted MOS reproduces human-rated system preferences. The results show that single models, ensembles, and domain adaptation do not ensure reliable comparisons across all conditions.
- A single MOS prediction model can be insufficient for reliable pairwise preferences even when its correlation with human-rated MOS is high.Source: §6 · p. 4
- Ensembling and domain adaptation provide partial improvements, but the best-performing model combination still leaves about 17% of judgments incorrect.Source: §6 · p. 4
- Domain adaptation can make comparisons highly reliable in the closed condition for some models and datasets, but its benefit is more limited in the more practical open condition.Source: §6 · p. 4
- SPA reveals disagreements that correlation-based evaluation alone does not expose; future work will examine more principled tie-threshold rules, including statistical-significance-based decisions.Source: §6 · p. 4