Source-linked AI summary
Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speech
Erica Cooper, Junichi Yamagishi
TL;DR
MOS ratings depend on the quality range represented in a synthesized-speech listening test, creating range-equalizing bias. The paper measures this effect by progressively narrowing tests toward higher-quality systems, finding stronger differences and a maximum 1.28-point drop for the worst system in the smallest top-system group.
Problem
Method
The authors conduct new listening tests that progressively halve the systems retained from BVCC, focusing on higher-quality regions while comparing results across zoom levels.
Source: §3.2 · p. 2Results
Takeaways & Limitations
To identify differences among similarly high-quality systems, researchers should keep the overall quality range in a listening test fairly small while recognizing that lower-rated systems may receive substantially lower scores.
Source: §5 · p. 4The subset choices relied on the original BVCC ranking, although rankings changed when similarly rated systems were examined more closely.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Mean Opinion Score (MOS) is a popular measure for evaluating synthesized speech. However, the scores obtained in MOS tests are heavily dependent upon many contextual factors. One such factor is the overall range of quality of the samples presented in the test -- listeners tend to try to use the entire range of scoring options available to them regardless of this, a phenomenon which is known as range-equalizing bias. In this paper, we systematically investigate the effects of range-equalizing bias on MOS tests for synthesized speech by conducting a series of listening tests in which we progressively "zoom in" on a smaller number of systems in the higher-quality range. This allows us to better understand and quantify the effects of range-equalizing bias in MOS tests.
1. Introduction
The paper investigates range-equalizing bias in MOS tests by progressively narrowing the evaluated systems toward higher quality. This exposes how test context changes ratings and system comparisons.
1. Introduction
- MOS tests are context-dependent, and range-equalizing bias describes listeners’ tendency to use the full rating scale regardless of sample quality.Relevant context includes listener characteristics, test environment, included systems, test sentences, and interface appearance.Source: §1 · p. 1
- The study revisits BVCC, which rates samples from 187 text-to-speech and voice conversion systems evaluated together in one large-scale MOS test.The dataset spans more than a decade of progress and includes broad system variety.Source: §1 · p. 1
- Because broad BVCC coverage limits fine-grained comparisons among similarly performing systems, the authors create progressively smaller subsets focused on higher-quality systems.The study also targets present-day comparisons and more difficult middle- and high-quality prediction cases.Source: §1 · p. 1
- Score distributions reveal range-equalizing bias, while their skew indicates that listeners retain some sense of absolute quality.The study also observes more significant differences and about a one-point MOS decrease for some systems.Source: §1 · p. 1
2. Related work
Prior work shows that perceptual speech ratings depend on many aspects of test context and are relative rather than absolute. This paper therefore focuses on systematically quantifying the effect of the represented quality range.
2. Related work
- Prior reviews identify recency, expectations, stimulus frequency, rating-category spacing, equipment, and interface appearance as influences on listening-test ratings.Anchoring techniques have been recommended to counteract range-equalizing bias.Source: §2 · pp. 1–2
- Even anchored MUSHRA tests remain context-sensitive because scores change when the balance of higher- and lower-quality samples changes.This occurs despite keeping the reference and anchor samples constant.Source: §2 · p. 2
- Scores also vary with presentation context and question wording, including whether a sample is heard alone or within paragraph or dialogue context.Scores from different presentation contexts do not necessarily correlate.Source: §2 · p. 2
- Listener count and sentence coverage can change rankings and significant differences, with statistical significance stabilizing only at 30 listeners in one revisited dataset.Older systems also lost about one MOS point when evaluated alongside modern systems, compressing differences between them.Source: §2 · p. 2
- These findings motivate a systematic study of how MOS changes when the overall quality range represented in a test becomes smaller.The paper treats MOS as a relative rather than absolute measure.Source: §2 · p. 2
3. Data and listening test design
The study uses balanced BVCC samples and new listening tests that progressively retain higher-rated systems while keeping test-set sizes similar. This design isolates changes associated with narrowing the represented quality range.
3.1. The BVCC dataset
- The BVCC materials contain 38 balanced samples per system from 187 systems, including text-to-speech, voice conversion, and natural-speech references.Samples cover multiple genres and, where relevant, source and target speakers.Source: §3.1 · p. 2
3.2. New listening tests
- New tests sort systems by original BVCC MOS and repeatedly halve the set, retaining the highest-rated half at each zoom level.The design progressively concentrates evaluation on higher-quality systems.Source: §3.2 · p. 2
- Each test uses 32 samples per system, maintains genre and speaker balance, and compensates for fewer systems by doubling samples per system within a listener’s set.Test sets therefore contain approximately the same total number of samples across zoom levels.Source: §3.2 · p. 2
- The 6% zoom condition contains the top 11 original-BVCC systems, five of which are natural speech.The overall design includes the original BVCC test as the 100% zoom comparison level.Source: §3.2 · p. 2, Table 1 · p. 2
- The study planned 200 native English-speaking listeners, but zoomed-out tests received fewer ratings per sample and zoomed-in tests fewer listeners because of budget constraints.Listeners could complete only one set from any zoom level to broaden participant variety and avoid cross-context contamination.Source: §3.2 · p. 2
4. Results
Zooming into progressively smaller sets of higher-quality speech systems reveals range-equalizing bias, more significant differences, weaker score correlations, and substantial MOS changes.
4.1. Distributions of scores of different listening tests
- Zooming into smaller higher-quality system sets reveals range-equalizing bias while listeners still preserve some sense of absolute quality.Top-system tests received many lowest-category ratings despite none in the corresponding original subset, while higher ratings remained more frequent overall.Source: §4.1 · pp. 2–3, Figure 1 · p. 3
4.2. Emergence of significant differences
- Narrower quality ranges generally produce more statistically significant system differences, with 25 significant pairs among the top 6% in both the 12% and 6% tests.The authors interpret this as wider system ranges compressing differences between similarly rated systems.Source: §4.2 · p. 3, Table 2 · p. 3
4.3. Correlations
- Correlations between the top 11 systems’ scores in zoomed tests and original BVCC scores decrease substantially as the evaluated range narrows.The two originally significant system pairs retain their order and significance; rank switches occur among systems without significant original differences.Source: §4.3 · p. 3, Table 3 · p. 3
4.4. Decrease in MOS ratings for some systems
- Some systems lose nearly an entire MOS point when reevaluated in finer-grained contexts, with the largest observed drop reaching 1.28 MOS points for BC2011-G.Score-drop regression has a slope close to zero across zoom levels, suggesting drops of up to about one point may occur regardless of zoom level.Source: §4.4 · p. 4, Figure 2 · p. 4, Table 4 · p. 4
5. Conclusions
The study finds clear range-equalizing bias in MOS tests and shows that narrowing the tested quality range can magnify apparent differences between systems. It recommends narrower quality ranges for distinguishing similar systems, while cautioning that this lowers scores for the weakest systems and that the subset selection relied on an uncertain original ranking.
5. Conclusions
- The worst system in the smallest high-quality group dropped by 1.28 MOS points, demonstrating strong contextual effects in MOS listening tests.The authors attribute this result to progressively zooming in on smaller groups of high-quality systems.Source: §5 · p. 4
- Researchers seeking finer distinctions between similar systems should keep the overall quality range fairly small, while recognizing that this can substantially lower scores for the group’s weakest systems.The authors warn that even relatively high-quality systems may receive lower ratings when judged within a narrow range.Source: §5 · p. 4
- The study’s top-system subsets were selected using the original BVCC ranking, which may not represent the true ranking because close-system differences were often nonsignificant and rankings changed under closer inspection.This limits how confidently the selected systems can be treated as ordered by quality.Source: §5 · p. 4
- Comparisons of Tacotron 2 MOS scores across studies suggest a decline from 4.53 in 2018 to 4.20 in 2020 and 3.01 in 2022, but differing test contexts prevent direct comparison.The authors interpret the overall decrease as consistent with possible range-equalizing bias and warn that narrow quality ranges can magnify apparent differences.Source: §5 · p. 4
- Future work will compare A/B and MUSHRA tests, assess finer-grained data for MOS predictors, examine lower- and middle-quality regions, and test different rating-scale increments.These studies are intended to clarify how listening-test formats and finer-grained data affect evaluation and prediction.Source: §5 · p. 4