PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation

Praveen Srinivasa Varadhan, Amogh Gulati, Ashwin Sankar, Srija Anand, Anirudh Gupta, Anirudh Mukherjee, Shiva Kumar Marepally, Ankur Bhatia, Saloni Jaju, Suvrat Bhooshan, Mitesh M. Khapra

arXiv:2411.12719v3cs.CLcs.LGcs.SDeess.AS

TL;DR

Modern TTS needs more consistent human evaluation because MOS misses subtle differences and CMOS is costly to scale, while MUSHRA can bias ratings toward matching human references. The paper evaluates MUSHRA with 492 listeners across Hindi and Tamil, proposes two refined variants, and finds that their combination improves reliability and granularity while releasing the 246,000-rating MANGO dataset.

  • Problem

    Human evaluation for modern TTS lacks standardization, while MOS struggles with similar systems and CMOS requires costly pairwise comparisons.

    Source: §1 · p. 1, §4.1 · p. 6

  • Method

    The paper analyzes 246,000 MUSHRA ratings from 492 listeners across Hindi and Tamil and proposes reference-obscuring and guideline-based MUSHRA variants.

    Source: §1 · p. 2, §3 · p. 3, §3.4 · p. 5

  • Results

    Both variants produce more reliable evaluations, while MUSHRA-DG additionally enables fine-grained fault isolation and their combination provides consistency and granularity.

    Source: §1 · p. 2, §9 · p. 16

  • Takeaways & Limitations

    The combined framework supports fairer assessment of modern TTS systems and more actionable diagnosis of pronunciation, prosody, and other quality factors.

    Source: §1 · p. 2, §9 · p. 16

    MUSHRA-DG takes nearly twice as long as the original MUSHRA test, although the authors regard the added time as worthwhile for richer diagnosis.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

Despite rapid advancements in TTS models, a consistent and robust human evaluation framework is still lacking. For example, MOS tests fail to differentiate between similar models, and CMOS's pairwise comparisons are time-intensive. The MUSHRA test is a promising alternative for evaluating multiple TTS systems simultaneously, but in this work we show that its reliance on matching human reference speech unduly penalises the scores of modern TTS systems that can exceed human speech quality. More specifically, we conduct a comprehensive assessment of the MUSHRA test, focusing on its sensitivity to factors such as rater variability, listener fatigue, and reference bias. Based on our extensive evaluation involving 492 human listeners across Hindi and Tamil we identify two primary shortcomings: (i) reference-matching bias, where raters are unduly influenced by the human reference, and (ii) judgement ambiguity, arising from a lack of clear fine-grained guidelines. To address these issues, we propose two refined variants of the MUSHRA test. The first variant enables fairer ratings for synthesized samples that surpass human reference quality. The second variant reduces ambiguity, as indicated by the relatively lower variance across raters. By combining these approaches, we achieve both more reliable and more fine-grained assessments. We also release MANGO, a massive dataset of 246,000 human ratings, the first-of-its-kind collection for Indian languages, aiding in analyzing human preferences and developing automatic metrics for evaluating TTS systems.

1 Introduction

Modern TTS evaluation lacks standardized human-testing practices, while MOS and CMOS struggle with subtle comparisons or scale poorly. This paper identifies two MUSHRA shortcomings, proposes two variants, and validates them with large-scale multilingual ratings.

Motivation

  • MUSHRA can scale better than MOS and CMOS, but modern high-quality TTS exposes unresolved reference and validity problems.MOS may miss subtle differences, while CMOS requires pairwise comparisons between all N systems.Source: §1 · pp. 1–2

Outcome

  • Combining the variants yields more reliable and fine-grained assessments without unfairly penalizing systems that surpass the reference in naturalness or prosody.Detailed scoring also provides actionable information about pronunciation, prosody, and other factors.Source: §1 · p. 2

Findings

  • The study identifies reference-matching bias and judgement ambiguity as two primary shortcomings of MUSHRA.These problems arise when raters align outputs with the human reference or interpret broad evaluation criteria inconsistently.Source: §1 · p. 2

Proposed variants

  • Two refined MUSHRA variants target these shortcomings: one removes explicit human-reference identification, and the other adds fine-grained scoring guidelines.The variants are designed to reduce unfair penalties and improve rating consistency.Source: §1 · p. 2

Contributions

  • 246,000 ratings from 492 listeners across Hindi and Tamil support a comprehensive assessment of MUSHRA for modern TTS systems.The evaluation covers three TTS systems and produces the MANGO dataset.Source: §1 · p. 2

2 Related Work

Prior work documents weaknesses in subjective speech evaluation and develops MUSHRA variants for different TTS assessment goals. These studies motivate more comprehensive and standardized evaluation frameworks.

Critiques of TTS Evaluation

  • MOS evaluations can vary with listener and context, show high rating variance, and depend on how scales, instructions, and evaluation details are reported.Prior studies also identify range-equalizing bias and insufficient methodological description.Source: §2 · p. 3

Emergence of Modern Tests

  • The paper positions its work as a systematic examination of MUSHRA reliability and alternative test designs after prior studies exposed limitations in existing protocols.This extends criticism beyond the mainly MOS-focused literature.Source: §2 · p. 3
  • MUSHRA variants have been adapted for imperfect transcripts, morphology, emotional strength, speaker similarity, accent similarity, naturalness, and expressiveness.These adaptations modify anchors, references, transcripts, or measured attributes for particular evaluation goals.Source: §2 · p. 3

Learnings from Human Evaluations in NLP

  • NLP evaluation research similarly calls for comprehensive, standardized frameworks and warns that average ratings and incomplete descriptions can distort interpretation.Other work emphasizes detecting subtle system differences and evaluating systems on shared documents or samples.Source: §2 · p. 3

3 MANGO: A Corpus of Human Ratings for Speech

MANGO is a large multilingual corpus built through standard MUSHRA-based TTS evaluation, with platform, synthesis, and annotation procedures designed for reliable human ratings. The framework enables simultaneous, fine-grained comparison of multiple systems against reference and anchor stimuli.

3.1 MUSHRA-Based Annotation Framework

  • Standard MUSHRA presents multiple stimuli with an explicit reference, anchor, and hidden reference on a continuous 0–100 scale.The scale is discretized into Excellent, Good, Fair, Poor, and Bad categories.Source: §3 · p. 3, §3.1 · p. 4
  • MUSHRA supports simultaneous multi-system comparisons with finer granularity and greater efficiency than isolated MOS or exhaustive pairwise CMOS testing.Reference-based calibration is intended to reduce scoring variability.Source: §3.1 · p. 4

3.2 Online Annotation Platform

  • The online platform was enhanced with session saving, listener breaks, consent and listening controls, and event tracking for page-level timing.These additions support participant management and analysis of test duration.Source: §3.2 · pp. 4–5

3.3 Synthesizing Speech Samples for Annotation

  • Speech samples were generated with FastSpeech2 plus HiFi-GAN, VITS, and fine-tuned StyleTTS2 using Hindi and Tamil IndicTTS data.Each language included female and male speaker recordings, with 20.17 hours for Hindi and 20.59 hours for Tamil.Source: §3.3 · p. 5

3.4 Annotation Process and Dataset Statistics

  • Native-language annotators received demonstrations, multi-session training, and a five-sample pilot review before rating 100 test samples.The process aimed to standardize understanding of the platform and evaluation criteria.Source: §3.4 · p. 5
  • MANGO comprises 246,000 human ratings collected from 492 participants across Hindi and Tamil MUSHRA evaluations and proposed variants.Demographic information was collected from most participants, while 21 declined to disclose details.Source: §3 · p. 3, §3.4 · p. 5

4 Key Insights on MUSHRA

The evaluation identifies reliability, bias, and scaling problems in conventional MUSHRA testing for modern TTS, while testing how design choices affect scores and rankings.

4.1 Is MUSHRA A Reliable Test?

  • CMOS finds VITS and ST2 close in quality to the reference in Hindi and Tamil, unlike conventional MUSHRA scores, indicating reference-matching bias.The authors argue that raters may align scores with the human reference, disadvantaging systems comparable to or better than human speech.Source: §4.1 · pp. 5–6

4.2 How reliable is the mean statistic in MUSHRA scores?

  • High within-rater and between-rater variance makes MUSHRA mean scores ambiguous, even when reported with confidence intervals.The authors hypothesize that raters prioritize different dimensions such as prosody, voice quality, or digital artifacts.Source: §4.2 · pp. 6–7, Figure 1 · p. 7

4.3 How sensitive is MUSHRA to number of listeners and utterances?

  • Using more than 30 listeners and 30 utterances yields rank correlations above 95% with the fully scaled Hindi evaluation, while fewer than 20 listeners fails to exceed 90%.When both listener and utterance counts are small, increasing listeners is more beneficial than increasing utterances; Tamil shows similar but more efficient trends.Source: §4.3 · p. 7, Figure 2 · p. 8, Figure 3 · p. 8

4.4 What is the impact of rejecting raters per standard MUSHRA protocol?

  • Rejecting raters who score the hidden reference ≤λ on more than 15% of utterances shifts scores but preserves system rankings.As λ increases, system scores decrease and reference scores increase, consistent with reference-matching bias.Source: §4.4 · p. 8, Figure 3 · p. 8

4.6 Does adding more systems affect scores?

  • With seven systems, XTTS receives 73.65, while duplicated VITS versions receive 68.99 and 68.47, and the original relative ranking remains consistent.The experiment finds no significant cognitive overload at n = 7, but the larger system count at which overload begins remains unknown.Source: §4.6 · p. 9

4.7 Do objective metrics align with human perception?

  • SI-SDR correlates strongly with MUSHRA scores at Pearson 0.76, whereas PESQ and STOI show weak alignment with human ratings.The authors attribute the mismatch partly to PESQ and STOI emphasizing intelligibility and signal distortions rather than naturalness, prosody, and timbre.Source: §4.7 · pp. 9–10

5 Rethinking MUSHRA

The paper identifies reference-matching bias and judgement ambiguity as central MUSHRA shortcomings, then proposes MUSHRA-NMR and MUSHRA-DG to address them.

5 Rethinking MUSHRA

  • The study frames these variants as responses to reference-matching bias and ambiguity caused by broad, subjectively interpreted rating criteria.Source: §5 · p. 10

MUSHRA-NMR

  • MUSHRA-NMR omits the explicitly identified ground-truth reference so listeners can assess TTS quality without matching outputs against it.Source: §5 · p. 10

MUSHRA-DG

  • MUSHRA-DG replaces a single broad naturalness judgement with detailed objective and perceptual dimensions combined through a weighted scoring formula.Raters record six error categories and rate liveliness, voice quality, and rhythm on 0–100 scales; weights can be adjusted for the use case.Source: §5 · p. 10
  • Only 1.3% of computed MUSHRA-DG scores were revised by raters, suggesting close alignment between the formula and their assessments.Source: §5 · p. 10
  • MUSHRA-DG preserves CMOS system rankings while significantly lowering score variance, indicating reduced ambiguity in naturalness ratings.Source: §5 · p. 10

6 Results

Across Hindi and Tamil evaluations, MUSHRA-NMR reduces reference-score gaps, MUSHRA-DG lowers rating ambiguity, and the combined variant preserves rankings while improving score alignment.

Does our proposed variant help mitigate the reference-matching bias?

  • MUSHRA-NMR reduces reference-matching bias by narrowing the gap between the best system and the reference, especially in Tamil, while preserving scaled-up MUSHRA rankings.The reference score itself decreased, indicating stricter ratings in the no-mentioned-reference condition.Source: Does our proposed variant help… · p. 11

How sensitive is MUSHRA-NMR to number of listeners and utterances?

6.2 Evaluations using MUSHRA-DG

  • MUSHRA-DG reduces score standard deviation by 43% in Hindi and 53% in Tamil compared with original MUSHRA while preserving CMOS-consistent rankings.Source: §6.2 · p. 12
  • Fine-grained MUSHRA-DG dimensions isolate different system weaknesses, including pronunciation, prosody, word-skipping, energy fluctuations, rhythm, and digital artifacts.For Hindi, FS2 performs well in pronunciation but has prosody and word-skipping weaknesses, while VITS and ST2 differ in energy fluctuations and rhythm.Source: §6.2 · pp. 12–13, Figure 5 · p. 13
  • MUSHRA-DG takes nearly twice as long as original MUSHRA because each sample is evaluated across multiple dimensions.The added time is presented as a trade-off for more comprehensive performance analysis and fault isolation.Source: §6.2 · p. 13, Figure 6 · p. 14

6.3 Evaluations using MUSHRA-DG-NMR

  • MUSHRA-DG-NMR brings most system scores closer to the reference and significantly reduces variance relative to MUSHRA-NMR.The combined variant predominantly places systems in the “Excellent” category, consistent with CMOS expectations.Source: §6.3 · pp. 13–14

7 Generalization to English

English evaluations reproduce the paper’s Hindi and Tamil findings: standard MUSHRA shows reference-matching bias and inconsistent ratings, while MUSHRA-DG-NMR improves score alignment without changing rankings.

7.1 Does the Reference-Matching Bias exist?

  • English CMOS results show ST2 and VITS comparable to or preferred over the reference, although standard MUSHRA places only the reference in the “Excellent” category.This discrepancy indicates that MUSHRA can fail to capture perceptual parity or superiority as modern TTS approaches reference quality.Source: §7.1 · p. 14, Table 7 · p. 15

7.2 Does Judgement Ambiguity Exist?

  • English ratings show both intra-rater inconsistency across utterances and wide inter-rater differences in how MUSHRA labels are applied.These patterns make the mean statistic an unreliable standalone measure of perceived quality.Source: §7.2 · pp. 14–15

7.3 Does our proposed variant (MUSHRA-DG-NMR) help?

  • MUSHRA-DG-NMR brings English system scores closer to the reference and preserves relative rankings, suggesting reduced reference-matching bias without ranking distortion.Source: §7.3 · p. 15
  • Fine-grained objective and subjective dimensions reveal quality trade-offs that a single score misses, such as VITS’s energy variations versus ST2’s digital artifacts and rhythmic advantage.Although English score variance does not significantly decrease, these dimensions identify specific strengths and weaknesses; the evaluation also requires additional time.Source: §7.3 · p. 15
  • English results expand the annotation dataset to 255,000 subjective annotations and validate the paper’s conclusions across high- and low-resource languages.Source: §7 · p. 14, §7.3 · p. 15

8 Future Work

Future work targets larger-scale comparisons, more efficient evaluation, demographic fairness, and validation of the refined variants in practical deployments.

8 Future Work

  • Because ITU-R recommends rating no more than 12 signals at once, future work should adapt MUSHRA-DG-NMR for tournament-style evaluations with larger system sets.Source: §8 · pp. 15–16
  • Additional directions include complementary objective metrics, scalable MUSHRA-DG studies, cultural and demographic bias analysis, and real-world deployment validation.Source: §8 · p. 16

9 Conclusion

The study identifies shortcomings in current MUSHRA evaluations and proposes two variants that improve reliability, fine-grained fault isolation, and support for TTS evaluation research.

  • The study identifies reference-matching bias and judgment ambiguity in MUSHRA evaluations of modern high-quality TTS systems.Source: §9 · p. 16
  • MUSHRA-NMR omits explicit human-reference identification, while MUSHRA-DG uses detailed guidelines to calculate scores systematically.Source: §9 · p. 16
  • Both variants produce more reliable evaluations, with MUSHRA-DG additionally enabling fine-grained fault isolation; the work also releases MANGO.MANGO is a large human rating dataset intended to support research in TTS evaluation.Source: §9 · p. 16

A.1 Instructions For MUSHRA

Participants received guidelines and a live demonstration of the MUSHRA rating page before taking the test.

  • Participants were shown MUSHRA guidelines, given a live demo of the rating page, and walked through the guideline sheet.Source: Figure 7 · p. 21

A.2 CMOS Evidence of Limitations of Reference Matching

The evaluation examines MUSHRA’s reference dependence, rater variability, sensitivity, and scoring ambiguity, then motivates refined procedures for modern TTS assessment. Evidence includes CMOS preferences, survey responses, detailed scoring guidelines, visualizations, and stated study limitations.

A.2 CMOS Evidence of Limitations of Reference Matching

  • StyleTTS2 was preferred over the human reference in 54% of Tamil cases and 43.8% of English cases, while VITS was preferred in 37.2% of English cases.These CMOS-derived preferences challenge treating the human reference as an invariant gold standard.Source: §A.2 · p. 21, Table 8 · p. 22

A.4 Sensitivity of MUSHRA in Tamil

  • Tamil rank correlations compare scores from subsets of listeners and utterances with scores using all listeners and utterances, with similar trends reported for Hindi.Source: §A.4 · p. 22, Figure 8 · p. 22

A.5 Post-Evaluation Survey

  • Among surveyed participants, 55% preferred systems closer to the reference, while 18% preferred systems that deviated from it and 11% rated solely by overall quality.Source: §A.5 · p. 22, Figure 9 · p. 23
  • Judgment ambiguity was reflected by competing priorities: 56% favored voice quality, while 33% favored better pronunciation over voice quality.Source: §A.5 · p. 24

A.6 Detailed Guidelines for MUSHRA-DG

  • MUSHRA-DG averages perceptual measures and subtracts penalties for word skips, pronunciation mistakes, speed irregularities, and other observable artifacts.Word skips incur 25 points, severe pronunciation mistakes 10 points, and mild pronunciation mistakes and other non-perceptual measures 5 points.Source: §A.6 · p. 24

A.7 Fault Isolation in English and Tamil

A.8 Limitations

  • The study is limited to Hindi and Tamil, omits English analysis, has missing demographic details for 21 of 492 participants, and does not cover all possible TTS-evaluation biases.The authors call for broader analysis across linguistic contexts and practical applications.Source: §A.8 · p. 24, §A.9 · p. 26

A.3 Visualizing MUSHRA Distributions

  • MUSHRA score visualizations show substantial within-rater and between-rater variance across systems and utterances.The figures cover Tamil, English, Hindi, and Tamil utterance-level distributions.Source: §A.3 · p. 22, Figure 12 · p. 26, Figure 13 · p. 27
Loading 2411.12719v3…