ProxyMOS: Label-Free Speech Quality Assessment by Multi-Teacher Distillation with Adaptive Routing

1. INTRODUCTION

• Predictor performance shifts across domains, with rank correlations falling from 0.77 to 0.47 for the same model between enhanced natural speech and Russian TTS.Collecting more human ratings to close that gap would reintroduce the cost that predictors were meant to avoid.Source: §1 · p. 1
• ProxyMOS learns to combine public MOS predictors on existing labeled sets, then uses their combined scores to label unlabeled corpora for distillation into one model.The approach joins per-utterance combination with distillation, a less-studied pairing of techniques with prior MOS-prediction precedents.Source: §1 · p. 1
• The paper benchmarks eight predictors across URGENT and Russian TTS, searches 26 teacher subsets under five rules, and introduces mos260, a 4,600-utterance Russian TTS benchmark.Its contributions also include ProxyMOS1, a student trained on 807k ensemble-labeled utterances that outperforms its teachers on both benchmarks.Source: §1 · p. 1

2.1. Overview

• The pipeline ranks public predictors on labeled data, fits an aggregation rule, applies it to unlabeled corpora, and trains one student with MSE on the resulting targets.Only teacher ranking and aggregation-rule fitting use human ratings, and only through small public sets.Source: §2.1 · pp. 1–2

2.2. Teacher pool and selection

• Eight public predictors are ranked by Spearman rho on URGENT, and the top five enter the ensemble search, with MOSNet retained as a deliberately weak member.Because output scales differ, predictions are z-scored to development-set human MOS statistics before RMSE and MAE are computed.Source: §2.2 · p. 2

2.3. Aggregation rules

• Five rules set teacher weights, including an adaptive two-layer MLP router that reads utterance-level teacher scores and emits normalized per-utterance weights.The router is trained with MSE on SOMOS and BVCC; all 26 subsets of the five teachers with at least two members are evaluated under every rule.Source: §2.3 · p. 2

2.4. Student: ProxyMOS

• The student is an OmniASR-W2V-300M wav2vec 2.0 encoder with attentive statistics pooling and a two-layer scalar-output head, trained against ensemble targets with MSE.The model is exported to FP32 and FP16 ONNX for deployment, and no teacher is needed at test time.Source: §2.4 · p. 2

3. DATA

• URGENT ranks teachers, BVCC and SOMOS fit the router, and three unlabeled corpora supply 807k utterances; mos260 is excluded from selection, fitting, and training.The unlabeled data include synthetic speech across generators and languages, plus natural Russian speech.Source: §3 · p. 2

3.1. The mos260 benchmark

• mos260 contains 4,600 Russian utterances spanning 38 synthesis conditions, with quality and intonation ratings collected from native listeners and filtered for rater reliability.The final data include 11,545 quality ratings on 4,355 utterances and 28,068 intonation ratings on 4,598 utterances; the quality-rated set is the evaluation set.Source: §3.1 · p. 2

4.1. Metrics and uncertainty

• 95% CI half-width for ρ is at most 0.010 on URGENT and 0.025 on mos260 for ρ ≥0.45, using Fisher z with Spearman correction.The reported sample sizes are N=6,900 for URGENT and N=4,355 for mos260.Source: §4.1 · p. 2

4.2. Teachers across domains

• The best teacher changes by domain: WhiSQA reaches ρ = 0.773 on URGENT, while Distill-MOS leads mos260 at ρ = 0.613.WhiSQA drops to 0.466 on mos260, and its gap from Distill-MOS grows from 0.025 on URGENT to 0.147 on mos260.Source: §4.2 · p. 3

4.3. Ensemble search

• ρ = 0.814 for the uniform four-model mean on URGENT, +0.041 over WhiSQA; routing holds ρ ≈0.636 on mos260 across K=2–5.The router is the only rule whose URGENT score still rises at K=5 (0.817); adding a fifth teacher brings cost without benefit on the held-out domain.Source: §4.3 · p. 3, Figure 2 · p. 3

4.5. System-level behaviour on mos260

• Across 38 mos260 conditions, human MOS and ProxyMOS means agree at ρ = 0.951 and r = 0.948; the student reproduces the human ordering of all six synthesis families exactly.Source: §4.5 · p. 3

4.4. Distilled student

• ρ = 0.802 on URGENT, above every teacher, while ProxyMOS reaches ρ = 0.636 on mos260 against 0.613 for Distill-MOS.The student retains 67% of the routed ensemble’s gain over the best teacher on URGENT and 100% on mos260.Source: §4.4 · p. 3, Table 2 · p. 4

4.5. System-level behaviour on mos260

• Scores compress toward the middle: Tortoise is 3.43 vs 2.94 and the natural reference 3.99 vs 4.45 (ProxyMOS vs human); intonation ρ is 0.50 vs 0.52 human-human.The paper identifies absolute calibration, rather than ranking, as the student’s weak point and says ProxyMOS tracks acoustic quality rather than prosody.Source: §4.5 · p. 4

4.6. Limitations

• mos260 covers one language, and a median of only three surviving quality ratings per utterance limits utterance-level agreement.The released benchmark includes each utterance’s rater count so users can weight items accordingly; targets also inherit the ensemble’s ceiling.Source: §4.6 · p. 4

• A wav2vec 2.0 student distilled from 807k routed targets outperforms every teacher on URGENT and the new Russian TTS benchmark mos260.The student runs in one forward pass.Source: §5 · p. 4
• Combine at least three SSL-based predictors with complementary training labels and prefer a learned router when the deployment domain is unknown.Only the router survived adding a weak member; distill the ensemble rather than deploying it.Source: §5 · p. 4
• The model, code, ONNX exports, and mos260 are released; future work includes disagreement-weighted targets, more benchmark languages, and system-level URGENT agreement.Source: §5 · p. 4

COMPLIANCE WITH ETHICAL STANDARDS

• mos260 ratings came from volunteer raters who gave informed consent, and the release contains audio and aggregated scores only.The authors report no competing interests and no funding.Source: COMPLIANCE WITH ETHICAL STANDARDS · p. 5

A. CONTENTS

• The supplement adds figures, rating rubrics and filtering details, training configuration, confidence intervals, and teacher-subset results.Source: A. CONTENTS · p. 6

B. ADDITIONAL FIGURES

• Supplementary figures show teacher confidence intervals, teacher–ensemble–student accuracy, prediction distributions, aligned and raw prediction densities, and top uniform ensembles.Source: B. ADDITIONAL FIGURES · p. 6

• Table S1 lists human quality scores, intonation scores, and ProxyMOS scores for every synthesis condition.Source: C. mos260 PER CONDITION · p. 6
• Fig. S1 plots the condition means against each other.Source: C. mos260 PER CONDITION · p. 6
• The per-condition table includes both human ratings and ProxyMOS predictions.It reports quality and intonation scores alongside ProxyMOS scores.Source: C. mos260 PER CONDITION · p. 6

• The MOS rubric ranges from studio-quality speech to non-speech, while the IntMOS rubric ranges from natural conversational delivery to unintelligible intonation.The rubric assigns scores from 0 to 5.Source: D. mos260 RATING PROTOCOL · p. 6
• Fig. S2 reports Spearman correlations with 95% confidence intervals for every model on URGENT and mos260.The student’s rows are shaded.Source: Figure S2 · p. 6
• Fig. S3 compares the best single teacher, the W+D+U+X ensemble under uniform and routed aggregation, and the student on both benchmarks.Source: Figure S3 · p. 6
• Figs. S4 and S5 show ProxyMOS prediction distributions and compare z-score-aligned model predictions with human MOS distributions.The aligned-density comparison covers URGENT and mos260.Source: Figure S4 · p. 6, Figure S5 · p. 7
• Raters were filtered using agreement, rating variance, and bias thresholds; MOS was recomputed after excluded raters’ scores were discarded.IntMOS ratings were not filtered.Source: D. mos260 RATING PROTOCOL · pp. 6–7
• Raw prediction densities show HuBERT-MOS, DNSMOS, and MOSNet collapsing to a narrow band, alongside near-zero correlation.Source: Figure S6 · p. 7

• Table S2 records the student's architecture, optimization, and training data.Source: E. ProxyMOS TRAINING CONFIGURATION · p. 7, Table S2 · p. 7
• The student is trained on targets from the routed W+D+U+X ensemble scores.Source: E. ProxyMOS TRAINING CONFIGURATION · p. 7
• Fig. S7 compares score distributions of the five best uniform-mean ensembles after z-score alignment against human MOS.Source: Figure S7 · p. 8

• Tables S4–S8 list 25 of 26 teacher subsets for each aggregation rule, with benchmark sides sorted independently by Spearman rho.The search report gives the top 10 and next 15 subsets; row indices therefore do not identify the same subset across benchmarks.Source: F. CONFIDENCE INTERVALS AND ALL… · p. 8
• Fig. S8 shows mos260 score distributions for the five best uniform-mean ensembles after z-score alignment against human MOS.Source: Figure S8 · p. 9
• Table S1 lists human quality and intonation scores and ProxyMOS scores for each mos260 synthesis condition, with in-house configurations pooled by training stage.Source: Table S1 · p. 9
• Table S3 reports Spearman correlations with 95% confidence intervals for every model on both benchmarks.Source: F. CONFIDENCE INTERVALS AND ALL… · p. 8, Table S3 · p. 10

URGENT

• Table S6 lists 25 of 26 teacher subsets under the 1/RMSE-weighted rule, sorted by their own Spearman rho.Source: Table S6 · p. 11
• Table S7 lists 25 of 26 teacher subsets under the MSE-optimised rule, sorted by their own Spearman rho.Source: Table S7 · p. 12
• Table S8 lists 25 of 26 teacher subsets under the adaptive-router rule, sorted by their own Spearman rho.Source: Table S8 · p. 12