Source-linked AI summary
ConformalMOS: Uncertainty-Aware MOS Prediction with Conformal Intervals and Ordinal Modeling
Kehinde Elelu, Joshua E. Siegel, Mohammadali Saffary, Tashfain Ahmed, Simeon Babatunde, Ebuka Okpala
TL;DR
MOS predictors often provide point estimates without uncertainty, despite human-rating variability and the challenge of reliable estimation. ConformalMOS combines ordinal-aware MOS prediction with conformal intervals; on BVCC, M2D2 achieves system-level MSE of 0.08, a 7.0% reduction from FUSE-MOS, with empirical coverage matching nominal targets.
Problem
Most MOS predictors report point estimates without quantifying uncertainty, while ratings are subjective and vary with acoustic conditions and listener bias.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The framework supplies reliability-aware MOS predictions with interval coverage guarantees under exchangeability.
Source: §1 · p. 1, §2.3 · p. 2Evaluation is limited to in-domain BVCC, leaving out-of-domain robustness untested.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Accurate prediction of human Mean Opinion Scores (MOS) is crucial for evaluating synthetic speech quality in text-to-speech (TTS) and voice conversion (VC). Most MOS predictors output only point estimates and ignore uncertainty, limiting reliability in model selection and deployment. We propose ConformalMOS, a framework that augments MOS prediction with conformal prediction to produce statistically valid prediction intervals with guaranteed coverage under exchangeability, alongside point estimates. During training, MOS labels are modeled in an ordinal-aware manner by converting one-hot targets into Gaussian-smoothed distributions, preserving score ordering. This improves uncertainty calibration, particularly when softmax confidence is overconfident on out-of-distribution speech. On BVCC datasets, ConformalMOS achieves competitive accuracy (MSE = 0.08) while delivering empirically validated coverage, enhancing reliability for real-world TTS and VC deployment.
1. Introduction
Human MOS prediction is difficult because ratings are subjective and noisy, while most predictors omit uncertainty estimates and existing probabilistic approaches lack formal interval-coverage guarantees. ConformalMOS combines conformal prediction with ordinal-aware label modeling and reports improved BVCC accuracy alongside calibrated uncertainty.
1. Introduction
- MOS ratings are subjective and vary with noise, accents, and listener bias, while even state-of-the-art systems show only moderate utterance-level correlation with human scores [3] [4].These factors make reliable automatic MOS estimation challenging.Source: §1 · p. 1
- Existing MOS predictors typically return point estimates without uncertainty, and some probabilistic approaches lack formal guarantees that intervals contain the true MOS.This limits reliability assessment in real-world conditions with unseen acoustic variations.Source: §1 · p. 1
- MSE = 0.08 on BVCC, a 7.0% reduction from FUSE-MOS, for the M2D2 backbone with α = 0.05, alongside calibrated uncertainty for low-confidence and out-of-distribution inputs.ConformalMOS uses conformal intervals [10] and Gaussian-smoothed ordinal labels [11].Source: §1 · p. 1
2. Methodology
ConformalMOS uses pretrained audio representations and an ordinal prediction head to produce point MOS estimates, then applies split conformal calibration to generate finite-sample valid intervals. Its training targets preserve score ordering through Gaussian smoothing, and calibration residuals determine interval width.
2. Methodology
- ConformalMOS combines an ordinal prediction head for point MOS estimates with post-hoc conformal calibration for statistically valid prediction intervals.The framework processes audio representations through a frozen upstream extractor before the prediction head.Source: §2 · p. 1, Figure 1 · p. 2
2.1. Upstream Model Extraction
- The upstream encoder maps waveform x to embeddings z, with temporal mean pooling producing a fixed-dimensional representation; the encoder remains frozen during training.This preserves pretrained knowledge and reduces overfitting.Source: §2.1 · p. 2
2.2. Prediction Head
- MOS is discretized into equal-width bins, and Gaussian-smoothed targets encode ordinal structure by giving neighboring scores partial correctness.The scalar MOS estimate is the expected value of the bin centers; training minimizes KL divergence with an auxiliary ℓ1 loss for stability.Source: §2.2 · p. 2
2.3. Conformal Calibration
- Split conformal calibration uses absolute residuals from a separate calibration set to define a scalar half-width, producing intervals clipped to the MOS range.The intervals satisfy finite-sample coverage under exchangeability; inference adds and subtracts the half-width from the point estimate.Source: §2.3 · p. 2
3. Experimental Setup
Experiments use the BVCC speech MOS dataset with standard VoiceMOS Challenge partitions and a separate calibration subset. The study evaluates point-prediction quality and conformal interval coverage, calibration, and width.
3.1. Dataset Preparation
- BVCC contains 7,106 English audio samples from 187 TTS and VC systems; the split reserves 1,066 samples each for testing and validation, plus 497 calibration samples.The remaining 4,477 development samples are used for model training.Source: §3.1 · pp. 2–3
3.2. Training Details
- Models train on frozen upstream encoders for up to 1,000 epochs with early stopping, using SGD with momentum and batch size 32.Most runs stopped after 300–400 epochs; training typically took 2 hours per run.Source: §3.2 · p. 3
3.3. Evaluation Metrics
- Point-prediction quality is assessed with MSE, LCC, SRCC, and Kendall Tau at both utterance and system levels.These metrics measure error, linear association, monotonic ordering, and ordinal association, respectively.Source: §3.3 · p. 3
- Interval quality is assessed using coverage against nominal 1 − α, calibration error, average interval width, and sharpness measured by RMS radius.Coverage is the proportion of test samples whose ground-truth MOS falls within the predicted interval.Source: §3.3 · p. 3
4. Experimental Results
On BVCC, ConformalMOS with M2D2 performs strongly at the system level and produces better-calibrated intervals than the Wav2Vec backbone. Interval width trades off against coverage, and evaluation is limited to in-domain data.
4.1. Evaluating ConformalMOS on BVCC
- ConformalMOS M2D2 models achieve the highest system-level LCC while maintaining the lowest MSE; the comparison is summarized in Table 1.The best-performing configuration is reported to surpass baselines at the system level while retaining competitive utterance-level accuracy.Source: Table 1 · p. 3, §4.1 · p. 4
- The evaluation uses in-domain BVCC data, leaving out-of-domain robustness untested; heuristic comparisons are omitted because they lack conformal prediction’s finite-sample coverage guarantee.Source: §4.1 · p. 4
- At α = 0.05, corresponding to 95% nominal coverage, the study selects a setting balancing interval width and reliability.Larger α narrows intervals but raises undercoverage risk; smaller α yields wider intervals for higher nominal coverage.Source: §4.1 · p. 4
- M2D2 retains strong utterance-level correlations close to UTMOS and FUSE-MOS, with a slight MSE increase; Wav2Vec underperforms across metrics.Source: §4.1 · p. 4
4.2. Backbone Effects on Conformal-MOS Calibration
- M2D2 coverage closely matches nominal targets with low calibration error, whereas Wav2Vec has larger errors, undercoverage at low α, and wider intervals.The Wav2Vec empirical coverage falls short of the 0.99 target at α = 0.01.Source: §4.2 · p. 4, Figure 2 · p. 4
5. Conclusion
ConformalMOS combines ordinal modeling with split conformal calibration for MOS prediction. BVCC experiments report improved system-level performance with competitive utterance-level accuracy, while calibration quality varies by backbone.
- ConformalMOS combines ordinal modeling and split conformal calibration to produce MOS prediction intervals with formal coverage guarantees.Source: §5 · p. 4
- On BVCC, the framework improves system-level performance while maintaining competitive utterance-level accuracy; M2D2 intervals match nominal coverage better than Wav2Vec intervals.Source: §5 · p. 4
- The reported coverage–sharpness trade-off is that increasing α narrows intervals but reduces coverage, and listener disagreement and rater-specific effects remain future work.Source: §5 · p. 4
7. Use of Generative AI Disclosure
The disclosure limits generative AI use to language editing and writing refinement. The authors state that they produced, reviewed, and verified the scientific work.