PaperlayerView exact revision on arXiv

Source-linked AI summary

MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network

Yichong Leng, Xu Tan, Sheng Zhao, Frank Soong, Xiang-Yang Li, Tao Qin

arXiv:2103.00110v1cs.SDeess.AS

TL;DR

MOS prediction commonly trains on utterance averages while discarding individual judges’ scores, leaving those ratings unused despite costly and limited MOS data. MBNet combines a mean-predicting subnet with a judge-conditioned bias subnet. Compared with MOSNet, it improves system-level SRCC on both VCC datasets.

  • Problem

    Prior MOS predictors used utterance-level averages and discarded individual judge scores, while MOS tests are costly and the available training data scarce.

    Source: §1 · p. 1

  • Method

    MBNet combines a mean subnet predicting utterance MOS with a bias subnet that predicts each judge’s deviation using the audio and judge identity.

    Source: §1 · p. 1

  • Results

    System-level SRCC rose from 0.922 to 0.949 on VCC 2018 and from 0.862 to 0.920 on VCC 2016 versus MOSNet.

    Source: §4.1 · p. 3

  • Takeaways & Limitations

    MBNet is a general framework for using judge scores and capturing personal preferences, without specific constraints on subnet architectures.

    Source: §2.2 · p. 2, §5 · p. 4

    Each audio’s scores come from a random subset of m judges among M total judges, with M>m.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Mean opinion score (MOS) is a popular subjective metric to assess the quality of synthesized speech, and usually involves multiple human judges to evaluate each speech utterance. To reduce the labor cost in MOS test, multiple methods have been proposed to automatically predict MOS scores. To our knowledge, for a speech utterance, all previous works only used the average of multiple scores from different judges as the training target and discarded the score of each individual judge, which did not well exploit the precious MOS training data. In this paper, we propose MBNet, a MOS predictor with a mean subnet and a bias subnet to better utilize every judge score in MOS datasets, where the mean subnet is used to predict the mean score of each utterance similar to that in previous works, and the bias subnet to predict the bias score (the difference between the mean score and each individual judge score) and capture the personal preference of individual judges. Experiments show that compared with MOSNet baseline that only leverages mean score for training, MBNet improves the system-level spearmans rank correlation co-efficient (SRCC) by 2.9% on VCC 2018 dataset and 6.7% on VCC 2016 dataset.

1. INTRODUCTION

MOS offers an overall human assessment of synthesized-speech quality, but collecting ratings is costly and prior predictors discarded individual judge scores. MBNet uses separate mean and bias subnets to use those scores and model judges’ personal preferences.

1. INTRODUCTION

  • MOS tests are costly and time-consuming because they require human judges to rate utterances.MOS is the arithmetic mean of judges’ scores for an utterance; system-level MOS averages utterance-level MOS values for a synthesis system.Source: §1 · p. 1
  • Prior MOS predictors used utterance-level averages and discarded individual ratings, despite scarce training data and the potential to multiply examples by using each judge score.High variance in human judgments makes those individual scores challenging to use stably.Source: §1 · p. 1
  • Different judges’ rating standards can make utterance-level MOS values incomparable across utterances rated by different judge groups.The authors say training directly on such MOS values without debiasing can hurt model performance.Source: §1 · p. 1
  • MBNet pairs a mean subnet predicting utterance MOS with a bias subnet predicting each judge’s deviation, whose output combines with the mean to predict that judge’s score.The bias subnet conditions on the audio spectrum and judge ID, while the mean subnet uses the spectrum.Source: §1 · p. 1, Figure 1 · p. 2

2. MEAN-BIAS NETWORK

The section describes why averaging judge scores can waste data and produce incomparable utterance-level MOS, then presents MBNet’s mean-and-bias framework and its training refinements.

2.1. Preliminary

  • Each audio has m judge scores drawn from a random subset of M judges, where M>m; a judge’s bias score is the delta between that judge’s score and the utterance mean.The total judge pool is larger than the number rating each utterance.Source: §2.1 · p. 2
  • Prior methods trained on utterance means and left individual ratings unused; because different audios may have different judges, their means can be incomparable.The paper notes M≫m and says direct use of these incomparable means can hurt training.Source: §2.1 · p. 2

2.2. MBNet Framework

  • MeanNet predicts the utterance mean from audio spectrum, while BiasNet uses spectrum and judge ID to predict a bias that combines with the mean to produce a judge score.Joint training implicitly sets BiasNet’s target to the bias score.Source: §2.2 · p. 2
  • Clipped MSE ignores errors within threshold τ, reducing pressure to fit discrete judge scores exactly and potentially improving generalization.The loss is used for both subnet objectives; the MeanNet’s zero-loss range is treated as a bias region handled by BiasNet through joint training.Source: §2.2 · p. 2
  • Repetitive padding fills batches with replicas of the original speech, based on the premise that repetitions retain the same MOS.It is proposed to stabilize batch-normalization mean and variance estimates.Source: §2.2 · p. 2, Figure 2 · p. 2
  • MBNet is a general framework for using MOS judge scores and capturing judge preferences, without restricting MeanNet or BiasNet architectures.Source: §2.2 · p. 2

3. EXPERIMENT SETTINGS

The experiments evaluate MBNet on VCC 2018 and VCC 2016, compare it with MOSNet, and specify the model and training configurations.

3.1. Dataset

  • VCC 2018 provides 20,580 audios from 38 systems, scored by 267 expert judges with four judges per audio; VCC 2016 tests generalization across 26,028 audios from 20 systems.The VCC 2018 split contains 13,580 training, 3,000 validation, and 4,000 test audios; for VCC 2016 testing, the VCC 2018 training and test sets are combined into a 17,580-audio training set.Source: §3.1 · pp. 2–3
  • Audios are down-sampled to 16 kHz and converted to 257-dimensional linear spectra by STFT before being input to MBNet.Source: §3.1 · p. 3

3.2. Model Details of MBNet and Baseline System

  • MeanNet uses four convolution blocks followed by a BLSTM and dense layers, while the smaller BiasNet uses two blocks and receives a judge-ID embedding.Both subnetworks produce frame-level scores and use global mean pooling; frame-level loss stabilizes output, with τ set to 0.5 and λ to 4.0 to emphasize BiasNet.Source: §3.2 · p. 3, Figure 3 · p. 3
  • MOSNet is the CNN-BLSTM baseline, trained directly on audio and utterance MOS without individual judge scores; MBNet has 12% more parameters.The authors report that adding 12% parameters directly to MOSNet yields almost the same performance as the baseline.Source: §3.2 · p. 3

3.3. Training Configuration

  • Experiments use Adam at a 10^-4 learning rate, select the lowest-validation-loss checkpoint across 50 epochs, and average metrics over 10 random seeds.Metrics include MSE, SRCC, and LCC; VCC 2018 reports utterance- and system-level results, while VCC 2016 reports system-level results only.Source: §3.3 · p. 3

4. RESULTS AND ANALYSES

MBNet improves on MOSNet on VCC 2018 and VCC 2016, including the system-level SRCC measure identified as most important for practical use. The analyses examine subnet roles, training choices, and the effect of judge identity.

4.1. MOS Prediction Results

  • 0.922 to 0.949 on VCC 2018 and 0.862 to 0.920 on VCC 2016: MBNet improves system-level SRCC over MOSNet.The paper identifies system-level SRCC as the most important metric for comparing speech-synthesis systems in practice.Source: §4.1 · p. 3, Table 1 · p. 4

4.2. Comparison with Other Methods

  • On VCC 2016, MBNet surpasses MOSNet variants using MTL, FL, GQT, and EL with 15% less training data.The comparison uses fewer training audios for MBNet than those systems.Source: §4.2 · pp. 3–4, Table 2 · p. 4

4.3. Ablation Study

  • Removing BiasNet hurts joint-trained MeanNet; MeanNet alone exceeds MOSNet, while BiasNet alone overfits to judges, supporting MeanNet’s role in generalization.The ablation results are reported using SRCC.Source: §4.3 · p. 4, Table 3 · p. 4
  • Repetitive padding outperforms zero-padding; clipped MSE may reduce utterance-level performance but improves system-level results by preventing overfitting to exact MOS.The ablation table reports SRCC only.Source: §4.3 · p. 4, Table 3 · p. 4

4.4. Bias Learned in BiasNet

  • 0.647 to 0.740 with correct judges, but 0.647 to 0.527 with random judges: utterance-level SRCC changes substantially with judge identity.The tested conditions are MeanNet alone, MeanNet plus BiasNet with random judges, and MeanNet plus BiasNet with the four correct judges.Source: Table 4 · p. 4, §4.4 · p. 4

5. CONCLUSION

MBNet combines a mean-predicting MeanNet with a BiasNet that captures judge-specific score deltas, without restricting the subnets’ architectures. VCC experiments show improvements over previous MOS prediction systems; future work includes stronger backbones and other MOS variations.

5. CONCLUSION

  • Experiments on the VCC dataset show significant improvements over previous MOS prediction systems.Source: §5 · p. 4
  • MBNet pairs MeanNet, which predicts each utterance’s mean, with BiasNet, which captures the delta to each specific judgement score.Source: §5 · p. 4
  • The MBNet framework has no restrictions on the architectures of MeanNet and BiasNet and fully utilizes the limited MOS dataset while capturing judge preferences.Source: §5 · p. 4
  • Future work will combine the method with a better MOS prediction backbone and consider predicting other MOS variations, including CMOS and SMOS.Source: §5 · p. 4
Loading 2103.00110v1…