Source-linked AI summary
MOSNet: Deep Learning based Objective Assessment for Voice Conversion
Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, Hsin-Min Wang
TL;DR
Existing VC objective metrics may not align with human judgments, while subjective naturalness and similarity ratings are expensive to collect. MOSNet uses deep neural architectures trained on large-scale VCC 2018 evaluations, achieving high system-level and fair utterance-level MOS correlation, with fair similarity-score correlation after modification.
Problem
Existing VC objective metrics are not always correlated with human perception, while subjective MOS and similarity evaluations are time-consuming and expensive.
Method
MOSNet uses CNN, BLSTM, and CNN-BLSTM architectures with fully connected layers to predict human naturalness ratings from speech.
Results
MOSNet shows high correlation with human MOS ratings at the system level and fair correlation at the utterance level, while its similarity variant is fairly correlated with human ratings.
Takeaways & Limitations
MOSNet could serve as a computational evaluator of VC-system naturalness and similarity, reducing reliance on expensive human ratings.
Takeaways & Limitations
MSE-based MOSNet predictions avoid some low and high scores and concentrate around the central MOS range, reflecting the Gaussian score distribution.
Abstract
from arXiv · showhide
Existing objective evaluation metrics for voice conversion (VC) are not always correlated with human perception. Therefore, training VC models with such criteria may not effectively improve naturalness and similarity of converted speech. In this paper, we propose deep learning-based assessment models to predict human ratings of converted speech. We adopt the convolutional and recurrent neural network models to build a mean opinion score (MOS) predictor, termed as MOSNet. The proposed models are tested on large-scale listening test results of the Voice Conversion Challenge (VCC) 2018. Experimental results show that the predicted scores of the proposed MOSNet are highly correlated with human MOS ratings at the system level while being fairly correlated with human MOS ratings at the utterance level. Meanwhile, we have modified MOSNet to predict the similarity scores, and the preliminary results show that the predicted scores are also fairly correlated with human ratings. These results confirm that the proposed models could be used as a computational evaluator to measure the MOS of VC systems to reduce the need for expensive human rating.
1. Introduction
VC evaluation commonly contrasts inexpensive objective metrics with costly human ratings, but objective measures may not track perceived naturalness and similarity. MOSNet addresses this gap with end-to-end deep neural predictors evaluated on VCC 2018 ratings.
- Motivation: Objective metrics such as MCD measure converted-speech quality automatically but may correlate poorly with human perception.They mainly quantify acoustic-feature distortion, whereas MOS and similarity scores represent perceived naturalness and speaker similarity.
- Motivation: Human MOS and similarity evaluations capture perceptual qualities but require many participants, making them time-consuming and expensive.
- Related work: Prior work includes intrusive and non-intrusive assessment models for enhanced and synthetic speech, including Quality-Net and neural naturalness predictors.Intrusive methods require a golden reference, while non-intrusive approaches predict quality without one.
- Proposed approach: MOSNet combines CNN, BLSTM, or CNN-BLSTM feature extraction with fully connected layers to predict speech naturalness from speech inputs.The architectures were selected for their ability to model perceptual features, temporal dependencies, and sequential characteristics.
- Reported findings: MOSNet achieves high system-level and fair utterance-level correlation with human MOS ratings, while its similarity-score variant is fairly correlated with human ratings.The models use large-scale VCC 2018 human evaluations as training and evaluation data.
2. The Voice Conversion Challenge Evaluation Data
VCC 2018 provides large-scale naturalness and similarity ratings for converted speech from diverse participating systems. The data show strong system-level consistency but greater utterance-level variability among listeners.
- Challenge data: VCC 2018 evaluated parallel and non-parallel VC systems using crowdsourced naturalness and speaker-similarity listening tests.The corpus selected professional US English speakers from the clean, noise-free DAPS dataset.
- Evaluation scale: 113,168 human evaluations covered 28,292 submitted audio samples, with each sample rated by four listeners.The evaluations comprised 82,304 naturalness assessments and 30,864 speaker-similarity assessments.
- Rating distribution: Mean utterance-level MOS values cluster around 3.0, while about half of submitted utterances have rating standard deviations above 1.The variation reflects listeners’ personal experiences and preferences during listening tests.
- Predictability: Listener ratings are more consistent and predictable at the system level than at the utterance level across LCC, SRCC, and MSE.Utterance-level MOS can be predicted only to a limited extent because subjective ratings vary between listeners.
3. MOSNet
MOSNet predicts speech naturalness from raw magnitude spectrograms using CNN, BLSTM, or combined CNN-BLSTM feature extractors followed by fully connected layers. Its training objective combines utterance-level and frame-level mean squared errors, while model configurations vary across these architectures.
- Model overview: MOSNet uses raw magnitude spectrograms as input and fully connected layers to generate predicted MOS.The models extract features through neural network architectures before prediction.
- Model architectures: The evaluated feature extractors are CNN, BLSTM, and CNN-BLSTM architectures.CNN-BLSTM combines convolutional and bidirectional recurrent processing for feature extraction.
- Model architectures: CNN expands its receptive field through stacked convolutional layers, while BLSTM integrates forward and backward temporal dependencies.The CNN configuration uses 12 convolutional layers with a final-neuron receptive field of 25 frames, approximately 400 ms.
- Objective function: MOS prediction is formulated as a regression task using spectral-feature sequences from speech utterances and VCC 2018 MOS evaluations as ground truth.The input is a sequence of spectral features extracted from each utterance.
- Model architectures: Table 2 specifies CNN, BLSTM, and CNN-BLSTM configurations using convolutional receptive-field size, channel count, stride, and frame count N.ReLU activations after convolutional layers are omitted from the table for brevity.
- Objective function: The objective combines utterance-level MSE with frame-level MSE, using α = 1 for the frame-level term.The ground-truth utterance MOS is used for every frame when computing frame-level MSE.
4. Experiments
MOSNet was evaluated across model architectures, prediction levels, frame-level objectives, and similarity scoring. CNN-BLSTM64 performed strongly at the system level, generalized across VCC datasets, and benefited from frame-level MSE.
- Comparison of different model architectures: CNN-BLSTM64 achieved the best utterance-level LCC of 0.624 and improved over the BLSTM1 baseline's 0.511.CNN-based models were more stable as batch size increased, while BLSTM1 degraded with larger batches.
- Evaluated results at the system level: CNN-BLSTM64 achieved a system-level LCC of 0.957, close to the human-evaluation LCC of 0.994.System-level predictions were highly correlated with human evaluations in VCC 2018.
- Evaluated results at the utterance level: Utterance-level predictions avoided some low and high MOS values, reflecting a limitation associated with MSE-based objectives.Predictions were mainly concentrated around MOS values from 2 to 3 and avoided many values between 3.5 and 4.
- Evaluated results at the system level: MOSNet achieved a system-level correlation of 0.917 when trained on VCC 2018 and tested on VCC 2016.The VCC 2016 speakers and participating systems differed from those in VCC 2018, creating a mismatched training-testing scenario.
- The effect of the frame-level MSE: With frame-level MSE, utterance-level LCC was 0.6424, versus 0.5604 without it.Training without frame-level MSE produced unstable frame-wise predictions that affected final utterance-level predictions.
- Experiments on similarity prediction: The modified MOSNet achieved 69.6% accuracy with scalar output, compared with 66.9% for the two-class model, and fairly correlated with human similarity ratings.Similarity labels merged scores 1 and 2 as same speaker and scores 3 and 4 as different speaker.
5. Conclusions
The paper presents MOSNet as a deep learning-based objective assessment model for voice conversion. It reports high system-level and fair utterance-level correlation with human MOS ratings, generalization from VCC 2018 to VCC 2016, and fair similarity prediction after modification.
- Conclusions: MOSNet is a deep learning-based quality assessment model for voice conversion trained on large-scale VCC 2018 human MOS evaluations.The paper describes it as the first end-to-end speech objective assessment model for VC.
- Conclusions: MOSNet predictions had high correlation with human ratings at the system level and fair correlation at the utterance level.These results were obtained for voice-conversion quality assessment.
- Conclusions: Applying the VCC 2018-trained model to VCC 2016 demonstrated decent generalization capability.The evaluation used a different challenge dataset from the training data.
- Conclusions: A slight modification enabled MOSNet to fairly predict converted-speech similarity relative to target speech.The paper identifies improved architecture and objective functions as future directions for stronger correlation with human ratings.