PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech Quality Assessment through MOS using Non-Matching References

Pranay Manocha, Anurag Kumar

arXiv:2206.12285v1eess.AScs.SD

TL;DR

Human MOS listening tests are costly, while automated MOS estimation is challenged by limited domain coverage and noisy labels. NORESQA-MOS estimates quality relative to non-matching references, and the paper reports stronger out-of-domain generalization than prior works trained on larger datasets. The authors also report that the approach can work with smaller models and combine with SSL pretraining.

  • Problem

    Human MOS testing is time- and resource-consuming, while domain-limited datasets and noisy ratings make it difficult to generalize across conditions and applications.

    Source: §1 · p. 1

  • Method

    NORESQA-MOS conditions MOS estimation on randomly chosen non-matching references and jointly predicts preference and relative MOS.

    Source: §1 · p. 1, §2 · p. 1, §2.2 · p. 2

  • Results

    NORESQA-MOS generalizes to out-of-domain datasets and outperforms prior works trained on much larger datasets.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The framework can be combined with SSL pretraining and provide good generalization with smaller models, making it more suitable for real-world use.

    Source: §1 · p. 1, §5 · p. 4

    The method assumes access to a small labeled MOS dataset and a clean speech database, whose recordings are treated as having perfect MOS.

    Source: §2.3 · p. 2

Abstract

from arXiv · show

Human judgments obtained through Mean Opinion Scores (MOS) are the most reliable way to assess the quality of speech signals. However, several recent attempts to automatically estimate MOS using deep learning approaches lack robustness and generalization capabilities, limiting their use in real-world applications. In this work, we present a novel framework, NORESQA-MOS, for estimating the MOS of a speech signal. Unlike prior works, our approach uses non-matching references as a form of conditioning to ground the MOS estimation by neural networks. We show that NORESQA-MOS provides better generalization and more robust MOS estimation than previous state-of-the-art methods such as DNSMOS and NISQA, even though we use a smaller training set. Moreover, we also show that our generic framework can be combined with other learning methods such as self-supervised learning and can further supplement the benefits from these methods.

1. Introduction

Human listening tests provide the gold-standard MOS, but they are costly and constrained by testing conditions. Existing automated approaches face limits in reference availability, domain coverage, and generalization; NORESQA-MOS uses non-matching references to address MOS estimation and reports stronger generalization.

1. Introduction

  • Human MOS listening tests are costly, difficult to scale, and require controlled listening environments and hardware.MOS is described as the de-facto metric for speech quality assessment through listening tests.Source: §1 · p. 1
  • Full-reference objective metrics require clean reference speech, correlate only to a limited degree with MOS, and can become obsolete in new scenarios.Examples include PESQ, POLQA, and VISQOL.Source: §1 · p. 1
  • MOS models trained on labeled datasets face limited domain coverage and noisy crowd-sourced ratings, complicating generalization across conditions and applications.The cited datasets often focus on particular domains, including TTS and voice conversion, telephony, or speech enhancement.Source: §1 · p. 1
  • NORESQA-MOS conditions MOS estimation on random non-matching references of known quality and reports high generalization across in-domain and out-of-domain datasets.The framework can use references from a labeled dataset or a clean set.Source: §1 · p. 1
  • Combining NORESQA-MOS with approaches such as SSL pretraining can provide computational benefits while improving generalization with smaller models.The paper presents the framework as usable with arbitrary speech recordings as NMR inputs.Source: §1 · p. 1

2. The NORESQA-MOS Framework

NORESQA-MOS assesses a test recording relative to a randomly chosen non-matching reference. Its shared model learns preference and relative-MOS outputs, is trained using labeled and optionally clean recordings, and estimates absolute quality by averaging over clean references.

2. The NORESQA-MOS Framework

  • Given a test recording and a randomly chosen reference, NORESQA-MOS predicts which recording is cleaner and their relative MOS rating.The reference need not match the test recording's content.Source: §2 · p. 1, Figure 1 · p. 2

2.1. Framework Design and Model Architectures

  • The architecture uses a shared base model, a downsampling block, and task-specific preference and relative-MOS output heads.Base models include a scratch-trained model and pretrained wav2vec2.0 models of different capacities; the downsampling block produces 32-dimensional frame representations.Source: §2.1 · p. 2

2.2. Training Tasks and Loss Functions

  • Multi-task training combines a preference classification task with relative-MOS quantification, and both output heads use attention pooling.The preference task predicts which input has the higher MOS; the relative-rating task predicts the absolute difference between the two MOS ratings using L1 loss.Source: §2.2 · p. 2

2.3. Training procedure

  • Training assumes a small labeled MOS dataset and a clean speech database, with clean recordings treated as having perfect MOS.The method augments recordings with waveform inversion, reversal, or time stretching selected to have none to minimal effect on MOS ratings.Source: §2.3 · p. 2

2.4. Usage: MOS Prediction

  • At inference, the model estimates a test recording's MOS relative to multiple clean non-matching references and averages the relative ratings.The averaging is used to obtain a lower-variance estimate of MOS.Source: §2.4 · p. 2

3. Experimental Setup

The experiments use a labeled MOS dataset and a clean reference set, with evaluation on diverse speech datasets. Comparisons include established no-reference systems and a direct-MOS variant of the proposed model.

3.1. Datasets and training

  • The labeled BVCC training dataset contains roughly 7,000 recordings rated by eight independent raters, and the clean NMR set comes from DAPS.The authors use the organizers' pre-created training, development, and test splits.Source: §3.1 · p. 2
  • Models use 3-second waveform excerpts, Adam with a 10^-4 learning rate, batch size 64, and 1,000 training epochs; evaluation uses 100 NMRs.Source: §3.1 · p. 2

3.2. Baselines

  • The baselines are DNSMOS, NISQA, and a same-architecture direct-MOS model; evaluation is at 16 kHz except NISQA, which predicts MOS at 48 kHz.The direct-MOS model predicts absolute MOS rather than using the NMR-based approach.Source: §3.2 · p. 2

4. Results

NORESQA-MOS embeddings capture audio-quality information, and the method is reported to generalize across diverse datasets and model capacities. It also improves utterance-level predictions, with consistent correlations and low errors.

4.1. Objective evaluations

  • PCA visualization shows local embedding structure, while UMAP shows global structure; the plots indicate that the embeddings capture audio-quality information.The evaluations also examine whether recordings with similar MOS cluster in the learned embedding space.Source: §4.1 · p. 2, Figure 2 · p. 3

4.2. Subjective evaluations

  • Figure 3 compares system- and utterance-level predictions from D-MOS, NORESQA-MOS, DNSMOS, and NISQA across in-domain and out-of-domain BVCC tasks.The scatter plots include the y = x line and fitted slope and intercept for each comparison.Source: Figure 3 · p. 3
  • On challenging BWE datasets, NORESQA-MOS scores lower MSE and higher correlations than DNSMOS and NISQA; SSL-Small generalizes better than D-MOS SSL-Big with 1/4 the trainable parameters.The authors also report small standard deviations (∼0.02 rating) across Tables 1–3 and use a fraction of the labeled examples used by DNSMOS or NISQA.Source: Table 1 · p. 3, §4.2 · p. 4, Table 2 · p. 4
  • Across datasets, utterance-level NORESQA-MOS predictions show consistent correlations and the lowest errors, with improvements on almost all datasets.The reported standard deviations are ∼0.15 rating and are expected to decrease as more NMRs are introduced.Source: §4.2 · p. 4, Table 4 · p. 4

5. Conclusions and future work

NORESQA-MOS estimates speech-signal MOS using non-matching references and is reported to generalize to out-of-domain datasets. The authors report better performance than prior works trained on larger datasets, and good generalization with smaller models; they identify additional attributes for future work.

  • NORESQA-MOS uses non-matching references to estimate speech-signal MOS, motivated by quality assessment independent of speech content.Source: §5 · p. 4
  • The method generalizes to out-of-domain datasets and outperforms prior works trained on much larger datasets, while smaller models also generalize well.The authors describe the smaller-model results as making the method more suitable for real-world uses.Source: §5 · p. 4
  • Future work will include additional speech-quality attributes, including noisiness and coloration.Source: §5 · p. 4
Loading 2206.12285v1…