Source-linked AI summary
Non-intrusive speech quality assessment using neural networks
Anderson R. Avila, Hannes Gamper, Chandan Reddy, Ross Cutler, Ivan Tashev, Johannes Gehrke
TL;DR
Subjective speech-quality assessment is accurate but costly and difficult to perform in real time, while established measures have limited performance on noise and reverberation. The paper evaluates three neural MOS estimators on a crowd-labeled speech dataset; all outperform the instrumental measures, with the Mel-frequency DNN approach strongest.
Problem
Subjective listening tests are costly and cannot be done in real time, while established measures developed for codec distortions perform poorly with noise, reverberation, or newer enhancement processing.
Source: §1 · p. 1Method
The authors create a crowd-labeled speech dataset and evaluate three non-intrusive neural MOS estimators using constant-Q, i-vector, and Mel-frequency features.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Speech-quality prediction can be useful for network design and development and for monitoring and improving customers’ QoE.
Source: §1 · p. 1The authors reserve evaluation on an extended dataset with network impairments for future work.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Estimating the perceived quality of an audio signal is critical for many multimedia and audio processing systems. Providers strive to offer optimal and reliable services in order to increase the user quality of experience (QoE). In this work, we present an investigation of the applicability of neural networks for non-intrusive audio quality assessment. We propose three neural network-based approaches for mean opinion score (MOS) estimation. We compare our results to three instrumental measures: the perceptual evaluation of speech quality (PESQ), the ITU-T Recommendation P.563, and the speech-to-reverberation energy ratio. Our evaluation uses a speech dataset contaminated with convolutive and additive noise, labeled using a crowd-based QoE evaluation, evaluated with Pearson correlation with MOS labels, and mean-squared-error of the estimated MOS. Our proposed approaches outperform the aforementioned instrumental measures, with a fully connected deep neural network using Mel-frequency features providing the best correlation (0.87) and the lowest mean squared error (0.15)
1. INTRODUCTION
The introduction motivates non-intrusive MOS estimation because subjective tests are burdensome and established measures have limited distortion coverage. It presents a crowd-labeled dataset and three neural approaches compared with instrumental measures.
1. INTRODUCTION
- Subjective listening tests are described as the most accurate quality assessment, but they require many listeners, time, expense, and cannot be done in real time.MOS averages participants’ scores on a 1-to-5 scale.Source: §1 · p. 1
- PESQ and POLQA were developed for codec distortions and perform poorly when speech has noise, reverberation, or newer enhancement processing.The introduction notes that many instrumental approaches also require clean reference speech, limiting them to synthetic datasets; non-intrusive methods use only corrupted speech and are expected to be less accurate.Source: §1 · p. 1
- The paper explores three neural architectures for MOS estimation and compares them with PESQ, P.563, and SRMR.The approaches use constant-Q features with a CNN, i-vectors with a DNN, and Mel-frequency features with a DNN.Source: §1 · pp. 1–2
2. THE AUDIO QUALITY EVALUATION DATASET
The dataset combines speech with room reverberation, additive noise, and processing, then uses crowd ratings to estimate perceived quality. Its 10,000 samples represent varied acoustic conditions.
2. THE AUDIO QUALITY EVALUATION DATASET
- The 10,000-sample dataset was generated from 2,010 clean speech files, with 670 each from male, female, and child speakers.The clean files were approximately 20 seconds long and sampled at 16 kHz.Source: §2 · p. 2
- Samples were convolved with room responses, mixed with office, home, or other noise, and half were processed by noise suppression and automatic gain control.The room-response library contained 120 RIRs; resulting SNRs were limited to 0–50 dB.Source: §2 · p. 2
- Ten judges rated each sample after training and qualification, and their scores were averaged to compute MOS.Fig. 2 summarizes the distortion and labeling procedure.Source: §2 · p. 2, Figure 2 · p. 3
- In the reported histograms of computed PESQ and averaged MOS, it is well visible that PESQ gives lower scores than human judges.Source: §2 · p. 3
3. PROPOSED FEATURES
The section presents three input representations for MOS estimation: constant-Q spectral features, i-vectors in total variability space, and Mel-frequency features.
3. PROPOSED FEATURES
- The constant Q transform provides a perceptually motivated spectral representation with improved time-frequency resolution and a fixed 240x220 input for the CNN.Short utterances are padded by replicating the last frame, while excess frames from long utterances are removed.Source: §3 · p. 3
- The i-vector framework maps utterance feature sequences into fixed-length vectors, projecting speaker and channel variability into a shared total variability subspace.The experiments use 400-dimensional i-vectors.Source: §3 · p. 3
- Mel-frequency inputs combine 26 MFCCs with pitch, VAD output, log-power energy, and first derivatives across a 25-frame context, producing a 1 × 1450 vector.Features are extracted from 512-sample frames with a 160-sample step at 16 kHz.Source: §3 · p. 3
4. PROPOSED NEURAL NETWORK MODELS
The paper evaluates convolutional and fully connected neural networks on the proposed feature inputs, with an additional approach for aggregating predictions across variable-duration audio.
4. PROPOSED NEURAL NETWORK MODELS
- The proposed CNN processes the constant-Q spectral input through convolutional and pooling layers, followed by a fully connected layer and an output unit.Its architecture uses two convolutional layers with 32 filters each, followed by two layers with 64 filters each.Source: §4 · pp. 3–4
- The paper uses two fully connected DNN architectures: one with 400 input units and hidden layers of 200 and 100 units, and one with four 1024-unit hidden layers for 1 × 1450 inputs.Both architectures use Adam optimization.Source: §4 · p. 4
- For variable-duration audio, the paper describes repeated fixed-length feature extraction with timeline statistics and adopts an extreme learning machine as an additional classifier.The mean or mode can summarize features across the timeline.Source: §4 · p. 4
5. EXPERIMENTAL SETUP
The experiment compares proposed MOS estimators with three speech-quality measures using correlation and squared error against subjective test-set MOS.
5. EXPERIMENTAL SETUP
- PESQ, P.563, and SRMR serve as benchmarks, including both intrusive and non-intrusive measures.SRMR is described as a candidate that has outperformed PESQ and P.563 especially for reverberant and dereverberated speech.Source: §5 · p. 4
- Performance is assessed using Pearson correlation and mean squared error between test-set MOS estimates and subjective MOS scores.The data split is 70% training, 15% validation and hyperparameter optimization, and 15% testing.Source: §5 · p. 4
6. EXPERIMENTAL RESULTS
All proposed models outperform the instrumental benchmarks, and the Mel-frequency DNN with ELM has the strongest reported results.
6. EXPERIMENTAL RESULTS
- 0.87 Pearson correlation and 0.15 MSE are achieved by Mel-frequency+DNN+ELM, outperforming P.563 at 0.55 correlation and 0.31 MSE.The paper also reports that the DNN+ELM approach surpasses PESQ and that all proposed models outperform the benchmarks.Source: Table 1 · p. 4, §6 · p. 4
- The results table reports MOS-estimation results for the benchmarks, CNN with constant-Q spectral features, DNN with i-vector features, and DNN with Mel-frequency features.The paper says PESQ’s distribution appears more aligned with SNR than with the MOS distribution.Source: Table 1 · p. 4, §6 · p. 4
7. CONCLUSION
The three non-intrusive neural-network approaches outperform PESQ, ITU-T P.563, and SRMR; the fully connected model using Mel-frequency features performs best. The work also develops a crowd-labeled audio-quality dataset and identifies future evaluation and training directions.
7. CONCLUSION
- All three proposed approaches outperform PESQ, ITU-T P.563, and SRMR; the fully connected model with Mel-frequency features has the best correlations and lowest mean squared errors.The i-vector DNN ranks next, followed by the constant-Q-spectrum CNN.Source: §7 · p. 4
- The work develops a crowd-sourced audio-quality dataset and compares three non-intrusive MOS-estimation approaches with three instrumental measures.Source: §7 · p. 4
- Future work will evaluate the methods on an extended dataset with network impairments and train a DNN using the raw signal.Source: §7 · p. 4