Source-linked AI summary
NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, Sebastian Möller
TL;DR
Speech-quality assessment needs practical single-ended prediction and diagnostic information beyond an overall MOS. NISQA combines CNN, Self-Attention, and Attention-Pooling in a multitask model trained across 81 datasets, and it reports reliable predictions for unknown data and real live phone calls.
Problem
Overall MOS does not reveal the cause of degradation, while subjective testing is costly and double-ended models require clean reference signals.
Method
NISQA uses an end-to-end multitask CNN–Self-Attention–Attention-Pooling network to predict MOS and Noisiness, Coloration, Discontinuity, and Loudness.
Results
NISQA outperforms other single-ended models on most datasets, predicts quality dimensions with RMSEs of 0.16–0.56, and gives reliable results on unknown data and live phone calls.
Takeaways & Limitations
Degradation decomposition provides more insight into the cause of underlying quality impairment than overall MOS alone.
Takeaways & Limitations
Comparisons involving P.563 and ANIQUE+ are limited to narrowband signals, which were downsampled to 8 kHz.
Abstract
from arXiv · showhide
In this paper, we present an update to the NISQA speech quality prediction model that is focused on distortions that occur in communication networks. In contrast to the previous version, the model is trained end-to-end and the time-dependency modelling and time-pooling is achieved through a Self-Attention mechanism. Besides overall speech quality, the model also predicts the four speech quality dimensions Noisiness, Coloration, Discontinuity, and Loudness, and in this way gives more insight into the cause of a quality degradation. Furthermore, new datasets with over 13,000 speech files were created for training and validation of the model. The model was finally tested on a new, live-talking test dataset that contains recordings of real telephone calls. Overall, NISQA was trained and evaluated on 81 datasets from different sources and showed to provide reliable predictions also for unknown speech samples. The code, model weights, and datasets are open-sourced.
1. Introduction
Speech quality is traditionally measured subjectively or predicted instrumentally, but existing approaches have important practical and diagnostic limitations. NISQA addresses these limitations by predicting overall quality and four perceptual dimensions using an end-to-end single-ended model trained on extensive subjective data.
- Motivation: Subjective speech-quality testing yields mean opinion scores but is costly and time consuming.Instrumental models were developed to automate prediction from speech signals.
- Motivation: Single-ended models require only degraded speech, enabling live-call monitoring and prediction without a clean reference.This contrasts with double-ended models such as POLQA, which compare degraded output with a clean reference.
- Motivation: Overall MOS indicates quality severity but does not identify the cause of degradation.The four dimensions are Noisiness, Coloration, Discontinuity, and Loudness, which are linked to technical root causes.
- Contribution: NISQA predicts overall MOS and four quality dimensions with one multitask neural network.The update uses subjective data only, trains end-to-end, and replaces the CNN-LSTM structure with a CNN–Self-Attention–Attention-Pooling network.
- Contribution: The model is trained and evaluated on 81 datasets from different sources.The paper also introduces eight new speech-quality datasets with subjective MOS and dimension ratings.
2. Method
NISQA processes segmented Mel-spectrograms through a CNN, Self-Attention time-dependency model, and Attention-Pooling stage. Shared feature extraction supports five prediction tasks, while attention-based pooling aggregates temporally weighted information for speech-quality estimation.
- Architecture: NISQA has four stages: Mel-Spec segmentation, a CNN framewise model, Self-Attention time-dependency modeling, and Attention-Pooling.Mel-spectrogram segments are converted into framewise features, temporally modeled, and pooled for quality prediction.
- Architecture: The selected architecture combines a CNN, Self-Attention, and Attention-Pooling based on comparisons of alternatives.The ablation study identifies this combination as the best-performing configuration among the tested stage designs.
- Input and framewise processing: Each 150 ms Mel-spectrogram segment uses 48 Melbands and overlaps neighboring segments by 73%.A 10-second speech signal produces approximately 250 segments with a 40 ms segment hop.
- Input and framewise processing: The CNN converts each 48 × 15 Mel-spectrogram segment into a 384-dimensional feature vector.It uses six convolutional layers and three max-pooling layers, with a feedforward network serving as a baseline.
- Pooling: Attention-Pooling masks padded time steps, normalizes attention scores, and computes a weighted average feature vector for final quality estimation.The weighted vector is passed through a fully connected layer; average- and max-pooling are baselines.
- Multidimensional model: The CNN and Self-Attention network are shared across tasks, while five separate pooling blocks predict MOS and four dimension scores.The five outputs correspond to overall MOS, Noisiness, Coloration, Discontinuity, and Loudness.
3. Datasets
NISQA uses training, validation, and test data assembled from multiple source pools, including eight newly created datasets. Independent tests include simulated network distortions, VoIP scenarios, and real conversational telephone calls.
- Dataset composition: The study includes 59 training datasets with 72,903 files, 18 validation sets with 9,567 files, and four test sets with 952 files.The sources include the POLQA Pool, the ITU-T P Suppl. 23 pool, older internal datasets, and eight newly created datasets.
- Dataset composition: Eight new datasets were created with overall-quality and quality-dimension ratings from varied speakers.These datasets expanded the subjective-data basis for model training and evaluation.
- Independent testing: Three independent test sets combine simulated distortions with live VoIP calls using Zoom, Skype, Google Meet, WhatsApp, and Discord.Simulated conditions include packet loss, warping, and low-bitrate transmission.
- Independent testing: NISQA TEST LIVETALK contains real telephone-call recordings made by talkers speaking directly into smartphones or laptops.Participants produced scenarios involving loud or quiet speech, loudspeaker use, and background music.
- Independent testing: The LIVETALK dataset contains 232 files from 58 conditions and eight German talkers.Each file received 24 laboratory ratings under ITU-T P.800.
4. Ablation Study
The ablation study identifies CNN-SA-AP—a CNN framewise model, Self-Attention time-dependency model, and Attention-Pooling—as the strongest combination. Self-Attention improves over omitting time-dependency modelling, while pooling differences are marginal.
- Framewise Model: CNN-SA-AP clearly outperforms FFN-SA-AP and Skip-SA-AP in the framewise-model comparison.The compared alternatives use a feedforward framewise network or no framewise network, respectively.
- Time-Dependency Model: CNN-SA-AP achieves the best results among the evaluated time-dependency models.The difference between Self-Attention and CNN-LSTM-AP is small, whereas combining Self-Attention and LSTM worsens results relative to either alone.
- Time-Dependency Model: Self-Attention or LSTM notably improves overall performance over CNN-Skip-AP without time-dependency modelling.The difference between Self-Attention and LSTM is small.
- Pooling: Attention-Pooling slightly outperforms Average-Pooling and Max-Pooling, although the performance differences are marginal.The comparison uses CNN for framewise processing and Self-Attention for time-dependency modelling.
5. Results
NISQA was trained on 59 datasets and validated on 18, then compared with single-ended and double-ended speech-quality models. It generally outperformed other single-ended models, performed better than POLQA on conversational-speech datasets, and predicted quality dimensions with RMSEs of 0.16–0.56.
- Training: NISQA was trained on 59 datasets and validated using the average PCC across datasets, with early stopping after validation PCC failed to increase for more than 10 epochs.Training used batch size 160, learning rate 0.001, Adam, and a bias-aware loss.
- Overall MOS: NISQA outperforms other single-ended speech-quality models on most datasets in overall MOS prediction.The comparison includes P.563, ANIQUE+, WAWEnets, POLQA, DIAL, and VISQOL.
- Evaluation Scope: NISQA’s live-talk test set contains no reference signals and can therefore only be compared with single-ended models.This constrains direct comparison with double-ended models for that dataset.
- Overall MOS: NISQA achieves better results than POLQA on test datasets containing conversational speech, while POLQA performs better on most POLQA-Pool datasets.The POLQA-Pool datasets contain typical ITU-T P.800 double sentences with a silent pause.
- Quality Dimensions: NISQA outperforms DIAL on most datasets with subjective dimension ratings and achieves RMSEs of 0.16–0.56.The predicted dimensions are the speech-quality dimensions evaluated in Table 5.
6. Conclusions
NISQA predicts overall MOS and four perceptual quality dimensions using CNN, Self-Attention, and Attention-Pooling. Across 81 datasets, it provides reliable predictions on unknown data and real live phone calls, while its code, weights, and datasets are open-sourced.
- Contributions: NISQA predicts overall MOS plus Noisiness, Coloration, Discontinuity, and Loudness.This degradation decomposition provides more insight into the cause of an underlying quality impairment.
- Architecture: The model combines a CNN, Self-Attention for time-dependency modelling, and Attention-Pooling for final time pooling.This is the architecture used for the presented speech-quality model.
- Evaluation: NISQA was trained and evaluated on 81 datasets and gave reliable results on unknown data and real live phone calls.The model focuses on distortions occurring in modern speech communication networks.
- Availability: The authors open-source the code, model weights, and speech-quality datasets.The weights can also be used to fine-tune the model for related prediction tasks.