Source-linked AI summary
A Pyramid Recurrent Network for Predicting Crowdsourced Speech-Quality Ratings of Real-World Signals
Xuan Dong, Donald S. Williamson
TL;DR
Objective speech-quality measures may not reflect human judgments of real-world signals, and large listener-rated real-world datasets are lacking. The paper collects crowdsourced ratings on two real-world corpora and trains an attention-based pyramid recurrent model to predict human MOS. The model outperforms baselines and recent data-driven approaches on prediction errors and correlations, with a noted exception for VOiCES SRCC against DNN.
Problem
Simulated speech may not capture real environments, objective scores do not always correlate with subjective ratings, and large public datasets of listener-rated real-world speech are lacking.
Source: §1 · p. 1Method
The authors collect 180,000 crowdsourced ratings from two real-world corpora and train an attention-based pyramid BLSTM model to predict human MOS.
Source: §1 · p. 1, §2.4 · p. 3Results
The proposed model outperforms baselines and recent data-driven approaches on prediction errors and correlations, with a VOiCES SRCC exception against DNN.
Source: §3.2 · pp. 3–4Takeaways & Limitations
The collected dataset will be made available to facilitate future research.
Source: §4 · p. 4
Abstract
from arXiv · showhide
The real-world capabilities of objective speech quality measures are limited since current measures (1) are developed from simulated data that does not adequately model real environments; or they (2) predict objective scores that are not always strongly correlated with subjective ratings. Additionally, a large dataset of real-world signals with listener quality ratings does not currently exist, which would help facilitate real-world assessment. In this paper, we collect and predict the perceptual quality of real-world speech signals that are evaluated by human listeners. We first collect a large quality rating dataset by conducting crowdsourced listening studies on two real-world corpora. We further develop a novel approach that predicts human quality ratings using a pyramid bidirectional long short term memory (pBLSTM) network with an attention mechanism. The results show that the proposed model achieves statistically lower estimation errors than prior assessment approaches, where the predicted scores strongly correlate with human judgments.
1. Introduction
The paper addresses limits in assessing speech quality in real environments: simulated data may not capture real-world conditions, and objective scores may not track human judgments. It proposes collecting real-world listener ratings and predicting them directly.
1. Introduction
- Subjective listening studies provide reliable quality assessments, but simulated stimuli may not capture real environments and are costly and time-consuming to collect.Listeners commonly use categorical or multi-stimuli rating protocols.Source: §1 · p. 1
- Reference-based objective measures compare clean and degraded signals, yet their scores do not always correlate with subjective quality ratings.Examples include PESQ, SDR, and POLQA.Source: §1 · p. 1
- Prior data-driven approaches often train on objective-score targets, while a human-MOS approach used simulated speech; the paper instead targets human ratings of real recordings.The authors collect 180,000 ratings from two real-world corpora and propose an attention-based pyramid BLSTM model.Source: §1 · p. 1
2. Methods
The methods combine crowdsourced MUSHRA listening tests on two real-world speech corpora with cleaning and averaging listener ratings to create MOS targets. An attention-based pyramid recurrent model predicts those targets from speech signals.
2.1. Crowdsourced listening study procedures
- Each MTurk HIT uses 15 MUSHRA trials, with a hidden clean reference, a low-pass anchor, and real-world test signals rated on a continuous 0–100 scale.Five crowdworkers complete each HIT; participants must be adult native English speakers with normal hearing.Source: §2.1 · p. 2
2.2. Speech material
- COSINE tests noisy shoulder or chest recordings against close-talk references, while VOiCES tests reverberant microphone recordings against foreground-speech references.COSINE includes everyday multi-party conversations; VOiCES recordings span two rooms of different sizes.Source: §2.2 · p. 2
- The study evaluates 18,000 signals from each corpus, using 3–6-second stimuli, and collects 180,000 human judgments.Across the study, 3,578 crowdworkers participated and 3,500 submissions were approved.Source: §2.2 · p. 2, §2.1 · p. 2
2.3. Data cleaning and MOS calculation
- Ratings are screened for outliers and worker bias, then the cleaned, scaled ratings for each stimulus are averaged to obtain its MOS.Outlier screening combines DBSCAN and isolation forests, discarding a rating only when both identify it as an outlier.Source: §2.3 · p. 2
2.4. Data-driven MOS quality prediction
- An attention decoder forms a context from encoder outputs and passes it through a fully connected layer to estimate utterance-level MOS.The pyramid encoder's shorter latent sequence also leaves fewer states for attention calculations.Source: §2.4 · p. 3
- The encoder stacks pyramid BLSTM layers that process sequences at multiple time resolutions, capturing short- and long-term dependencies in speech.The implementation uses three pBLSTM layers with reduction factor M = 2, reducing time resolution by a factor of 8.Source: §2.4 · p. 3
3. Experiments and analysis
Experiments compare pBLSTM+Attn with baseline and non-intrusive quality measures on two corpora. It generally achieves stronger predictions, with one reported VOiCES correlation exception.
3.1. Experimental setup
- Signals are framed at 40 ms, normalized as log-magnitude spectrograms, split 70/10/20 by dataset, and evaluated with five-fold cross-validation.The reported measures are MAE, RMSE⋆, Pearson correlation γ, and Spearman rank correlation ρ.Source: §3.1 · p. 3
3.2. Prediction of subjective quality
- Average correlations are ρ = 0.89 and γ = 0.88 for pBLSTM+Attn, versus ρ = 0.53 and γ = 0.52 for BLSTM; results are statistically significant (p-value < 0.0001).The Table I comparison reports results on both corpora together.Source: §3.2 · pp. 3–4, Table I · p. 4
- All data-driven methods outperform conventional P.563 and SRMR by a good margin; six non-intrusive methods are compared using the same data split as the proposed model.P.563 and SRMR outputs are polynomially mapped to the 0–10 MOS scale for MAE and RMSE⋆ calculations.Source: §3.2 · p. 4
- pBLSTM+Attn achieves the highest performance in prediction error and correlations against recent data-driven methods, except for DNN SRCC on VOiCES; Table II compares methods by corpus.Similar trends are reported on VOiCES as on COSINE.Source: §3.2 · p. 4, Table II · p. 4
- PCC reaches 0.88 versus AutoMOS 0.76, Quality-Net 0.81, and NISQA 0.84; RMSE⋆ is 0.52 on COSINE and 0.61 on VOICES, below listed alternatives.The reported advantage over NISQA in MAE and RMSE⋆ on COSINE is not statistically significant at p < 0.01 (p-values 0.047 and 0.078, respectively).Source: §3.2 · p. 4, Table II · p. 4
- Figure 3 plots predicted against true MOS for COSINE and VOiCES; most predictions scatter along the diagonal, indicating high correlation with human assessments.COSINE is orange and VOiCES is green in the figure.Source: §3.2 · p. 4, Figure 3 · p. 4
4. Conclusions
The paper presents an attention-based pyramid recurrent model for predicting human MOS ratings of real-world speech, reporting stronger performance than baseline and state-of-the-art approaches.
4. Conclusions
- The proposed model demonstrates superiority over baseline models and several state-of-the-art methods for speech-quality evaluation.The passage does not provide quantitative results.Source: §4 · p. 4
- An attention-based pyramid recurrent model is trained to estimate MOS directly from real-world speech signals.Source: §4 · p. 4
- Crowdsourced listening studies support the model’s accuracy and robustness in real-world environments.Source: §4 · p. 4
- The collected dataset will be made available to facilitate future research.Source: §4 · p. 4