Source-linked AI summary
An Open source Implementation of ITU-T Recommendation P.808 with Validation
Babak Naderi, Ross Cutler
TL;DR
Subjective speech-quality assessment is commonly conducted in controlled laboratories, while P.808 specifies a crowdsourcing approach based on ACR. This paper implements and extends P.808 for AMT, integrating qualification and environment checks into rating tasks and adding DCR and CCR; across runs, DMOS removes the observed MOS offset.
Problem
Laboratory speech-quality tests control external factors but may not reflect typical listening systems and environments, while objective measures are not reliable replacements for listening-opinion tests.
Source: §1 · p. 1Method
The authors provide an open-source AMT implementation of P.808 with integrated worker qualification and environment checks, plus DCR and CCR test procedures.
Source: §1 · p. 1, §3.2 · p. 2, §3.2.1 · p. 2Results
Across runs, bias between MOS values is common, while DMOS successfully removed that offset.
Source: §4.2 · p. 4Takeaways & Limitations
The implementation provides an open-source way to conduct P.808 crowdsourced tests, process responses, and run DCR and CCR procedures.
Source: §1 · p. 1, §3.2 · p. 2, §3.2.4 · p. 3
Abstract
from arXiv · showhide
The ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. We provide an open-source implementation of the ITU-T Rec. P.808 that runs on the Amazon Mechanical Turk platform. We extended our implementation to include Degradation Category Ratings (DCR) and Comparison Category Ratings (CCR) test methods. We also significantly speed up the test process by integrating the participant qualification step into the main rating task compared to a two-stage qualification and rating solution. We provide program scripts for creating and executing the subjective test, and data cleansing and analyzing the answers to avoid operational errors. To validate the implementation, we compare the Mean Opinion Scores (MOS) collected through our implementation with MOS values from a standard laboratory experiment conducted based on the ITU-T Rec. P.800. We also evaluate the reproducibility of the result of the subjective speech quality assessment through crowdsourcing using our implementation. Finally, we quantify the impact of parts of the system designed to improve the reliability: environmental tests, gold and trapping questions, rating patterns, and a headset usage test.
1. Introduction
Speech quality is subjective, so listening-opinion tests remain important despite the convenience of objective measures. The paper presents an open-source P.808 implementation for crowdsourced assessment, with scripts for running tests and processing responses.
1. Introduction
- PESQ, SRMR, and P.563 correlate poorly with listening opinion and are not reliable replacements for listening-opinion tests.Laboratory tests following ITU-T Rec. P.800 control external factors but do not reflect typical listening systems and environments.Source: §1 · p. 1
- Crowdsourcing collects subjective assessments online from geographically distributed, demographically diverse workers using varied listening devices and environments.ITU-T Rec. P.808 specifies a crowdsourcing listening-opinion test using the ACR method.Source: §1 · p. 1
- The paper provides an open-source P.808 implementation with scripts for creating and executing tests, cleansing data, and analyzing answers.The stated aim is to reduce misinterpretation and operational errors and make subjective assessment more accessible without a dedicated test laboratory.Source: §1 · p. 1
2. Related work
The paper compares open-source systems for crowdsourced subjective speech-quality assessment and distinguishes them by deployment and reliability features.
2. Related work
- The compared systems differ in whether they require separate servers or run directly on crowdsourcing platforms such as AMT.The paper describes direct platform use as more convenient.Source: §2 · p. 1, Table 1 · p. 2
- The paper identifies hearing and environmental tests, lab-based validation, and reproducibility across times and raters as system requirements.It defines validation as comparison with a lab study showing accuracy, and reproducibility as consistent results at different times with different raters.Source: §2 · p. 1
- The authors report that their solution is the first to meet all of the requirements they identify.Those requirements include hearing and environmental tests, validation, and reproducibility.Source: §2 · p. 1
3. P.808 implementation
The implementation runs P.808 crowdsourcing tests through AMT without a dedicated webserver and provides scripts for test creation, data cleansing, and analysis. Its extensions integrate qualification and periodic environment checks into rating tasks and add DCR and CCR procedures.
3.1. Tools
- Scripts support test creation, answer post-processing, data cleansing, rating reconstruction and aggregation, and AMT interactions.The post-processing script generates reports and statistics including MOS, standard deviations, valid-vote counts, and 95% confidence intervals.Source: §3.1 · p. 2
- The master script takes clip, training, trapping, and gold-standard stimulus URLs and generates the task HTML, input URLs, and parser configuration.The generated task includes hearing, environment-suitability, headphone-use, trapping, and gold-standard checks.Source: §3.1 · p. 2
3.2. Extensions
- The implementation extends P.808 with DCR and CCR procedures and general extensions intended to speed testing and increase validity.The authors describe it as an exact implementation of P.808 alongside these extensions.Source: §3.2 · p. 2
3.2.1. Integrated rater qualification
- Integrating worker qualification as a section within the rating job made the entire process scalable and reduced execution time by 4–5×, without a specified comparison baseline.The qualification section appears only once, as part of a worker’s first rating task. The post-processing script checks qualification eligibility and rejects fraudulent responses involving other browsers or manipulated browser storage.Source: §3.2.1 · p. 2
3.2.2. Headset detection
- WebRTC device-name parsing detects a headset device in 40.1% of sessions on average, but failure to detect one does not establish that no headphone is connected.The API omits a wide variety of devices without microphones.Source: §3.2.2 · p. 3
3.2.3. Periodic environment test
- A temporary environment-suitability certificate that expires after 30 minutes reduced participants’ overall working time by 40%.When a certificate is absent, the environment test is injected into the rating assignment; a passed test stores the certificate in browser local storage.Source: §3.2.3 · p. 3
3.2.4. Extensions in test procedures
- DCR rates annoyance on a five-point scale after reference-then-processed playback, while CCR uses randomized presentation and a seven-point relative-quality scale.The post-processing script accounts for presentation order and calculates DMOS for DCR and CMOS for CCR.Source: §3.2.4 · p. 3
4. Validation
The implementation was evaluated against P.800 laboratory MOS and across repeated crowdsourcing runs, then examined for reliability differences between submissions that passed or failed integrated filters. The results report strong agreement with the laboratory benchmark and greater consistency for passed submissions, with filter effects qualified by the study design.
4.1. Validity
- PCC = 0.954, SRCC = 0.923, and RMSE = 0.237 (0.214 after first-order mapping) across ten samples of 96 votes per condition compared favorably with P.800 lab MOS.The comparison used the ITU Supplement 23 English dataset, with 200 files across 50 channel-degradation conditions; Figure 3 plots MOS by condition.Source: §4.1 · p. 3, Figure 3 · p. 3
4.2. Reproducibility
- Across five runs, MOS values had PCC = 0.994 and SRCC = 0.94; ICC was 0.719 for MOS and 0.907 for DMOS.The runs used unique raters on separate days; DMOS removed the between-run offset observed in MOS values.Source: §4.2 · pp. 3–4, Figure 4 · p. 4
4.3. Accuracy improvements
- Submissions passing integrated filters were more consistent across runs, with significant improvements reported for gold stimulus, environment test, and all criteria together.The analysis compared passed and failed submissions using ICC, average PCC, and average SRCC; Table 3 summarizes the filtering criteria's effect on reliability.Source: §4.3 · p. 4, Table 3 · p. 4
P.808 implementation.
The filter-impact analysis cannot isolate each filter’s exact effect on reliability, and the headset result is inconclusive because headset detection succeeded in only 40% of cases.
P.808 implementation.
- The headset filter did not significantly improve SRCC, but its result is inconclusive because headset detection succeeded in only 40% of cases.Source: §P.808 · p. 4
- Unequal pass/fail group sizes and varying ratios across filters prevent conclusions about each filter’s exact impact on reliability.The reported effects are examples from a typical use case, not exact estimates.Source: §P.808 · p. 4
- Only eight submissions failed trapping questions across all five runs, so this dataset cannot establish that trapping questions are not beneficial.Prior studies cited in the paper found that their presence affected participant behavior and encouraged higher-quality responses.Source: §P.808 · p. 4
5. Conclusions
The authors conclude that their open-source P.808 implementation correlates well with laboratory results and produces reproducible results. It reduces test execution time, adds DCR and CCR procedures, and has been used in internal tests and a challenge; they identify several extensions for future work.
5. Conclusions
- The open-source implementation correlates well with a lab study and produces reproducible results, while reducing test execution time and adding DCR and CCR procedures.Source: §5 · p. 4
- The tool has been used in dozens of internal tests and the INTERSPEECH Deep Noise Suppression challenge, with costs low enough for daily machine-learning development.Source: §5 · p. 4
- Future work proposes P.835 templates for separate speech, noise, and overall-quality MOS values, continuous environment monitoring, and extending run ratings through anchoring and mapping.Source: §5 · p. 4
- Training-section trapping questions significantly reduced submissions that later failed rating-procedure trapping questions.Source: §5 · p. 4