Source-linked AI summary
ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
Michael Chinen, Felicia S. C. Lim, Jan Skoglund, Nikita Gureev, Feargus O'Gorman, Andrew Hines
TL;DR
Perceptual-quality estimation needs to cover evolving speech and audio processing conditions, including lower bitrates and generative models. ViSQOL v3 addresses this through real-world feedback, C++ tooling, and production-oriented validation, while remaining bounded for substantially different data.
Problem
Existing quality models face extended domains such as lower-bitrate codecs and generative audio models that are not well handled by current full-reference speech metrics.
Method
ViSQOL v3 incrementally improves the existing framework using real-world Google production feedback and reimplements it as an open-source C++ tool.
Results
Real-world integration yielded usability and performance improvements, and Hangouts Meet reproduced reliable results across several hundred testbed calls.
Takeaways & Limitations
ViSQOL can support practical production and integration-testing applications when its use case is investigated carefully and, where needed, models are retrained.
Takeaways & Limitations
ViSQOL is not guaranteed to provide meaningful absolute MOS values for cases significantly different from its original design, often requiring additional metrics for validation.
Abstract
from arXiv · showhide
Estimation of perceptual quality in audio and speech is possible using a variety of methods. The combined v3 release of ViSQOL and ViSQOLAudio (for speech and audio, respectively,) provides improvements upon previous versions, in terms of both design and usage. As an open source C++ library or binary with permissive licensing, ViSQOL can now be deployed beyond the research context into production usage. The feedback from internal production teams at Google has helped to improve this new release, and serves to show cases where it is most applicable, as well as to highlight limitations. The new model is benchmarked against real-world data for evaluation purposes. The trends and direction of future work is discussed.
I. INTRODUCTION
ViSQOL v3 extends an established perceptual-quality framework through real-world feedback, practical tooling improvements, and attention to application blind spots and emerging audio models.
- Motivation: Existing objective metrics assess recorded-audio quality, while PESQ and POLQA are widely used for speech and general audio.Other examples include PEAQ and PEMO-Q; many metrics require commercial software.
- Motivation: Advances in denoising, compression, and generative audio models motivate quality estimation beyond the domains represented by existing datasets.Lower useful bitrates and waveform-generating models create extended analysis challenges.
- Contribution: ViSQOL v3 introduces incremental improvements based on independent Google teams’ real-world feedback rather than fundamental end-to-end DNN changes.The analysis targets problems encountered in the wild and seeks improvements to usability and performance.
- Scope: The paper reports that real-world validation is a practical compromise until more diverse subjective-score datasets are available.The improvements address blind spots in existing datasets and are not expected to improve results on those datasets.
- Practical tooling: The release reimplements ViSQOL in C++ and places the code on GitHub to reduce production, server, and licensing barriers.C++ was selected as a widely available and extensible language that can be wrapped in other languages.
II. CASE STUDIES AND USER FEEDBACK
Google teams integrated ViSQOL into production and testing workflows, using case studies to assess reliability and sensitivity to real-world network and audio conditions.
- Integration: The ViSQOL release resulted from production and integration-testing cases initiated independently by Google teams.Teams consulted developers or used documentation when integrating the tool.
- Hangouts Meet: Hangouts Meet uses ViSQOLAudio with 48 kHz reference and degraded samples to assess call audio quality under replicated adverse network conditions.The testbed reliably reproduces network conditions for evaluation.
- Hangouts Meet: ViSQOL MOS correlated with Meet’s internal no-reference metric, whose scale runs from 0 to 1 with lower scores indicating better quality.The comparison was used to investigate reliable behavior for this use case.
- Network sensitivity: ViSQOL scores decreased as network conditions worsened, ranging from 4.21–4.28 in good conditions, 4.04–4.16 in medium conditions, and 3.72–3.94 in extremely challenging conditions.The scenarios represented near-perfect calls, potentially problematic calls with good audio, and perceptually degraded calls, respectively.
- Network sensitivity: Figure 1 shows higher ViSQOL scores for unaffected audio blocks and successful detection of degradations in affected blocks.The affected blocks are 1 and 4; unaffected blocks are 2 and 3.
- Deployment: Several hundred testbed calls reproduced the results, after which ViSQOL became a main objective audio-quality metric for Hangouts Meet.The deployment followed positive comparison and reliability findings.
B. Opus Codec
ViSQOL and POLQA were used to assess Opus speech and music across bitrate and complexity settings, including low-bitrate and noisy-reference cases. ViSQOL v3 better matched expectations for speech, while music analysis exposed sensitivity and retraining limitations.
- Opus benchmarking covered speech and music across varying bitrates and computational complexities using ViSQOL and POLQA.
- Speech: ViSQOL v3 produced speech MOS trends that more closely matched expectations across bitrate and complexity settings after the reported improvements.
- Music: ViSQOLAudio discriminated monotonically between music bitrates but was relatively insensitive to complexity differences, limiting automated regression testing without retraining.
- Noisy references: ViSQOL was not designed for noisy references, yet produced behavior similar to clean-reference cases for both speech and audio.
- Evaluation boundary: Formal listening tests were recommended to interpret POLQA–ViSQOLAudio differences and train a model representing low-bitrate ranges.
C. Other Findings
Other product teams adapted ViSQOL to use cases that often differed from its training data. Practical findings included activity-based segmentation for ViSQOLAudio and a thresholding issue on silent references.
- Other teams adapted ViSQOL for products whose use cases often differed substantially from the training data.
- One team required a rig for continuous automated testing because it analyzed a network loop with digital and analog interfaces.
- ViSQOLAudio performed poorly on silent-reference segments because averaging and missing log-scale thresholding amplified sensitivity to small ambient-noise differences.
- Extracting 3 to 10 second segments with known activity was recommended to address averaging effects.
III. DESIGN AND IMPROVEMENTS
The design section presents the shared ViSQOL and ViSQOLAudio program flow and identifies the new system components introduced in the v3 release.
- Figure 6 summarizes the overall program flow and highlights the new components introduced in the system.
- The diagram uses white fill for inputs and outputs, blue fill for processing components, thick edges for new components, and distinct fills for speech-only and audio-only components.
A. General Design
ViSQOL and ViSQOLAudio share core processing components while retaining speech- and music-specific analysis. Practical revisions addressed dependencies, test findings, voice activity detection, and windowing.
- The speech and audio algorithms share the gammatone spectrogram and NSIM calculation by design.
- Speech-specific processing includes voice activity detection, while general-audio processing analyzes higher bands up to 24 kHz.
- Minor component changes addressed practical issues such as dependency modifications, case-study findings, and test failures.
- The VAD uses a simple energy-based implementation considered sufficient for the clean-reference requirement.
- Window sizes were updated to 80 ms with a 20 ms hop after an issue was found in previous-version windowing.
B. C++ Library and Binary
ViSQOL v3 removes MATLAB dependence through an open-source C++ implementation available as both a binary and library. Supporting features include protobuf-based control, error checking, language wrapping, CSV batch processing, conformance tracking, and extensive testing.
- C++ Library and Binary: ViSQOL v3 is implemented in C++ with only open-source dependencies and distributed as a binary or library.The code was placed on GitHub to support public use and external contributions.
- C++ Library and Binary: The library adds protobuf support and error checking for finer control and enables wrappers in languages such as Python.The binary depends on this library.
- C++ Library and Binary: Outputs include per-frequency-band and per-frame NSIM values, a conformance version number, and CSV-based batch processing.The conformance number increments when MOS changes for known files.
- C++ Library and Binary: Testing used Google Test to check correctness, detect regressions, improve edge-case stability, and verify conformance to known scores.The implementation included 23 test classes with multiple tests.
C. Fine-scaled Time Alignment
ViSQOL v3 adds fine-scale time alignment and silence thresholds to address patch misalignment and noise sensitivity in log-amplitude spectrograms. The thresholding uses absolute and relative amplitude floors before NSIM calculation.
- Fine-scaled Time Alignment: An additional alignment step offsets corresponding time-domain regions using lag from cross-correlation of aligned patches.The gammatone spectrogram is then recomputed for sample-aligned patch audio.
- Silence Thresholds: Silence thresholds address excessive NSIM sensitivity to ambient noise in low-level log-amplitude spectrogram regions.A near-silent reference against very low ambient noise could otherwise receive a low NSIM despite perceptual transparency.
- Silence Thresholds: The thresholded amplitude y_t,f(x) applies a global absolute floor and a relative per-frame floor to an input spectrogram.The relative floor may be higher for high-amplitude frames.
E. NSIM to MOS Model
The updated NSIM processing requires a new support vector regression model mapping frequency-band NSIM to MOS. The authors also provide tooling for users to train tailored SVR models, while noting poor behavior on very low-bitrate audio.
- NSIM to MOS Model: A new SVR model maps frequency-band NSIM values to MOS after the preceding changes affect NSIM scores.The model uses libsvm and is trained with four-way cross-validation loss minimization.
- NSIM to MOS Model: Training used TCDAudio14, CoreSV14, and AACvOpus15, with a grid search to minimize four-way cross-validation loss.The resulting model was observed to be too specific to the training data.
- NSIM to MOS Model: Users can train their own SVR model from CSV input files containing subjective scores for degraded/reference pairs.The documented libsvm grid-search procedure is intended to tailor models to users’ data.
IV. DISCUSSION
The discussion finds that ViSQOL can support varied real-world applications, but out-of-distribution use requires careful investigation and sometimes retraining or additional metrics. Generative audio may require fundamental algorithmic redesign and larger subjective-score datasets.
- IV. DISCUSSION: The discussion follows case-study feedback and reflects on trends and promising areas for future work.
- IV. DISCUSSION: ViSQOL supports various applications, but developers must investigate each use case because new audio types may be out of distribution.In some cases, users can retrain a model to match new data.
- IV. DISCUSSION: Developers found useful correlation in their use cases, although meaningful absolute MOS is not guaranteed for substantially different conditions.Additional metrics often helped validate ViSQOL’s application.
- IV. DISCUSSION: Generative audio may require fundamental algorithm redesign, including different spectrogram representations or DNNs.
- IV. DISCUSSION: A larger subjective-score dataset could enable new development while requiring reconsideration of tools such as ViSQOLAudio’s support vector regression.The existing regression approach is intended for datasets on the order of hundreds of points.
V. CONCLUSION
ViSQOL v3 integrates real-world feedback from Google teams into improvements over the previous version and is released for use on GitHub, while identifying directions for future work.
- Google teams’ integration of ViSQOL v3 into real-world problems yielded insights and improvements over the previous version.
- The release is available for use on GitHub.
- Future work includes DNN-based approaches, a more general model, and accounting for new generative audio approaches.