Source-linked AI summary
TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio
Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, Buye Xu
TL;DR
Speech assessment is important, but human evaluations are difficult to scale and common objective metrics rely on clean reference speech. TorchAudio-Squim offers deep-learning estimators for objective metrics and MOS in TorchAudio; the authors report that its methods outperform prior state-of-the-art methods.
Problem
Human listening tests are difficult to scale, and common objective metrics require clean reference speech that is usually unavailable in real-world use.
Source: §1 · p. 1Method
Results
The authors report that TorchAudio-Squim’s subjective and objective methods outperform prior state-of-the-art methods.
Source: §5 · p. 4Takeaways & Limitations
TorchAudio-Squim makes deep-learning speech-quality and intelligibility assessment available through TorchAudio for speech systems and research on non-intrusive assessment.
Source: §5 · p. 4The NORESQA-MOS approach uses a clean speech signal as a non-matching reference, though any clean signal can serve.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Measuring quality and intelligibility of a speech signal is usually a critical step in development of speech processing systems. To enable this, a variety of metrics to measure quality and intelligibility under different assumptions have been developed. Through this paper, we introduce tools and a set of models to estimate such known metrics using deep neural networks. These models are made available in the well-established TorchAudio library, the core audio and speech processing library within the PyTorch deep learning framework. We refer to it as TorchAudio-Squim, TorchAudio-Speech QUality and Intelligibility Measures. More specifically, in the current version of TorchAudio-squim, we establish and release models for estimating PESQ, STOI and SI-SDR among objective metrics and MOS among subjective metrics. We develop a novel approach for objective metric estimation and use a recently developed approach for subjective metric estimation. These models operate in a ``reference-less" manner, that is they do not require the corresponding clean speech as reference for speech assessment. Given the unavailability of clean speech and the effortful process of subjective evaluation in real-world situations, such easy-to-use tools would greatly benefit speech processing research and development.
1. INTRODUCTION
Speech assessment must balance subjective evaluation’s reliability against its cost and objective metrics’ dependence on clean reference speech. TorchAudio-Squim addresses the need for readily usable estimators by integrating deep-learning tools for both metric classes into TorchAudio.
1. INTRODUCTION
- Human listening tests are reliable but difficult to scale, while common objective metrics require clean references that are often unavailable in real-world use.Objective metrics can also fail to correlate well with subjective assessments.Source: §1 · p. 1
- Prior work developed reference-less estimators, but their use remained uncommon because simple, readily integrated tools and inference models were lacking.The paper also identifies open-source tools as support for future metric-estimation research.Source: §1 · p. 1
- The system’s design targets real-world use with non-intrusive, deep-neural-network estimators for both subjective and objective metrics.The authors also intend to update models, reduce computational load, and extend metric coverage over time.Source: §1 · p. 1
- TorchAudio-Squim releases speech-assessment estimators within TorchAudio, with the current version supporting three objective metrics and one subjective metric.Integrating the components into TorchAudio is intended to ease metric use in speech-system design and development.Source: §1 · pp. 1–2
2. TORCHAUDIO-SQUIM OVERVIEW
TorchAudio-Squim provides objective and subjective speech-assessment models through TorchAudio. Its objective model estimates three metrics, while its subjective model estimates MOS using NORESQA-MOS.
2. TORCHAUDIO-SQUIM OVERVIEW
- The objective model estimates PESQ, STOI, and SI-SDR with a single network; PESQ denotes Wideband-PESQ throughout the paper.The authors describe the objective-estimation approach as novel.Source: §2 · p. 2
- The subjective model estimates human-listener MOS ratings using NORESQA-MOS, which takes a clean speech sample from a database as a reference.MOS ratings range from 1 to 5.Source: §2 · p. 2
- TorchAudio defines model architectures in torchaudio.models and pretrained end-to-end assessment pipelines in torchaudio.pipelines.Source: §2 · p. 2
3. SUBJECTIVE METRICS: SYSTEM DESCRIPTION
The subjective estimator uses NORESQA-MOS to predict MOS from test speech and a non-matching clean-speech reference. The released model is the one previously described in.
3. SUBJECTIVE METRICS: SYSTEM DESCRIPTION
- NORESQA-MOS predicts MOS from a test speech sample paired with a clean speech sample from a database as a non-matching reference.The authors state that any clean speech signal can serve as the reference and that this does not reduce the model’s utility compared with pure reference-less approaches.Source: §3 · p. 2
- The TorchAudio-Squim MOS model is the same NORESQA-MOS model analyzed and evaluated in.The authors refer readers to for its details.Source: §3 · p. 2
4. OBJECTIVE METRICS: SYSTEM DESCRIPTION
The objective-metric system uses a shared time-domain speech representation and estimates STOI, WB-PESQ, and SI-SDR through separate metric branches. Experiments report improved estimation with multi-task learning and outperformance of the compared baselines.
4. OBJECTIVE METRICS: SYSTEM DESCRIPTION
- A time-domain network uses DPRNN blocks for sequential modeling, then sends the learned representation to metric-specific transformer branches for estimating three objective metrics.The system proposes both a DPRNN-based architecture and a multi-task training strategy.Source: §4 · p. 2
4.1. Sequential modeling with dual-path RNN blocks
- Four DPRNN blocks use bidirectional LSTMs to model intra-chunk and inter-chunk dependencies, with residual connections in each sub-block.A convolutional layer encodes overlapped frames, which are divided into overlapped chunks before block processing.Source: §4.1 · p. 2
4.2. Multi-objective learning with transformer blocks
- Each metric branch applies a transformer with multi-head self-attention to the shared representation, then produces a scalar metric estimate.The transformer has two linear layers; the first uses ReLU. STOI and WB-PESQ estimates are transformed to their stated value ranges.Source: §4.2 · pp. 2–3
4.3. Facilitating speech assessment via multi-task learning
- A secondary reference-signal estimation branch provides supervisory information and regularization for the shared layers during joint training.The metric and reference-estimation losses are combined as a weighted sum.Source: §4.3 · p. 3
4.4.1. Data and Setup
- The DNS Challenge 2020 experiments use noisy speech mixtures and enhanced mixtures, with roughly 364,500 training, 14,600 validation, and 22,800 test samples.Additive-noise mixtures span SNRs from -15 to 25 dB; training signals are truncated to 5 seconds.Source: §4.4.1 · pp. 3–4
4.4.2. Results and discussion
- Using MAE loss instead of MSE yields significantly better performance, while the tested MTL weights achieve almost the best performance across all three scores.The reported weights are w1 = 1, w2 = 2, w3 = 0.5, and w0 = 2.Source: §4.4.2 · p. 4, Table 1 · p. 4
- Our model significantly outperforms the compared baselines on MAE, PCC, and SRCC; MTL improves WB-PESQ MAE/PCC/SRCC from 0.168/0.942/0.951 to 0.142/0.958/0.963.The model’s estimates are also more densely distributed near the diagonal than MOSA-Net and AMSA in Fig. 3.Source: Figure 3 · p. 3, §4.4.2 · p. 4, Table 2 · p. 4
- Joint estimation of multiple metrics does not degrade and slightly improves performance over separate models, while shared modules make training more computationally efficient.The authors attribute the improvement to correlated metrics regularizing one another’s estimation branches.Source: §4.4.2 · p. 4, Table 3 · p. 4
5. CONCLUSIONS
TorchAudio-Squim provides non-intrusive speech quality and intelligibility assessment through TorchAudio. Its subjective and objective metric methods are reported to outperform prior state-of-the-art methods.
5. CONCLUSIONS
- TorchAudio-Squim provides non-intrusive speech quality and intelligibility assessment through deep-learning methods released as part of TorchAudio.Source: §5 · p. 4
- Its methods support estimation of both subjective and objective speech metrics and are reported to outperform prior state-of-the-art methods.Source: §5 · p. 4
- The authors plan to continue improving the models, extend coverage to other speech assessment metrics, and explore more computationally efficient models.Source: §5 · p. 4