Source-linked AI summary
Is Semantics Enough for Speech Mean Opinion Score Prediction?
Tianyu Lan, Yufei Shi, Yang Ai, Honghao Sun, Huipeng Du, Zhenhua Ling
TL;DR
Human-rated MOS is the gold standard for speech naturalness, but its cost motivates automatic predictors, many of which rely on semantic-focused SSL representations that may miss acoustic detail. The paper benchmarks SSLs, acoustic-only NACs, and unified NACs across in-domain and OOD datasets. Unified semantic-acoustic representations generally achieve a higher upper bound in matched-language settings, while SSLs remain more resilient in cross-lingual evaluation.
Problem
Human MOS ratings are costly, and semantic-focused SSL representations may overlook fine-grained acoustic details relevant to speech-quality prediction.
Source: §1 · p. 1Method
The study benchmarks SSL, acoustic-only NAC, and unified NAC representations using a common predictor across BVCC and OOD datasets.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 3Results
Unified semantic-acoustic representations generally outperform single-modal models on BVCC and SOMOS, while SSLs show stronger generalization on cross-lingual BC2019.
Source: §4 · p. 4, §3.3 · p. 4Takeaways & Limitations
The findings indicate that semantics alone may not fully capture speech quality and that fine-grained acoustic information provides complementary benefits for MOS prediction.
Source: §4 · p. 4Unified NACs’ advantage diminishes for unseen languages, where SSL features can generalize comparably or better.
Source: §1 · p. 1, §3.3 · p. 4
Abstract
from arXiv · showhide
Mean Opinion Score (MOS) is the gold standard for evaluating synthesized speech naturalness. However, current automatic MOS predictors are dominated by self-supervised learning (SSL) models that prioritize high-level semantics, potentially compromising their ability to capture critical acoustic details. In this paper, we systematically investigate representations from three paradigms: SSLs, acoustic-only neural audio codecs (NACs), and unified NACs that integrate semantics into reconstruction-based architectures. Extensive benchmarking on the standard BVCC and multiple out-of-domain (OOD) datasets demonstrates that features synergizing semantic understanding with fine-grained acoustic modeling achieve a higher performance upper bound in speech quality assessment. Ultimately, our findings highlight that semantics alone are not enough; a dual focus on semantic content and acoustic fidelity is essential for robust MOS prediction.
1. Introduction
Automatic MOS prediction is important because human ratings are costly, but semantic-focused SSL representations may miss acoustic details relevant to speech quality. The paper compares SSL, acoustic-only NAC, and unified NAC representations and finds that combining semantics with acoustic modeling generally raises the performance upper bound, with a cross-lingual exception.
1. Introduction
- Human-rated MOS remains the gold standard for speech naturalness, but manual annotation is prohibitively costly, motivating efficient automatic evaluation.Most current state-of-the-art MOS predictors use semantic-based SSL backbones.Source: §1 · p. 1
- SSL pre-training objectives encourage high-level contextual abstraction that can overlook noise and waveform distortions, potentially limiting quality assessment.The paper hypothesizes that this acoustic blind spot constrains representations’ upper bound, particularly for utterance-level naturalness MOS.Source: §1 · p. 1
- The study compares SSLs, acoustic-only NACs, and unified NACs using BVCC in-domain evaluation and SOMOS matched-language and BC2019 cross-lingual OOD evaluation.Predictors are evaluated with frozen encoders to probe the representations’ intrinsic information content.Source: §1 · p. 1
- Unified representations combining semantic understanding and fine-grained acoustic modeling generally establish a higher MOS-prediction upper bound than semantic-focused SSLs or acoustic-only baselines.The advantage persists in matched-language settings across frozen and fine-tuning protocols, while SSL features can match or outperform unified NACs in cross-lingual OOD scenarios.Source: §1 · p. 1
2. MOS Prediction Framework
The framework compares feature representations from three model paradigms through a common downstream MOS predictor. Experiments use frozen and fine-tuning protocols to distinguish intrinsic representation information from performance after adaptation.
2. MOS Prediction Framework
- The framework uses an upstream feature extractor followed by a downstream prediction network, with the upstream pre-training objective posited to shape latent information content.Source: §2 · pp. 1–2
2.1. Upstream Feature Extractors
- Unified NACs combine codec reconstruction objectives with semantic modeling, while SSLs emphasize contextual semantics and acoustic-only NACs preserve waveform details.The study uses Wav2Vec 2.0, HuBERT, and WavLM as SSLs; EnCodec and DAC as acoustic-only NACs; and SpeechTokenizer and Xcodec as unified NACs, using continuous unquantized encoder representations.Source: §2.1 · p. 2
2.2. Downstream Prediction Network
- The common downstream network processes upstream features with an FFN and four self-attention layers, then concatenates mean-pooled attention features with max-pooled FFN features.A linear regression head with sigmoid activation scales predictions to the MOS range [1, 5].Source: §2.2 · p. 3
2.3. Investigation Protocols
- The frozen setting trains only the downstream network to probe pretrained representations, while fine-tuning unfreezes upstream encoders for end-to-end MOS optimization.Source: §2.3 · p. 3
3. Experimental Setup and Results
The paper evaluates MOS prediction across three datasets and finds benefits from combining semantic and acoustic information, with cross-lingual performance favoring SSLs.
3.1. Datasets
- BVCC, SOMOS, and BC2019 provide in-domain and zero-shot evaluation, with English and Mandarin samples and ratings from multiple listeners.BVCC has 7,106 English utterances; zero-shot tests use 3,000 SOMOS-clean samples and 540 BC2019 utterances.Source: §3.1 · p. 3
3.2. Implementation Details and Evaluation metrics
- The implementation uses a shared backend MOS predictor, trains for up to 30 epochs, and reports MSE, LCC, SRCC, and KTAU averaged across three runs.The checkpoint with the highest validation SRCC is selected.Source: §3.2 · p. 3
3.3. Experimental Results and Analysis
- Xcodec-wavlm remains the top-performing model, while frozen BVCC results favor unified NACs and WavLM over standard SSLs and acoustic-only NACs.In fine-tuning, the gap between unified NACs and SSLs narrows, consistent with MOS prediction being semantic-dominant.Source: §3.3 · pp. 3–4, Table 2 · p. 4
- On matched-language SOMOS, unified NACs outperform SSL baselines; on unseen-language BC2019, their advantage diminishes as SSLs match or surpass them.The paper attributes the cross-lingual shift to SSL continuous latent spaces versus unified NACs' discrete codebooks biased toward pre-training language.Source: §3.3 · p. 4, Table 3 · p. 4
- Across 100k training iterations, semantic-module gradients are consistently larger than acoustic-module gradients, and semantic-acoustic models retain the highest performance upper bound.The gradient analysis is sampled at 1k-step intervals.Source: §3.3 · pp. 3–4, Figure 2 · p. 3
4. Conclusion
Unified representations generally outperform single-modal models on BVCC and SOMOS, while SSLs show stronger cross-lingual generalization; multi-scale fusion appears beneficial.
4. Conclusion
- Unified representations generally outperform single-modal models on BVCC and SOMOS, while SSLs show stronger cross-lingual generalization.The conclusion identifies improving unified NACs' cross-lingual adaptability as future work.Source: §4 · p. 4
- Multi-scale fusion appears beneficial for robust speech quality assessment.Source: §4 · p. 4