Source-linked AI summary
CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Models
Alef Iury Siqueira Ferreira, Pedro Lustosa Rege Botelho, Fernanda Silva, Daniel Casanova, Rafael Faustino, Frederico Oliveira, Arlindo Galvão Filho, Anderson da Silva Soares
TL;DR
MOS prediction with SFMs raises the question of which layers to use and how to combine them reliably across backbones and datasets. The paper benchmarks layer strategies and evaluates per-layer adapters before pooling; calibrated aggregation is the strongest, most consistent frozen-backbone approach in the final comparison.
Problem
Method
Results
A+M is the strongest and most consistent frozen-backbone aggregation method; Wav2BERT A+M reaches 0.388/0.749 in average utterance-level MSE/SRCC.
Source: §4 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Speech Quality Assessment (SQA) is essential for modern speech technologies, and recent non-intrusive SQA predictors increasingly rely on Speech Foundation Models (SFMs). However, because SFMs expose representations from many layers, it remains unclear which depths are most informative for MOS prediction and how multi-layer information should be combined reliably across backbones and datasets. We benchmark ten SFMs on four MOS datasets under three regimes: full fine-tuning, last-layer probing with a frozen encoder, and naive cross-layer weighted aggregation. We find that the best layer is strongly backbone- and dataset-dependent, and that naive weighted fusion can be unstable across settings. We further evaluate a layer-calibrated aggregation variant that applies per-layer adapters before pooling, which improves the robustness of multi-layer fusion and narrows the gap to full fine-tuning while keeping the backbone frozen.
1. Introduction
MOS listening tests remain the standard but are costly, time-consuming, difficult to scale, and affected by annotator variability. This study examines how to use SFM layers for MOS prediction and evaluates per-layer calibration before aggregation.
1. Introduction
- MOS listening tests remain the standard for quality assessment but are expensive, time-consuming, difficult to scale, and subject to annotator variability and bias.These constraints motivate automatic, non-intrusive MOS prediction as a practical alternative.Source: §1 · p. 1
- Prior MOS prediction work uses either full SFM fine-tuning or lightweight predictors on frozen representations, leaving open how best to exploit internal representations.Other work combines SFM embeddings with complementary acoustic features to improve robustness.Source: §1 · p. 1
- Earlier and intermediate SFM layers often retain acoustic and phonetic cues, while deeper layers become more shaped by pre-training objectives, making layer choice relevant to MOS cues.Prior studies have found early-to-mid layers can outperform the final layer for MOS prediction [15] [16].Source: §1 · p. 1
- The study benchmarks ten SFMs on four MOS datasets, comparing probing and aggregation strategies with full fine-tuning and evaluating per-layer adapters before fusion.Its contributions include showing depth varies by backbone and dataset, naive weighted fusion is not generally reliable, and calibration can improve frozen-backbone aggregation.Source: §1 · p. 1
2. Related Works
Prior MOS prediction research established SFMs as effective backbones and found that early-to-mid layers can outperform final layers. This work extends layer selection research by testing whether calibrated multi-layer fusion is more reliable across backbones and datasets.
2. Related Works
- Recent SQA systems increasingly use SFMs, including fine-tuned wav2vec 2.0, HuBERT, Whisper-based systems, UTMOS, and MOSA-Net+ [11].MOSA-Net+ combines Whisper embeddings with spectral features to improve robustness.Source: §2 · p. 1
- Layer-wise research finds intermediate layers retain acoustic and phonetic information, and early-to-mid layers can outperform final layers in MOS prediction [15] [16].Source: §2 · p. 2
- This work moves beyond single-layer selection to test robust multi-layer use, drawing on speaker-verification approaches including learnable weighting, multi-scale aggregation, and per-layer adapters [17].It compares layer selection, weighted aggregation, and fine-tuning, then examines per-layer calibration for MOS prediction.Source: §2 · p. 2
3. Methods
The methods isolate how SFM layer utilization affects non-intrusive MOS prediction. They compare frozen single- and multi-layer strategies with full fine-tuning, including adapters that calibrate each layer before pooling.
3. Methods
- The benchmark includes ten publicly available SFMs that differ in objective, supervision, scale, and pre-training data, evaluated across four MOS datasets.The datasets cover different languages, speech-generation conditions, and rating scenarios.Source: §3 · p. 2
- The frozen-backbone strategies are Last Layer, Best Layer, and Weighted Sum, which respectively use the final layer, a selected intermediate layer, or learnable normalized weights over all layers.Weighted Sum assumes representations from different depths are compatible enough for direct fusion.Source: §3 · p. 2
- A+M is the layer-calibrated aggregation strategy: independent adapters transform each layer before concatenation and mean pooling, without full backbone adaptation.Each adapter uses a linear projection, layer normalization, ReLU, and a second linear layer.Source: §3 · p. 2
- Full Fine-Tuning jointly optimizes the SFM and regression head, while A+M transforms layer representations before pooling into a fixed-dimensional utterance representation.Source: §3 · p. 2
- After layer selection or aggregation, masked mean pooling produces an utterance embedding for an MLP regressor; models are trained with MSE and evaluated using MSE and SRCC.The protocol screens candidates for 20 epochs, then extends detailed final comparisons to 100 epochs.Source: §3 · p. 2
- Figure 1 plots normalized SRCC by encoder layer and dataset for five selected models, marking the top three layers in each row.Strong layers often appear in early-to-mid portions of the network, but vary across backbones and datasets.Source: Figure 1 · p. 2
4. Results and Discussion
The results show that useful MOS information varies across model depth, backbones, and datasets, making naive layer fusion unreliable. Per-layer calibration makes frozen-backbone aggregation more robust, with Wav2BERT A+M the best overall configuration.
4. Results and Discussion
- 0.388/0.749 in average utterance-level MSE/SRCC and 0.052/0.932 in average system-level MSE/SRCC: Wav2BERT A+M was the best overall configuration and strongest consistent frozen-backbone aggregation method.A+M was also more effective than LL, BL, or naive WS in this setting.Source: Table 2 · p. 3, §4 · p. 4
- Layer-wise results often favor early-to-mid encoder depths, but the best layer varies across backbones and datasets, with broad strong regions in some models and sharper peaks in others.The global mean importance curve shows the same early-to-mid tendency.Source: Figure 2 · p. 4, §4 · p. 4
- The 100-epoch comparison retained five strong model families from screening; 300M multilingual variants were preferred over 1B counterparts when the larger models lacked a clear SRCC advantage to justify added analysis and training cost.The retained families were WavLM-L, HuBERT-L, XLSR-300M, MMS-300M, and Wav2BERT.Source: §4 · p. 4
- Best-layer selection improves over last-layer probing most clearly for XLSR-300M and Wav2BERT, while gains are smaller for WavLM-L and HuBERT-L.The results indicate that intermediate-layer usefulness is backbone-dependent rather than universal.Source: §4 · p. 4
- Naive weighted sum is competitive for Wav2BERT, MMS-300M, and XLSR-300M but weaker for WavLM-L and HuBERT-L, while full fine-tuning remains a strong baseline.Directly combining hidden states can be too restrictive when representations from different depths are insufficiently compatible.Source: §4 · p. 4
- WavLM-L benefits markedly from A+M, HuBERT-L gains more moderately, and XLSR-300M gains less because its high-performing region is already stable; MMS-300M results are mixed.For MMS-300M, A+M improves some ranking metrics but is less consistently favorable on MSE as strong layers shift across datasets.Source: §4 · p. 4
5. Conclusion
Across ten backbones and four datasets, the final layer is often not the most informative for MOS prediction, and naive cross-layer fusion is not reliably effective across backbones. Per-layer adapters followed by pooling improve multi-layer fusion robustness and yield the strongest overall frozen-backbone results.
5. Conclusion
- Per-layer adapters followed by pooling improve multi-layer fusion robustness and yield the strongest overall frozen-backbone results, notably with Wav2BERT and WavLM-L.The authors propose exploring other compact fusion designs and broader cross-domain generalization in future work.Source: §5 · p. 4
- Across ten backbones and four datasets, the final layer is often not the most informative representation for MOS prediction, with strong performance frequently emerging in early-to-mid layers.Source: §5 · p. 4
- Naive cross-layer weighted fusion is not a reliable general solution because its effectiveness depends strongly on the backbone.Source: §5 · p. 4
- The authors conclude that MOS prediction depends not only on stronger backbones but also on using their hierarchical representations more effectively.Source: §5 · p. 4
6. Use of Generative AI Disclosure
The authors used large language models only to polish portions of the manuscript. They take full responsibility for its content.
6. Use of Generative AI Disclosure
- The authors used generative AI tools, specifically large language models, to polish portions of the manuscript.Source: §6 · p. 5
- The tools were used for minor editing tasks such as improving grammar, clarity, and phrasing.Source: §6 · p. 5
- The authors take full responsibility for the manuscript's content.Source: §6 · p. 5