Source-linked AI summary
Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech
Tsubasa Ochiai, Marc Delcroix, Nahomi Kusunoki, Rintaro Ikeshita, Naohiro Tawara, Naoyuki Kamo, Tetsuji Ogawa, Shoko Araki
TL;DR
Non-intrusive MOS predictors are widely used to evaluate and rank speech enhancement systems, but it was unknown whether increasing their scores at test time improves perceived quality. This paper modifies enhanced signals to raise multiple predictors’ average scores. Across the studied challenge systems, scores rose without corresponding listener-rated improvement, and the optimization could distort challenge rankings.
Problem
Non-intrusive MOS predictors are widely used to evaluate and rank speech enhancement systems, but whether raising their scores at test time improves perceived quality had not been studied.
Source: §1 · p. 1Method
The authors optimize each enhanced signal at test time to increase the average scores of multiple MOS predictors while keeping the SE system fixed.
Source: §1 · p. 1, §3.2 · p. 2Results
Across URGENT 2026 systems, predicted scores rose substantially while other measures, including reference-based metrics and an unseen predictor, did not improve correspondingly, and listeners rated optimized signals no better than originals.
Source: §5.1 · p. 3, §5.2 · p. 3, §6 · p. 4Takeaways & Limitations
Test-time optimization can move challenge rankings by several places, so the authors recommend not evaluating systems with predictors used for optimization and keeping ranking predictors undisclosed.
Source: §1 · p. 1, §6 · p. 4Because the listening test differed from the challenge in evaluated data and protocol, the authors draw no conclusion about the relative standing of baird and RICK2000.
Source: §5.2 · p. 3
Abstract
from arXiv · showhide
Non-intrusive MOS predictors are widely used instead of subjective listening tests to evaluate and rank speech enhancement (SE) systems. If they accurately reflect perceived quality, raising their scores should lead to higher-quality speech. We present the first comprehensive analysis of test-time optimization for the SE task, which directly modifies the enhanced signal to raise the average of multiple MOS predictor scores. On seven systems from the URGENT 2026 challenge, we find that 1)~all the optimized predicted scores increase while reference-based metrics remain nearly unchanged, 2)~a non-optimized predicted score does not increase, and 3)~a MUSHRA listening test shows no improvement in perceived quality. These findings reveal a risk that such optimization can distort evaluations, e.g., biasing comparisons of SE systems regardless of their perceived quality. We believe these findings can inform future evaluation practices: they suggest that predictors used for optimization should not be used for evaluation, and that challenges should keep the predictors used for ranking undisclosed.
1. INTRODUCTION
Non-intrusive MOS predictors are used to evaluate and rank speech enhancement systems, but whether raising their scores at test time improves perceived quality had not been examined. The paper tests this question and finds that predicted-score gains can occur without better listener ratings, with potential consequences for evaluation.
1. INTRODUCTION
- Subjective listening tests are reliable but costly, so non-intrusive MOS predictors are widely used to compare systems and contribute to official challenge rankings.These predictors take a speech signal without its clean reference and predict the MOS listeners would assign.Source: §1 · p. 1
- The paper asks whether directly modifying enhanced signals to raise predicted MOS scores at test time also improves the quality listeners perceive.Prior work had added score-increasing losses during training, while this procedure can modify each test utterance without retraining the SE system.Source: §1 · p. 1
- Test-time optimization increases the objective predictors’ scores without improving perceived quality, raising a risk of distorted evaluations and challenge rankings.The authors recommend not evaluating systems with predictors used for optimization and keeping ranking predictors undisclosed.Source: §1 · p. 1
2. RELATED WORK
Prior work showed that speech-quality predictors could be manipulated, but did not provide a comprehensive test-time optimization analysis for enhanced speech that included listener ratings. A concurrent challenge study also left perceived quality untested.
2. RELATED WORK
- Earlier attacks targeted DNSMOS or UTMOS on clean or noisy speech; this study instead examines jointly optimizing multiple predictors on enhanced speech and evaluates listener ratings.Lin et al. constrained perturbations to more than 30 dB below signal level and found listeners could not distinguish perturbed from original speech in an AB test.Source: §2 · pp. 1–2
- Klement et al. raised DNSMOS from 2.83 to 4.18 on a REAL-TSE challenge submission while task-specific metrics stayed unchanged, but did not test perceived quality.The present paper addresses that open question with a listening test.Source: §2 · p. 2
3. TEST-TIME OPTIMIZATION OF ENHANCED SPEECH
The method optimizes an additive signal perturbation to increase the average of multiple predicted MOS scores while penalizing deviation from the original enhanced signal. It is applied independently to each utterance, with the SE system and predictors held fixed.
3.1. Preliminaries
- Given an observed waveform, the SE system produces an enhanced signal and MOS predictors produce scores for that signal.The SE system and each predictor are represented by functions with their respective parameters.Source: §3.1 · p. 2
3.2. Optimization objective
- The objective adds a signal z to the enhanced output and optimizes z to raise the average predictor score while penalizing the modified signal’s distance from the original.The trade-off weight λ controls the balance; only z is optimized, with the SE system and predictors fixed, independently for each utterance and without a reference signal.Source: §3.2 · p. 2
3.3. Optimization procedure
- The procedure initializes z with small random values, then uses gradient descent for a fixed number of iterations, backpropagating through the predictors while updating only z.The predictors remain unchanged throughout the procedure.Source: §3.3 · p. 2
4. EXPERIMENTAL DESIGN
The experiments apply multi-predictor test-time optimization to URGENT 2026 systems and assess metrics, rankings, and perceived quality. The evaluation uses a non-blind test set and a MUSHRA-style listening test, while the optimization differentiates through four challenge MOS predictors.
4. EXPERIMENTAL DESIGN
- The study evaluates URGENT 2026 submissions across non-intrusive and reference-based metrics, listener ratings, and challenge rankings.The ranking is recomputed after replacing each system’s metrics on modified signals while leaving the other leaderboard entries unchanged.Source: §4 · p. 2, §4.1 · p. 3
4.1. Data and evaluation
- The analysis uses 1,023 utterances from the non-blind test set and seven systems; it reports all 12 challenge metrics and additionally evaluates Distill-MOS as an unseen predictor.The blind set used for final challenge ranking was unavailable; metric ranks are averaged within four categories before deriving the overall rank.Source: §4.1 · p. 2
- The MUSHRA-style test rates five original or modified signals for each of 10 English 48 kHz utterances, with 11 listeners retained after screening.The tested systems are baird and RICK2000; the test uses clean and noisy signals as reference and anchor, respectively.Source: §4.1 · p. 3
4.2. Optimization
- Optimization targets DNSMOS, NISQA, UTMOS, and SCOREQ, using a multi-resolution L1 distance and 500 gradient-descent iterations.Predictor operations were replaced with PyTorch versions for gradients, which could differ slightly from the scoring functions; the learning rate starts at 5 × 10^-5 and decays by 0.99 per iteration.Source: §4.2 · p. 3
5. RESULTS AND DISCUSSION
Across objective, listening-test, and leaderboard evaluations, test-time optimization raises MOS-predictor scores without evidence of improved perceived quality. Its effects on challenge rankings depend on how those rankings are used.
5.1. Objective evaluation
- The four optimized MOS predictors rise substantially across all seven systems, while reference-based metrics generally remain unchanged or decrease.At λ = 0.005, reference-based metrics move by no more than 0.01; at λ = 0.05, predictor scores rise further and reference-based metrics degrade more.Source: §5.1 · p. 3
- Distill-MOS, which is not in the objective, changes by at most 0.03 in either direction despite substantial gains on the four optimized predictors.The gains therefore do not generalize to a predictor outside the objective.Source: §5.1 · p. 3
- When only DNSMOS is optimized, its score rises more than under joint optimization while the other predicted scores decrease.This disagreement is the stated reason for jointly optimizing the signal for all four predictors.Source: §5.1 · p. 3
5.2. Subjective evaluation
- Every optimized MUSHRA condition has a negative ΔMUSHRA, and the authors conclude that optimization raises predicted scores without improving perceived quality.For “baird” at λ = 0.005, the equivalence bound indicates a change of at most 1.8 points in either direction; the authors draw no conclusion about the two systems’ relative standing because their test differs from the challenge’s in data and protocol.Source: §5.2 · p. 3, Table 2 · p. 4
5.3. Impact on the challenge ranking
- At λ = 0.05, every system reaches rank 1 or 2 in the non-intrusive category, and the baseline’s overall rank rises from 13 to 6.Averaging that category with three others reduces but does not remove the effect on overall rank.Source: §5.3 · p. 4, Table 1 · p. 4
- The ranking effect is indirect when a challenge uses rankings to select systems for listening tests, but direct when rankings determine final placement.The paper gives URGENT 2026 as an example of the former and REAL-TSE 2026 and AVSE 2026 as examples of the latter.Source: §5.3 · p. 4
6. CONCLUSION
Test-time optimization raises targeted MOS predictor scores without corresponding improvement in other measures or listener-rated quality, and can shift challenge rankings. The findings suggest undisclosed ranking predictors and listening tests for final decisions.
6. CONCLUSION
- Optimized predicted scores increased substantially, but other measures—including reference-based metrics, downstream metrics, and an unseen predictor—did not improve correspondingly, and listeners rated optimized signals no better.The authors say these results suggest that the raised predicted scores do not correspond to better perceived quality.Source: §6 · p. 4
- Test-time optimization moved challenge rankings by several places, revealing a risk of distorting challenge outcomes.Using multiple predictors for ranking does not eliminate the risk when all their scores are jointly optimized.Source: §6 · p. 4
- The authors suggest challenge organizers keep ranking predictors undisclosed and base final decisions on listening tests.They plan future work investigating how MOS predictors can be made robust against this optimization.Source: §6 · p. 4