Source-linked AI summary
How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys
Ali Alavi, Donald S. Williamson
TL;DR
Speech quality predictors increasingly act as rewards, but no agreed protocol measures how easily they can be hacked. This paper corrects the reference and attacker sampling used to measure hackability, benchmarks common predictors, and audits patching; predictor vulnerability varies widely, while patching has narrow benefits and costs.
Problem
No agreed protocol compares predictor hackability under a common perturbation strength and rule, although predictors increasingly serve as speech-generation rewards.
Source: §I · p. 1Method
The paper measures hacks relative to the codec round trip, reports the worst case across trained attackers, benchmarks four predictors, and audits an attack–detect–patch loop.
Source: §I · pp. 1–2, §III · p. 2Results
Corrected benchmarking found NISQA 0.90 against UTMOS 0.06; patching hardens only its trained attack space, and random perturbations match it.
Source: §VI · p. 4Takeaways & Limitations
Choosing a reward model is itself a robustness decision, and patching benefits should be judged against attack-space transfer and the random-perturbation baseline.
Source: §VI · p. 4Patching hardens the predictor only in the attack space used for training.
Source: §VI · p. 4
Abstract
from arXiv · showhide
Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack. Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four (0.31 to 0.08 for one defence). Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several. Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%. We then audit a closed attack-detect-patch loop. It hardens the predictor only in its own attack space, by less than the spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out of domain. Enhancers post-trained against patched predictors hack them far less (PESQ -0.03 versus -0.23). Code, preregistration and run outputs are released.
I. INTRODUCTION
Speech quality predictors increasingly serve as both evaluation metrics and optimization rewards, but practitioners lack evidence about how much score an optimizer can extract without improving audio. This paper proposes a common hackability protocol and benchmarks predictors, showing that measurement reference and attacker choice affect conclusions.
I. INTRODUCTION
- Published attack results are not directly comparable because they vary in perturbation strength, attack rule, and evaluation setup.The paper notes that predictors can appear robust simply because they were attacked more gently.Source: §I · p. 1
- Hackable predictors can reward degraded speech, promote the wrong systems in selection workflows, and offer no assurance that higher scores mean better sound.The paper cites speech enhancement, TTS, voice conversion, challenge ranking, and hearing-aid fitting as settings where learned quality predictors are used.Source: §I · p. 1
- Referencing the processing chain’s output rather than raw input separates perturbation effects from the predictor’s score shift on codec-processed audio.The codec itself shifts predictor scores, so a raw-input reference charges that shift to the attack.Source: §I · p. 1
- Five attackers differing only by random seed reached success rates from 0.00 to 0.38 against one fixed predictor, making one attacker only a sample.The proposed protocol reports the worst case over several trained attackers.Source: §I · p. 2
- 90% of utterances were hacked for NISQA, the most hackable of four predictors benchmarked at one perturbation size.The paper also audits closed-loop patching, reporting narrow hardening, a random-perturbation baseline that matches it, and an out-of-domain cost.Source: §I · p. 2
II. RELATED WORK
Prior work documents quality-metric gaming, attacks predictors, and explores defenses, but the cited approaches do not provide the closed-loop predictor defense audited in this paper.
II. RELATED WORK
- Quality metrics can be gamed under single-metric optimization, while composite rewards are a common remedy and an independent panel can diagnose hacking.Asonitis et al. [1] diagnose hacking when an optimized score rises as an independent panel falls, but do not modify the predictor.Source: §II · p. 2
- Earlier predictor defenses include one-pass adversarial training for DNSMOS and one-pass reward-model hardening for TTS, without a hack detector and downstream generator audit.The cited work also includes gradient-norm regularization for image quality models.Source: §II · p. 2
- Related closed-loop and robustness approaches address LLM reward models, reward-model updates, ensembles, agent benchmarks, and music post-training.The paper positions its speech study alongside Adv-RM [15] and these other lines of work.Source: §II · p. 2
III. METHOD
The method trains a policy to perturb neural-codec latents, detects hacks against the codec round trip using a score rule and independent quality measures, then patches the predictor with flagged pairs. Repeated rounds retrain the attacker against the updated predictor.
III. METHOD
- The attacker uses a policy to propose fixed-size perturbations in continuous codec-latent space and trains with GRPO using the target predictor’s scores.Its time-invariant action is one C-dimensional offset per utterance, chosen because frame-rate actions did not learn at tested sizes.Source: §III · p. 2
- An output is flagged only when its score exceeds the zero-perturbation round trip and an independent panel measure degrades relative to that same reference.The panel includes ESTOI, SI-SDR, SpeechBERTScore, and Whisper word error rate; the corrected reference isolates perturbation effects from codec round-trip loss.Source: §III · p. 2, Figure 1 · p. 2
- Each flagged pair prefers the round trip over the attacked output in a ranking-loss patch, and subsequent rounds retrain the attacker against the patched predictor.The replay buffer retains hacks from prior rounds; a multi-space variant combines codec-latent, DAC-latent, and waveform-PGD attack pairs.Source: §III · p. 2, Figure 1 · p. 2
IV. EXPERIMENTAL SETUP
Experiments evaluate a trained SSL-MOS predictor, multiple attackers and defenses, and a post-trained enhancer, using preregistered seeds, metrics, and decision rules with stated amendments and exploratory analyses.
IV. EXPERIMENTAL SETUP
- SSL-MOS was trained on 4,974 BVCC training utterances, with human agreement evaluated across 13 test sets.The test sets include BVCC test, Voice-MOS’22, NISQA sets, SOMOS, TMHINT-QI, VoiceMOS’23 tracks, and URGENT’24/’25 quality-assessment sets.Source: §IV · pp. 2–3
- Table I covers hackability at ϵ=0.5 in EnCodec latents, using five independently trained attackers on 500 BVCC-test utterances and reporting the worst attacker.“Worst” denotes the largest success rate among the five attackers.Source: Table I · p. 3
- The EnCodec-24 kHz latent attacker used ϵ = 0.5, and the detector combined a score threshold with margins for independent measures.The detector threshold was τ = 0.3 MOS; defenses shared a 4,500-step budget, including random latent perturbations without RL or a detector.Source: §IV · p. 3
- A 2.6 M-parameter enhancer was post-trained with GRPO for 2,500 steps and evaluated on an 824-utterance test set.The policy added per-frame offsets on 32 ERB bands without a KL penalty.Source: §IV · p. 3
- Seeds, primary metrics, and decision rules for E1–E3 were preregistered, with one validation-based amendment; multi-space results and ablations were exploratory.The corrected ceiling reference was a post-hoc re-analysis of stored outputs, and results are mean ± sd over seeds.Source: §IV · p. 3
V. RESULTS
Under the corrected reference, the four predictors span a wide range of hackability, while the measured effect of patching depends on attack space and comes with out-of-domain costs. Downstream enhancement results show less PESQ degradation against patched predictors, but the random-pair baseline also matches the patch’s in-space result.
V. RESULTS
- Referencing the codec round trip changes measured hackability: DNSMOS is 0.183 versus 0.136 and NISQA 0.839 versus 0.896 under naive versus corrected references.One defence shifts from 0.306 to 0.082; attacker-seed rates also span 0.000–0.382 on R3.Source: §V · p. 3
- In-space ASR falls from 0.226±0.205 at R0 to 0.050 at R1 and 0.017 at R2, while random latent pairs reach 0.019±0.016 without RL or a detector.The effect is not monotone, and the R0-to-R3 difference changes sign across seeds.Source: §V · p. 3, Table II · p. 4
- EnCodec-trained patches do not transfer to DAC or PGD, whereas multi-space training reaches DAC ASR 0.016 and PGD ASR 0.841±0.089 but has the largest loss of human agreement.Out-of-domain mean system SRCC falls from 0.731 at R0 to 0.637 at R3 and 0.552 for Rmulti; TMHINT-QI is the worst noisy-speech case.Source: §V · pp. 3–4
- 89.6% of utterances are hacked for NISQA, versus 20.5% for SSL-MOS, 13.6% for DNSMOS OVRL and 5.7% for UTMOS at one perturbation size.Worst-case rates across five attackers are 0.916, 0.252, 0.200 and 0.076, respectively.Source: §V · p. 3, Figure 2 · p. 4
- Against R0, enhancement raises reward by +0.263 while PESQ falls −0.226; against R3, reward rises +0.269 with PESQ changing −0.034.R2 shows the same PESQ change (−0.034), and the random-pair baseline again matches the result (+0.019).Source: §V · p. 4, Table III · p. 4
VI. CONCLUSION
Corrected hackability estimates depend on referencing the processing chain’s own output and evaluating multiple attackers; common predictors then differ by more than an order of magnitude. Patching hardens only the attack space it trains on and matches a random-perturbation baseline, while the conclusions are bounded by study limitations.
VI. CONCLUSION
- NISQA is hacked on 0.90 of utterances versus 0.06 for UTMOS under the corrected protocol, a spread of more than an order of magnitude.Hackability depends on referencing the processing chain’s own output and reporting the worst case over several attackers.Source: §VI · p. 4
- Patching on a predictor’s own hacks hardens only that attack space, and a random-perturbation baseline matches it.Source: §VI · p. 4
- The benchmark ranks predictors only against a mild black-box attack family; the patch study uses three seeds and one predictor family.No listening test was run; the patch study used 46 A100-hours.Source: §VI · p. 4