Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges

Motivation

• Speech quality depends on fine-grained acoustic perception, while human listening tests are slow and costly and neural MOS predictors offer limited evidence about what shaped a score.MOS predictors may also fail to generalize to new listening-test contexts.Source: §1 · p. 1

Research question and approach

• The audit asks whether judges use acoustic cues justified by the target criterion, defining an acoustic shortcut as an irrelevant or human-overweighted cue shaping quality judgments.It tests six speech LLM judges using controlled, one-factor comparisons and human preference calibration when a factor may reasonably affect perceived quality.Source: §1 · p. 2

A controlled acoustic audit

• The study manipulates intensity, content richness, and emotional delivery, tests pointwise and pairwise judgments and rationales, and releases stimuli and evaluation tools as SpeechJudgeAudit.The audit includes human recordings and generated speech.Source: §1 · p. 2

Judges turn salient acoustic cues into quality preferences

• Judges reward louder audio, prefer content-rich speech more strongly than human listeners, and map emotional delivery into quality preferences, with effects clearest in pairwise comparison.Pointwise scores often make these effects less visible; judges may reward easy-to-detect acoustic changes instead of improved target quality.Source: §1 · p. 2

Rationale underspecification and implications

• Rationales rarely name the manipulated cue behind changed judgments, relying instead on limited broad rubric language that makes them less useful for debugging and development.The authors recommend controlling salient acoustic properties, comparing pointwise and pairwise behavior, and grounding training rationales in localized acoustic evidence.Source: §1 · p. 2

2.1 Speech LLM Judges

• Speech judges range from general-purpose zero-shot models to specialized fine-tuned systems that return scores, preferences, or rationales, but prior work had not systematically tested controlled acoustic shortcuts.Specialized systems target either naturalness or broader speech quality.Source: §2.1 · p. 3

2.2 From LLM-Judge Bias to Acoustic Shortcuts

• Text judges can favor position, length, verbosity, or model identity over target quality, while speech adds cues such as loudness, duration, and emotional delivery.Prior controlled audio studies report shortcut use in related tasks, motivating audits of acoustic effects on quality judgments.Source: §2.2 · p. 3

2.3 Rationale Faithfulness and Acoustic Evidence

• Prior rationale-faithfulness work tests whether predictions follow stated reasoning; this audit instead asks whether rationales connect judgment changes to the acoustic evidence in the input.A rationale can match the final preference yet still omit the cue that changed the judgment.Source: §2.3 · p. 3

3 Methodology

• The audit varies a targeted speech property while holding potential confounds fixed, then tests its effect on judgments and whether rationales name the acoustic change.This controlled design is applied across six speech LLM judges.Source: §3 · pp. 3–4

3.1 Judges and Audit Scope

• The six judges include specialized and zero-shot general-purpose models targeting naturalness or overall quality, with pointwise, pairwise, and rationale outputs.The emotional-delivery test uses human recordings; other tests use synthesized speech from four TTS systems spanning different architectures and output types.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Controlled Acoustic Manipulations

• The controlled factors are intensity, content richness, and emotional delivery, with duration controls separating richer wording from longer playback.Intensity changes RMS while holding other signal properties fixed; content pairs preserve intended meaning and speaker conditions, and emotion comparisons hold speaker and sentence constant.Source: §3.2 · p. 4

3.3 Evaluation and Analysis

• Pointwise scores are compared within judge, pairwise tests counterbalance A/B order, and human preferences calibrate content-richness and emotion judgments against listener responses.Rationale analysis checks vocabulary adaptation and explicit naming of manipulated cues using prespecified lexicons and frequent-word comparisons.Source: §3.3 · pp. 4–5

• Pairwise judges favor louder and content-richer speech, while emotional delivery also shifts quality preferences; human calibration reveals clearer judge-specific mismatches for richness and Happy delivery.Source: §4.1 · p. 5, §4.2 · p. 6, §4.3 · p. 6

4.1 Judges Reward Louder Audio

• At +6 dB, all five pairwise judges favor the louder clip, although pointwise scores give a less consistent picture and the signal-level change does not improve underlying speech.Human listeners also favor the +6 dB clip (0.612), confirming perceptual salience without making loudness valid quality evidence.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 Judges Overweight Content Richness

• All five pairwise judges prefer the meaning-equivalent richer clip; duration and repetition controls do not reproduce the preference, which remains at matched duration.Pointwise scores show no consistent richer-speech advantage; human listeners are less decisive, with a mean preference of 0.547 whose confidence interval crosses chance.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 Emotional Delivery Shapes Quality Judgments

• For emotional delivery, Sad loses to Neutral for four of five judges, while Happy beats Neutral for three; human calibration indicates a clearer judge-specific mismatch for Happy.The human mean favors Neutral for Happy (preference 0.407), while the judge mean is 0.531; Sad preference is 0.217 for listeners and 0.403 for judges.Source: §4.3 · p. 6, Figure 3 · p. 6

4.4 Pairwise Comparison Makes Acoustic Shortcuts More Visible

• Pairwise comparisons more consistently expose loudness and content-richness preferences than pointwise scores, but can also make invalid or overweighted cues more decisive.With 1,000 ms of boundary silence, UniSRM's pairwise preference for the padded clip falls to 0.069 although its pointwise scores are nearly unchanged; other judges show mixed effects or ties.Source: §4.4 · pp. 6–7

5.1 Rationales Draw on a Limited Recurring Vocabulary

• Across content richness, intensity, boundary silence, and emotional delivery, rationales repeatedly use broad terms such as naturalness, prosody, and clarity rather than manipulation-specific descriptions.A rationale may notice acoustic differences yet treat them as naturalness defects without considering whether they express the intended emotion.Source: §5.1 · p. 7

5.2 Rationales Rarely Name the Relevant Acoustic Change

• Rationales rarely name changed loudness, silence, content richness, or emotion; repetition is a partial exception, mentioned by four judges at rates of 5.7%–27.9%.These findings do not establish that every rationale is factually incorrect or unfaithful, but generic explanations can limit their value as debugging signals.Source: §5.2 · pp. 7–8, Figure 5 · p. 8

• Within the stimulus pools, the audit finds no stable preferences by speaker gender, accent, or Qwen-family self-preference, and no broad low-level descriptors explaining the main effects.These analyses do not provide fairness guarantees; the authors bound the claim to specific cues made salient or overweighted by the evaluation protocol.Source: §6 · p. 8
• Before using speech judges for comparison, data selection, or reward modeling, evaluators should normalize loudness, trim boundary silence when irrelevant, and report counterbalanced pairwise results alongside pointwise scores when available.Human preference calibration is especially important when content richness or emotional delivery may plausibly affect perceived quality.Source: §6 · p. 8
• Future judge-training data should ground rationales in localized audible evidence and cover diverse speakers, recording conditions, emotions, and synthesis systems.Reasoning-oriented training objectives could reward links between scores and localized evidence, but should themselves be audited because consistent rationales may still omit the changed cue.Source: §6 · p. 8

Conclusion

• Judges reward louder audio, favor content-rich speech more than human listeners, and map emotional delivery into quality preferences; these effects are clearest in pairwise comparison.Pairwise comparison can expose small acoustic differences but can also make shortcuts more decisive.Source: Conclusion · p. 9
• Rationales rarely name the acoustic changes that alter judgments, instead relying on broad rubric terms such as “naturalness” and “prosody.”This leaves the rationales acoustically underspecified.Source: Conclusion · p. 9
• Reliable speech judges need both robustness to acoustic shortcuts and rationales that connect decisions to the acoustic evidence behind them.Source: Conclusion · p. 9

7 Limitations

• The controlled audit uses English stimuli from selected TTS systems, VCTK speakers, and ESD recordings, so its null results do not establish broad fairness guarantees.The audit does not cover all languages, accents, recording conditions, speakers, or modern TTS systems.Source: §7 · p. 9
• Because judges target different criteria, content richness and emotional delivery may affect perceived quality, so the study calibrates these shifts against human preferences rather than labeling every shift bias.Source: §7 · p. 9
• The human preference study covers headline conditions and distinguishes main judge–listener mismatches, but cannot estimate fine-grained preferences across many languages, speakers, emotions, and synthesis domains.Source: §7 · p. 9
• The rationale audit concerns user-visible wording: rationales do not reliably identify manipulated acoustic changes, but this omission does not show that judges lack internal cue representations.Source: §7 · p. 9

A.2 Stimulus Pools

• The stimulus pools include 80 LibriTTS utterances for loudness and silence, 149 meaning-equivalent ASSET pairs for content richness, and matched emotional recordings from ESD.Richness controls use padding or repetition, while emotion comparisons match speaker and text.Source: §A.2 · pp. 11–12

A.3 Prompts and Reproducibility

• SPEECHJUDGEAUDIT releases the controlled stimuli, manifests, prompt templates, and analysis scripts for applying the tests to additional speech judges.For ESD, it provides a selected file list and preparation script rather than redistributing recordings.Source: §A.3 · p. 12

B.1 Statistical Reporting

• Pairwise effects are manipulated-clip preference rates with ties scored 0.5, while pointwise effects are within-judge score differences; significance markers and low-power labels qualify table results.Source: §B.1 · pp. 12–13

B.2 Loudness and Silence Results

• Every pairwise judge prefers +6 dB clips, while pointwise scoring often hides loudness effects; UniSRM’s pointwise scores barely change but pairwise comparison strongly rejects silence padding.Source: Table 4 · p. 13, Table 5 · p. 13, §B.2 · p. 13

B.3 Content-Richness Results

• Every pairwise judge prefers richer speech; silence-padding and repetition do not reproduce that effect, while transcript matching can confound the comparison.Gemini-3.1’s explicit position/length instruction leaves its richer-speech preference near 0.70.Source: §B.3 · p. 13, Table 6 · p. 13, Table 8 · p. 14

B.4 Emotional Delivery Results

• Four of five pointwise judges respond significantly to emotion; pairwise, Sad loses to Neutral for four, while Happy beats Neutral for UniSRM, SQ-LLM, and Gemini-3.1.Pairwise comparisons use matched clips from the same speaker and text.Source: §B.4 · p. 14

B.5 Pointwise and Pairwise Comparison

• Pairwise comparison often exposes directional preferences that pointwise scores leave weak, noisy, or absent.Source: §B.5 · p. 14

B.6 Position-Bias Diagnostics

• Order-counterbalanced A/B and B/A trials guard against mistaking presentation-order effects for acoustic preferences.Source: §B.6 · p. 14

• For Sad versus Neutral, GRM calls the Sad delivery flat, robotic, and unnatural, assigning it 2 versus 9 without contextualizing those cues as Sad delivery.Source: §C.1 · p. 15
• Rationales rarely name manipulated changes in loudness, silence, duration, or emotion; repetition is a partial exception, but detection remains incomplete.Source: §C.1 · p. 15
• The audit counts explicit acoustic terms with narrow cue-specific lexicons, excluding generic prosody words from the emotion counts.Source: §C.1 · p. 15

• Fifteen listeners completed 449 valid judgments across 60 pairs, choosing the better overall naturalness and speech quality or a tie.Source: §D.1 · pp. 15–16
• Position-bias diagnostics justify order counterbalancing and are not themselves acoustic-shortcut results.Source: Table 11 · p. 16
• Listeners found higher loudness salient but not valid evidence of better TTS quality; content richness was near chance for humans despite consistent judge preference.Source: §D.1 · p. 16
• Sad-versus-Neutral showed strong listener agreement in favor of Neutral, while Happy-versus-Neutral was more split even as three of five judges favored Happy.Source: §D.1 · p. 16

• Table 13 separates how often rationales mention repetition from how often repeated speech loses when repetition is or is not named.Source: Table 13 · p. 17
• Table 14’s human brackets are bootstrap confidence intervals, while judge brackets show spread across five judges rather than sampling uncertainty.Source: Table 14 · p. 17
• Listener quality-control diagnostics report majority agreement and item-mean MAE.Source: Table 15 · p. 17
• The supporting diagnostics are within-pool evidence and should not be read as causal mechanisms or broad fairness guarantees.Source: §E.1 · p. 16, Table 16 · p. 17