PaperlayerView exact revision on arXiv

Source-linked AI summary

SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences

Mingyue Huo, Shivam Mehta, Bhavin Jawade, Yinghong Lan, Haoqi Li

arXiv:2609.34582v2cs.AIcs.SD

TL;DR

Diagnostic speech judging requires comparing candidates across perceptual dimensions and grounding verdicts in audible cues, but human annotation is costly and zero-shot model supervision is unreliable. SPEECHCRITIC uses limited human judgments to calibrate acoustic hints for a machine labeler, then examines training signals for specialized judges. The calibrated hints improve dimension-level agreement and better match human TIE behavior.

  • Problem

    Diagnostic speech judging needs multidimensional, evidence-grounded comparisons, while collecting consistent human judgments and explanations is costly and zero-shot model labels are unreliable.

    Source: §1 · p. 2

  • Method

    SPEECHCRITIC uses roughly 300 human-labeled comparisons to select and calibrate acoustic measurements as non-binding A/TIE/B hints for a machine labeler.

    Source: §1 · p. 2

  • Results

    Calibrated hints improve dimensional macro-F1 by 6.3 points and reduce TIE-rate MAE by 10.4 points versus labeling without hints.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    SFT, OPD, and RL shape judge behavior differently, and human evaluation finds RL can improve rationale grounding in specific, localized acoustic cues without direct rationale rewards.

    Source: §7 · p. 10

    Persuasive rationales can accompany incorrect verdicts, so explanation quality alone does not establish reliable judgment.

    Source: §6.2 · p. 9, §7 · p. 10

Abstract

from arXiv · show

Human speech conveys rich perceptual information, such as emotion and speaker identity, yet most automatic speech quality judges reduce it to a single naturalness score. We study diagnostic speech judges: given two candidates, a diagnostic judge decides which is better, along which perceptual dimensions (e.g., timbre, emotion, timing) they differ, and which audible cues support its decision. Learning such judges is challenging: expert annotation is costly, and simply prompting a frontier audio-language model to produce labels is unreliable: our probing reveals substantial errors and unstable instruction following. We introduce SpeechCritic, which learns a diagnostic judge in a reference-conditioned cross-lingual setting from only about 300 human-labeled comparisons. Rather than replacing the frontier model, SpeechCritic calibrates it with these labels: for each dimension, it selects the acoustic measurements that agree with human judgments, maps them to A/Tie/B probabilities, and passes these to the model as non-binding hints alongside the audio. Compared with the same model labeling without hints, this raises dimension-level agreement with humans by 6.3 points and cuts the mismatch with human Tie rates by 10.4 points. We then train a 7B judge on this supervision and find that different training signals shape different judge behaviors: SFT establishes the task, OPD transfers the teacher's dimension-level strengths and weaknesses, and RL helps most on clear-cut comparisons where human raters agree. Notably, human listeners also find that RL makes rationales cite more specific, localized acoustic cues, although it never directly rewards rationale text. Finally, we show that the pipeline is language-pair agnostic by instantiating it on both English-Japanese and English-Spanish. Together, these results demonstrate a path from limited human preferences to a diagnostic speech judge.

1 INTRODUCTION

The paper frames diagnostic speech judging as multidimensional, evidence-grounded comparison and asks whether limited human judgments can support scalable supervision. SPEECHCRITIC calibrates machine-generated labels with human-selected acoustic evidence, then studies how SFT, OPD, and RL shape trained judges.

1 INTRODUCTION

  • Diagnostic judging compares two speech candidates across perceptual dimensions and supports its verdicts with audibly verifiable cues.The reference-conditioned cross-lingual setting evaluates speaker characteristics, emotion, timing, pronunciation or accent, and audio artifacts.Source: §1 · p. 2
  • Human annotation is costly, while zero-shot audio-language model judgments show weak human agreement, skewed preferences, and inconsistent dimension-level decisions.These outputs are unreliable as training supervision, motivating the question of whether limited human judgments can support scalable specialized judging.Source: §1 · p. 2
  • Human-calibrated hints improve dimensional agreement by 6.3 percentage points and reduce mismatch in TIE behavior by 10.4 points.The pipeline selects reliable acoustic measurements, maps them to A/TIE/B probabilities, and supplies them as non-binding hints while the labeler listens to the audio.Source: §1 · p. 2
  • SFT, OPD, and RL shape different judge behaviors, and the framework is evaluated for rationale grounding and transfer across target languages.The contribution set describes scaling roughly 300 human-labeled comparisons into more than 10,000 verdict-and-rationale comparisons.Source: §1 · p. 2

2 RELATED WORK

Prior speech evaluators either prompt audio-language models or train dedicated evaluators, while much work emphasizes naturalness or low-level quality. This paper instead focuses on multidimensional comparisons with rationales grounded in audible evidence, using human-screened acoustic measurements as non-binding guidance.

2 RELATED WORK

  • Existing speech-evaluation work includes prompted audio-language models and dedicated evaluators, but much of it emphasizes naturalness or low-level acoustic quality.Related approaches also address instruction-driven assessment, multilingual judging, spoken dialogue, and multi-task evaluation.Source: §2 · p. 3
  • Evaluator supervision can be generated or distilled, but speech evaluators must perceive audio and ground verdicts and rationales in audible evidence.Related approaches include model-generated feedback, AI feedback, human-written principles, and synthetic evaluator data.Source: §2 · p. 3
  • Speech measurements estimate individual perceptual properties but do not jointly provide comparative verdicts and audio-grounded rationales.SPEECHCRITIC screens candidate measurements against held-out human judgments, calibrates retained signals into A/TIE/B hints, and gives them as non-binding guidance to a raw-audio labeler.Source: §2 · p. 3

3 DIAGNOSTIC SPEECH JUDGING: TASK AND HUMAN EVALUATION

The task is reference-conditioned cross-lingual comparison: a judge evaluates two target-language candidates against a source-language reference and produces structured verdicts with localized acoustic rationales. Human ratings establish the evaluation setting, while zero-shot model probing reveals shortcomings in human alignment and rubric consistency.

3 DIAGNOSTIC SPEECH JUDGING: TASK AND HUMAN EVALUATION

  • The judge compares two target-language candidates against a source-language reference, with matched source and target transcripts and a rubric.The primary experiments use an English reference and Japanese candidates.Source: §3 · p. 3
  • The structured response gives verdicts for five dimensions and a binary Overall verdict, with rationales citing localized, audibly verifiable observations.The dimensions are Speaker, Emotion, Timing, Pronunciation/Accent, and Audio Artifacts.Source: §3 · pp. 3–4
  • The study constructs more than 10,000 comparisons from heterogeneous speech synthesis systems and collects three human judgments per comparison.The benchmark includes 315 development and 290 test comparisons; only development data calibrate the supervision pipeline, and the test set is held out from calibration, training, and model selection.Source: §3 · p. 4
  • A randomly sampled panel listener agrees with the panel-majority label at 85.8% for Overall and 78.8% for averaged dimensional macro-F1.These are label-reliability references, not model-performance upper bounds.Source: §3 · p. 4
  • Zero-shot models remain below human agreement in Overall accuracy and dimensional macro-F1, and no model performs consistently across all five dimensions.Qwen2.5-Omni-7B predicts invalid Overall TIE on 78.3% of comparisons; Figure 2 covers three proprietary and five open-source models on the English–Japanese test set.Source: §3 · p. 4, Figure 2 · p. 4

4 METHOD: SCALING SUPERVISION FROM LIMITED HUMAN JUDGMENTS

The section uses limited human judgments to select and calibrate acoustic metrics, then supplies probabilistic hints to a machine labeler that still evaluates the audio. On held-out comparisons, the hints improve dimension-level alignment and TIE-rate calibration without detectable change in Overall accuracy.

4.1 SELECTING DOMAIN METRICS

  • Metrics are oriented so larger values favor a candidate, then selected by grouped out-of-fold macro-F1; no candidate beats the class-frequency baseline for Audio Artifacts.Table 1 summarizes retained measurements and their out-of-fold performance on the human-labeled development set.Source: §4.1 · p. 5, Table 1 · p. 5

4.2 CALIBRATING THE SELECTED METRICS

  • A final mapping is refit on all development comparisons and learned with cross-entropy to represent uncertainty in calibrated human choices rather than force a hard label.The standardization and regression parameters are then frozen.Source: §4.2 · pp. 5–6

4.3 GENERATING SCALABLE DIAGNOSTIC SUPERVISION

  • 6.3 points higher dimensional macro-F1 and a 10.4-point reduction in TIE-rate MAE follow from calibrated hints, with no detectable change in Overall accuracy.Both confidence intervals for dimensional macro-F1 and TIE-rate MAE exclude zero; the Overall-accuracy interval crosses zero.Source: Table 2 · p. 6, §4.3 · p. 6
  • For each unlabeled comparison, frozen mappings provide non-binding hints for four supported dimensions; Audio Artifacts is unknown and Overall receives no hint.The machine labeler checks the hints against the audio and may override them before generating verdicts and rationales.Source: §4.3 · p. 6

5 TRAINING SIGNALS FOR DIAGNOSTIC SPEECH JUDGES

The section compares SFT, OPD, and RL as distinct ways to train a diagnostic speech judge from imperfect machine-generated supervision. Each signal shapes different behavior rather than forming a uniform accuracy ladder.

5 TRAINING SIGNALS FOR DIAGNOSTIC SPEECH JUDGES

  • SFT raises zero-shot Qwen2.5-Omni-7B Overall accuracy from 13.10% to 69.54% and establishes rubric following, dimensional distinctions, and response structure.Because SFT uses fixed machine-generated targets, it remains exposed to their errors.Source: §5 · p. 6
  • RL raises Overall accuracy from 69.54% to 71.84% under the gated reward, mainly on high-consensus cases (+3.9 points), while dimensional macro-F1 falls from 52.37% to 47.64%.The overall gain is absent on low-consensus cases, and dimensional macro-F1 decreases further under the dimension-only reward.Source: Table 3 · p. 7, §5 · p. 7
  • Privileged Qwen3 OPD raises Overall accuracy from 56.78% to 73.79%, but dimensional macro-F1 is 44.61% versus 52.15% for SFT-teacher OPD.With the privileged teacher fixed, changing student initialization yields no consistent improvement, indicating that teacher choice and conditioning dominate initialization.Source: Table 3 · p. 7, §5 · p. 7
  • Figure 4 indicates that more supervision is not always helpful for SFT.Source: Figure 4 · p. 8
  • Overall, SFT establishes the task, OPD transfers the teacher’s judgment profile, and RL sharpens the verdict behavior encoded by its reward.The scalable supervision remains imperfect because it inherits errors from the machine labeler.Source: §5 · pp. 6, 8

6 UNDERSTANDING DIAGNOSTIC JUDGE BEHAVIOR

The section tests how supervision choices affect judge behavior and whether the pipeline transfers across languages. It finds that supervision quantity and adaptation choices matter, verdict rewards can improve rationale evidence, and judges transfer between the two studied target languages.

6.1 RQ1: HOW SHOULD SCALABLE SUPERVISION BE USED?

  • SFT performance initially improves with more machine-generated comparisons, then declines with the full pool; the authors use 80%, about 8,000 comparisons, in later experiments.The fixed-target training recipe and epochs are held constant as the supervision amount varies.Source: §6.1 · p. 8
  • Extending SFT adaptation to the audio path raises Overall accuracy from 69.54% to 72.41%, while dimensional macro-F1 remains unchanged.The audio projector alone is insufficient, and most of the Overall-accuracy gain comes from the encoder.Source: §6.1 · p. 8

6.2 RQ2: DOES BETTER JUDGMENT IMPLY BETTER DIAGNOSTIC REASONING?

  • With identical Overall verdicts, listeners prefer SFT+RL rationales for audible-evidence grounding in 11 of 28 judgments, versus one preference for SFT and 16 ties.The two-sided exact sign test gives p = 0.006; rationale text receives no direct reward.Source: Figure 5 · p. 8, §6.2 · p. 9

6.3 RQ3: DOES SPEECHCRITIC SCALE ACROSS LANGUAGES?

  • Monolingual judges transfer strongly across English–Japanese and English–Spanish, while joint training incurs no clear loss relative to either monolingual model.The authors treat VOX-DUB as an external reference because its dimensions and protocol differ from theirs.Source: Table 5 · p. 9, §6.3 · p. 9

6.2 RQ2: DOES BETTER JUDGMENT IMPLY BETTER DIAGNOSTIC REASONING?

  • Human preference for Qwen3-teacher rationales often occurs when a dimension-level verdict is wrong or both models choose the wrong Overall winner.Those rationales average about 488 tokens, compared with 322 for SFT-teacher OPD, so fluency or persuasiveness is not diagnostic reliability.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion presents SpeechCritic as a way to learn acoustically grounded diagnostic speech judges from limited human preferences. Its findings connect scalable supervision and training choices to judge decisions and explanations, while noting that verdict correctness and rationale grounding require separate evaluation.

7 CONCLUSION

  • The select–calibrate–scale pipeline turns roughly 300 human-labeled comparisons into more than 10,000 verdict-and-rationale training comparisons, improving agreement with human dimension judgments and reflecting human ties better.The resulting judges and supervision also transfer across Japanese and Spanish.Source: §7 · pp. 9–10

A.1 ANNOTATION PROTOCOL AND PROMPT

  • Raters choose an Overall A/B winner, then judge five dimensions independently with A, TIE, or B; Overall has no TIE option.The protocol excludes translation accuracy and makes comments optional.Source: §A.1 · pp. 13–14

A.2 ANNOTATION RESULTS AND BENCHMARK COMPOSITION

  • The benchmark uses development and test comparisons with a strict Overall majority; most received three annotations, while 30 shared anchors received 20.High- and low-consensus counts use only standard three-rater outcomes; comparisons without a strict Overall majority are excluded.Source: §A.2 · p. 14

A.3 DO THE FIVE DIMENSIONS EXPLAIN OVERALL PREFERENCE?

  • 93.0% out-of-fold accuracy on 302 English–Japanese development comparisons shows that equal-weight dimension votes capture most Overall choices, without establishing exhaustiveness or causality.The more flexible model with dimension-specific preference and TIE-rate features reaches 92.1% accuracy; equal-weight analysis reaches 82.6% on 293 English–Spanish comparisons.Source: §A.3 · pp. 14–15

A.4 AGREEMENT ANALYSIS

The agreement analysis treats human disagreement as a reference for interpreting model scores, not as a strict ceiling. It also distinguishes benchmark-specific TIE behavior from general properties of speech evaluation and describes how comparisons are constructed and held out.

A.4 AGREEMENT ANALYSIS

  • 85.82% Overall accuracy, 78.84% dimensional macro-F1, and 86.07% dimensional TIE recall are the English–Japanese test-set human agreement reference, not an independent-rater ceiling.The estimator is mildly optimistic because the sampled rater also contributes to the panel-majority label.Source: §A.4 · p. 15
  • Calibrated Gemini moves its dimensional TIE profile toward humans, and the OPD judge moves closer still overall, but the effect varies by dimension and does not establish comparison-level correctness.The reported table covers all 290 English–Japanese test comparisons, whereas the calibrated-versus-uncalibrated labeling-policy comparison uses a 272-comparison subset.Source: §A.4 · p. 15, Table 7 · p. 16
  • Raw pairwise agreement and Krippendorff’s α describe annotation reliability but are not directly comparable to model accuracy or macro-F1; three-rater leave-one-out agreement is downward biased on low-consensus items.The 20-rater anchors support a less biased leave-one-out diagnostic but are too few to represent the full test distribution.Source: §A.4 · p. 16

B.1 CONSTRUCTING DIVERSE SPEECH COMPARISONS

  • Comparisons are formed from randomly paired candidates drawn from diverse synthesis outputs, with speaker-level train/development/test splits and randomized, balanced A/B positions.Human development judgments select and calibrate metrics; test judgments remain untouched until final evaluation, and the natural–synthesized controls are secondary.Source: §B.1 · p. 16

B.2 SELECTING DOMAIN METRICS FOR HUMAN-CALIBRATED HINTS

The section describes how candidate domain metrics are screened to select evidence for human-calibrated hints.

B.2 SELECTING DOMAIN METRICS FOR HUMAN-CALIBRATED HINTS

  • Fourteen prespecified metrics are screened using grouped out-of-fold macro-F1 against a class-frequency baseline, with expert listening pilots when automatic validation is inconclusive.The strongest supported metric for each dimension is retained for the calibration step.Source: §B.2 · p. 17

C.1 TRAINING OBJECTIVES AND IMPLEMENTATION DETAILS

The appendix specifies the training procedures, release contents, evaluation protocol, and language-agnostic judge instructions. These details define how the SFT, OPD, and RL experiments are run and how verdicts and acoustic evidence are assessed.

C.1 TRAINING OBJECTIVES AND IMPLEMENTATION DETAILS

  • SFT trains on fixed machine-generated verdicts and rationales, OPD matches teacher and student next-token distributions on student-generated trajectories, and RL rewards parsed verdicts rather than rationale text.The principal SFT setting uses 7,840 comparisons; controlled RL reward comparisons use 651 labeled prompts.Source: §C.1 · p. 18

C.2 CODE RELEASE

  • The released configuration-driven code supports SFT, OPD, and RL training, but does not include trained checkpoints.Users initialize from specified public base models or compatible checkpoints.Source: §C.2 · p. 18

C.3 EVALUATION METRICS AND STATISTICAL PROTOCOL

  • Overall accuracy uses binary A/B labels, while dimensional macro-F1 averages across five dimensions after excluding items without a strict human-vote majority.A predicted Overall TIE is incorrect; reported seed variation is a sample standard deviation, not a confidence interval.Source: §C.3 · pp. 18–19

C.4 LANGUAGE-AGNOSTIC JUDGE SYSTEM PROMPT

  • The judge is instructed to assess candidates against the reference, judge each dimension independently, and ground rationales in concrete, localized audible cues.It must not judge translation meaning or invent numerical measurements, and should return an Overall A/B verdict that addresses trade-offs.Source: §C.4 · p. 20

D ADDITIONAL EXPERIMENTAL RESULTS

The additional experiments test supervision scale, training and teacher choices, test-time voting, and external cross-lingual validity. Results show trade-offs across verdict accuracy and dimensional diagnosis, with benchmark label distributions and rationale-study reliability requiring careful interpretation.

D ADDITIONAL EXPERIMENTAL RESULTS

  • Performance improves over smaller fractions of machine-generated supervision but declines when the full pool is used, showing that more imperfect supervision is not necessarily better.The SFT recipe and number of epochs are held fixed.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 21
  • Target and input controls suggest performance does not hinge on any single textual field, while a natural-speech control tests whether comparisons extend beyond synthesized pairs.These are single-run controls, so differences are treated as sensitivity evidence rather than model rankings.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 21
  • Privileged task information makes Qwen3 the strongest teacher for Overall accuracy, while changing student initialization has a smaller effect.Audio-encoder updating can shift the Overall–dimensional balance without removing the teacher-specific profile transferred by OPD.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 21
  • MeanBlend, CorrectGated, and SumJSD do not uniformly dominate stronger single-teacher configurations; Spanish rows test the teacher choices on a second target language.Qwen2.5 better preserves the dimensional rubric, whereas Qwen3 provides stronger Overall-verdict supervision; MeanBlend uses three seeds, while the other controls are single-run.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 21
  • 71.03% from audio-path adaptation plus gated RL is below audio-adapted SFT at 72.41% and LLM-only SFT followed by RL at 71.84%.The two interventions correct only five of the same errors (Jaccard overlap = 0.13); starting RL from OPD gives the strongest composed result, while encoder updating during RL offers no consistent additional gain.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 21
  • Majority voting usually beats an average sampled response but does not reliably beat greedy decoding, and paired-bootstrap significance in Overall accuracy appears only for SFT.Across five representative judges, draw-to-draw agreement provides a correctness AUC of 0.58–0.68, making it a more useful confidence signal than added diagnostic capability.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 21

D.1 EXTERNAL CROSS-LINGUAL EVALUATION

  • 38.7 and 38.8 mean macro-F1 for English–Spanish and joint-language judges are close to Gemini-3.1-Pro’s 40.0, and both judges exceed Gemini on pronunciation.No evaluated model beats Always-TIE on voice, where gold labels are 80% TIE and inter-rater reliability is low; VOX-DUB is an out-of-distribution check with no Overall verdict and an approximate naturalness–Timing mapping.Source: §D.1 · pp. 21, 23, Table 10 · p. 23

E.1 PROTOCOL

  • The rationale evaluation compares anonymous responses after listeners hear the reference and candidates, with model identities and automatic scores hidden.The protocol includes 244 unique English–Japanese test comparisons plus 22 repeated assignments for quality control, and paired models in each comparison share the same Overall verdict.Source: §E.1 · p. 23, Table 11 · p. 23

E.2 PAIRWISE RESULTS

Listeners generally found SFT and SFT+RL rationales similar, but directional preferences favored RL for evidence grounding. The study’s limited repeat agreement bounds how broadly to interpret these preferences.

E.2 PAIRWISE RESULTS

  • 11 of 12 directional evidence-grounding judgments favored SFT+RL over SFT (91.7%; two-sided exact sign test p = 0.006), while 16 found no meaningful difference.Most judgments found the rationales similar overall, but perceived differences favored RL, especially for audible-evidence grounding.Source: §E.2 · p. 24
  • Qwen3-teacher OPD rationales received stronger grounding preference than SFT-teacher OPD rationales despite often accompanying incorrect verdicts.Textual detail or persuasiveness is not treated as evidence of diagnostic correctness.Source: §E.2 · p. 24
  • Repeated-comparison agreement was 27% for overall rationale preference and 41%–64% across the four criteria, so the study supports large coherent patterns rather than a complete quality ranking.A text-only GPT-5.4 grader was not used to establish grounding because its localization measures were weakly predictive or no better than chance.Source: §E.2 · p. 24

Related papers

Loading 2609.34582v2…