When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models

Joonyong Park; Jerry Li

Source-linked AI summary

When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models

Joonyong Park, Jerry Li

arXiv:2608.31035v1Aug 31, 2026cs.CLcs.SDeess.AS

TL;DR

The paper asks when learned perceptual predictors can guide RL for codec-based speech generation while preserving human alignment. It studies constrained GRPO and matched Best-of-N reranking, finding uneven transfer and supporting predictor–axis–base diagnostics for reward selection.

Problem
Learned perceptual rewards may not remain aligned with listener judgments when optimized for codec-based speech generation.
Method
The study applies GRPO with a CER-zone hard constraint, compares it with Best-of-N reranking, and evaluates machine gains using human tests and reward diagnostics.
Results
Predictor gains transfer unevenly to listeners: ANIMESCORE is a strong positive case, UTMOS shows modest average preference shift, and LIKABILITY fails on average despite calibrated high-confidence regions.
Takeaways & Limitations
Subjective speech rewards should be analyzed as predictor–axis–base tuples using diagnostics before multi-reward post-training.
Takeaways & Limitations
The experiments do not fully disentangle reward-model architecture, perceptual axis, and base-model distribution, so findings are diagnostics rather than causal claims about axis difficulty.

Abstract

from arXiv · show

Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.

1 Introduction

Codec-based TTS enables RL post-training over acoustic tokens, but learned perceptual rewards may diverge from listener judgments. This study introduces constrained GRPO, compares it with reward-based reranking, and evaluates transfer using human studies and reward diagnostics.

Codec-based TTS converts waveforms into discrete acoustic codes that speech language models generate autoregressively from text and optional reference audio.
Learned perceptual predictors offer scalar rewards for listener-defined attributes, but policies can increase proxy scores without preserving human-perceptual alignment.
The study uses GRPO with a CER-zone hard constraint and compares it with Best-of-N reranking under the same reward gate.
Multi-rater human tests show that machine-level reward gains transfer unevenly across predictor–axis–base tuples.
The paper evaluates reward-gap calibration, base-output spread, domain match, and within-zone signal strength to assess subjective rewards for RL and future multi-reward post-training.
The work characterizes when predictor-based RL remains aligned with human perception in codec-based speech language models.

2 Related Work

Related work frames codec-based speech generation as autoregressive modeling over discrete acoustic tokens and surveys RL, preference optimization, and learned perceptual predictors. It also motivates testing whether proxy-score improvements correspond to listener preference.

Codec-based speech language models formulate TTS as autoregressive generation over discrete acoustic tokens produced by neural audio codecs.
Existing TTS post-training methods use RL, preference optimization, and differentiable reward prediction for objectives including intelligibility, speaker similarity, stability, and prosody.
Learned predictors such as MOSNet, UTMOS, and NISQA convert subjective speech judgments into scalar quality estimates that can serve as proxy objectives.
Predictor scores may be miscalibrated, out-of-domain, insensitive to relevant differences, or vulnerable to overoptimization, motivating direct listener evaluation.

3 Constrained Perceptual GRPO

The method combines perceptual rewards with a hard CER-zone gate and group-relative optimization. It monitors constraint-aware validation behavior rather than selecting checkpoints by raw predictor scores alone.

The codec speech language model samples acoustic-token sequences from text, decodes them into waveforms, and applies perceptual predictors to the decoded speech.
The CER constraint measures transcription error between the ASR transcript of generated speech and the input text.
CER-zone reward template: The reward template assigns normalized predictor reward plus a CLEAN bonus, predictor reward in the FEASIBLE zone, and a fixed penalty in the VIOLATE zone.
CER-zone reward template: The method sets τl = 0.10, τh = 0.30, b = 0.5, and ρ = 1.0, preventing perceptual reward from numerically compensating for transcription failure.
GRPO optimization and stopping: GRPO samples K = 4 rollouts per prompt, subtracts an adaptive in-reward KL penalty, and normalizes rewards within each rollout group.
GRPO optimization and stopping: Checkpoints maximize constraint-aware validation reward while monitoring CER violations and KL drift rather than relying on raw predictor scores alone.

4 Experimental Setup

The study evaluates codec-based TTS with learned perceptual rewards, CER-gated GRPO, Best-of-N reranking, and machine- and human-level tests.

Base speech model: Llasa with XCodec2 provides the codec-based TTS backbone, tokenizer, and waveform decoder.
Reward predictors: Three learned perceptual predictors support the main experiments, with MSP-Dim arousal used only for a training-dynamics diagnostic.
Reward predictors: ANIMESCORE measures anime-like speaking style, UTMOS predicts MOS-like naturalness on [1, 5], and LIKABILITY predicts likability from scores on [1, 6].
Data and evaluation: GRPO uses 900 Japanese Wikipedia-derived prompts, with 100 disjoint validation prompts and 50 held-out Japanese test prompts spanning five prompt groups.
Evaluation protocol: First-shot evaluation uses one fixed-seed generation, while CER-retry regenerates outputs above CER 0.30 and applies the same selection rule to both systems.
Evaluation metrics: Machine evaluation reports target and cross-axis predictor shifts, CER statistics, and violation rates; human tests use 50 paired items per axis and five ratings per item.
Baselines: Best-of-N samples N base-model candidates and selects the highest-scoring CER-gated candidate, contrasting sample selection with policy optimization.

5 Machine-Level Behavior and Baselines

Single-reward GRPO produces axis-specific machine shifts rather than universal quality gains, while CER gating and Best-of-N expose reward- and deployment-dependent trade-offs.

Cross-axis specificity: The optimized axis receives the largest positive shift in every single-reward run, indicating axis-specific movement rather than generic quality improvement.
Cross-axis specificity: No single subjective reward acts as a universal surrogate for all desired speech properties.
Policy optimization versus Best-of-N reranking: Best-of-N reward-selected samples already exist in the base model’s support, whereas GRPO tests whether selection behavior can be amortized into the policy.
Policy optimization versus Best-of-N reranking: GRPO is not uniformly better than reranking; the value of policy optimization depends on the reward axis and deployment trade-off.

6 Human Evaluation

Human transfer is uneven across reward axes: ANIMESCORE succeeds on average, while LIKABILITY fails on average despite calibrated high-confidence regions. Reward gaps predict pooled listener choices better than residual CER gaps, but calibration varies by axis, and GRPO is not clearly preferred over Best-of-8.

6.1 Axis-level transfer: ANIMESCORE is the clearest average-transfer success, while LIKABILITY fails on average and UTMOS produces only a modest aggregate human preference shift.LIKABILITY's average failure masks a calibrated high-confidence region.
6.2 Comparison between GRPO and Best-of-N in human preference: Across all three axes, human preference between GRPO and Best-of-8 is near chance: HWR is 52.0 for ANIMESCORE, 46.0 for UTMOS, and 48.0 for LIKABILITY.Listeners do not clearly prefer GRPO over the strong Best-of-8 baseline.
6.3 Reward-gap calibration: A one-standard-deviation increase in signed reward gap favoring GRPO increases the odds of choosing GRPO by 1.93×, whereas the CER gap is not predictive.The pooled vote-level logistic regression uses within-axis standardized reward and CER gaps with axis fixed effects and item-clustered robust standard errors.
6.3 Reward-gap calibration: Reward-gap calibration is heterogeneous: LIKABILITY and UTMOS have significant reward-gap slopes, while ANIMESCORE's positive slope is not significant.Calibration is therefore a local confidence diagnostic within a predictor–axis–base tuple, not the sole explanation of average transfer.
6.3 Reward-gap calibration: Retaining all 50 items despite residual CER-threshold violators changes vote-level HWR by +1.9 pp for ANIMESCORE, −2.1 pp for UTMOS, and −4.5 pp for LIKABILITY, without changing qualitative conclusions.The protocol retains these items to avoid post-hoc filtering by a metric tied to the RL objective.

7 Discussion: Diagnostics for Predictor–Axis–Base Tuples

Subjective speech rewards should be evaluated as predictor–axis–base tuples, because machine reward gains can transfer unevenly to listeners. Reward-gap calibration, base-output spread, domain validation, and within-zone signal provide practical screening diagnostics, but they are evidence-supported heuristics rather than causal rules.

Reward-gap calibration: Reward-gap calibration distinguishes average listener transfer from within-axis agreement between signed reward advantages and listener choices.Pooled analyses found signed reward gaps predictive of GRPO-side preference, while per-axis calibration remained heterogeneous.
Reward-gap calibration: Likability failed on average but showed a significant per-axis reward-gap slope, indicating informative predictions in high-confidence regions.Few items reached a reward-gap regime considered perceptually reliable under this predictor–axis–base tuple.
Base-output spread: ANIMESCORE had the widest base-output score spread and the strongest positive transfer, whereas LIKABILITY had narrower spread and failed on average.Base-model score spread is proposed as a screening signal for predictor resolution on the target distribution.
Within-zone signal under constraints: A subjective predictor can be meaningful as an evaluator yet ineffective as an RL reward when its within-zone variation is too small relative to the CER penalty.In the VAD-AROUSAL run, validation reward improved mainly through fewer CER violations, while validation arousal stayed within seed-level variation; no human A/B study was conducted.
Practical screening: Practical screening checks include reward-gap calibration, base-output spread, domain coverage, and within-zone variation before scaling subjective-reward RL.These diagnostics are intended to identify rewards likely to produce low-confidence transfer, not to establish causality.

Conclusion

The study finds that learned subjective-reward gains do not reliably transfer to human-perceptual gains in codec-based speech language models. Transfer and calibration must be assessed alongside predictor resolution, domain validation, and within-zone signal, while Best-of-N results position GRPO as policy-level amortization of reward-based selection.

Conclusion: Predictor gains alone are insufficient: average human transfer and within-axis reward-gap calibration can diverge.The conclusion also identifies predictor resolution, domain validation, and within-zone signal under intelligibility constraints as relevant considerations.
Conclusion: ANIMESCORE provides a strong in-domain positive case, UTMOS shows high machine–human agreement with modest average preference shift, and LIKABILITY fails on average but aligns in high-confidence regions.VAD-AROUSAL fails as a training-time reward under the constrained scaffold.
Conclusion: Best-of-N reranking indicates that reward-selected samples often already exist in the base model’s support, so GRPO amortizes selection into policy-level movement.The conclusion does not characterize GRPO as uniformly perceptually superior to reranking.
Conclusion: The findings support practical screening heuristics for subjective speech rewards before full RL or multi-reward speech post-training.The proposed scope is diagnostic rather than a universal rule for reward success.

Limitations

The experiments do not fully disentangle reward-model architecture, perceptual axis, and base-model distribution, and their generality is constrained by compute and annotation budgets. Results should therefore be interpreted as diagnostics for specific predictor–axis–base tuples.

Scope: The study cannot fully disentangle reward-model architecture, perceptual axis, and base-model distribution.Reward models differ in training data, score scale, target construct, and base-model coverage.
Scope: The results should be read as diagnostics for predictor–axis–base tuples rather than causal claims about intrinsic difficulty across perceptual axes.This scope boundary follows from differences among the reward models and their coverage.
Evaluation constraints: Compute and annotation budgets limited the evaluation to one decoding configuration, fixed reward axes, Best-of-8 reranking, and relatively few listeners per item.The paper proposes broader models, listener populations, reranking budgets, seeds, and rollout measurements as extensions.

Ethical Considerations

The paper frames controllable synthetic-speech post-training around evaluation and diagnostics because target-style generation can lower the cost of nonconsensual use. Released artifacts are restricted to research-oriented materials, with guidance against impersonation, voice cloning, and unauthorized style imitation.

Ethical scope: Controllable synthetic speech can support creative and accessibility applications but may also reduce the cost of generating speech in a target style without consent.The authors therefore frame the work around evaluation and diagnostics rather than deployment.
Artifact release: The release includes code, prompts, generated audio samples, and reward scores, but excludes weights and training corpora whose licenses or copyright status prevent redistribution.The stated release boundary is intended for research use.
Responsible use: Although ANIMESCORE targets style rather than speaker identity, the paper notes that style and identity can interact in downstream use.Responsible-use guidance excludes impersonation, voice cloning, and unauthorized style imitation.
Future considerations: Future work should examine style-control rewards across demographic groups, listener communities, and culturally specific notions of expressiveness or likability.This scope is identified as a broader consideration for subjective speech rewards.

Use of AI Assistance

The paper documents AI-assisted language polishing, LaTeX editing, and presentation brainstorming, while authors verified and revised the scientific content. It also specifies the evaluation, reward-predictor, and training infrastructure used throughout the study.

Use of AI Assistance: AI assistants supported language polishing, LaTeX editing, and brainstorming presentation of results.The authors state that all scientific claims, experiments, analyses, and final manuscript content were verified and revised by them.
Implementation: Training used verl with vLLM-backed rollouts on one H100 GPU, while reward predictors ran on a second H100.The separate predictor GPU isolated codec decoding and predictor inference from the rollout engine.
Reward Predictor Implementation: Each reward predictor decoded generated XCodec2 tokens into a 16 kHz mono waveform and returned a scalar score.The documented predictors covered anime-likeness, naturalness, likability, and arousal axes.
Evaluation: The held-out test set was excluded from reward-model training, GRPO training, and checkpoint-selection validation.This separation defines the evaluation-set boundary for machine and human testing.
CER Robustness: Residual CER violations were assessed after retry, excluding item pairs with either side above τh = 0.30 in the clean-only robustness check.The union count records pairs excluded when either compared output remained above the threshold.
Reward-Gap Analysis: The pooled logistic analysis found reward gaps significant while CER-gap differences were statistically indistinguishable from zero.Adding standardized reward gaps improved fit, whereas adding CER gaps did not; the analysis therefore treated reward gaps as stronger predictors of listener preference.

I VAD-Arousal Training-Only Negative Result

The VAD-AROUSAL run produced only a small validation-arousal increase despite improved validation reward. The evidence is limited to training dynamics, because no human A/B study evaluated this axis.

Scope: No human A/B study was conducted for VAD-AROUSAL, so the result concerns training dynamics rather than perceptual transfer.This scope boundary limits what can be concluded about listener alignment for the arousal reward.
Arousal Outcome: Validation arousal remained between 0.61 and 0.64 during early training, with a net change of approximately +0.014.The reported change indicates little movement on the target arousal metric during this period.
Training Dynamics: Validation reward improved from −0.34 to −0.05 while the violation rate fell from 0.62 to 0.45.The paired changes show reward improvement alongside fewer constraint violations.
Interpretation: The authors interpret the reward gain as primarily moving samples out of the VIOLATE shelf rather than increasing arousal within the feasible zone.Because per-rollout reward decompositions were not retained, this remains a training-dynamics diagnostic rather than causal attribution of individual updates.
Checkpoint Selection: For ANIMESCORE, raw validation reward rose from +0.625 to +0.955 between steps 1400 and 1710, but constraint-aware reward decreased from 2.584 to 2.556.The violation rate also increased from 0.26 to 0.31, motivating selection of step 1400.