Source-linked AI summary
RAWD-TTS: Ratio-Free Reward Alignment for Discrete-Diffusion Voice Cloning
Maxim Maslov, Kirill Borodin, Vasilii Kudryavtsev, Nikita Vasiliev, Grach Mkrtchian
TL;DR
Supervised acoustic-token prediction does not directly optimize decoded speech intelligibility or speaker identity, and discrete-diffusion sampling complicates reward alignment. RAWD-TTS scores generated samples and weights their masked-token reconstruction with group-relative advantages. On 500 Russian CV3-Eval prompts, joint alignment reduces WER by 24.0% at the reward-selected checkpoint and 19.0% at the final checkpoint, while improving speaker similarity over base OmniVoice.
Problem
Supervised acoustic-token prediction does not directly measure intelligibility or speaker preservation in decoded speech, while discrete diffusion couples token choices with reveal positions.
Source: §1 · p. 1Method
RAWD-TTS scores decoded voice-cloning samples with recognition and speaker rewards, then uses group-relative advantages to weight masked-token reconstruction of those samples.
Source: §1 · p. 1, §3.1 · p. 2Results
24.0% WER reduction at the reward-selected checkpoint and 19.0% at the final checkpoint on Russian CV3-Eval, with speaker similarity improving over base OmniVoice.
Source: §6 · p. 4Takeaways & Limitations
Single-reward and supervised controls reveal recognition–identity trade-offs, supporting joint waveform rewards as a practical alignment objective for discrete TTS.
Source: §6 · p. 4Reported checkpoints are reward-selected on the evaluated prompts, and training uses stochastic rollouts without CFG while evaluation uses CFG 2 and greedy tokens.
Source: §5.3 · p. 4
Abstract
from arXiv · showhide
Zero-shot text-to-speech synthesizes new utterances in a speaker's voice from a short reference recording. Voice cloning requires accurate content and preserved speaker identity, but supervised acoustic-token prediction does not directly optimize these waveform-level properties. Reward-based post-training addresses this mismatch, but in discrete diffusion, token choices and reveal positions jointly define the sampling trajectory, complicating alignment. We introduce RAWD-TTS (Ratio-free Advantage-Weighted Denoising), which scores decoded samples with recognition and speaker rewards and uses group-relative advantages to weight masked-token reconstruction of those samples, without reverse-trajectory likelihoods or target audio. On 500 Russian CV3-Eval voice-cloning prompts, joint alignment reduces word error rate from 3.18% to 2.42% at the reward-selected checkpoint (24.0% relative) and to 2.58% at the final checkpoint (19.0%), while WavLM speaker cosine rises from 0.733 to 0.748 and 0.755. Controlled experiments characterize recognition-identity trade-offs and the effects of corruption count, group composition, and weighting.
1. INTRODUCTION
Zero-shot TTS must preserve both requested words and an unseen speaker’s identity, but supervised token loss does not directly measure either decoded-waveform property. RAWD-TTS weights reconstruction of generated samples with group-relative reward advantages, adapting reward alignment to discrete-diffusion voice cloning.
1. INTRODUCTION
- Supervised acoustic-token loss does not directly measure intelligibility or speaker preservation in the decoded waveform, motivating task-specific reward-based post-training.Recent flow-matching TTS alignment methods optimize recognition and speaker rewards through denoising trajectories [1] [2].Source: §1 · p. 1
- Discrete-diffusion TTS jointly determines token choices and reveal positions, so a final waveform cannot simply be assigned the product of autoregressive token probabilities.Trajectory-based optimization can model transitions, but requires a likelihood appropriate to the actual sampling process.Source: §1 · p. 1
- RAWD-TTS adapts signed final-sample denoising to conditional multi-codebook TTS, preserving voice-cloning conditioning and normalizing reconstruction across codebooks and utterances.The paper also reports controlled studies of reward composition, corruption count, group allocation, and weighting, with released training code.Source: §1 · p. 1
- RAWD-TTS uses generated tokens as waveform-rewarded samples and reconstruction targets, with group-relative advantages weighting reconstruction losses as the policy-update path.Training requires reference speech and target text, not target audio.Source: Figure 1 · p. 1, §1 · p. 1
2. RELATED WORK
Prior reward-alignment work spans flow-matching and autoregressive speech synthesis, while ratio-free weighted denoising and related methods provide precedents for diffusion optimization. RAWD-TTS adapts this weighting principle to multi-codebook speech with waveform rewards and studies its empirical behavior.
2. RELATED WORK
- Flow-matching and autoregressive TTS methods study recognition and speaker rewards, but their transitions differ from categorical masked-token updates.F5R-TTS applies group-relative policy optimization [1], FlowTTS-GRPO studies multi-objective waveform rewards [2], and Koel-TTS aligns an autoregressive model [5].Source: §2 · p. 1
- Ratio-free weighted denoising in wd1 and Advantage Weighted Matching [8] are methodological precedents; UDM-GRPO retains a ratio-based objective [9].Source: §2 · p. 1
- The weighting principle follows wd1 and AWM, while RAWD-TTS contributes its adaptation to multi-codebook speech with waveform rewards and the resulting empirical study.SEPO, d1 [11], and MaskGRPO [12] also address discrete diffusion using score entropy, approximate sequence likelihood, or estimated importance weights, respectively.Source: §2 · p. 2
3. METHOD
RAWD-TTS samples voice-cloning rollouts, scores their decoded speech with recognition and speaker rewards, and uses group-relative advantages to weight reconstruction of final samples. Its signed surrogate replaces native trajectory likelihood with a masked-diffusion evidence bound, without a likelihood-ratio constraint.
3.1. Voice-cloning rollouts and terminal rewards
- For each prompt, the native voice-cloning sampler generates acoustic tokens that are decoded into waveforms; frozen recognition and speaker models score independent stochastic rollouts.The prompt contains target text, reference audio, and its transcript, with reference and text conditioning fixed within each group.Source: §3.1 · p. 2
- Joint training sums recognition and speaker rewards with coefficients (1, 1), while single-reward runs set one coefficient to zero; each reward component lies in [0, 1].The recognition reward applies the same Russian text normalization to both texts, and its coefficient 2.5 is empirical.Source: §3.1 · p. 2
3.2. Final-sample denoising surrogate
- Each rollout’s detached advantage weights reconstruction of its final tokens after K forward corruptions, with one corruption ratio shared across each prompt group.Text, reference tokens, and padding are never masked; empty target masks are redrawn, then a target token is forced masked if needed.Source: §3.2 · p. 2, §3.1 · p. 2
- The reconstruction reduction averages masked tokens within codebooks before combining codebooks, so longer utterances do not receive larger weights merely for containing more tokens.Gradients accumulate across groups and corruption states before one optimizer step; training uses neither KL regularization nor a supervised anchor loss.Source: §3.2 · p. 2
3.3. Interpretation and weighting control
- The surrogate uses a masked-diffusion evidence lower bound in place of the native sampler’s intractable trajectory likelihood, and K controls forward-corruption estimation independently of reverse decoding steps.Because a bound replaces trajectory likelihood, the objective is signed rather than an unbiased policy-gradient estimator and imposes no likelihood-ratio constraint on native position selection.Source: §3.3 · p. 2
- The wd1-inspired control retains the reconstruction reduction without rescaling its weights, changing both relative coefficients and gradient scale.The weighting control is inspired by wd1.Source: §3.3 · p. 2
4. EXPERIMENTAL SETUP
Training uses Russian voice-cloning prompts paired with target text and initializes all systems from base OmniVoice. Evaluation tests 500 Russian CV3-Eval prompts with both training scorers and independent evaluators, while checkpoint selection and decoding settings define the reported comparisons.
4.1. Training data and implementation
- Training uses 8,000 Russian voice-cloning prompts from YouTube, filtered to single-speaker 2–8-second references, and pairs each with target text from a different eligible record.The 10.96 hours of reference audio average 4.93 seconds; target-record audio is not used for training or rewards.Source: §4.1 · pp. 2–3
- All systems initialize from 0.6B base OmniVoice with eight frozen-codec codebooks and LoRA; controlled comparisons use B = 4, G = 8, and K = 3.Joint K = 5 changes the corruption count; the K = 5 run took 9.25 GPU-hours for 2,000 iterations on one RTX 6000 Ada Generation GPU.Source: §4.1 · p. 3
4.2. Evaluation protocol
- Evaluation uses all 500 Russian CV3-Eval prompts and reports Whisper WER and WavLM-ECAPA cosine, plus independent Qwen3-ASR WER, ERes2Net cosine, and DistilMOS.Independent scorers do not participate in training or checkpoint selection; DistilMOS is a predicted score, not a listening-test result.Source: §4.2 · p. 3
- Primary runs train for 2,000 updates, while corruption and group-size ablations use 1,000; one training run is conducted per configuration.Table 1 reports reward-selected checkpoints separately from unselected final checkpoints, and K = 5 has a coarser evaluation grid than K = 3.Source: §4.2 · p. 3
- The SFT control uses real target audio and transcripts, unlike randomized-target reward-alignment training; controlled alignment comparisons hold the backbone fixed.The evaluation figure tracks WER and speaker cosine on all 500 prompts across updates, showing single-reward optima and the reward-selected joint checkpoint.Source: §4.2 · p. 3, Figure 2 · p. 3
5. RESULTS AND DISCUSSION
Joint RAWD-TTS improves recognition while retaining speaker similarity, but single-reward alignment exposes a recognition–identity trade-off. Checkpoint behavior is non-monotonic, ablations show configuration effects, and the evidence is limited in scope.
5.1. Recognition and speaker-identity trade-off
- Selected Joint K = 5 attains 2.42% Whisper WER and 0.748 WavLM cosine; its unselected 2,000-update final checkpoint records 2.58% WER (19.0% relative reduction) and 0.755 cosine.Source: Table 1 · p. 3, §5.1 · p. 4
- Recognition-only alignment lowers WER from 3.18% to 2.35% but lowers speaker cosine from 0.733 to 0.705; speaker-only reaches 0.763 cosine as WER rises to 3.34%.Joint K = 3 gives 2.41% WER and 0.736 cosine.Source: §5.1 · p. 4
- After selected update 1,300, joint WER rises from 2.41% to 2.82% at update 2,000 while cosine improves from 0.736 to 0.743.Training-batch rewards rise steadily over the run, so they do not imply monotonic benchmark improvement.Source: §5.1 · p. 4
5.2. Hyperparameter ablations
- At 1,000 updates, K = 5 improves both metrics over K = 1 and K = 3, while similarity is non-monotonic with K and group composition trades prompt diversity against within-prompt comparisons.At BG = 32 and K = 3, (B, G) = (4, 8) gives lower WER than (8, 4) or (2, 16); one run per setting does not establish reliability.Source: §5.2 · p. 4, Table 2 · p. 4
5.3. Limitations
- Evidence is limited to one backbone, Russian speech, and one training run per configuration.Evaluation uses CFG 2 while training uses stochastic rollouts without CFG, so the reported gains transfer to the guided evaluator rather than directly optimizing guided generation.Source: §5.3 · p. 4
6. CONCLUSION
The conclusion presents RAWD-TTS as an adaptation of signed final-sample denoising for discrete-diffusion voice cloning. On Russian CV3-Eval, joint alignment improves recognition and speaker similarity over base OmniVoice, with gains also reflected by independent evaluators.
6. CONCLUSION
- RAWD-TTS adapts signed final-sample denoising to discrete-diffusion voice cloning.On CV3-Eval Russian, it uses five corruption views for joint alignment.Source: §6 · p. 4
- Joint alignment reduces WER by 24.0% at the reward-selected checkpoint and by 19.0% at the final checkpoint, while improving speaker similarity over base OmniVoice.Independent evaluators are consistent with both gains.Source: §6 · p. 4
- Single-reward and supervised controls reveal recognition–identity trade-offs, supporting joint waveform rewards as a practical alignment objective for discrete TTS.Source: §6 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The study used previously collected public speech recordings and collected no new human-subject data. The authors report that ethical approval was therefore not required and declare no conflicts of interest.
7. COMPLIANCE WITH ETHICAL STANDARDS
- The study used only publicly available, previously collected Balalaika and CV3-Eval speech recordings.Source: §7 · p. 4
- No new human-subject data were collected, so ethical approval was not required; the authors declare no conflicts of interest.Source: §7 · pp. 4–5