Source-linked AI summary
Preference Optimization for Non-Verbal Vocalization Synthesis
Haoyang Li, Chenglin Xu, Junchuan Zhao, Yuang Cao, Liumeng Xue, Yiwen Guo, Eng Siong Chng
TL;DR
Faithful NV generation remains insufficiently understood in expressive TTS, especially regarding how preference signals and DPO design choices affect NV and lexical accuracy. The paper introduces weighted NV-CER and systematically evaluates preference construction and objectives, finding an effective standard-DPO setup supported by objective, LLM-based, and human evaluations. These results provide practical guidance for NV-aware TTS post-training.
Problem
Preference optimization for NV-capable TTS has limited evidence about its specific effects on NV realization and the design choices governing performance.
Method
The paper uses NV-ASR-derived preference signals, weighted NV-CER over pinyin and NV-tag sequences, and systematic studies of preference pairs and DPO objectives.
Results
Objective, LLM-based, and human evaluations support improvements over baselines and establish an effective setup using standard DPO.
Takeaways & Limitations
Weighted NV-CER provides controllable emphasis on NV realization versus lexical fidelity without modifying the underlying optimization algorithm.
Abstract
from arXiv · showhide
Non-verbal vocalizations (NVs), such as laughter, coughs, and sighs, are essential for expressive TTS, but the effectiveness of preference optimization for NV generation remains poorly understood. We systematically study preference optimization for NV-capable TTS, focusing on preference signals, preference-pair construction, and DPO-based optimization objectives. We formulate an NV-aware character error rate (NV-CER) by treating NV tags as distinct output symbols and computing a weighted pinyin-based CER over both verbal and non-verbal content, enabling controllable optimization of NV realization without modifying the underlying optimization algorithm. Experiments on Emilia-NV and the augmented NV-Bench covering 18 NV types reveal how different design choices affect NV realization and lexical fidelity, and establish an effective setup using standard DPO. Objective, LLM-based, and human evaluations provide converging evidence for our findings, offering practical insights into NV-aware post-training for expressive TTS.
I. INTRODUCTION
NV generation is essential yet underexplored in expressive TTS, and existing preference-optimization studies provide limited evidence about its specific effects. The paper addresses this gap by systematically studying NV-aware signals, pair construction, and DPO objectives.
- Non-verbal vocalizations convey emotion, discourse structure, and speaker intent, but faithful generation remains challenging and relatively underexplored in TTS.
- Existing NV-aware preference-optimization studies mainly report overall synthesis quality or omit NV evaluation, leaving their impact on NV generation unclear.
- Manual preference pairs that contrast speech with and without NVs encourage NV occurrence rather than faithful realization and were reported to degrade CER.
- The paper introduces weighted NV-CER over pinyin lexical tokens and distinct NV tags, enabling control of NV-versus-lexical emphasis without changing the optimization algorithm.
- Systematic experiments examine preference signals, preference-pair construction, and DPO loss formulations, with objective, LLM-based, and human evaluations supporting practical NV-aware post-training principles.
2) NV-Aware Character Error Rate:
NV-CER measures lexical and non-verbal correctness in one token-level error rate. It uses pinyin for Mandarin lexical content while preserving NV tags as independent symbols.
- NV-CER measures both lexical and NV correctness by computing edit distance over combined reference and predicted token sequences.
- Mandarin lexical tokens are represented with pinyin to reduce sensitivity to homophones and character variations, while NV tags remain independent symbols.
- The weighted edit distance uses token-dependent insertion and deletion costs, with substitution cost equal to the maximum cost of the two tokens.
3) Controllable NV Preference Strength:
The framework controls NV preference strength by weighting NV-tag errors, then constructs DPO pairs from multiple synthesized candidates ranked by NV-CER. Alternative construction strategies are evaluated experimentally.
- A higher NV-tag weight increases the contribution of NV errors to NV-CER and gives them greater influence in preference-signal construction.
- Setting wNV = 1 assigns equal cost to lexical and NV tokens, while larger values emphasize NV accuracy relative to lexical accuracy.
- The pretrained TTS model independently generates K candidate utterances, each transcribed by NV-ASR and scored against the reference with NV-CER.
- The lowest-NV-CER candidate becomes the preferred response x+, and the highest-NV-CER candidate becomes the rejected response x−.
- The study also tests ground-truth-based preference selection and manipulated synthetic speech as alternative preference-pair construction strategies.
2) Preference Optimization Objective:
The paper uses DPO to increase the relative likelihood of preferred over rejected speech responses, while comparing standard DPO with SFT-augmented and alternative-ranking objectives.
- DPO directly increases the trainable model’s relative likelihood of the preferred response over the rejected response.
- In the DPO objective, πθ and πref denote the trainable and reference TTS models, while β controls preference strength.
- The study compares standard DPO with combinations of DPO and conventional supervised fine-tuning, as well as alternative preference rankings.
III. EXPERIMENTS
Experiments train on Emilia-NV and evaluate expressive Mandarin speech with objective metrics covering intelligibility, perceptual quality, and speaker similarity. The evaluation includes NV-Bench test sets and compares loss and preference-pair configurations.
- Datasets: Emilia-NV provides 573.4 hours of Mandarin speech annotated with 18 predefined NV tags.The tags cover laughter, breathing, hesitation, and other vocal events.
- Datasets: NV-Bench includes single-label and multi-label Mandarin evaluation subsets, with 50 utterances per NV category in the single-label subset and 391 multi-label utterances.NV-Bench covers 15 of the 18 training tags.
- Objective metrics: NV-CER jointly measures lexical content and NVs, while CER and PCER separately measure verbal and non-verbal generation.NV-CER uses wNV = 1 in the reported intelligibility evaluation.
- Objective metrics: DNSMOS P.835 and UTMOS assess perceptual speech quality, while SECS measures speaker similarity from speaker-embedding cosine similarity.SECS embeddings are extracted with Resemblyzer.
- Ablations: The ablation tables examine loss formulation, preference-pair construction, and training-epoch effects.Table I covers loss and preference-pair construction, while Table II covers training epochs with Syn+/Syn− pairs.
2) LLM-based multi-rater:
The study combines LLM-based multi-rater evaluation with human A/B/Tie preference testing to assess NV realization, naturalness, quality, expression, and lexical accuracy. Both procedures compare systems while reducing evaluation-order and identity biases.
- LLM-based multi-rater: Gemini-2.5-Pro rates NV accuracy, NV perceptual effect, naturalness, quality, and expression using NVVSpeech Challenge Track 2 rubrics.Three independent simulated raters evaluate each sample.
- LLM-based multi-rater: Systems for each utterance are jointly presented with anonymized labels and randomized presentation orders to mitigate system-identity and positional biases.
- Human preference: Eleven native-speaking volunteers conduct A/B/Tie comparisons between the SFT baseline and preference model on the multi-label NV-Bench set.Human criteria include NV accuracy, NV naturalness, lexical accuracy, and overall naturalness.
C. Implementation Details
Implementation constructs diverse synthetic preference pairs, computes NV-aware signals, and tests alternative losses, pair-selection strategies, and NV-tag weights. The setup uses an SFT-initialized CosyVoice2 model and an NV-capable ASR evaluator.
- Model initialization: A pretrained CosyVoice2-0.5B model is SFT-trained on Emilia-NV for 4 epochs to create the common Base initialization.SFT uses a learning rate of 1 × 10−5.
- Training data: Preference optimization uses an approximately 100-hour Emilia-NV subset that prioritizes utterances containing underrepresented NV classes.
- Preference pairs: Eight candidate utterances are generated per training sample with CosyVoice2’s default Repetition Aware Sampling strategy before preference-pair construction.Candidate generation uses topp = 0.9 and topk = 30.
- Preference signals: The NV-ASR model recognizes all 18 training-set NV tags and supplies the transcription basis for NV-CER.
- Loss formulation: The study tests DPO alone against a combined DPO-plus-SFT objective, weighting the SFT loss by 0.2.
- Preference pairs: Preferred responses are selected from top-ranked synthetic, ground-truth, or conditional hybrid candidates, while rejected responses can include NV-free synthetic speech.
- Preference signals: Preference signals compare NV-CER, conventional pinyin CER, a recognition-quality composite, and alternative NV-tag weights.Varying wNV explicitly changes the relative importance assigned to NV errors.
IV. RESULTS AND DISCUSSION
Results favor synthetic best-versus-worst preference pairs and standard DPO, while NV-CER-based signals consistently outperform conventional alternatives across single-tag and multi-tag evaluation. Increasing NV emphasis improves NV metrics with a modest lexical trade-off.
- A. Loss and Preference Pair Ablation: Syn+/Syn− achieves the best performance in the loss and preference-pair ablation and is used in subsequent experiments.GT/Hybrid preferred responses reduce performance, while SynNoNV underperforms because NV-free speech is a weak negative example.
- A. Loss and Preference Pair Ablation: DPO alone becomes unstable after the first or second epoch, whereas adding SFT substantially improves training stability but weakens the DPO effect for Syn+/Syn−.Subsequent experiments therefore use DPO, with DPO+SFT retained as a robustness check.
B. Investigation on the impact of NV-Weight tuning
Weighted NV-CER shifts preference optimization toward NV-related accuracy while preserving lexical performance across tested weights.
- Increasing wNV improves PCER but causes a modest increase in CER.The weighting changes the relative emphasis on NV-related and lexical errors.
- CER remains below the Base and SFT models across all tested NV-tag weights.
- Weighted NV-CER modulates NV and lexical error emphasis without changing the underlying preference optimization framework.
C. Comparison with Baselines
NV-CER-based preference signals outperform baselines across single-tag and multi-tag evaluations, with gains supported by objective, LLM-based, and human assessments. The improvements target NV and lexical accuracy while preserving speaker similarity and perceived naturalness.
- NV-CER consistently improves lexical and NV-related metrics over Base, SFT, and conventional ASR-CER signals on the single-tag test set.SFT slightly degrades performance from Base, while NV-CER outperforms conventional ASR-CER.
- Combining NV-CER with UTMOS achieves similar transcription accuracy while slightly improving UTMOS and DNSMOS.
- The same trends hold for DPO and DPO+SFT, while speaker similarity remains largely unchanged.
- NV-CER-based preference signals consistently outperform baselines on the multi-tag test set using objective and LLM-based metrics.
- Human evaluations favor the preference model over SFT for NV accuracy at 55.5% versus 44.5% and lexical accuracy at 57.2% versus 42.8% among non-tied judgments.No significant preference is observed for NV naturalness or overall naturalness, and the paper reports converging objective, LLM-based, and human evidence.