Source-linked AI summary
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Jingbin Hu, Qirui Zhan, Yuang Cao, Ziyu Zhang, Yunxiang Chen, Houdun Liu, Su Feng, Bengu Wu, Lei Xie, Liumeng Xue
TL;DR
Controllable non-verbal vocalization generation remains a challenge for continuous autoregressive speech models. The paper constructs LALM-ranked bilingual preference pairs and trains with RSFT followed by anchored Flow-DPO. On NVVSpeech Challenge Track 2, it reaches 75.80, 1.84 above the VoxCPM2 baseline.
Problem
Method
The method combines bilingual prompts, stochastic rollouts, and LALM-ranked same-prompt pairs with RSFT followed by SFT-anchored Flow-DPO.
Source: §2 · p. 2, §2.2 · p. 2, §2.3 · p. 3Results
75.80 FinalTrack2Score, a +1.84 improvement over the VoxCPM2 baseline on NVVSpeech Challenge Track 2.
Source: §4 · p. 4Takeaways & Limitations
The reported gains are concentrated in NVV Accuracy and NVV Perceptual Effect, while Overall Quality remains nearly unchanged.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
We propose a preference optimization framework with Large Audio-Language Model (LALM) feedback for controllable non-verbal vocalization (NVV) generation in continuous autoregressive speech models. To construct preference data without human preference annotation, we build a bilingual prompt corpus by combining NVV-injected real transcripts with LLM-generated semantically aligned prompts, perform stochastic model rollouts, and use a LALM to rank candidate utterances and form same-prompt chosen--rejected pairs. We then adopt a two-stage optimization strategy: Rejection Sampling Fine-Tuning (RSFT) first adapts the model to LALM-selected high-scoring samples, followed by Anchored Flow-DPO, which formulates pairwise preference optimization using utterance-level flow-matching loss and retains the chosen-sample flow-matching objective as an SFT anchor. This design enables DPO-style preference learning without explicit sequence likelihoods while preserving direct supervision on preferred realizations. On the official 1,600-utterance NVVSpeech Challenge Track~2 test set, our method achieves a Final Track2Score of \textbf{75.80} (79.39 ZH / 72.21 EN), outperforming the VoxCPM2 baseline by \textbf{+1.84}. The improvements are mainly driven by higher NVV Accuracy and NVV Perceptual Effect, while Overall Quality remains stable.
2. Method
The method adapts preference optimization to continuous autoregressive speech by using utterance-level flow-matching loss as a preference surrogate. It combines bilingual LALM-ranked preference data with RSFT followed by SFT-anchored Flow-DPO.
2.1. Overview and Continuous Autoregressive Formulation
- Utterance-level flow-matching loss serves as a preference surrogate because the decoder lacks the explicit sequence log-likelihoods required by conventional DPO.Patch-level losses are aggregated because preference labels apply to complete utterances.Source: §2.1 · p. 2
2.2. LALM-Based Preference Data Construction
- The bilingual corpus combines NVV-tagged real transcripts that create conflict-rich prompts with LLM-generated prompts whose contexts align semantically with target NVV tags.The two sources provide challenging control cases and realistic expressive scenarios.Source: §2.2 · p. 2
- For each prompt, the LALM scores stochastic rollouts on five dimensions, and the highest- and lowest-scoring candidates form same-prompt preference pairs.The chosen samples are also reused for RSFT supervision.Source: §2.2 · p. 2
2.3. Preference Alignment for Continuous Autoregression
- RSFT first fine-tunes on LALM-selected chosen samples, then initializes Flow-DPO, which uses chosen–rejected comparisons relative to a frozen reference model.RSFT alone does not use the relative information in rejected samples.Source: §2.3 · pp. 2–3
- Anchored Flow-DPO retains chosen-sample flow-matching supervision alongside its pairwise objective, encouraging fit to preferred realizations during preference optimization.The pairwise objective alone does not require low flow-matching loss on the chosen sample.Source: §2.3 · p. 3
3. Experiments
On the official 1,600-utterance test set, the proposed system outperforms VoxCPM2, with gains concentrated in NVV-specific metrics. Ablations examine RSFT, Flow-DPO, and the SFT anchor.
3.1. Experimental Setup
- Evaluation covers 1,600 official-test utterances, split evenly between Chinese and English, using Gemini 2.5 Pro rather than the preference-construction LALM.Table 1 covers main results and ablations; Table 2 reports subjective evaluation.Source: §3.1 · p. 3, Table 1 · p. 4, Table 2 · p. 4
3.2. Main Results
- 75.80 FinalTrack2Score beats VoxCPM2’s 73.96 by +1.84; Chinese rises from 77.62 to 79.39 and English from 70.30 to 72.21.Table 1 reports the official test-set results.Source: Table 1 · p. 4, §3.2 · p. 4
- NVV Accuracy and NVV Perceptual Effect improve in both languages, while Overall Quality changes only slightly.Accuracy rises from 4.52 to 4.60 for Chinese and 4.37 to 4.45 for English; Perceptual Effect rises from 4.41 to 4.48 and 3.90 to 3.99, respectively.Source: §3.2 · p. 4
- In evaluation by 12 Chinese and 12 English native listeners, the proposed model outperforms the baseline across A/P/N/Q/E, with preference scores rising from 2.71 to 2.80 in Chinese and 3.07 to 3.19 in English.Table 2 reports subjective evaluation results using the same metrics as Table 1.Source: §3.2 · p. 4, Table 2 · p. 4
3.3. Ablation Study
- RSFT alone raises Final score from 73.96 to 75.00 (+1.04), with gains in both Chinese and English.Chinese rises from 77.62 to 78.54 and English from 70.30 to 71.46.Source: §3.3 · p. 4
- Direct Flow-DPO without the SFT anchor reaches 75.43 Final score, a +1.47 improvement over baseline, exceeding RSFT alone’s +1.04.The paper attributes the richer supervision to relative contrast between chosen and rejected samples.Source: §3.3 · p. 4
- Initializing Flow-DPO from the RSFT checkpoint raises Final score from 75.43 to 75.69 (+0.26), with a notable Chinese Track2Score increase from 78.73 to 79.41.Chinese NVV Perceptual Effect rises from 4.42 to 4.50.Source: §3.3 · p. 4
- Adding the SFT anchor yields the best Final score, 75.80, and improves English Track2Score from 71.97 to 72.21 while retaining top Chinese NVV Accuracy of 4.60.The authors describe pairwise Flow-DPO as driving the primary gains, with RSFT initialization and anchoring playing complementary roles.Source: §3.3 · p. 4
4. Conclusion
The paper presents LALM-guided preference optimization for continuous autoregressive NVV generation and reports a 75.80 FinalTrack2Score, +1.84 over VoxCPM2.
4. Conclusion
- The framework combines an RSFT cold start with SFT-anchored Flow-DPO for controllable NVV generation in continuous autoregressive speech models.It uses utterance-level flow-matching loss for likelihood-free DPO-style alignment while retaining direct supervision on preferred samples.Source: §4 · p. 4
- 75.80 FinalTrack2Score is +1.84 over VoxCPM2 on the NVVSpeech Challenge Track 2 test set.The conclusion reports improved NVV accuracy and perceptual quality.Source: §4 · p. 4