NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech

• NVV tags let users specify vocal events inline, but scarce diverse annotations [5] leave models prone to omitting or confusing them after fine-tuning [6].Source: §1 · p. 1
• Unlike discrete-token TTS, continuous AR-FM TTS generates acoustic patches through a flow-matching head, and Flow-GRPO adapts likelihood-ratio optimization using stochastic differential equation sampling [17, 18].Prior continuous AR-FM work also optimized reward-weighted flow losses or preference objectives [6] [19].Source: §1 · p. 1
• Prior direct-gradient work backpropagated differentiable rewards through image diffusion and flow models and explored TTS objectives including intelligibility, speaker identity, and emotion [26].The introduction asks whether NV-ASR reward gradients can improve tag-following in continuous AR-FM TTS.Source: §1 · p. 1
• NVAlign freezes an NVV-aware ASR model as a differentiable reward, then uses a two-step gradient surrogate to update the backbone and flow head.Fidelity penalties and reference-velocity regularization [27] help preserve speech quality and speaker similarity.Source: §1 · p. 1
• NVAlign improves NVV tag-following over SFT and Flow-GRPO on NVV-SuperBench and in human listening evaluations.The framework jointly adapts the autoregressive backbone and flow head.Source: §1 · p. 1

2.1. Non-verbal vocalizations.

• NVAlign backpropagates frozen NV-ASR target-tag probabilities through speech generation, unlike Hu et al.’s evaluator-selected or paired training for NVV control [6].Annotated NVV datasets support recognition and controllable generation [5] [31], and dedicated benchmarks assess recognition, generation, and perceptual quality [34].Source: §2.1 · p. 1
• The overview shows differentiable NVV rewards and fidelity penalties guiding joint LoRA adaptation of the autoregressive backbone and flow head, with decoder and evaluators frozen.Reference-velocity regularization is not shown in the figure.Source: Figure 1 · p. 2

2.2. Reward and preference optimization.

• TTS post-training methods feed reward feedback through reward-weighted flow matching [19], causal-value gradients, likelihood-ratio updates, or preferred/rejected speech pairs.These approaches are GROW, VGPO, FlowTTS-GRPO, and ARDM-DPO, respectively.Source: §2.2 · p. 2
• Direct-gradient precedents include LeapAlign’s two-step backward-trajectory surrogate, speech objectives differentiated through diffusion or codec relaxations, and SR-FD’s VoxCPM2 adapter optimization [26].NVAlign extends direct-gradient speech optimization to NVV tag-following in continuous AR-FM TTS.Source: §2.2 · p. 2

3.1. Supervised initialization

• NV-ASR and TTS models are fine-tuned on NVV-annotated speech; the NV-ASR uses Qwen3-Omni-30B [39], and TTS retains auxiliary losses including stop prediction.The TTS backbone conditions on tagged text, a speaker prompt, and preceding acoustic patches, while its flow head predicts patch velocities.Source: §3.1 · p. 2

3.2. NVV reward and fidelity penalties

• The frozen NV-ASR scores each target transcript token under teacher forcing; the reward requires every token in a tag to score highly and sums across vocalizations.Fidelity penalties bound speaker consistency, speaker-prompt similarity, and predicted MOS, and also penalize excessive waveform peaks and low signal levels.Source: §3.2 · pp. 2–3

3.3. Direct-gradient post-training

• Two flow-head evaluations approximate gradients through the full sampling trajectory, sending frozen NV-ASR reward and fidelity gradients through the decoder to the flow head and autoregressive backbone.The method recomputes backbone conditioning from preceding patches and uses a straight-through latent connector to preserve sampled forward values while routing gradients through the estimates.Source: §3.3 · p. 3
• Post-training anchors the flow field to the SFT model’s reference velocities and updates backbone LoRA adapters, flow-head input projections, and the flow head.The reference velocities are evaluated at sampled states using the SFT flow head and the adapted backbone’s conditioning.Source: §3.3 · p. 3

4.1. Datasets and experimental setup

• The study fine-tunes VoxCPM2, dots.tts, and an NVV-aware NV-ASR model on 834.5 hours from six public datasets, using a unified 39-tag inventory and excluding benchmark-text and speaker overlap.Because only 18.9 hours of the public data are English, the production study uses a separate proprietary English NVV dataset and NV-ASR model. Flow-GRPO and NVAlign share SFT initialization; Flow-GRPO updates only the flow head, while NVAlign jointly updates the flow head and AR backbone.Source: §4.1 · p. 3
• Evaluation combines NVV-SuperBench and balanced human listening sets with tag-recovery and sound-event detectors, speaker-similarity and speech-quality metrics, and Mandarin CER and English WER.Three blinded raters judge target sounds at marked positions; human tag accuracy requires unanimous agreement, and naturalness MOS is averaged over raters and clips.Source: §4.1 · pp. 3–4

4.2. Results and analysis

• NVAlign raises listening-set accuracy from 43.2% to 47.6% for VoxCPM2, 63.2% to 65.4% for dots.tts, and 39.8% to 54.0% for English production.Whisper and SED detection rates also increase across all three systems. Paired 95% confidence intervals exclude zero for both English gains over SFT, both Mandarin VoxCPM2 gains over Flow-GRPO, and the English listening-set gain over Flow-GRPO.Source: Table 1 · p. 4, §4.2 · p. 4
• On the NVV-SuperBench human subset, VoxCPM2 gains 7.5 percentage points over SFT and 15.8 over Flow-GRPO, while dots.tts scores 2.7 points below SFT.Confidence intervals include zero for the other reported comparisons.Source: Table 1 · p. 4, §4.2 · p. 4
• SIM, UTMOS, and DNSMOS rise across all systems, while human naturalness MOS stays within 0.1 of SFT; English benchmark human tag-following rises from 33.7% to 53.7% as Gemini PE decreases.Relative to SFT, Mandarin CER increases by 1.26 and 0.13 points for VoxCPM2 and dots.tts, respectively; English WER increases by 0.07 on the listening set and 1.33 on NVV-SuperBench.Source: Table 1 · p. 4, §4.2 · p. 4
• Flow-head-only NVAlign reaches 0.441 reward versus 0.392 for Flow-GRPO and 0.397 for SFT; joint backbone and flow-head adaptation further improves reward and Gemini ratings.Removing velocity regularization raises reward from 0.468 to 0.488 but lowers Gemini accuracy/PE from 3.54/3.34 to 2.88/1.94; subsequent removals reduce UTMOS and then peak/level ratings, while removing speaker terms drops SIM from 0.796 to 0.360.Source: Table 2 · p. 4, §4.2 · p. 4

• Sparse NVV supervision can limit the NV-ASR reward, whose optimization may exploit recognition shortcuts or degrade prosody because tag likelihood does not capture the full speech distribution.Source: §5 · p. 4
• Fidelity penalties and reference-velocity regularization constrain reward-driven drift, but adding objectives makes gradient balance part of the optimization problem.Source: §5 · p. 4
• The formulation assumes discrete NVV events and excludes continuous or overlapping vocalizations.Source: §5 · p. 4

6. CONCLUSION

• NVAlign improves human tag-following across VoxCPM2, dots.tts, and an English production system while increasing Whisper tag-recovery and SED detection rates.It backpropagates NV-ASR rewards through a two-step flow surrogate and constrains updates with fidelity and reference-velocity penalties.Source: §6 · p. 4

• Human listening evaluations were conducted through Podonos, and no institutional ethics approval, exemption, or waiver was obtained.Source: §7 · p. 5