Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations

1. Introduction

• Existing systems still cannot reliably place a requested NVV at a specified position with convincing acoustic realization.NVVs convey affect, attitude, turn-taking cues, and physiological state, but conventional TTS systems prioritize lexical content and may filter them during data curation.Source: §1 · p. 1
• The system combines dedicated tokens for 16 NVV labels and adapted stop prediction with bilingual pre-training, targeted augmentation, frequency-aware rebalancing, and continued SFT.At inference, it selects decoding configurations and uses multi-metric Best-of-N sampling.Source: §1 · p. 1
• Targeted augmentation yields the largest gains for severely underrepresented NVV categories, while robust candidate selection balances NVV correctness, lexical fidelity, and perceptual quality.The challenge evaluates natural realization of tagged events while maintaining intelligibility, naturalness, audio quality, and expressiveness.Source: §1 · p. 1
• The system ranked first in Mandarin and overall, and second in English, among participating Track 2 systems.The revised official protocol combines LALM evaluation with subjective listening in Mandarin and English.Source: §1 · p. 1

2.1. System overview

• The system pre-trains DiTAR on bilingual lexical and NVV-annotated speech, then synthesizes underrepresented events, balances tag and language distributions, and continues SFT before inference-time decoding search and Best-of-N sampling.Figure 1 summarizes the training and inference pipeline.Source: §2.1 · pp. 1–2, Figure 1 · p. 2

2.2. NVV-aware DiTAR Architecture

• DiTAR groups continuous VAE latents into acoustic patches, uses causal language-model conditions for LocDiT to generate the next patch, and incorporates 16 dedicated NVV tokens into the shared text embedding and attention space.The stop predictor labels only the final valid patch as stop and intermediate patches, including NVV patches, as continue.Source: §2.2 · p. 2

2.3. Bilingual pre-training and long-tail-aware SFT

• Bilingual pre-training trains the NVV embeddings and continuous speech generator; continued SFT adds synthetic examples for weak categories and uses tag-frequency-aware resampling, English upsampling, and caps on speaker and utterance repetition.Synthetic NVVs span sentence-initial, medial, and final positions, with varied contexts and voices.Source: §2.3 · pp. 2–3

2.4. Inference-time optimization

• Inference optimization searches four decoding configurations on development data, then generates five candidates per tagged transcript and selects among them with normalized multi-metric scores.The two-stage strategy aims to improve reliability without model updates, at the cost of additional inference computation.Source: §2.4 · p. 3

3.1. Training and evaluation setup

• Evaluation combines local measures of lexical fidelity, NVV realization and localization, and audio quality with official language-specific scores based on LALM evaluation and subjective listening.The NVV-aware DiTAR reduces Mandarin CER from 6.267% to 5.603% and English WER from 3.159% to 2.949% versus the official baseline, while DNSMOS and detector precision improve.Source: §3.1 · p. 3, §3.2 · p. 3, Table 1 · p. 3

3.2. Overall results

• Long-tail-aware SFT reduces CER and WER to 5.211% and 2.711%, while achieving detector precision of 0.878 and recall of 0.275.Decoding-configuration search further reduces CER and WER to 5.098% and 2.597%, raises CA-F1 to 0.582, and achieves NTD of 0.068.Source: §3.2 · p. 3
• Best-of-N achieves the best evaluated CER (5.012%), WER (2.194%), DNSMOS (3.221), and CA-F1 (0.589), with NTD increasing slightly from 0.068 to 0.069.The results indicate a favorable balance among lexical fidelity, audio quality, and NVV realization.Source: §3.2 · p. 3, Table 4 · p. 4

3.3. Long-tail and per-tag analysis

• English cry gains 23.30 points and Mandarin cry 4.63 points; yawn, laugh, and burp improve in both languages, while sneeze degrades in both.Snore is nearly unchanged, and the results indicate that added data is most effective when missing coverage is the main bottleneck.Source: §3.3 · p. 4, Table 2 · p. 4

3.4. Ablation Study

• With a fixed checkpoint, w = 2.0 and τ = 0.9 yields CER of 5.098%, WER of 2.597%, and CA-F1 of 0.582; w = 2.5 and τ = 0.7 yields the lowest NTD.The selected setting balances metrics rather than optimizing one alone; w = 2.0 and τ = 0.7 has the highest DNSMOS, by a small margin over the selected setting.Source: §3.4 · p. 4, Table 3 · p. 4
• 62.786 overall, with final scores of 61.775 for Mandarin and 63.797 for English, places the system first overall, first in Mandarin, and second in English among participating systems.The scores combine the revised official evaluation components and exceed the baseline by 1.355 overall, 2.075 for Mandarin, and 0.634 for English.Source: Table 5 · p. 4, §3.4 · p. 4

3.5. Discussion

• Targeted augmentation helps data-scarce categories most, whereas low-diversity synthetic data yields inconsistent gains; decoding optimization and multi-metric Best-of-N improve performance without model updates but add inference cost.The second-place English rank and lower English LALM score leave English NVV control as an important direction.Source: §3.5 · p. 4

• The system achieved an official weighted bilingual score of 62.786 and ranked first overall among participating Track 2 systems.Source: §4 · p. 4
• The system combined continuous-latent modeling, long-tail-aware training, decoding search, and multi-metric Best-of-N selection.Source: §4 · p. 4
• Future work will improve English NVV data and event-level selection metrics.Source: §4 · p. 4