InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

1. INTRODUCTION

• Existing ITTS systems often leave the links among instructions, target text units, and acoustic changes implicit, with individual-character control less explored.They often generate speech directly from text and instructions without identifying which units or attributes should change [10], while emphasizing utterance- or segment-level characteristics [12].Source: §1 · p. 1
• Character-level acoustic attributes enable precise, inspectable control, but typically require users to specify attributes and can become cumbersome as utterances and control dimensions grow.This leaves a gap between natural-language instruction control and explicit character-level acoustic realization.Source: §1 · p. 1
• InstCharVoice identifies instruction-relevant characters and predicts duration, boundary, energy, pitch, and tone for each before generating corresponding speech tokens.An LLM-assisted pipeline generates natural-language instructions and character-level grounding labels from text, speech, and acoustic attributes.Source: §1 · p. 1
• The framework combines an annotation pipeline linking instructions to target characters and attributes with keyword prediction and grounding-aware loss weighting for localized control.The paper reports improved keyword-level acoustic control alongside competitive speech naturalness and character-level controllability.Source: §1 · pp. 1–2
• Experiments show improved instruction following and keyword-level acoustic control over representative ITTS systems, with competitive speech naturalness and explicit character-level controllability.The framework connects instruction specification, character-level grounding, acoustic planning, and speech realization.Source: §1 · pp. 1–2

2.1. Character-Level Grounded Instruction Annotation

• On WordVoice-5A-zh, an approximately 2,546-hour Chinese-speech corpus, Qwen3-Omni generates and refines instructions using character-level acoustic cues, then labels the character-attribute pairs addressed by each instruction.The resulting annotations support both utterance-level instruction following and character-level acoustic grounding.Source: §2.1 · p. 2

2.2. Character-Level Instruction-to-Speech Modeling

• The autoregressive model generates character-level speech chunks, predicting each character's grounding label and acoustic attributes before its aligned speech tokens.The grounding head provides auxiliary training supervision and is not fed back into the generation sequence.Source: §2.2 · pp. 2–3
• Training sequences combine instruction and content tokens, a speaker embedding, leading silence, and character chunks containing user-specified attributes, all five attributes, and aligned speech tokens.User-specified attributes are randomly provided or masked during training, enabling prediction of complete attributes from partial specifications.Source: §2.2 · p. 2
• At inference, five attribute-specific heads predict complete attributes conditioned on any provided attributes, and the model generates aligned speech tokens for waveform synthesis.Unspecified attributes are inferred from the instruction and generation context; WordVoice-FM synthesizes the waveform from the generated attributes and tokens.Source: §2.2 · p. 3

2.3. Training Objectives

• Training jointly optimizes speech-token, character-grounding, and five-dimensional attribute prediction, using uncertainty-based weighting to balance the objectives.The five attributes are duration, boundary, energy, pitch, and tone; duration uses aligned-token counts, while energy, pitch, boundary, and tone use 20, 20, 5, and 7 categories, respectively.Source: §2.3 · p. 3
• Grounding-aware reweighting adjusts attribute loss for grounded versus non-grounded pairs, with attributes lacking grounded pairs trained without that reweighting.The model first trains for five epochs without reweighting, then for three additional epochs with it enabled.Source: §2.3 · p. 3

3.1. Evaluation of Grounded Instruction Annotations

• 84.50% of cue-refined annotations received consistent judgments, up from 35.75%, while inconsistent and uncertain ratings fell to 9.25% and 6.25%.The evaluation used 40 randomly sampled utterances, each assessed by ten native Chinese speakers.Source: Figure 3 · p. 3, §3.1 · p. 3

3.2. Main Comparison

• InstCharVoice achieves the highest mean NMOS and IMOS, the highest DNSMOS, and the lowest errors across all five keyword-level acoustic metrics among the compared systems.These results cover subjective speech ratings, speech quality, and localized acoustic control.Source: §3.2 · p. 3, Table 2 · p. 4, Table 3 · p. 4
• InstCharVoice has lower SMOS than FireRedTTS3 and CosyVoice3, while its CER is 0.69 percentage points above the best result of 2.89%.The comparison is influenced by Seed-VC conversion applied to Qwen3-TTS and FireRedTTS3.Source: Table 3 · p. 4, Table 2 · p. 4, §3.2 · p. 4

3.3. Ablation Study

• Removing either keyword prediction or grounding-aware loss weighting degrades keyword-level control across all five attributes; removing both yields the highest errors.Removing grounding-aware loss weighting causes a larger degradation than removing keyword prediction.Source: Table 4 · p. 4, §3.3 · p. 4

3.4. Character-Level Control Comparison

• InstCharVoice has slightly higher duration and energy errors but lower pitch, boundary, and tone errors than WordVoice, with comparable overall character-level control.The comparison supplies both systems the same text, reference speech, and complete character-level acoustic attributes, without natural-language instructions.Source: Table 5 · p. 4, §3.4 · p. 4

• InstCharVoice improves instruction following and keyword-level acoustic accuracy over representative ITTS systems, with higher naturalness scores.Source: §4 · p. 4
• The framework uses an LLM-assisted annotation pipeline and an autoregressive model to ground instructions in characters and predict acoustic attributes before speech tokens.Keyword prediction and grounding-aware loss weighting further improve localized control.Source: §4 · p. 4
• Character-level control performance remains comparable to WordVoice under explicit attribute conditioning.Source: §4 · p. 4
• The framework models five measurable attributes but not abstract factors such as emotion or speaking style; evaluation with partial specifications and more complex instructions remains future work.Gaps also remain in acoustic realization accuracy, intelligibility, and speaker similarity.Source: §4 · p. 4