SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS

1. INTRODUCTION

• Existing methods do not jointly formulate inline prosody as a clause-level change from the preceding same-speaker clause while modeling temporal scope and RVQ codebook allocation.The paper identifies this gap as limiting coherent prosodic progression in long-form synthesis.Source: §1 · p. 1
• Pitch, Energy, and Speed instructions target changes relative to the preceding clause from the same speaker, while Pause uses an absolute duration interval.This avoids imposing one absolute acoustic target across speakers and supports paragraph-level progression from locally controlled clauses.Source: §1 · p. 1
• The evaluation uses target-voice-adapted systems and held-out utterances from those speakers, rather than testing zero-shot speaker generalization.Source: §1 · p. 1
• In Qwen3-TTS, Speed and Pause primarily affect sequence length through q0, whereas Pitch and Energy depend on information distributed across residual codebooks.Standard instruction fine-tuning uses text-token tags without explicitly modeling Pitch/Energy tag timing or codebook-specific influence.Source: §1 · p. 1
• RVQ diagnosis finds Energy effects saturate in early residual codebooks while Pitch transfer accumulates across a deeper prefix, motivating SCIC’s temporal routing and tag-specific codebook weighting.SCIC injects localized conditioning into MTP; Speed and Pause remain on q0, and multi-reward GDPO post-training jointly targets control and quality.Source: §1 · p. 1

2.1. Overall framework

• SCIC conditions MTP prediction of q1–q15 using frame-level tag activation and tag-specific weights, while the Main Talker predicts q0 and provides ht.Its Router is trained with balanced frame-level losses, and the combined objective adds Router loss to generation loss.Source: §2.1 · p. 2, §2.4 · p. 2, Figure 1 · p. 3

2.2. Speaker-relative inline control formulation

• Pitch, Energy, and Speed instructions measure change against the preceding clause from the same speaker, while Pause specifies an absolute duration interval.Pitch, Energy, and Speed use direction-normalized logarithmic changes; the reported Pause levels are L1 [0.50, 0.75], L2 [0.75, 1.00], and L3 [1.00, 1.25] seconds.Source: §2.2 · p. 2

2.3. RVQ codebook diagnosis

• RVQ Codebook Diagnosis constructs duration-matched speech pairs, exchanges projected codebook contributions, and measures attribute transfer relative to the complete codec-reconstructed change.Forward intervention inserts transformed contributions into the original reconstruction; reverse intervention removes them from the transformed reconstruction.Source: §2.3 · p. 2
• Energy transfer saturates within q1–q3, while Pitch approaches saturation around q1–q10, motivating attribute-specific rather than uniform codebook weights.The diagnosis exchanges projected contributions between duration-matched original and attribute-transformed speech; its transfer ratios describe latent interventions, not additive information percentages.Source: §2.3 · p. 2

2.4. Scope- and codebook-aware instruction conditioning

• The Temporal Instruction Router predicts each Pitch/Energy tag’s frame-level activation, with absent tags inactive and aligned clause spans providing binary supervision.The Router loss balances positive and negative frames separately for each supervised tag type.Source: §2.4 · p. 2
• Tag-Specific Codebook Weighting scales localized Pitch/Energy signals across residual codebooks before MTP predicts q1–q15.Each tag’s weights are zero-initialized, and SCIC combines generation loss with a weighted Router objective.Source: §2.4 · p. 2

2.5. Multi-reward GDPO post-training

• Multi-reward GDPO post-training rewards bounded requested-direction changes and Pause interval matches while including CER and speaker-similarity rewards.Objectives are standardized across responses before weighted aggregation to prevent reward-scale differences from dominating optimization.Source: §2.5 · pp. 2–3

3.1. Experimental setup

• The supervised corpus contains approximately 1,000 hours of Chinese speech from 30 speakers, with evaluation utterances disjoint from supervised and post-training data.Systems are initialized from a live-streaming Qwen3-TTS checkpoint; Instruction SFT uses text-token tags, SCIC-SFT adds routing and codebook weighting, and SCIC-GDPO adds multi-reward post-training.Source: §3.1 · p. 3

3.2. Main results

• SCIC-GDPO achieves the best Pause ACC at all three levels, and all systems’ mean detected durations fall within their target intervals.Pause ACC measures whether detected silence falls inside the requested duration interval.Source: §3.2 · pp. 3–4
• Relative to Instruction SFT, SCIC-SFT improves Pitch, Energy, Speed, and Pause by 7.31, 10.53, 0.51, and 0.67 percentage points; SCIC-GDPO gains are 9.65, 11.30, 2.97, and 3.87 points.CER remains within 1.63–1.68%, while speaker similarity remains essentially unchanged.Source: §3.2 · p. 3, Table 2 · p. 4
• On 200 long-form texts, HMOS rises from 3.244 for Zero-Shot and 3.504 for SFT without instructions to 4.042 with paragraph-level instructions.The passage characterizes the result as a clearer expressive hierarchy for the complete system.Source: §3.2 · p. 4
• Relative to SCIC-SFT, SCIC-GDPO increases median direction-normalized change magnitudes for both directions of Pitch, Energy, and Speed while remaining below frozen training references.The figure shows median speaker-relative change magnitudes; Down values are sign-flipped so positive means the requested direction.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Temporal and codebook-axis ablations

• Replacing the soft Router with a binary Router reduces Pitch accuracy and increases Boundary MAE from 295 to 375 ms; removing codebook weighting lowers Pitch and Energy accuracy.The Router primarily controls instruction scope, whereas codebook weighting controls attribute allocation across residual predictions.Source: §3.3 · p. 4, Table 3 · p. 4

• SCIC combines temporal routing and tag-specific codebook weighting for speaker-relative inline prosody control, while multi-reward GDPO improves control and long-form listening indicates a clearer expressive hierarchy.The conclusion reports comparable intelligibility and speaker similarity.Source: §4 · p. 4