Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction

• Direction-following TTS transforms a reference utterance according to a direction while preserving its speaker identity and linguistic content.It differs from conventional style-conditioned TTS by modeling a relative transformation between pre-modification and post-modification utterances.Source: §1 · p. 1
• Training requires triplets containing a pre-modification utterance, direction text, and corresponding post-modification utterance for the same script.Source: §1 · p. 1
• Large speech corpora usually provide only one reading per script, while smaller multi-recording corpora generally lack modification directions.This limits coverage of relative stylistic changes and robust speaker-preserving transformation across diverse speakers.Source: §1 · p. 1
• The proposed pipeline uses impression-controllable TTS to generate paired readings and an LLM to describe their estimated impression differences as natural-language directions.The resulting pseudo-triplets train a transformation from pre-modification to post-modification utterances.Source: §1 · p. 1

• Pseudo data and recorded data provide complementary sources: synthesis offers scalability, while interactive recordings capture authentic human responses.Source: §2 · p. 1

2.1. Pseudo paired utterance generation

• For each script and reference, an impression-controllable zero-shot TTS model synthesizes baseline and style-modified utterances as pre-modification and post-modification speech.Source: §2.1 · p. 1
• The impression representation contains 13 continuous antonym-based voice-impression dimensions, including pitch, clarity, calmness, emotionality, and fluency.Two added axes are fluent–hesitant and emotional–neutral.Source: §2.1 · p. 2

2.2. Filtering and impression estimation

• Generated pairs are filtered using speaker-embedding similarity and speaking-rate differences, while near-identical embeddings are removed because they provide little directional guidance.Source: §2.2 · p. 2
• The impression estimator computes pre- and post-modification vectors, and their difference is defined as ∆I = Ipost − Ipre.Source: §2.2 · p. 2
• An LLM generates a plausible performance direction from the relative impression difference rather than labeling either utterance with an absolute style.The generated direction describes the transformation from pre-modification to post-modification speech.Source: §2.3 · p. 2

• The style refiner modifies speech embeddings, and inference adds the predicted modification to the pre-modification embedding.Keeping the backbone fixed focuses learning on direction-dependent style refinement.Source: §3 · p. 2
• The target modification is ∆ = epost − epre, treated as a local additive approximation without assuming global linearity.Different speakers may require different embedding shifts for the same direction.Source: §3 · p. 2
• Rectified flow matching models a direction-conditioned stochastic vector field because deterministic regression would otherwise collapse multiple plausible shifts into a conditional mean.Source: §3 · p. 2
• The refiner predicts velocity conditioned on the pre-modification embedding, direction representation, and time, matching the target velocity ∆ − x0.Auxiliary losses encourage directional consistency and modification-magnitude alignment.Source: §3 · p. 2

4.1.1. Pseudo data

• Pseudo data were generated from 1,600 Japanese speakers by applying impression control to randomly selected dimensions and sampling control values from −2 to +2.Source: §4.1.1 · p. 2
• Filtering retained pairs with ECAPA-TDNN cosine similarity from 0.80–0.95 and speaking-rate ratios from 0.85–1.15, yielding 74,619 utterance pairs.Source: §4.1.1 · p. 2
• Up to five LLM-generated performance directions were paired with each valid speech pair to form pseudo triplets.Source: §4.1.1 · p. 2
• The final pseudo dataset contained 350,617 triplets and 127.6 hours of unique speech, with subsets covering 200, 400, and 800 speakers.Source: §4.1.1 · p. 3

4.1.2. Recorded data

• Recorded data comprised 8.9 hours and 6,899 utterance pairs from interactive sessions with two professional voice actors.Source: §4.1.2 · p. 3

4.2. Training and evaluation conditions

• The comparison trains separate models with pseudo-only, recorded-only, combined, and different pseudo-speaker-scale configurations.Source: §4.2 · p. 3

Evaluation setup

• Evaluations covered naturalness, speaker similarity, and direction alignment for two seen and two unseen speakers.The objective evaluation generated 15,000 direction-conditioned utterances per speaker.Source: §5.1 · p. 3

Objective evaluation

• Pseudo conditions produced more robust speaker similarity than Recorded for unseen speakers, while increasing pseudo-data diversity reduced extreme speaker drift.The Full condition remained more stable than Recorded while balancing robustness and variability.Source: §5.1 · p. 3
• Direction refinement preserved naturalness, with scores comparable to recorded utterances across configurations.Source recorded utterances scored 2.67±0.01 for seen speakers and 2.97±0.01 for unseen speakers.Source: §5.1 · p. 3
• Full and Recorded achieved higher direction alignment than Pseudo conditions, while Full retained improved speaker robustness relative to Recorded.Increasing pseudo-data scale further stabilized speaker identity preservation without substantially degrading alignment.Source: §5.1 · p. 4

Subjective evaluation

• Subjective results showed lower speaker similarity for Recorded, whereas Full and Pseudo-all achieved higher similarity scores for seen and unseen speakers.Pseudo-all slightly outperformed Full, likely because its more conservative modulation produced smaller changes between utterances.Source: §5.2 · p. 4
• Recorded achieved the highest subjective direction alignment, followed by Full and Pseudo-all; Full balanced expressive variation and identity stability.Participants rated alignment on a 5-point scale and assigned a score of 1 when speaker identity changed significantly.Source: §5.2 · p. 4

Analysis of modulation

• Recorded pairs showed larger absolute F0 changes than pseudo-generated pairs, helping explain Pseudo-all’s more conservative modulation.Mean ln F0 changes were 0.14±0.15 for Recorded and 0.05 ± 0.08 for Pseudo.Source: §5.2 · p. 4

Overall findings

• Pseudo-all alone enabled stable identity-preserving refinement with reasonable direction alignment, while Full provided the most favorable overall trade-off.Subjective alignment results were broadly consistent with the LLM-based objective evaluation.Source: §5.2 · p. 4

Conclusion

• Direction-following TTS models relative performance modification using pseudo triplets built through impression-controlled synthesis and LLM-based direction generation.The triplets address the scarcity of paired direction data.Source: §6 · p. 4
• Pseudo data improves speaker robustness, recorded data enhances direction alignment, and combining them achieves the best balance between stability and expressiveness.Future work will investigate integrating human expressiveness with pseudo data.Source: §6 · p. 4
• Pseudo data alone enables stable identity-preserving refinement with reasonable direction alignment, supporting the effectiveness of pseudo-triplet construction.This conclusion is reported alongside the complementary benefits of recorded data.Source: §6 · p. 4