Source-linked AI summary
DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS
Ambuj Mehrish, Abhinaba Roy, Alex Ivanov, Tawsif Ahmed, Dorien Herremans
TL;DR
Zero-shot TTS references entangle speaker identity with accent, limiting independent requests. DEFINE conditions on separate speaker and accent exemplars and a guidance weight; it matches cascade accent transfer on seen and out-of-domain accents while better preserving speaker identity.
Problem
Zero-shot TTS can reproduce an unseen speaker from a short reference, but the reference also carries accent, with no independent mechanism to request a different accent for that speaker.
Source: §1 · p. 1Method
DEFINE specifies speaker identity and target accent from separate audio examples, using prototype-supervised exemplar conditioning and a single inference-time weight to control accent strength.
Source: §1 · p. 1, §3.2 · p. 2Results
DEFINE matches Seed-VC on seen accents (ACC 0.196 vs. 0.188), is statistically indistinguishable on out-of-domain accents (0.087 vs. 0.083), and better preserves speaker identity.
Source: §4.1 · p. 3, §5 · p. 4Takeaways & Limitations
A single zero-shot TTS model can independently control speaker identity and accent from audio exemplars, including for out-of-domain accents.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Zero-shot text-to-speech (TTS) can reproduce an unseen speaker from a short reference recording, but typically entangles speaker identity and accent within the same reference. We introduce DEFINE, an end-to-end framework that decouples these factors by conditioning speaker identity and target accent on separate audio exemplars. A single inference-time guidance weight continuously controls accent strength without retraining. Built on F5-TTS with parameter-efficient LoRA adaptation, DEFINE maps short accent exemplars into a conditioning space using an exemplar encoder supervised through learned accent prototypes, requiring neither accent labels at inference time nor post-synthesis waveform conversion. On seen accents, increasing accent guidance improves accent-probe accuracy from 6.5% to 19.6%. More importantly, a single DEFINE model generalizes accent control beyond its training accent set: on seen and out-of-domain accents, though not on held-out accents, it matches the accent transfer performance of a two-model TTS-voice-conversion cascade while achieving higher speaker similarity and comparable predicted speech quality. These results demonstrate that speaker identity and accent can be independently controlled from audio exemplars within a single zero-shot TTS model, including for accents unseen during training.
1. INTRODUCTION
Zero-shot TTS references can entangle speaker identity and accent, leaving no independent way to request a target accent for a reference speaker. DEFINE separates them using speaker and accent exemplars, with a guidance weight for adjusting accent strength.
1. INTRODUCTION
- Earlier approaches use accent labels or embeddings, or convert speech after synthesis; neither naturally unifies reference-based voice and accent specification with post-training strength control.Zero-shot TTS training data often pairs each speaker with a single native accent, giving little incentive to represent identity and accent independently.Source: §1 · p. 1
- DEFINE specifies speaker identity with reference audio and target accent with a separate short exemplar, without requiring an accent label at inference.The exemplar is encoded into a conditioning representation rather than passed through a separate waveform-conversion stage.Source: §1 · p. 1
- A single inference-time weight w controls accent strength, with w=0 removing accent conditioning and larger values strengthening its influence.The control is available after training.Source: §1 · p. 1
- Prototype anchoring uses learned per-accent prototypes to supervise an exemplar encoder whose representation conditions an F5-TTS backbone adapted with LoRA.This provides direct supervision where flow-matching alone gives only indirect supervision to the encoder.Source: §1 · p. 1
2. RELATED WORK
Related work covers zero-shot TTS, explicit accent conditioning, and post-hoc accent conversion. DEFINE’s overview instead shows accent exemplars conditioning the synthesizer directly.
2. RELATED WORK
- Zero-shot TTS uses short references to reproduce unseen speakers, but those references jointly specify identity and speaking characteristics, implicitly coupling accent with voice.Related approaches include discrete speech representations, flow matching over masked acoustic features, and explicit speaker encoders.Source: §2 · p. 1
- Explicit accent conditioning uses labels or learned representations, but such methods are generally restricted to training accents and require labelled data and adaptation for new accents.Some methods also control accent strength.Source: §2 · p. 1
- Post-hoc accent conversion modifies synthesized or recorded speech, often through a separate voice-conversion model that adds a transformation stage where speaker characteristics must be preserved.DEFINE’s overview depicts accent exemplars being encoded and injected into the TTS model instead.Source: §2 · p. 2, Figure 1 · p. 2
3. METHOD
DEFINE adapts F5-TTS to condition speaker identity and target accent separately, using accent exemplars and learned prototypes. Its inference-time guidance weight continuously adjusts accent strength.
3.1. Backbone and accent injection
- DEFINE builds on F5-TTS conditional flow matching over mel spectrograms and adapts its unchanged backbone with rank-16 LoRA, training 6.3M of 343M parameters.The reference audio carries speaker identity through in-context infilling; the accent is represented by a separate 128-dimensional embedding.Source: §3.1 · p. 2
- RMS-scaled shifts inject accent conditioning into timestep and text embeddings, reaching every transformer block; setting a=0 leaves embeddings unchanged and trains an accent-free branch with pdrop=0.15.Without RMS scaling, the added term is about 100 times smaller than the timestep embedding and is ignored during training.Source: §3.1 · p. 2
3.2. Exemplar encoder and prototype anchoring
- A frozen XLS-R encoder at layer 15 processes short clips from speakers absent from training; clips are pooled into accent embeddings, with K=1 training and K=1 or 3 inference pooling giving similar accuracy.The encoder uses speech-region crops and a masked mean of frame features before mapping clips to the accent-conditioning space.Source: §3.2 · p. 2
- Prototype anchoring addresses weak flow-matching supervision: learned per-accent prototypes supervise encoder alignment, while mixed prototype/exemplar conditioning, contrastive grouping, and speaker-adversarial loss shape representations; prototypes are discarded at inference.The prototype/exemplar mixture uses a Bernoulli choice with π=0.5; the stop-gradient prevents the flow-matching loss from updating the exemplar encoder.Source: §3.2 · p. 2
3.3. Accent guidance at inference
- A continuous inference-time guidance weight controls accent strength: w=0 removes accent conditioning, and higher w strengthens it to trade off accent strength against speaker similarity.DEFINE is reported at w=15, selected on validation items disjoint from the evaluation grid; it infers accent from exemplars without an accent label.Source: §3.3 · p. 3, §3.2 · p. 3
4. EXPERIMENTAL SETUP AND RESULTS
On seen accents, DEFINE’s guided accent accuracy matches the cascade with higher speaker similarity; it is statistically indistinguishable from the cascade on out-of-domain accents but trails on held-out accents. In the listening test, listeners preferred DEFINE over both comparison systems.
4. EXPERIMENTAL SETUP AND RESULTS
- Evaluation covered 1,764 items: 1,134 seen, 378 held-out, and 252 out-of-domain, using an accent probe, speaker-similarity measure, WER, and predicted quality.Training combined 137k real recordings with 94.7k deliberately mismatched prompt–target pairs; held-out and out-of-domain accents were excluded from training.Source: §4 · p. 3
4.1. Results
- 0.196 ACC at w=15 matches the Seed-VC cascade’s 0.188 on seen accents, with 0.024 higher SPK and equivalent predicted quality; on held-out accents, the cascade leads 0.175 to 0.127.On out-of-domain accents, DEFINE is statistically indistinguishable from the cascade (0.087 vs. 0.083).Source: §4.1 · p. 3
- 0.214 exemplar-driven ACC on seen accents and 0.111 on out-of-domain accents are the peaks at w=19; SPK falls monotonically as guidance increases.Table 2 reports exemplar-driven ACC and SPK alongside seen-accent prototype-lookup ACC, which is a bound rather than an exemplar result.Source: Table 2 · p. 3, §4.1 · p. 4
- Without prototype anchoring, no guidance weight improves on w=0 (ACC 0.079); the contrastive prototype term adds 0.053, while cross-attention at w=23 yields UTMOS 1.52 and SPK 0.22.Mean-pooling three exemplars changes ACC by +0.000 on seen and +0.003 on held-out accents, and adds 0.019 SPK.Source: §4.1 · p. 4, Table 3 · p. 4
- Prototype lookup reaches 0.354 ACC on seen accents versus 0.196 from exemplars, while nearest-prototype accuracy drops from 0.873 on training speakers to 0.540 on unseen speakers.The authors identify speaker overfitting in the encoder, rather than conditioning capacity or guidance, as the bottleneck.Source: §4.1 · p. 4
- DEFINE’s out-of-domain result does not rely on Seed-VC-generated training data; its reported SPK is 0.624–0.654 versus 0.600–0.603 for the baseline.Held-out and out-of-domain accents were excluded from the converted pool.Source: §4.1 · p. 4
4.2. Listening test
- DEFINE received 77 of 110 votes (0.70) against the cascade’s 23 and F5-TTS’s 10; listeners chose it most often for 9 of 11 participants.With guidance off, votes fell below chance; the two active guidance settings were not separable (p=0.69).Source: §4.2 · p. 4, Figure 2 · p. 4
5. CONCLUSION
DEFINE separates speaker identity from accent using distinct audio exemplars, with a post-training scalar controlling accent strength. It matches the conversion cascade on seen accents, is statistically indistinguishable on out-of-domain accents, and better preserves speaker identity.
5. CONCLUSION
- DEFINE separates speaker identity and accent using distinct audio exemplars, while a post-training scalar controls accent strength without retraining.With a single inference pass and no accent labels, it matches the conversion cascade on seen accents, is statistically indistinguishable on out-of-domain accents, and better preserves speaker identity.Source: §5 · p. 4