Source-linked AI summary
Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning
Yoomee Cho, Jisun Lee
TL;DR
Cross-lingual zero-shot TTS can inherit a reference speaker’s accent, and single operating points leave the identity cost of accent reduction unreported. The paper introduces AAG, which estimates an accent direction from the model’s predictions for one synthetic voice in two languages. AAG leaves the reweighting curve and reaches near-native dubbing accent while retaining speaker similarity.
Problem
In cross-lingual zero-shot TTS, a reference can lend its pronunciation to target speech, while single operating points leave the identity cost of accent reduction unreported.
Source: §1 · p. 1Method
Results
AAG leaves the reweighting curve; on OmniVoice, accent reaches 4.28 versus 4.33 for a reference-free voice while retaining two thirds of the speaker similarity that reweighting gives up.
Source: §6 · p. 4Takeaways & Limitations
A premise test and the reach of a model’s curve indicate whether and roughly how much an inference-time fix can gain, including the no-gain prediction for X-Voice.
Source: §1 · p. 1, §5 · p. 3The identity–accent curve argument uses a working factorization of reference evidence into speaker and accent, explicitly presented as a model rather than a derivation.
Source: §2 · p. 1
Abstract
from arXiv · showhide
In cross-lingual zero-shot text-to-speech, the accent of the reference leaks into the target speech. We propose accent analogy guidance (AAG), a training-free sampler term that subtracts an accent direction estimated from the model's own predictions for one synthetic voice rendered in both languages, so the voice cancels and only the accent remains. By a blind LLM accent judge on real dubbing data, reweighting classifier-free guidance between reference and text, and its variants, stay near one identity-accent trade-off curve; we score a method by its speaker similarity above that curve at equal accent ($\Delta$SIM). Across four open TTS models AAG lies above the curve: on OmniVoice $\Delta$SIM is +0.11 to +0.27 on three test sets (accent 3.51 to 4.28 on a 1-5 scale at speaker similarity 0.29, where reweighting keeps 0.02); MaskGCT and CosyVoice 2 also lie above their curves, and on F5-TTS it is more native than any reweighting setting. An LLM-free language-ID measure and a twelve-listener panel agree. A premise test and the reach of a model's own curve indicate in advance whether and roughly how much AAG can gain, predicting the one model where it gains nothing (X-Voice).
1. INTRODUCTION
Cross-lingual zero-shot TTS can copy a reference speaker’s pronunciation into target speech, while inference-time guidance changes trade speaker identity against accent. The paper proposes AAG and evaluates methods relative to that trade-off.
1. INTRODUCTION
- When reference and target languages differ, zero-shot TTS can copy the reference’s pronunciation, making dubbed speech foreign-accented.Training-time remedies add language or accent representations, while inference-time remedies manipulate classifier-free guidance.Source: §1 · p. 1
- Inference-time remedies rescale the same reference evidence, so accent and speaker move together.Results reported only as single operating points leave the identity cost of reducing accent unreported.Source: §1 · p. 1
- AAG estimates an accent direction from a proxy voice rendered in both languages, and the paper evaluates speaker similarity at equal accent against the model’s reweighting curve.The introduction reports that seven inference-time variants stay less than 0.05 above the curve, while AAG reaches 0.11–0.30 across three sets.Source: §1 · p. 1
2. GUIDANCE AND THE IDENTITY–ACCENT CURVE
Dual-weight guidance combines text and reference evidence, with the reference residual modeled as shared speaker-and-accent evidence. The section defines a reweighting curve as a baseline for comparing speaker similarity at equal accent.
2. GUIDANCE AND THE IDENTITY–ACCENT CURVE
- Dual-weight guidance scales the text evidence with a_t and the reference evidence with a_s when sampling target speech.The reference residual is the evidence added by the reference beyond the text-only prediction.Source: §2 · p. 1
- The working factorization treats reference evidence as speaker and source-accent factors raised together by a_s, predicting a speaker–accent trade-off curve.The curve is the upper convex hull of reweighting settings and the no-reference endpoint; ΔSIM measures similarity above it at equal accent.Source: §2 · p. 1
- Fig. 1 plots WavLM SIM against judge accent; its shaded upper hull marks reweighting-reachable settings, and ΔSIM is the horizontal gap to the hull.Black crosses show Sec. 2 variants, while red stars and dashes show AAG settings and its β sweep.Source: Figure 1 · p. 2
- AAG scores +0.30, +0.11, and +0.15 on the pilot and public sets (held-out +0.27), beating each set’s best variant by +0.09 to +0.28.None of seven variants lies 0.05 or more above the reweighting hull; five of 17 variant–set pairs are significantly above it, all by under 0.05 and at low accent.Source: §2 · p. 2
3. ACCENT ANALOGY GUIDANCE
AAG estimates accent direction from the model’s predictions for one proxy voice in two languages, so the voice cancels while the accent direction remains. The section also describes diagnostics for when AAG may help, proxy selection, and parameter tuning.
3. ACCENT ANALOGY GUIDANCE
- Contrasting the model’s predictions for one proxy voice rendered in both languages estimates the accent direction while holding speaker identity fixed.Averaging this difference over M proxies supplies the direction used in AAG.Source: §3 · p. 2
- With β = as, AAG guides toward the counterfactual reference “same speaker, target-language accent,” represented as a direction rather than audio.The shared voice in the two conditions is why identity cancels.Source: §3 · p. 2
- The premise test checks whether cloning from a source-language proxy rendition loses accent relative to its target-language rendition, while curve reach measures retained SIM at accent 4.0 where available.The table reports premise, reach, and AAG ΔSIM across test sets; its caption identifies F5-TTS and CosyVoice 2 as models whose curves reach accent 4.0.Source: Table 1 · p. 2, §3 · p. 2
- Proxy sets use native-accented target-language samples and same-voice source-language clones, with two proxies per language pair and gender and no test speakers.For models other than OmniVoice, proxies are selected by accent movement in the premise test, measured without test data.Source: §3 · p. 2
- The design pilot chose β = 5 at as = 3, above the counterfactual reading’s β = as, but β still needs a small per-model sweep.The two prospective uses of this choice were not significant.Source: §3 · p. 2
- Each proxy adds two network evaluations per step, taking 3.3 versus 2.0 seconds per OmniVoice line on an RTX 3090.Source: §3 · p. 3
4. EXPERIMENTAL SETUP
The experiments test AAG across public TTS models and dubbing datasets, using fixed OmniVoice settings after a pilot. Accent, speaker similarity, intelligibility, and naturalness are evaluated with automated measures, with human-expert comparisons for accent validation.
4. EXPERIMENTAL SETUP
- Experiments use public checkpoints of OmniVoice, MaskGCT, F5-TTS, X-Voice, and CosyVoice 2, with AAG applied at different stages of their generation pipelines.Default CFG scales are (3, 3) for OmniVoice and F5-TTS and (3.5, 3.5) for MaskGCT and X-Voice; OmniVoice also uses the dubbing-tuned (2, 5) setting.Source: §4 · p. 3
- After a pilot of 49 dubbing lines from 15 videos, OmniVoice settings were fixed and each test set was run once without further tuning.Tests include 36 held-out Korean lines, 50 Zeroth-Korean lines, and 50 THCHS-30 development or test lines for the Chinese settings, using Gemini 2.5 translations and generally two seeds.Source: §4 · p. 3
- Accent is scored by a blind Gemini 2.5 Pro judge, whose ratings are compared with expert ratings; Whisper language-ID logits provide an LLM-free measure, alongside WavLM speaker similarity, Whisper WER, and UTMOS naturalness.On real speech, the accent judge has Spearman 0.74 agreement with experts, while Whisper language ID has AUC 0.995 for separating the tested native and non-native groups.Source: §4 · p. 3
5. RESULTS
AAG improves accent at equal speaker similarity on OmniVoice and transfers above or beyond reweighting curves on other models. Controls and independent evaluations support the accent-specific effect, while also showing proxy sensitivity and audible trade-offs.
OmniVoice
- On held-out dubbing, accent rises from 3.51 to 4.28 at SIM 0.294, with ∆SIM +0.27 above the hull’s 0.02.The accent score reaches within 0.05 of the no-reference voice’s 4.33; the tuned setting’s SIM is 0.429.Source: §5 · pp. 3–4
Other models
- Across MaskGCT, F5-TTS, and CosyVoice 2, AAG is more native than every reweighting setting in at least one setting, while X-Voice gains nothing.The X-Voice exception matches the failed premise that accent depends on the reference language when voice is fixed.Source: §5 · p. 4
Controls
- Wrong-language proxies remove the pilot gain, indicating the effect is language-specific rather than a general pull toward native voices.Spanish proxies for Korean lines yield accent 3.49 versus 4.33 and ∆SIM +0.03, which is not significant.Source: §5 · p. 4
Independent evaluations
- Whisper language ID follows the judge across 45 pilot settings (Spearman 0.97), while twelve listeners rate AAG as more native and closer to the reference than equal-accent reweighting.Listeners chose AAG in 85% of decided answers for both comparisons; against the tuned setting, they judged the tuned voice closer in 77%.Source: §5 · p. 4
Controls
- AAG’s gain is not simply a per-token cut of the reference residual, and results vary with the proxy set.The parallel component retains at most 37% of the gain; two alternative proxy sets match the held-out gain but give about half on Zeroth.Source: §5 · p. 4
6. CONCLUSION
The paper presents AAG as a training-free way to improve accent while retaining speaker identity beyond reweighting curves. Its results transfer across several models, with a diagnostic predicting the exception and limitations involving measurement, proxies, and tuning.
6. CONCLUSION
- AAG contrasts the model’s predictions for one synthetic voice rendered in two languages to isolate an accent direction, without training.The conclusion reports its use in token, logit, and flow samplers.Source: §6 · p. 4
- On OmniVoice, AAG reaches accent 4.28 versus 4.33 for a reference-free voice while retaining two thirds of the speaker similarity reweighting gives up.Speaker identification remains 69–99% across sets, and WER stays within one point.Source: §6 · p. 4
- The stated limitations are reliance on an LLM judge, proxy-set sensitivity, and a per-model β sweep.Language ID disagrees on THCHS, Spanish, and F5.Source: §6 · p. 4