PaperlayerView exact revision on arXiv

Source-linked AI summary

Phrase-Localized Language-Contrastive Guidance: Training-Free Localized Accent Control for Code-Switching Text-to-Speech

Che Hyun Lee, Sangkwon Park, Donghun Kang, Dongwook Lee, Youngho Cho, Heeseung Kim, Sungroh Yoon

arXiv:2609.01016v1cs.CLcs.SD

TL;DR

Cross-lingual code-switching TTS often gives embedded foreign phrases the matrix language’s accent, because global guidance couples local pronunciation to the reference voice. LCG uses self-attention-derived masks and independent localized language guidance without retraining, improving phrase nativeness while preserving speaker-related qualities across a 12-direction benchmark.

  • Problem

    Zero-shot multilingual TTS suffers accent leakage in embedded foreign phrases, while global guidance cannot selectively control localized language transitions.

  • Method

    LCG derives phrase masks from internal self-attention probing and applies independent language-contrastive guidance over localized acoustic regions without model retraining.

  • Results

    LCG reduces overall MER from 0.564 to 0.445, raises embedded phrase language accuracy from 0.233 to 0.518, and receives a 75.5% native-speaker preference across 488 ratings.

  • Takeaways & Limitations

    Across 12 directions, LCG robustly enforces native accents while preserving global naturalness, speaker identity, and intra-utterance consistency.

  • Takeaways & Limitations

    Evaluation primarily uses synthetic code-switching scripts and monolingual reference voices, so stability across spontaneous real-world conversation remains bounded.

Abstract

from arXiv · show

Current speech synthesis struggles with code-switching, which mixes a foreign language phrase into a primary language utterance, causing the phrase to be spoken with the primary language's accent rather than its native one. We propose Phrase-Localized Language-Contrastive Guidance (LCG), a training-free inference framework that restores a native accent to code-switched phrases in cross-lingual text-to-speech. LCG replaces the single language guidance applied across the whole utterance with a separate guidance for each region, so each part is guided by its own language. To choose where to apply this localized guidance, we propose a self-attention probing technique that finds the phrase boundaries without external alignments. Together, these components generate speech in which each region carries the accent of its own language, requiring no fine-tuning or auxiliary models. Across diverse language pairs, LCG robustly increases the nativeness of the code-switched phrase while suppressing accent leakage, and preserving overall speaker identity and naturalness.

1 Introduction

Zero-shot multilingual TTS can preserve speaker identity across languages but often leaks the reference language’s accent into embedded foreign phrases. LCG addresses this by deriving phrase boundaries internally and applying localized language guidance while preserving the surrounding utterance.

  • Motivation: Zero-shot multilingual TTS still suffers cross-lingual accent leakage when the reference voice language differs from the target text.The resulting foreign accent is especially problematic in code-switched utterances.
  • Motivation: Intra-utterance code-switching requires native pronunciation for the embedded phrase alongside carrier fluency and speaker consistency.The speaker profile can lock pronunciation into the primary language, making localized style shifts difficult.
  • Motivation: Uniform classifier-free guidance globally amplifies reference-voice bias and flattens local language transitions into the matrix accent.Because standard guidance uses one scale across the sequence, it cannot selectively control individual segments.
  • Proposed approach: LCG applies localized guidance over self-derived phrase boundaries, enforcing native pronunciation while preserving matrix-carrier quality and speaker identity.The framework uses internal probing rather than external phrase alignments.
  • Contributions: The framework combines Phrase-Localized LCG, internal attention-based phrase localization, and a balanced 1,200-utterance benchmark spanning 12 directions across 5 languages.The authors report native-accent enforcement across all directions while preserving global naturalness.

2 Related Works

Related work has improved multilingual and code-switching speech synthesis through model training and specialized architectures, while inference-time guidance remains largely global. LCG connects localized region control with internally probed acoustic-text boundaries and diffusion denoising.

  • Multilingual and Code-Switching Speech Synthesis: Multilingual foundation models clone speaker identities across unseen single-language tracks but frequently struggle when multiple languages mix within one sentence.Earlier code-switching TTS methods encode capabilities through bilingual posteriorgrams, cross-lingual embeddings, fine-tuning, or specialized diffusion architectures.
  • Inference-Time Control and Spatial CFG: Standard speech-side classifier-free guidance applies one scalar across the entire sequence, lacking spatial resolution for localized phonetic shifts.Global timestep schedules likewise do not isolate localized structural boundaries.
  • Inference-Time Control and Spatial CFG: LCG uses frame-level span masks dynamically derived from internal self-attention probing instead of predetermined token selection or external alignment models.The broader pipeline extracts masks during inference and applies localized scale adjustment.
  • Attention Probing and Sequence Alignment: Mask refinement combines a dual-tag union with margin dilation to maximize recall at conservative phrase boundaries.The refinement is designed to tolerate alignment shifts and minor boundary errors.
  • Attention Probing and Sequence Alignment: Bi-directional denoising in discrete diffusion models can absorb minor boundary-mask errors without acoustic or identity discontinuities.This provides a model-compatible buffer for localized phrase control.

3 Method

LCG localizes accent control in three stages: self-attention extracts frame masks, boundary refinement expands recall, and independent language-contrastive steering guides phrase regions. It operates during inference on a discrete diffusion TTS model without auxiliary alignment models or fine-tuning.

  • Phrase Localization: The framework extracts per-frame phrase masks by probing decoder self-attention and identifying the text token receiving maximum attention for each acoustic frame.Frames whose argmax token falls within the embedded phrase span are marked for localized control.
  • Phrase Localization: Pilot probing finds sharp, language-agnostic alignment-tracking spikes in middecoder layers, with the {L8, L12} max-pooled ensemble maximizing worst-case boundary recall.The configuration is selected on Korean KSS and English LJSpeech pilot utterances to reduce script-specific bias.
  • Mask Refinement: The mask is refined by uniting matrix- and phrase-conditioned attention masks, then applying symmetric graded dilation to recover residual boundary frames.The union adds no computational overhead because both attention tensors are already computed during parallel conditional inference.
  • Mask Refinement: With k = 4, dilation produces an effective ±10-frame window; transient spillover into matrix regions is absorbed during iterative denoising while preserving carrier naturalness.The method deliberately favors recall because missed phrase frames revert to the matrix accent, whereas false positives are more easily overridden.
  • Language-Contrastive Guidance: The Swap baseline routes phrase-conditioned logits inside the mask and matrix-conditioned logits outside it, but couples contrastive strength to the global CFG scale.LCG replaces this fixed coupling with an independent parameter λ that scales the localized contrastive direction m(cphr − csrc).

4 Experimental Setup

The evaluation uses a balanced synthetic code-switching corpus, objective and subjective metrics, and sweeps guidance, mask refinement, and mask sources. It also examines the λ trade-off between localized accent enforcement and global naturalness.

  • Dataset: The corpus contains 1,200 synthetic utterances spanning five languages and twelve balanced directional code-switching tasks.Each utterance includes multiple embedded phrases.
  • Experimental Axes: Experiments sweep localized guidance strength λ while fixing global text guidance at γ = 2.0, with λ = 0 as Baseline and λ = 3 as the coupled Swap baseline.The tested λ values are 0, 3, 5, 7, 9, 11, and 13.
  • Metrics: Objective evaluation measures mixed transcription errors, embedded-phrase language accuracy and identification confidence, plus speaker naturalness and alignment.Whisper-based MER uses WER for Latin scripts and CER for Japanese and Korean embedded phrases.
  • Guidance Sweep: Figure 4 evaluates native-pronunciation saturation and the trade-off between foreign accent enforcement and global speech naturalness, with the Pareto envelope maximized at λ = 7.The plotted metrics include LIDe and UTMOS.
  • Metrics: Crowdsourced tests separately assess global utterance quality with MOS and perceived foreign-accent nativeness using isolated phrase preference judgments.The setup includes global quality MOS and Phrase Nativeness AB Preference.

5 Results

Across twelve directions, LCG improves embedded-phrase language control and mixed-error performance while preserving speaker identity, with dynamic masks outperforming an offline alignment oracle. λ = 7 provides the main steering trade-off, and human listeners prefer the resulting phrase nativeness with only a small MOS decrease.

  • 5.1 Macro Performance: MER decreases from 0.564 to 0.445, while LAe rises from 0.233 to 0.518 and LIDe rises from 0.247 to 0.588 with LCG at λ = 7.The improvements are averaged over all twelve cross-lingual directions.
  • 5.2 Spatial Localization: LCG at λ = 7 outperforms the forced-alignment oracle, reducing MER from 0.472 to 0.445 and increasing LAe from 0.459 to 0.518.The comparison supports the value of dynamically refined boundaries.
  • 5.2 Spatial Localization: MER improves from 0.455 to 0.445 when refinement progresses from the raw argmax mask to k = 4 + XU, while radius changes affect MER by only 0.002 and LIDe by 0.006.The dual-tag union is identified as the optimal M4+XU configuration.
  • 5.3 Localized Guidance Intensity: Foreign-accent metrics rise sharply through λ = 7 and then saturate, whereas UTMOS degrades with increasing λ and drops sharply beyond λ = 7.The study identifies λ = 7 as the functional optimization knee.
  • 5.4 Human Perceptual Evaluation: 75.5% of 488 native-speaker ratings favor LCG over the unguided Baseline for isolated embedded phrases.Preference rates are 75.7% for JA →EN and 75.2% for KO →EN.
  • 5.4 Human Perceptual Evaluation: Global Quality MOS is 3.931 ± 0.092 for LCG versus 4.007 ± 0.110 for Baseline, a numerically small drop below 0.08 with overlapping 95% confidence intervals.The evaluation uses complete utterances on a 5-point scale.

6 Conclusion

The paper presents LCG as a training- and module-free inference framework for phrase-level code-switching that combines self-attention masks with localized language-contrastive guidance. Evaluations on a balanced 1,200-utterance, 12-direction benchmark show native-accent enforcement while the method preserves speaker identity and absorbs minor boundary errors during denoising.

  • 6 Conclusion: LCG dynamically extracts masks from self-attention layers and decouples localized accent control from global text guidance without model retraining.The framework is designed for phrase-level code-switching.
  • 6 Conclusion: Objective and human evaluations on a balanced 1,200-utterance, 12-direction benchmark confirm robust enforcement of native accents.The conclusion summarizes both automated and perceptual evidence.
  • 6 Conclusion: Expanded masking aligns with iterative discrete-diffusion denoising to absorb minor boundary errors while preserving the matrix carrier and speaker identity.The paper characterizes this as a scalable zero-training-cost alternative for controllable cross-lingual speech synthesis.

Limitations

The framework’s evaluation is limited by constrained language and data coverage, dependence on discrete diffusion backbones, required phrase-span inputs, and synthetic evaluation conditions.

  • Data and linguistic scope: The evaluation corpus contains 1,200 utterances and does not exhaustively cover language typology, leaving deeply low-resource or structurally divergent families open for study.The benchmark focuses on major high-resource language pairs.
  • Model scope: LCG is tightly coupled to discrete diffusion language model backbones, while its robustness for autoregressive speech generation remains unverified.The authors note that autoregressive error accumulation could affect generalization.
  • Input requirements: LCG requires the embedded phrase’s character span, which may need to be supplied when script boundaries or language identifiers cannot determine it automatically.
  • Evaluation realism: Evaluation uses synthetically authored code-switching scripts paired with monolingual reference voices, so performance across spontaneous conversational code-switching may vary.Human-authored transcript results preserve the same ordering, but single-word insertions show narrower margins and noisier phrase-level measurements.

Ethical Considerations

The paper discusses ethical risks from precise accent steering, including deceptive multilingual deepfakes and exaggerated or stereotypical accent patterns, alongside safeguards in perceptual evaluation.

  • Risks: Precise, training-free foreign-accent steering could be misused to create deceptive multilingual deepfakes.
  • Representation: Aggressive contrastive scaling can produce exaggerated phonetic patterns or stereotypical accent caricatures that risk cultural insensitivity.The paper frames native-like rendering as a design target rather than a universal ideal because real speech may be matrix-language accented.
  • Participant disclosure: Human annotations were crowdsourced through AMT with disclosed synthetic-audio research use, voluntary participation, and task-based compensation.
  • Evaluation design: Quality MOS evaluates naturalness, prosodic continuity, and processing anomalies across four audio conditions on a five-point scale.
  • Evaluation design: Phrase-nativeness comparisons use only oracle cases with perfect language accuracy to isolate phonetic accent steering from token-generation failures.
  • Evaluation design: The evaluation separates full-utterance naturalness from phrase nativeness because a single global metric conflates distinct perceptual dimensions.
  • Reported outcomes: LCG preserves global carrier naturalness: all three execution patterns have statistically indistinguishable Quality MOS results with overlapping 95% confidence intervals.
  • Reported outcomes: Localized guidance is preferred over the unguided baseline for phrase nativeness across all language configurations with p < 0.001.

A.4 Phrase Nativeness AB Test: Per-Direction Analysis

LCG is strongly preferred for localized phrase nativeness in both evaluated source-language directions, showing consistent gains across Japanese-to-English and Korean-to-English code-switching.

  • Per-direction results: 75.7% preference in JA →EN and 75.2% in KO →EN yield a 75.5% overall preference margin for LCG.The result is highly statistically significant against 50% chance performance: z = 10.01, p < 0.001.
  • Per-direction results: The preference pattern remains consistent regardless of the source matrix language direction, with native validators favoring more authentic phrasal pronunciations.

B Implementation Details

The implementation uses OmniVoice’s discrete diffusion backbone and self-attention probing, requiring an eager attention backend that adds measurable but still modest synthesis overhead.

  • Backbone and decoding: LCG uses the public 0.81 B-parameter k2-fsa/OmniVoice discrete DLM with 28 decoder layers, an 8-codebook codec, and 32 denoising steps.The reported default configuration uses γ = 2 and λ = 7 with M4+XU refinement.
  • Attention probing: LCG consumes decoder self-attention weights, so fused kernels that do not materialize attention require recomputation or an eager implementation.All compared guidance conditions use the same eager backend for fair contrasts.
  • Backend configuration: The implementation benchmark uses SDPA as the reference because the OmniVoice implementation runs on SDPA rather than FlashAttention.
  • Deployment benchmark: The deployment benchmark measures synthesis cost over 50 approximately 38-second utterances using a single NVIDIA B200 GPU in bfloat16.Timing excludes two warmup utterances per condition and uses the main decoding configuration.
  • Deployment cost: LCG incurs 2.26× total overhead, decomposed into 1.61× from the eager backend and 1.40× from the third guidance branch.The third branch remains below the 1.5× FLOPs expected from one additional branch.
  • Deployment cost: Despite the overhead, LCG synthesizes about 19× faster than real time, adding 1.1 seconds per approximately 38-second utterance.Only 2 of 28 decoder layers are probed, leaving eager-backend cost potentially reducible.

D Robustness to the Attention Weighting

The study tests whether value-norm weighting changes attention-based phrase-mask selection and finds that the selected configuration and operating-point recall remain effectively stable.

  • Motivation: Raw attention can misrepresent token importance when value vectors have small norms, while attention sinks may attract weight without carrying much content.These concerns motivate testing value-norm-weighted attention rather than relying solely on raw weights.
  • Method: The method treats attention as text-to-audio alignment, excludes transcript-external sink positions, and selects layer/head settings empirically by forced-alignment recall.The argmax is restricted to transcript token columns, while configuration selection is based on recall against forced-alignment ground truth.
  • Evaluation: Twelve layer/head configurations were re-scored on KSS and LJSpeech pilot sets of 100 utterances each using per-head value-norm weighting before pooling.The re-scoring follows the Figure 3 setup and evaluates masks against forced-alignment ground truth.
  • Results: {L8, L12} with head-max pooling remains Rank-1 in worst-case recall under both raw and value-norm-weighted attention.The configuration-selection criterion is worst-case recall across the two pilot domains.
  • Results: Recall at the operating point is nearly identical under both weightings, with the residual difference smaller than the boundary tolerance supplied by margin dilation.The reported result indicates no recall cost from raw attention in this setting.

E Evaluation on Human-Authored Code-Switching Text

The paper evaluates code-switching behavior on human-authored Korean–English transcripts and compares LCG with an external zero-shot TTS system. On the real-text evaluation, LCG improves embedded-phrase language accuracy while preserving global quality and speaker similarity, whereas the external comparison is explicitly non-controlled.

  • Evaluation setup: The main benchmark uses synthetic mixed-language text to control phrase length, direction coverage, density, position, and reference voice.Speech is synthesized by the systems being compared, allowing controlled diagnosis of code-switching behavior.
  • Evaluation on human-authored text: The real-text evaluation samples 100 Korean–English utterances each from long-phrasal and single-word strata, using transcripts only and no recorded audio.The long-phrasal stratum contains at least four-word English insertions, while the single-word stratum reflects a common natural code-switching regime.
  • Results: Baseline < Swap < LCG holds on both embedded-phrase nativeness metrics in both strata of the human-authored evaluation.The ordering is reported for LAe and LIDe across the long-phrasal and single-word regimes.
  • Results: +0.154 embedded-phrase language accuracy over the baseline is achieved overall, while UTMOS stays flat and speaker similarity is unchanged.The gain is largest for long phrasal insertions; single-word segments remain noisier but still improve with LCG.
  • External reference comparison: MOSS-TTS reaches LAe 0.280 vs. 0.233 and LIDe 0.302 vs. 0.247 against the unguided baseline, but the comparison is a cross-backbone reference point.Its results vary substantially by language direction, so they do not isolate the guidance mechanism.
  • Human evaluation: The human evaluation includes a Phrase Nativeness AB test on isolated phrase segments and a Global Quality MOS test on full utterances.The MOS task uses a 5-point scale with labeled reference clips to anchor ratings.
Loading 2609.01016v1…