PaperlayerView exact revision on arXiv

Source-linked AI summary

Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data

Thanathai Lertpetchpun, Thanapat Trachu, Jihwan Lee, Tiantian Feng, Dani Byrd, Shrikanth Narayanan

arXiv:2603.07534v1cs.CL

TL;DR

Current TTS systems are biased toward dominant English varieties because suitable accented speech data are limited, leaving fine-grained accent control unresolved. Accent Vector derives a controllable parameter shift by fine-tuning multilingual TTS on native or regional speech, then scaling or combining it. Experiments report accent control across languages and mixed-accent synthesis, while identifying evaluation and modeling constraints.

  • Problem

    Limited accented speech data leave current TTS systems predominantly modeling mainstream English, while existing methods provide limited fine-grained control over accent variation.

  • Method

    Accent Vector fine-tunes a multilingual TTS model on native or regional speech and extracts a task-space parameter shift whose magnitude and combinations control accent characteristics.

  • Results

    Experiments across four settings demonstrate accented English synthesis, non-English generalization, fine-grained accent-strength control, and blending of two accents.

  • Takeaways & Limitations

    Accent Vector enables accent-specific synthesis without accent-specific speech datasets while supporting continuous strength control and mixed-accent generation across languages.

  • Takeaways & Limitations

    Automatic evaluation relies on pretrained proxies with domain biases and mismatch for many cross-lingual accent combinations, so absolute scores require caution.

Abstract

from arXiv · show

Accent is an integral part of society, reflecting multiculturalism and shaping how individuals express identity. The majority of English speakers are non-native (L2) speakers, yet current Text-To-Speech (TTS) systems primarily model American-accented English due limited accented data. We propose \textit{Accent Vector}, a controllable representation that enables accent manipulation in multilingual TTS without requiring accented training data. \textit{Accent Vector} is derived by fine-tuning a TTS system on native speech of a different language (i.e. non-English) and computing task vectors capturing accent characteristics (i.e. in English). By scaling and interpolating the vector, we achieve fine-grained control over accent strength and generate mixed-accent speech. In addition, it generalizes beyond English, enabling accent control across multiple languages. Objective and human evaluations confirm the effectiveness of Accent Vector for fine-grained and compositional accent control.

1. Introduction

Accent Vector addresses the scarcity of accented English TTS data by deriving controllable accent representations from native-language speech. It supports accent-strength control, multilingual generalization, and mixed-accent synthesis.

  • Motivation: Limited large-scale, high-quality datasets leave current TTS systems disproportionately optimized for American English and weaker on underrepresented varieties.Existing accented L2 English datasets are generally insufficient for training high-quality TTS systems.
  • Motivation: Existing data-efficient accent-generation methods induce accent characteristics from native-language resources but offer limited strength control and narrow linguistic coverage.They often focus on pronunciation mapping or duration modeling, leaving scalable fine-grained control unresolved.
  • Approach: Accent Vector is a controllable parameter shift derived by fine-tuning multilingual TTS on native or regional speech associated with a target accent.For L2 accents, the model uses speech from the speaker’s native language rather than large-scale accented English data.
  • Capabilities: The framework enables explicit accent-strength control, generalizes beyond English, and supports composition of multiple accents for mixed-accent speech.The paper highlights applications to speakers shaped by extended exposure to two linguistic environments.

2. Background and Related work

The paper builds on task vectors and multilingual zero-shot TTS, contrasting Accent Vector with label-, transliteration-, rule-, and duration-based accent-generation methods. Its novelty is unified parameter-level control across segmental and suprasegmental features, languages, and accent combinations.

  • Task Vector: Task vectors represent parameter differences between fine-tuned and pretrained models, and approximate linearity permits scaling and addition for controllable adaptation.Accent Vector applies this parameter-shift formulation to accent-specific speech characteristics.
  • XTTS: XTTS is a multilingual zero-shot TTS model supporting 17 languages, using transcript, reference speech, and a language-ID token to synthesize speech.Its encoder predicts discrete acoustic codes generated by a VQ-VAE before decoding.
  • Related Work: Related approaches use accent labels, transliteration, phonetic rules, or target-language duration modeling, but face metadata, fixed-realization, scalability, or coverage limitations.Native-speech methods reduce dependence on accented recordings but may primarily model duration and have narrow validation scope.
  • Accent Vector: Accent Vector provides unified model-level control over phoneme realization, duration, and prosody without accent-specific English datasets.It also supports linear composition and demonstrates effectiveness across German, Hindi, English, French, Spanish, and Mandarin accents.

3. Method

Accent Vector is extracted as the parameter difference between a fine-tuned and pretrained multilingual TTS model, then arithmetically modified to control accent strength and combine accents. The procedure supports accented synthesis across base languages and mixed-accent speech.

  • Fine-tuning: The framework fine-tunes a multilingual TTS model using target-accent speech while conditioning on the base language and its transcripts.For Spanish-accented English, Spanish reference speech and transcripts are used while the language ID is set to English.
  • Obtaining Accent Vector: The Accent Vector is computed as the difference between fine-tuned parameters θft and pretrained parameters θpre.This parameter shift encodes accent-specific characteristics in the model parameter space.
  • Cross-lingual control: The same extraction procedure can generate accents across languages by swapping the base language while preserving fine-tuning and vector extraction.Examples include British-accented Mandarin or Spanish speech.
  • Accent strength: At inference, adding α · τaccent to pretrained parameters provides continuous accent-strength control, with larger α producing stronger accent characteristics.The transcript remains in the base language, while the accent is introduced through modified model parameters.
  • Mixed accents: Multiple Accent Vectors are combined by weighted addition, with coefficients αi controlling each accent’s relative contribution.This supports nuanced mixed-accent synthesis, such as speech influenced by a native language and British English.
  • Accent Vector arithmetic: Figure 2 summarizes Accent Vector computation, scaling for accent strength, and interpolation for mixed-accent synthesis.The figure presents the arithmetic operations underlying the framework’s control mechanisms.

4. Experimental Setup

Experiments use XTTS-v2 with LoRA fine-tuning and speech datasets representing selected accents and languages. The data are restricted to representative dialects and filtered for recording quality and duration.

  • Model and training: XTTS-v2 is fine-tuned with LoRA for 60,000 steps, reducing trainable parameters from 378M to approximately 8M.LoRA uses 16 ranks and is applied to all encoder-module linear layers.
  • Datasets: Training data come from VCTK, Common Voice, IndicVoices-R, and KeSpeech for English, Spanish, Hindi, German, French, and Mandarin accents.These sources provide language- and accent-specific speech for training and evaluation.
  • Dialect selection: Each language is represented by a single selected dialect or regional variety to model a more consistent accent.Examples include England English, Peninsular Spanish, France French, and selected German regional speakers.
  • Data cleaning: Utterances are retained only when DNSMOS exceeds 3.4 and duration is at least 3 seconds.The duration filter is intended to exclude samples with insufficient information for reliably characterizing accents.

5. Evaluation Metrics

The evaluation measures both accentedness and utility, combining automatic accent and language-identification proxies with speech-quality, intelligibility, speaker-similarity, and human judgments. Cross-lingual accent assessment remains approximate, so LID is used as a complementary proxy.

  • Accentedness: Accentedness is evaluated with VoxProfile accent classification and a Spoken Language Identification model.VoxProfile reports target-accent posterior probability and cosine similarity between synthesized and representative real-speech embeddings.
  • Cross-lingual evaluation: Cross-lingual accent evaluation is approximate because available accent classifiers generally do not recognize rare cross-lingual accent configurations.An ECAPA-TDNN LID model trained on VoxLingua107 is therefore included as a complementary proxy.
  • Utility: Utility is evaluated using WER, CER, UTMOS, and speaker similarity.Transcripts are generated with Whisper-medium, and error rates are computed only for utterances shorter than 30 seconds.
  • Human evaluation: The human evaluation examines perceived accent, accent strength, and naturalness using 70 samples rated by 16 listeners.Listeners include native and fluent English speakers with varied native languages, all living in the US.

6. Results and Discussion

Accent Vector shifts synthesized English toward six target accents while preserving speaker similarity, although accentedness increases ASR errors and Mandarin shows the smallest accent-probability improvement.

  • Accent Shift Effectiveness: The framework evaluates accented English synthesis across six accents: British, Spanish, Hindi, German, French, and Mandarin.Accent probability, accent similarity, WER/CER, speaker similarity, and naturalness are reported.
  • Accent Shift Effectiveness: Across all six accents, accent probability and accent similarity increase relative to the pretrained baseline while speaker similarity remains around 0.9.The result indicates accent modification without substantial loss of speaker identity.
  • ASR Degradation and Accentedness: Accented speech has higher WER and CER than standard English synthesis, partly because ASR systems are trained predominantly on standard American English.Pronunciation differences such as “very” realized as “wery” can produce systematic ASR mismatches.
  • Mandarin Accent: Prosodic and Phonological Constraints: Mandarin exhibits the smallest relative improvement in accent probability among the evaluated accents.The paper attributes this difficulty to differences between Mandarin tonal, syllable-timed prosody and English stress-timed rhythm and intonation.
  • Mandarin Accent: Prosodic and Phonological Constraints: Mandarin-accented English also yields consistently higher WER and CER because phonological differences can cause English phoneme substitutions or omissions.These deviations are especially problematic for ASR systems trained mainly on native English speech.

6.2. English-Accented Non-English Speech

Accent Vector transfers English-accent characteristics to Spanish, German, and Mandarin while retaining relatively strong naturalness and speaker similarity. Scaling its coefficient provides progressively stronger accents, accompanied by higher recognition errors and slightly lower predicted naturalness.

  • English-Accented Non-English Speech: The framework tests British-accent transfer to Spanish, German, and Mandarin by fine-tuning on English speech while setting the language token to each target base language.The selected languages span Romance, Germanic, and East Asian language groups.
  • Accent Transfer Results: English accent probability increases substantially across all base languages after fine-tuning, with LID assigning higher English-language probabilities as complementary evidence.VoxProfile provides accent probability, while LID serves as a proxy for accent leakage when direct classifiers are unavailable.
  • Accent Transfer Results: WER and CER increase because English-accented Spanish and Mandarin are rarely represented in ASR training data and are therefore more out-of-domain.This compounds the general ASR degradation associated with accented speech.
  • Accent Transfer Results: The cross-lingual configuration achieves relatively strong naturalness compared with the non-English-accented English setting while preserving speaker similarity.The paper attributes this partly to the high UTMOS of the VCTK fine-tuning corpus and English-oriented UTMOS evaluation.
  • Controlling Accent Strength: Increasing the task-vector coefficient α from 0 to 1 in increments of 0.2 produces progressively stronger British- and Hindi-accent characteristics.The monotonic trend supports smooth accent-intensity control through linear scaling in Accent Vector space.
  • Controlling Accent Strength: As α increases, WER rises while UTMOS gradually declines, revealing a trade-off between accent strength and automatically evaluated speech quality.Stronger accent realization deviates further from the ASR model’s training distribution.

6.4. Mixed Accent

Accent Vectors can be combined and coefficient-scaled to synthesize mixed accents while controlling each accent’s relative strength. Objective and human evaluations indicate that the method shifts speech toward intended accents, though some combinations favor British accent probability and listeners confuse certain European accents.

  • Mixing Multiple Accents: At coefficient 0.5, both target-accent probabilities generally increase in mixed-accent synthesis, except for Spanish + British and Mandarin + British.In those exceptions, British accent probability increases at the expense of the other accent.
  • Mixing Multiple Accents: Mixed-accent speech has lower WER than speech synthesized with a single non-English accent alone.
  • Mixing Multiple Accents: Adjusting α and 1 − α provides fine-grained control over the relative strengths of Spanish and British accents.Figure 4 evaluates Spanish and English accent probabilities across coefficient values, with the pretrained model shown as a reference.
  • Subjective Evaluation: Human listeners distinguish accents significantly better than the 14% random-chance level, while generated accents remain noticeably salient and moderately natural.Naturalness ratings range from 2 (“slightly natural”) to 4 (“quite natural”).
  • Subjective Evaluation: Listeners most frequently confuse German-, French-, and Spanish-accented English, potentially reflecting United States–based listener bias.The confusion matrix compares intended target accents with perceived accents from human responses.
  • Subjective Evaluation: Subjective evaluation confirms that the method shifts speech toward the intended target accent, consistent with objective evaluation trends.

6.6. Limitation

Accent Vector’s effectiveness is constrained by evaluation-proxy bias, domain mismatch, fine-tuning data quality, and linguistic distance between languages. The approximately linear parameter-shift formulation may also inadequately represent complex suprasegmental phenomena.

  • Evaluation Proxies and Domain Mismatch: Objective scores rely on pretrained evaluation proxies with domain biases and limited coverage of cross-lingual accent combinations.Absolute automatic scores therefore require extra caution, particularly for cross-lingual accent transfer.
  • Data and Modeling Constraints: Effectiveness depends on fine-tuning data quality and linguistic distance, with weaker Mandarin gains linked to recording quality and English–Mandarin differences.The reported Mandarin limitations involve substantial prosodic and phonological differences from English.
  • Data and Modeling Constraints: An approximately linear parameter shift may be insufficient to capture complex suprasegmental phenomena, particularly tonal variations.

7. Conclusion and Future Work

The conclusion presents Accent Vector as a practical multilingual TTS framework for controllable accent manipulation without accent-specific speech datasets.

  • Conclusion and Future Work: Accent Vector uses task-space parameter differences to control accent strength continuously and compose mixed-accent speech without accent-specific speech datasets.Experiments across multiple languages report target-accent shifting, speaker-identity preservation, fine-grained control, and accent mixing.

9. Generative AI Use Disclosure

The authors disclose that generative AI tools were used only for language editing, manuscript polishing, and code-writing assistance.

  • Generative AI Use Disclosure: Generative AI tools supported language editing, manuscript polishing, and code writing, while the research ideas, experiments, analyses, and scientific contributions remained human-generated.
Loading 2603.07534v1…