Source-linked AI summary
UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech
Shuhei Kato
TL;DR
Multilingual LLM-TTS can struggle with Japanese pronunciation when it processes raw text without explicit phoneme or prosody input. UtterTune adds LoRA-based phoneme-level pronunciation and pitch-accent control. The authors report improved naturalness and accent correctness while maintaining speaker similarity in zero-shot synthesis.
Problem
Raw-text multilingual LLM-TTS can learn Japanese pronunciation inaccurately, and users cannot control pitch accent when G2P and accent estimation are implicit.
Source: §1 · p. 1Method
UtterTune freezes the CosyVoice 2 backbone, adds LoRA updates, and uses special tokens to delimit Japanese kana input encoding pronunciation and pitch accent.
Source: §3.2 · p. 3, §3.3 · p. 3Results
Naturalness rose from 3.44 to 3.88 MOS, accent correctness rose from 0.498 to 0.899, CER stayed below 7%, and speaker similarity was not degraded.
Source: §6 · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving performance in the others. Although LLM architectures have enabled TTS models to achieve remarkable naturalness, accurately modeling grapheme-to-phoneme (G2P) mapping and prosody remains challenging, especially when the model omits an explicit G2P module and directly processes minimally encoded text (e.g., byte-pair encoding). UtterTune leverages low-rank adaptation to enable the control of segmental pronunciation and pitch accent at the phoneme level for Japanese speech, the target language in this paper, while maintaining naturalness and speaker similarity in a zero-shot setting. Objective and subjective evaluations confirm its effectiveness.
1 Introduction
Multilingual LLM-TTS can sound natural but may mispronounce Japanese when pronunciation and prosody are learned from raw text without an explicit text frontend. UtterTune adds LoRA-based control of Japanese segmental pronunciation and pitch accent while aiming to preserve other model capabilities.
1 Introduction
- Raw-text multilingual LLM-TTS may omit explicit phoneme and prosody markers, leaving pronunciation to be learned from speech-text data that may not cover a language’s diversity.The authors identify Japanese as one language for which this data reliance fails.Source: §1 · p. 1
- Japanese pronunciation is difficult to learn from data because kanji have multiple readings and word combinations can have special readings, while pitch-accent nuclei can shift across linked words.The text notes more than 2,000 everyday-use kanji and over 6,000 kanji in Japanese Industrial Standards.Source: §1 · p. 1
- Unlike conventional Japanese TTS, models with implicit G2P and accent estimation offer no frontend output to edit; kana can correct segmental pronunciation, but users cannot control accent.UtterTune is introduced as an adaptation that supports editing and control, including native-like pitch accent.Source: §1 · pp. 1–2
- UtterTune applies LoRA to CosyVoice 2 for Japanese segmental pronunciation and pitch-accent control, using only a small set of additional parameters.The authors report that its LoRA update file is less than 0.5% of the base model and that it preserves naturalness and speaker similarity in zero-shot evaluation.Source: §1 · p. 2
2 Tokyo Japanese Pronunciation and Its Notation
The section describes Tokyo Japanese segmental and pitch-accent features and the notation used to represent them for speech synthesis.
2 Tokyo Japanese Pronunciation and Its Notation
- Tokyo Japanese segmental pronunciation can be written in hiragana or katakana, which are interchangeable in pronunciation; most kana characters represent one mora.A single mora can contain multiple phonemes, such as a consonant followed by a vowel.Source: §2 · p. 2
- A Tokyo Japanese accent phrase has at most one accent nucleus, and its pitch generally falls from high to low starting at that nucleus.An accent phrase generally begins low-high, except when the nucleus is at the first mora, when it begins high-low; nuclei may shift when words combine.Source: §2 · p. 2
- The standard synthesis notation represents segmental pronunciation in katakana, marks accent nuclei with apostrophes, and marks accent-phrase boundaries with slashes.Source: §2 · p. 2
3 Proposed Method
UtterTune adds Japanese pronunciation control to CosyVoice 2 by combining LoRA updates with tagged phoneme-and-accent input. The model is trained to honor this notation while retaining its ordinary text-input mode.
3.1 Base Model
- CosyVoice 2 processes raw text with a shared BPE tokenizer and generates speech tokens with a unified text-speech language model before a vocoder produces the waveform.Its raw-text design requires pronunciation to be learned from data; the cited CosyVoice 2 paper reports weaker Japanese performance than English and Chinese.Source: §3.1 · p. 2
3.2 LoRA Integration in Transformer Layers
- UtterTune freezes CosyVoice 2 weights and adds rank-r LoRA matrices to every text-speech LM self-attention block’s Q/K/V/O projections.For each weight W, the effective weight is W + α · BC; the trainable updates comprise less than 0.5% of the base model.Source: §3.2 · p. 3
3.3 Phoneme Inputs with Special Tokens
- The <PHON_START> and <PHON_END> tokens delimit a kana sequence that encodes segmental pronunciation and accent, including apostrophe-marked nuclei and slash-marked phrase boundaries.Training converts a portion of input sentences to this notation, teaching the model to map phonemic sequences with accent information to speech tokens; untagged input remains ordinary text mode.Source: §3.3 · p. 3
- The paper records that its two-token target-span formulation appeared in August 2025, before an essentially identical PronSteering mechanism was presented in June 2026.The authors identify this correspondence to record precedence for their formulation.Source: §3.3 · p. 3
4 Experiments
The experiments adapt CosyVoice 2 using Japanese speech-transcription pairs and evaluate general synthesis and targeted pronunciation control in zero-shot settings. They define test sets, training settings, baselines, and objective and subjective measures.
4.1 Training and Validation Data
- 15,097 Japanese speech-transcription pairs from JSUT and JVS were used for adaptation, with 754 utterances held out for validation.One randomly selected noun per transcript was converted to phonemic form, with estimation errors modified by a native-speaker author.Source: §4.1 · p. 3
4.2 Evaluation Data
- The evaluation used two 50-sentence test sets: general easy-to-read Japanese and a stress test with one difficult-to-read word per sentence.Both sets produced 2,100 stimuli per system using zero-shot synthesis with the same 42 speakers; Test Set 2 varied difficult words from 2–6 morae and accent nuclei from first to last mora.Source: §4.2 · pp. 3–4
4.3 Fine-Tuning Configuration
- Fine-tuning used rank 16, alpha 64, dropout 0.05, AdamW at 1 × 10^-4, batch size 8, and 20,000 steps, completed in under one hour on an RTX 4090.Only LoRA layers and embeddings for the two special tokens were updated; training used next-token cross-entropy without auxiliary losses.Source: §4.3 · p. 4
4.4 Baseline Systems
- The official CosyVoice 2-0.5B checkpoint served as the baseline, with an additional kana-input baseline on Test Set 2 replacing the target difficult word with katakana.Source: §4.4 · p. 4
4.5 Evaluation Metrics
- Evaluation measured naturalness with UTMOSv2 and a native-Japanese-speaker MOS test, and speaker similarity with ERes2Net cosine similarity.UTMOSv2 scores below 2.0 were excluded, with less than 2% of samples dropped per test set and system.Source: §4.5 · p. 4
- Intelligibility was assessed by CER against input text using Whisper large-v3, while native Japanese listeners judged whether the target word’s pitch accent matched instructions.Samples with CER above 0.5 were omitted; accent ratings covered 126 stimuli per model with at least two ratings per sample.Source: §4.5 · pp. 4–5
5 Results and Discussion
UtterTune improves instructed Japanese pitch-accent correctness while retaining naturalness and speaker similarity; kana input alone improves segmental readings but leaves accent errors.
5.1 Naturalness and Speaker Similarity
- 3.44→3.88 crowd-sourced MOS on Test Set 1; a paired Wilcoxon test found a significant gain (p < 0.0001).The UTMOSv2 results were also reported to show no naturalness degradation.Source: §5.1 · p. 5
- On Test Set 2, UTMOSv2 differences were nonsignificant, with 99% confidence intervals below 0.06 for every system pair.The authors considered the three systems equivalent.Source: §5.1 · p. 5
- Across both test sets, ERes2Net cosine similarity stayed around 0.69 with no significant system differences.The authors interpret this as retaining the ability to maintain the prompt speaker’s voice characteristics.Source: §5.1 · p. 5
5.2 Effect of Phoneme Tags and Kana Transcription
- Kana transcription reduced CER from 0.1005 to 0.0595, but accent correctness remained low at 0.498 because pitch patterns were often wrong.Source: §5.2 · p. 5
- 0.899 accent correctness with CER 0.0626; the CER difference from the kana baseline was significant but negligible in effect size.The reported p-value was < 0.005 and Cliff’s delta was 0.115.Source: §5.2 · p. 5
- Outside the phoneme tags, accent correctness was 0.796 for the baseline and 0.829 for UtterTune, with no significant difference.The check used 50 sentences with one katakana word containing an accented nucleus and all 42 speakers; the authors conclude control did not leak beyond the tags.Source: §5.2 · p. 5
- The authors release the kana-plus-accent-tagged training data and difficult, katakana, and easy evaluation sets.The passage also points to open-source kana-output ASR and joyo-kanji reading benchmark resources as useful foundations for future Japanese pronunciation evaluation.Source: §5.2 · p. 5
- Even native Japanese speakers find Tokyo Japanese accent identification challenging when listening.Source: §5.2 · p. 5
6 Conclusion
UtterTune uses a small LoRA-based update to add phoneme-level pronunciation control to multilingual LLM-TTS, improving Japanese pronunciation while retaining naturalness and speaker similarity. Further work is proposed for other dialects and languages and for code switching.
6 Conclusion
- UtterTune raised MOS from 3.44 to 3.88, kept everyday-sentence CER below 7%, and raised stress-test accent correctness from 0.498 to 0.899 without degrading speaker similarity.Source: §6 · p. 6
- The method freezes the 0.5-B-parameter CosyVoice 2 backbone and updates rank-16 LoRA matrices plus two tokens, adding less than 0.5% parameters with a few GPU-hours of training.Source: §6 · p. 6
- The authors state that LoRA need not be applied during synthesis, so UtterTune in principle does not degrade performance in other languages.Source: §6 · p. 6
- Future work includes other Japanese dialects, other languages, and verification during code switching.Source: §6 · p. 6