Source-linked AI summary
Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech
Shuhei Kato
TL;DR
Raw-text TTS can guess rare words, while Japanese listeners need both the reading and pitch accent to be right. This paper trains a frozen backbone on its own output to install both controls without recorded speech. Screened raters judged the tag correct on 0.80 to 0.93 of unseen difficult words across four backbones, though listeners preferred unedited renditions on three backbones for words needing no edit.
Problem
Released raw-text TTS lacks a user-writable reading-and-accent channel, while prior evaluated pitch-accent controls relied on annotated recordings.
Source: §1 · p. 1Method
The frozen backbone’s output for a plain carrier sentence teaches the same model to follow a tagged reading with pitch accent, using no recorded speech.
Source: §1 · p. 1, §2.2 · p. 2Results
Screened raters judged the tag correct in both reading and accent on 0.80 to 0.93 of difficult unseen words across four backbones, versus 0.38 to 0.60 for plain kana.
Source: §5 · p. 4Takeaways & Limitations
A small text inventory and self-generated supervision can install joint reading-and-accent control for words unseen during training across four backbones.
Source: §5 · p. 4On three backbones, listeners preferred the unedited rendition for words that needed no edit.
Source: §4.5 · p. 4, §5 · p. 4
Abstract
from arXiv · showhide
Text-to-speech that reads raw text has no lexicon: a rare word is read as guessed, and a native Japanese listener accepts a word only if its reading and pitch accent are both right. A known remedy installs a reading-and-accent channel into a released model, but it needs many recordings. This paper removes the recordings: the frozen backbone reads a sentence containing a common word it already says correctly, and that output serves as the teacher for the same sentence with the word replaced by an annotated reading with its pitch accent. Screened raters judged the tag right on 0.80 to 0.93 of unseen difficult words on four backbones spanning autoregressive, diffusion, and encoder-decoder synthesis; plain kana, which cannot express an accent, got 0.38 to 0.60. On words needing no edit, naturalness is non-inferior on one backbone; on the other three, listeners prefer the unedited rendition by 0.19 to 0.26.
1. INTRODUCTION
Raw-text TTS can guess rare words, and Japanese listeners need both the reading and pitch accent to be right. Prior accent-control approaches rely on annotated recordings; this paper installs control in a released model through self-distillation without recorded speech.
1. INTRODUCTION
- Rare words in end-to-end TTS are guessed, and a wrong Japanese pitch accent can make listeners search for a word that was never uttered.The paper frames the unresolved need as installing reading and accent control in a released model without the recording cost.Source: §1 · p. 1
- Other approaches require target-word audio, address only encoder–decoder backbones, alter a released model without adding accent, or cannot be added after release.GRAFT [4], SonoEdit [5], and FlowEdit [6] need a clip for each target word; SpeechSplice needs an encoder–decoder backbone and none addresses accent. Built-in input control [8] [9] cannot be added to an already released model, while ASR-ranked preference optimization [10] adds no user-written reading or accent channel.Source: §1 · p. 1
- Earlier evaluated pitch-accent control used annotated recordings: 7,596 pairs for Kurihara et al. and 15,097 for UtterTune [2], each on one architecture.Input-side controls include prosodic symbols and special tokens carrying a katakana reading with accent [2]; Sarashina2.2-TTS reports about 4,000 hours of supervised fine-tuning [3].Source: §1 · p. 1
- The paper proposes installing reading-and-accent control in a frozen backbone by self-distillation from text and a public front-end, with no recorded speech.It also describes registering a word once so later occurrences are corrected automatically, and explicitly makes no claim of language independence or novelty for low-rank adaptation or tag tokens [14].Source: §1 · p. 1
2. METHOD
The method replaces a target word with an accented reading tag and trains the frozen backbone on its own rendition of the same utterance. A text inventory, screened synthetic teachers, and a small adapter provide the training signal without recorded speech.
2.1. Control signal
- The target span is replaced by tags around its katakana reading; an apostrophe marks the pitch fall, a slash marks an accent-phrase boundary, and an unmarked reading is flat.The notation follows JEITA IT-4006, and only the tag spelling changes across backbones.Source: §2.1 · p. 1
2.2. Self-distilled teachers
- The frozen base’s output for a plain carrier sentence is the target for the same sentence with the word replaced by its reading and pitch accent.A take is kept only if it passes the degeneration guard and ASR detects the target reading; failed takes are redrawn up to five times, without best-of-n selection, error-rate threshold, or per-word cap.Source: §2.2 · p. 2
- Teachers above 0.22 seconds per character of the plain carrier are discarded; the same rate threshold is applied across backbones.Across 23,549 carriers, this rule leaves 12,870 for Sarashina, 13,751 for CosyVoice 2, 13,901 for Irodori, and 18,781 for T5Gemma.Source: §2.2 · p. 2
- Text-only backbones use the base’s own carrier rendition, while Irodori and CosyVoice 2 use the same 2,000 synthetic voices as reference latents or prompt clips.No recording enters the process.Source: §2.2 · p. 2
2.3. The text set, and why odaka shapes it
- The 471-word inventory uses up to 50 carriers per word, balances domains, mora-initial consonants, and accent types, and keeps training and evaluation vocabularies disjoint.Odaka words are over-sampled at 15%, with a particle immediately after each target so the accent fall is audible.Source: §2.3 · p. 2
2.4. Training
- The tagged student input is paired with the base’s plain-carrier output; the evaluated setup uses rank-16 LoRA and only tag-token embedding rows while keeping the synthesis stack frozen.Figure 1 measures the share of unseen difficult words judged correct in both reading and accent, comparing the tag with plain kana.Source: §2.4 · p. 2, Figure 1 · p. 2
3. DEPLOYMENT: REGISTER ONCE, APPLIED AUTOMATICALLY
A registry maps each surface form to an accented reading, and a router applies it at synthesis time. Once registered, a word is corrected in later sentences without per-sentence markup, retraining, or an exemplar.
3. DEPLOYMENT: REGISTER ONCE, APPLIED AUTOMATICALLY
- The registry maps a surface form to its accented reading, and the router finds registered surfaces at morpheme boundaries before applying the entry’s reading and pitch accent.The router wraps only matched spans and absorbs a following suffix when the front-end attaches one.Source: §3 · p. 2
- If no registered surface aligns with a morpheme boundary, the router falls back to the unedited base and records that the edit did not fire.Source: §3 · p. 2
- Registering a word once corrects every later occurrence in any sentence, without per-sentence markup, retraining, or an exemplar.Unlike per-word editors [5] [6], the deployed system needs no exemplar.Source: §3 · p. 2
4. EXPERIMENTS
The experiments test pronunciation control across four TTS backbones using unseen difficult words, screened listeners, and words already read correctly by the base. The tag significantly outperforms plain kana on reading-plus-accent judgments across all four backbones, while naturalness non-inferiority holds only for Sarashina.
4.1. Setup
- The setup tested autoregressive, diffusion, and encoder–decoder backbones, using a 319-item test split by word-reading connected component and 42 held-out voices.The test items came from a benchmark filtered for dictionary G2P misreadings; the evaluation plan and materials were finalized before teacher audio review.Source: §4.1 · pp. 2–3
4.2. Rejecting degenerate output, identical in every arm
- The rate rule retained 12,870–18,781 teachers across backbones, and the final guard yielded 4 Sarashina redraws, none exhausted, and 32 T5Gemma redraws.CosyVoice 2 and Irodori had no redraws; seven T5Gemma outputs remained inaudible and were scored as misses. The Sarashina rule was selected using the first 100 test items; for the other backbones it was fixed beforehand.Source: §4.2 · p. 3
4.3. Does control transfer to unseen words?
- Against no edit, the tag improved reading accuracy by +0.364 on Sarashina, +0.502 on CosyVoice 2, +0.282 on Irodori, and +0.357 on T5Gemma.The objective 319-item test included words absent from every training carrier. ASR ignores accent, unlike the listening evaluation, which scores reading and accent together.Source: §4.3 · p. 3
- The only recorded-data comparison was CosyVoice 2: without recordings or labels, the method trailed UtterTune by 0.072 on the same 319 items and voices.UtterTune was trained on 15,097 hand-corrected recorded pairs; the other three backbones had no comparable corpus or adapter.Source: §4.3 · p. 4
4.4. Listening evaluation
- The listening test used blind, randomized whole-sentence stimuli under held-out voices, with raters admitted through a 20-item test requiring at least 18 correct.The test covered four backbones and compared tag with kana on 50 difficult words; Sarashina also included no edit.Source: §4.4 · p. 4
- All four tag-versus-kana contrasts were significant for reading-and-accent judgments; on Sarashina, the tag also led no edit, but not significantly.Four admitted raters judged 50 difficult words per backbone; Fleiss’ κ was 0.89, with 13 tied stimuli excluded.Source: §4.4 · p. 4
4.5. Naturalness
- Naturalness non-inferiority was met only for Sarashina; on the other three backbones, listeners preferred unedited renditions when both versions gave the same reading.The easy set contained words the base already read correctly, and the registered non-inferiority margin was 0.15.Source: §4.5 · p. 4, Table 2 · p. 4
5. CONCLUSION
The paper adds reading-and-pitch-accent control to frozen TTS backbones through self-generated supervision, without recorded speech. It reports control of unseen words across four backbones, alongside a naturalness trade-off and limits in language and evaluation scope.
5. CONCLUSION
- 0.80 to 0.93 of difficult unseen words were judged right with the tag, compared with 0.38 to 0.60 for plain kana.The selected native listener judged reading and accent together.Source: §5 · p. 4
- Self-distillation on words the backbone already says correctly yields joint reading-and-accent control for unseen words across all four backbones, without recorded speech.A registry applies the control wherever a registered word appears.Source: §5 · p. 4
- On three backbones, listeners still preferred the unedited rendition for words that needed no edit.The study reports non-inferiority on one backbone; it was conducted in one language and used four raters.Source: §5 · p. 4
- The self-generated supervision uses the model’s own outputs, which must first be checked for additional lessons they might impart.Source: §5 · p. 4
Compliance with ethical standards
The study reports ethical safeguards for its listening studies, including informed consent, voluntary participation, and compensation. No institutional ethics committee review was sought.
Compliance with ethical standards
- No institutional ethics committee review was sought because the author had no committee to which to apply.Source: Compliance with ethical standards · p. 5
- The listening studies collected only non-invasive judgments on synthesized speech, with no audio, biometric, or personally identifiable data gathered.Source: Compliance with ethical standards · p. 5
- Participants received a written task, data-use, and compensation description, gave informed consent, and could stop at any time while still being paid.Every candidate took the same screening and was paid for it, regardless of outcome.Source: Compliance with ethical standards · p. 5