PaperlayerView exact revision on arXiv

Source-linked AI summary

Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis

Lianbo Liu, Shiao Zhu, Kai Washizaki, Reo Yoneyama, Haesung Jeon, Mengjie Zhao, Yusuke Fujita, Hao Shi, Nao Yoshida, Yuan Gao, Roman Koshkin, Yukiya Hono, Yui Sudo

arXiv:2606.25369v1cs.SDcs.AIcs.CLeess.AS

TL;DR

Japanese remains under-explored in LLM-based TTS, and its context-dependent kanji readings pose a language-specific challenge. Sarashina2.2-TTS combines scaled speech training and targeted augmentation with a kanji-reading benchmark and kana-based metric. It reports improved kanji-reading accuracy, competitive general-sentence pronunciation, and stable pronunciation across prompt languages.

  • Problem

    Most LLM-based TTS systems focus on English and Chinese, while Japanese-specific linguistic challenges remain under-addressed.

    Source: §1 · p. 1

  • Method

    Sarashina2.2-TTS combines large-scale speech training and targeted augmentation with a benchmark covering Joyo kanji readings and Kana-CER evaluation in kana space.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Sarashina2.2-TTS outperforms baselines across CER-based Joyo benchmark metrics, improves reading accuracy with targeted augmentation, and maintains stable pronunciation across prompt languages.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Kana-CER avoids orthographic variation in pronunciation evaluation, and kanji-level benchmark annotations support fine-grained error attribution.

    Source: §1 · p. 2

    Kana-ASR may mis-transcribe highly colloquial or stylistically extreme pronunciations that remain intelligible to humans.

    Source: §4.1.1 · p. 7

Abstract

from arXiv · show

While large language model (LLM)-based text-to-speech (TTS) systems have achieved high-quality speech synthesis, most existing systems focus on English and Chinese. Japanese, however, remains under-explored, and its unique linguistic challenges, such as widespread context-dependent kanji polyphony, have yet to be adequately tackled. Here we introduce Sarashina2.2-TTS (https://github.com/sbintuitions/sarashina2.2-tts), a Japanese-centric LLM-TTS system that tackles these challenges through a dual approach: data strategy and evaluation methodology. First, we scale training to approximately 361k hours of speech, incorporating a balanced mix of Japanese and English data. Furthermore, we design a targeted data augmentation pipeline covering all 2,136 Joyo (regular-use) kanji designated by Japan's Agency for Cultural Affairs to efficiently address kanji polyphony disambiguation. Second, we introduce the Joyo Kanji Yomi Benchmark (https://github.com/sbintuitions/JoyoKanji-Yomi-Benchmark), covering all 2,136 Joyo kanji and their 4,378 readings. Alongside this benchmark, we propose Kana-CER, a metric that compares synthesized speech against reference readings in the kana space, eliminating orthographic variations to directly measure pronunciation correctness. Experiments demonstrate that our targeted data augmentation significantly improves reading accuracy. Overall, Sarashina2.2-TTS achieves state-of-the-art kanji-level reading accuracy and matches top baselines on general sentence-level pronunciation, while delivering the highest speaker similarity in zero-shot Japanese speech synthesis. Furthermore, cross-lingual evaluation reveals that Sarashina2.2-TTS is the only system that maintains stable Japanese pronunciation regardless of the prompt language, confirming that our balanced training approach improves cross-lingual robustness.

1 Introduction

Japanese TTS must resolve context-dependent readings across thousands of polyphonic kanji, yet existing systems and evaluation methods leave gaps in language-specific coverage and diagnosis. Sarashina2.2-TTS combines scaled, targeted training with a kanji-focused benchmark and kana-based metric, and reports strong benchmark and cross-prompt results.

1 Introduction

  • Context-dependent readings make kanji polyphony a central Japanese TTS challenge; the 2,136 Joyo kanji collectively have 4,378 recognized readings.The kanji “生” alone has 12 readings, with the appropriate reading determined by context.Source: §1 · pp. 1–2
  • Existing multilingual systems often devote little training data to Japanese and lack strategies for covering infrequent kanji readings.The introduction identifies both limitations as part of the data-strategy gap.Source: §1 · p. 2
  • Standard CER and WER can count equivalent pronunciations written in different forms as errors, while existing benchmarks lack kanji-level annotations for identifying misreadings.Japanese consequently appears as an outlier in multilingual evaluations using these metrics.Source: §1 · p. 2
  • The Joyo Kanji Yomi Benchmark covers all 2,136 regular-use kanji and 4,378 readings, while Kana-CER compares pronunciations in kana to avoid orthographic variation.The benchmark includes 13,095 native-speaker-verified sentences with sentence- and kanji-level annotations.Source: §1 · p. 2
  • Sarashina2.2-TTS outperforms all baselines across CER-based metrics on the Joyo Kanji Yomi Benchmark, matches the best baseline’s pronunciation accuracy on JSUT, and maintains stable Japanese pronunciation across prompt languages.It also achieves the highest speaker similarity in zero-shot Japanese speech synthesis.Source: §1 · pp. 2–3

2 Sarashina2.2-TTS

Sarashina2.2-TTS separates linguistic token generation from acoustic reconstruction. Prompt text and reference speech condition semantic-token generation, while prompt-derived acoustic conditions support waveform synthesis and zero-shot voice cloning.

2 Sarashina2.2-TTS

  • The backbone LLM autoregressively generates semantic tokens from prompt text, target text, and semantic tokens extracted from reference speech; an acoustic stage converts them into a waveform.The acoustic stage uses a flow-matching decoder to produce a mel-spectrogram and a vocoder to convert it to audio.Source: §2 · p. 3, Figure 1 · p. 3
  • Prompt-derived conditions across both stages enable zero-shot voice cloning, with acoustic reconstruction conditioned on a speaker embedding and reference mel-spectrogram.The acoustic stage also conditions on prompt semantic tokens.Source: §2 · p. 3, Figure 1 · p. 3
  • The model uses S3Tokenizer V2, whose ASR-supervised single-codebook tokens primarily encode phonemic content and distinguish fine-grained kanji-reading differences.The tokenizer produces tokens at 25 Hz.Source: §2 · p. 3
  • Training uses a teacher-forced causal language-modeling objective, computing cross-entropy loss only over semantic-token positions.Training samples are naturally contiguous multi-sentence speech, and inference preserves the sequential prompt–target arrangement.Source: §2 · p. 4
  • For the open-source release, the decoder and vocoder are adopted from CosyVoice 2; this work trains only the backbone LLM and reports results using that configuration.The acoustic-stage components may differ in internal deployments.Source: §2 · p. 4

3 Data Strategy

Sarashina2.2-TTS pairs a large multilingual speech corpus with targeted synthetic data to address residual reading errors. PronSteering supplies controlled pronunciations for synthesizing and filtering examples covering Joyo kanji readings.

3.1.1 Data Composition

  • The training corpus contains approximately 361k hours: 194k Japanese hours across seven domains and 167k English hours, with the languages intentionally balanced for bilingual synthesis and cross-lingual prompting.Japanese-domain diversity spans formal narration and broadcast speech as well as spontaneous conversation and customer-service calls, broadening prosodic patterns and kanji usage.Source: §3.1.1 · pp. 4–5

3.1.2 Preprocessing

  • Audio preprocessing includes standardization, source separation, diarization, segmentation, enhancement, transcription, language identification, and quality filtering with DNSMOS and dual-ASR verification.Semantic tokens are described as resilient to minor artifacts from source separation and enhancement, allowing less stringent filtering and higher data retention.Source: §3.1.2 · p. 5

3.2 Targeted Synthetic Data Augmentation

  • Large-scale training leaves residual errors on rare proper nouns, place names, and other infrequent readings, motivating targeted synthetic speech–text augmentation.The augmentation pipeline is built on PronSteering, a text-side pronunciation-control mechanism.Source: §3.2 · p. 5

3.2.1 Pronunciation Steering (PronSteering)

  • PronSteering marks a target kanji with a delimited control fragment containing its kana reading and pitch-accent tags, isolating the instruction from surrounding text.The tags use “[” for pitch rise and “]” for pitch fall; the internal model uses PronSteering for synthesis, while the open-source release does not include the capability.Source: §3.2.1 · pp. 5–6

3.2.2 Augmentation Pipeline

  • The augmentation pipeline generates kanji-specific sentences, verifies them, adds dictionary-derived reading and prosody controls, synthesizes diverse prompts, and filters examples using Kana-ASR.Prosody tags are successfully annotated for 97.5% of samples; the remaining 2.5% are synthesized with reading control only.Source: §3.2.2 · p. 6
  • The pipeline covers all 2,136 Joyo kanji and 4,378 readings, producing approximately 280k synthetic samples; 95.1% are retained for Stage 2 training.The retained set amounts to about 320 hours of speech.Source: §3.2.2 · p. 6

4 Evaluation Framework

The evaluation framework combines kana-based scoring, which targets pronunciation rather than spelling, with a benchmark that localizes errors to individual Joyo kanji readings.

4.1 Kana-CER

  • Kana-CER compares kana-ASR transcripts with kana-form reference readings, avoiding orthographic variation when evaluating pronunciation.It is applied at sentence and target-kanji granularity, alongside standard CER for comparison with prior work.Source: §4.1 · p. 7

4.1.1 Kana-ASR Model

  • Kana-ASR is fine-tuned to transcribe kana and achieves 0.979% Kana-CER on real JSUT recordings.Because it lacks semantic compensation, it can mis-transcribe highly colloquial or stylistically extreme speech that humans understand; evaluation uses reading-style prompts to encourage standard pronunciation.Source: §4.1.1 · p. 7

4.2 Joyo Kanji Yomi Benchmark

  • The Joyo Kanji Yomi Benchmark addresses sentence-level metrics’ inability to identify which kanji reading was mispronounced.It covers the Joyo kanji and readings with contextually disambiguated sentences, kana annotations marking each target, and three retained samples per pair after native-speaker review.Source: §4.2 · pp. 7–8
  • Its protocol aligns Kana-ASR output to each marked target segment and computes Kana-CERkanji, with clipping to 1.0 limiting outlier influence.The benchmark also tallies errors per reading, enabling diagnosis of difficult readings to inform targeted data synthesis.Source: §4.2 · p. 8

5.1 Setup

The setup evaluates the Sarashina2.2-TTS backbone against Japanese-capable LLM-TTS baselines across three benchmarks, using a two-stage training procedure and repeated sampling.

5.1 Setup

  • The 24-layer Transformer decoder is initialized from Sarashina2.2-0.5B-Instruct, and only the backbone LLM is trained in this work.The open-source configuration adopts its decoder and vocoder from CosyVoice 2; reported experiments use that configuration.Source: §5.1 · p. 8
  • Stage 1 trains the backbone on the 361k-hour corpus, then Stage 2 fine-tunes on higher-quality natural data mixed with targeted synthetic data for long-tail readings.Stage 1 uses a constant 1×10−4 learning rate; Stage 2 decays it linearly from 1×10−4 to 1×10−6.Source: §5.1 · p. 8
  • Comparisons use T5Gemma-TTS, Qwen3-TTS, FishAudio S1-mini, and FireRedTTS-2, with five random seeds and mean ± standard deviation reported.The latter three baselines are primarily multilingual systems that support Japanese.Source: §5.1 · pp. 8–9
  • Evaluation spans the Joyo Kanji Yomi Benchmark, JSUT basic5000, and the Japanese subset of CV3-Eval.Joyo and JSUT use reading-style prompts; CV3-Eval measures zero-shot speaker similarity using cosine similarity between CAM++ speaker embeddings.Source: §5.1 · pp. 8–9

5.2 Kanji Reading Accuracy

Sarashina2.2-TTS Stage 2 leads the Joyo Kanji Yomi Benchmark on aggregate and per-reading accuracy, while the benchmark’s per-reading analyses show where systems struggle and how targeted synthesis improves results.

5.2.1 Overall Results

  • Kana-CERkanji is 7.83 and Kana-CER† kanji is 5.45 for Sarashina2.2-TTS Stage 2, outperforming all baselines on the Joyo Kanji Yomi Benchmark.The next-best system excluding Stage 1, T5Gemma-TTS, has Kana-CER† kanji of 8.55, 57% higher.Source: §5.2.1 · p. 9
  • Stage 2’s targeted data synthesis improves all metrics on both benchmarks compared with Stage 1.The paper attributes the improvement to its synthetic data augmentation strategy.Source: §5.2.1 · p. 9
  • Standard CER is consistently higher than Kana-CERsent across almost all systems and benchmarks, consistent with orthographic variation inflating standard CER.For Stage 2 on JSUT, the gap is 5.11 points (2.91 vs. 8.02).Source: §5.2.1 · p. 9

5.2.2 Per-Reading Analysis

  • Per-reading errors distinguish unlearned rare readings, context-dependent mistakes, and occasional Kana-ASR transcription noise.Examples include 15/15 errors for rare readings of 坂 and 六, context-sensitive errors for 正 and 生, and a 1/15 error for 駄 that may reflect ASR confusion.Source: §5.2.2 · pp. 9–10, Table 3 · p. 10
  • Common readings are near-universally correct, while rare readings vary widely across systems; Stage 2 has the lowest error counts in most representative cases.The per-reading analysis uses kanji-reading pairs and counts mispronounced trials out of 15.Source: §5.2.2 · p. 10, Table 4 · p. 10
  • Stage 2 leads across the per-reading error distribution, with 81.5% of readings fully correct versus 79.8% for Stage 1 and 79.7% for T5Gemma-TTS.Stage 1 ranks second at every threshold, and Stage 2 shifts the distribution toward lower error counts through targeted augmentation.Source: §5.2.2 · pp. 10–11, Figure 2 · p. 11

5.3 Cross-Prompt Evaluation

The evaluation tests pronunciation across diverse Japanese speaking styles and prompt languages. Sarashina2.2-TTS performs competitively across styles and is the only system reported to avoid degradation with non-Japanese prompts.

5.3 Cross-Prompt Evaluation

  • The cross-prompt evaluation uses 12 prompts spanning narration, news, podcasts, rakugo, horse-race commentary, and English accents, with standard CER chosen because stylistic diversity can destabilize Kana-ASR.The test set is JSUT, and CER is computed with Whisper large-v3-turbo.Source: §5.3 · p. 11

5.3.1 Cross-style Robustness

  • Across nine Japanese prompts, Stage 2 has better CER metrics than most baselines but slightly lags FishAudio S1-mini in both mean and standard deviation.The CER evaluation uses Whisper large-v3-turbo.Source: §5.3.1 · p. 11, Table 5 · p. 12
  • The authors hypothesize that expressive prosody cloning may increase Whisper transcription difficulty and inflate apparent CER, rather than reflect a true pronunciation difference.Horse-race commentary is given as an example of an acoustically challenging style.Source: §5.3.1 · p. 11

5.3.2 Cross-lingual Robustness

  • The cross-lingual comparison measures CER for Japanese and non-Japanese prompt groups and defines degradation as their relative CER difference.The calculation is (non-Japanese − Japanese) / Japanese × 100%.Source: Table 6 · p. 12, §5.3.2 · p. 12
  • Sarashina2.2-TTS is the only system without cross-lingual CER degradation (−0.2%), while Qwen3-TTS, FireRedTTS-2, and FishAudio S1-mini degrade by +328%, +168%, and +113%.T5Gemma-TTS shows +18% degradation when prompted with non-Japanese speech.Source: §5.3.2 · p. 12
  • The authors hypothesize that the cross-lingual difference relates to training-language balance: Japanese is 53.7% of Sarashina2.2-TTS training hours, unlike English-dominant baselines.They propose that non-Japanese-dominated training may make prompt acoustics more likely to bias target pronunciation decisions.Source: §5.3.2 · p. 12

5.4 Speaker Similarity

On CV3-Ja zero-shot evaluation, Sarashina2.2-TTS has the highest speaker similarity scores across both stages.

5.4 Speaker Similarity

  • Stage 1 scores 75.64 and Stage 2 scores 74.75 on SIM, the highest scores across systems on CV3-Ja zero-shot evaluation.Both stages substantially outperform all baselines.Source: §5.4 · p. 12
  • Pronunciation accuracy and speaker similarity do not correlate straightforwardly among the baselines.Qwen3-TTS ranks second in SIM at 69.86 despite poor pronunciation accuracy, while T5Gemma-TTS has strong pronunciation but SIM of 50.59.Source: §5.4 · p. 12
  • Qwen3-TTS scores 69.86 SIM despite poor pronunciation accuracy, whereas T5Gemma-TTS has strong pronunciation but scores 50.59 SIM.Source: §5.4 · p. 12

5.5 Speech Quality

Automatic MOS predictors assess whether pronunciation-focused improvements come at the cost of speech quality. Sarashina2.2-TTS scores highest on two predictors, and the two stages perform comparably across quality metrics.

5.5 Speech Quality

  • Sarashina2.2-TTS achieves the highest scores on UTMOS v2 and DNSMOS while remaining competitive on the other reported quality metrics.The evaluation uses automatic MOS predictors on the CV3-Ja subset.Source: §5.5 · pp. 12–13, Table 8 · p. 13
  • Stage 1 and Stage 2 perform comparably across quality metrics, with the paper reporting that synthetic data augmentation improves pronunciation accuracy without compromising perceived speech quality.Source: §5.5 · p. 13
  • All synthesized systems score above the reference recordings, a pattern attributed to automatic MOS predictors favoring clean, consistent TTS output over real-world variability.Source: §5.5 · p. 13

6 Conclusion

Sarashina2.2-TTS combines large-scale speech training with targeted kanji-reading augmentation and kana-based evaluation. The experiments report stronger kanji-level accuracy, stable pronunciation across prompt languages, and open-source resources for further research.

6 Conclusion

  • Stage 2’s synthetic data improves kanji reading accuracy on the Joyo Kanji Yomi Benchmark and JSUT, achieving the best kanji-level accuracy among evaluated systems.The system also outperforms all baselines across CER-based metrics on the Joyo Kanji Yomi Benchmark while maintaining highly competitive general-sentence pronunciation accuracy.Source: §6 · p. 13
  • Sarashina2.2-TTS maintains stable pronunciation regardless of prompt language, while most evaluated multilingual systems degrade under non-Japanese prompts.The system is trained on 361k hours of multi-domain speech and applies targeted augmentation covering all regular-use kanji readings.Source: §6 · p. 13
  • The consistent gap between standard CER and Kana-CER across systems indicates that orthographic variation inflates conventional Japanese TTS evaluation metrics.Kana-CER and the Joyo Kanji Yomi Benchmark support kana-based evaluation and systematic kanji-level error attribution.Source: §6 · p. 13
  • The authors open-source the model weights, Joyo Kanji Yomi Benchmark, Kana-ASR model, and evaluation scripts to facilitate future Japanese speech-synthesis research.Source: §6 · p. 13
Loading 2606.25369v1…