Source-linked AI summary
Training-Free Pronunciation Transcription via Text-Constrained Acoustic Rescoring
Hikaru Asano, Yotaro Kubo, So Kuroki
TL;DR
Pronunciation transcription matters for preparing TTS training data at scale, but G2P and S2P use only text or speech, while trained ST2P requires costly pronunciation annotations. This paper combines lexical candidates and frozen S2P acoustic scoring in a training-free inference pipeline. It reports lower error than evaluated baselines on Japanese and improvements across Spanish, French, and preliminary English.
Problem
G2P and S2P use only text or speech, while ST2P uses both but requires costly pronunciation-annotated training data for a task important to TTS preparation at scale.
Source: §1 · p. 1Method
The training-free pipeline generates text-constrained pronunciation candidates and uses frozen S2P models to select among them by greedy search over acoustic scores.
Source: §1 · p. 1, §3.2 · p. 2Results
The method achieves lower CER than evaluated baselines on three Japanese corpora with reference and ASR transcripts, and improves over four open MLLMs and text-only specialists in Spanish, French, and preliminary English.
Source: §1 · p. 1, §4.2 · p. 3Takeaways & Limitations
The pipeline combines lexical and acoustic clues at inference without pronunciation triplets or model training, and is reported as inexpensive while achieving low transcription error rates.
Source: §1 · p. 1Trained ST2P approaches require costly pronunciation-annotated training data, which limits their scalability.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Accurate and efficient pronunciation transcription is essential for preparing text-to-speech training data at scale. Existing approaches have different limitations: grapheme-to-pronunciation (G2P) and speech-to-pronunciation (S2P) methods each capture only partial information, using only text or only speech, while speech-and-text-to-pronunciation (ST2P) methods use both but require costly pronunciation-annotated data. To address this problem, we propose a training-free ST2P pipeline that integrates both lexical and acoustic information at inference time. Lexical resources and G2P tools generate text-constrained candidates, and a left-to-right greedy search selects the best one using whole-sequence negative log-likelihoods from frozen pretrained S2P models. On three Japanese corpora, our method reduces Character Error Rate (CER) from 0.60--1.40\% (text-only baseline) to 0.04--0.17\% with reference transcripts, and 0.64--1.58\% with ASR transcripts. It outperforms all baselines, including a trained ST2P model and commercial multimodal LLMs. Our greedy search method is 3--3.5$\times$ faster than beam search at similar CER, and the cascade is 2$\times$ faster than direct decoding ensuring the efficiency and accuracy. In Spanish, French, and preliminary English, it also surpasses four open multimodal LLMs and the best traditional methods.
1. INTRODUCTION
Pronunciation transcription supports applications such as TTS data preparation, where low cost and low error rates matter at scale. The paper proposes a training-free ST2P pipeline combining text-constrained candidates with acoustic scoring, and reports improvements across Japanese and other languages.
1. INTRODUCTION
- Pronunciation transcription supports TTS data preparation, language education, and study of low-resourced languages, making low cost and low error rates important at scale.The figure compares real-time factor with character error rate on JVS-dev.Source: §1 · p. 1, Figure 1 · p. 1
- G2P uses transcripts and S2P uses speech, but each discards the other clue; ST2P uses both yet requires costly pronunciation-annotated training data.These approaches differ in input, and the annotation requirement limits scalability.Source: §1 · p. 1
- The proposed training-free ST2P pipeline generates text-constrained pronunciation candidates and selects among them using frozen S2P acoustic scores at inference time.It requires neither audio–text–pronunciation triplets nor model training.Source: §1 · p. 1, Figure 2 · p. 2
- On three Japanese corpora, the proposed method achieves lower CER than evaluated baselines with reference and ASR transcripts, while the plot reports the lowest CER at low computational cost.The authors also report improvements on Spanish, French, and preliminary English, including over four open MLLMs and text-only specialists.Source: §1 · p. 1, Figure 1 · p. 1
2. TASK AND TRAINING-FREE SETTING
Given speech and its transcript, the task is to estimate the pronunciation actually spoken, rather than the canonical reading. The transcript may be reference text or ASR output, and the pronunciation alphabet depends on the language and available resources.
2. TASK AND TRAINING-FREE SETTING
- The target is the pronunciation sequence realized in the speech, not its canonical reading.The input is speech x together with transcript w.Source: §2 · p. 2
- The transcript can come from reference text or ASR output, with ASR enabling an automatic pipeline.Source: §2 · p. 2
- The pronunciation alphabet varies with language and available resources, such as kana for Japanese and IPA otherwise.Source: §2 · p. 2
3. METHOD
The method generates text-constrained pronunciation candidates and scores complete candidate sequences with frozen S2P models. It uses left-to-right greedy search, an optional two-model cascade, and a margin check to select readings.
3.1. Candidates and frozen acoustic scores
- Candidate readings from lexical resources are evaluated as whole pronunciation sequences under frozen S2P models, with an optional penalty favoring more reliable candidates.The acoustic score is the candidate’s negative log-likelihood under a frozen S2P model; the rendering maps pronunciations into the model’s alphabet and tokens.Source: §3.1 · p. 2
3.2. Greedy search and cascade
- The left-to-right greedy search approximates the exponentially growing joint assignment space by selecting and committing one candidate per span.At each span, candidates are scored as complete sequences with earlier choices fixed and later spans at their defaults.Source: §3.2 · p. 2
- A second frozen S2P model can be combined with the first, and the cascade reruns the search with fused scores only if the first pass changes a default reading.Otherwise, the first-pass result is retained.Source: §3.2 · p. 2
3.3. Margin check
- A length-scaled margin check reverts changed readings to their defaults when their score advantage does not meet the threshold per pronunciation symbol.Setting τ = 0 disables the check.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments evaluate the pipeline on Japanese corpora and across languages, measuring accuracy alongside inference cost. Results report improvements over evaluated baselines, including trained ST2P and open multimodal systems.
4.1. Japanese resources and protocol
- Japanese evaluation uses three corpora, 512 common utterances per corpus, and reference or Whisper large-v3 transcripts, with normalized-kana CER as its metric.The protocol includes text-only, audio-only, text-conditioned, released ST2P, and multimodal LLM baselines.Source: §4.1 · p. 3, Table 1 · p. 3
4.2. Japanese results
- The cascade outperforms all baselines on three Japanese corpora with reference and ASR transcripts, including a trained ST2P system requiring pronunciation-labeled speech.It also outperforms CTC-only and AR-only rescoring on all corpora.Source: §4.2 · pp. 3–4, Table 1 · p. 3
- Removing the gate or cascade raises CER by up to 0.022 or 0.016, while unrelated or silent audio raises CER to 12.4–16.2%.Beam search performs similarly to greedy search, consistent with mostly local span-level acoustic evidence.Source: §4.2 · p. 4, Table 2 · p. 4
4.3. Cost and accuracy (Fig. 1)
- Greedy search is 3–3.5× faster than beam search at comparable CER, and the cascade is twice as fast as kana-whisper decoding.The cascade runs kana-whisper only on selected spans and scores G2P candidates with a single forward pass rather than stepwise decoding.Source: §4.3 · p. 4
4.4. Multilingual evaluation
- With language-specific lexical resources and scorers, the pipeline improves on the specialist baseline in Spanish, French, and preliminary English, and both configurations outperform every open MLLM.English results are preliminary; PFER scores are not directly comparable across languages.Source: §4.4 · p. 4, Table 3 · p. 4
5. CONCLUSION
The paper concludes that its training-free ST2P pipeline is faster than Japanese baselines and works across Spanish, French, and English.
- The training-free ST2P pipeline, built from open models and lexical resources, is faster and outperforms all Japanese baselines, and also works for Spanish, French, and English.Source: §5 · p. 4