Source-linked AI summary
GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech
Antonis Asonitis, Francesco Verdini, Aref Farhadipour, Vijeta Avijeet, Pierre-Edouard Honnet, Marzieh Razavi, Juan Pablo Zuluaga Gomez
TL;DR
Text can underdetermine pronunciation, and phoneme-conditioned systems do not provide a direct acoustic handle for controlling a chosen word. GRAFT binds a spoken example of a chosen word, encoded with the model’s own codec, into the prompt and uses voice-conversion training to separate hint-speaker identity from pronunciation. D-PER is reduced by 22–39% over the text-only base across five languages with speaker similarity and naturalness preserved; across 30 difficult words and 600 comparisons, GRAFT led the strongest baseline by 150 Bradley-Terry points.
Problem
Text can underdetermine pronunciation, and phoneme-conditioned systems do not provide a direct acoustic handle for controlling a chosen word.
Source: §I · p. 1Method
GRAFT binds a spoken example of a chosen word, encoded with the model’s own codec, into the prompt and uses voice-conversion training to separate hint-speaker identity from pronunciation.
Source: §I · pp. 1–2, B. Per-word audio binding · p. 3Results
D-PER is reduced by 22–39% over the text-only base across five languages with speaker similarity and naturalness preserved; across 30 difficult words and 600 comparisons, GRAFT led the strongest baseline by 150 Bradley-Terry points.
Source: §VI · p. 6, §VII · p. 7Takeaways & Limitations
A non-expert can prescribe a word’s pronunciation by saying it, without phonetic notation or a lexicon entry.
Source: §I · p. 1, §VII · p. 6GRAFT is shown on one 0.6B base model and single-word hints; tonal-language transfer, multi-word spans, and scaling remain future work.
Source: §VI · p. 6
Abstract
from arXiv · showhide
We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling. Existing systems reach high intelligibility and naturalness but inherit the ambiguity of text and mispronounce rare proper nouns, loanwords and technical terms. Even phoneme-conditioned models offer no direct acoustic handle for per-word pronunciation. GRAFT controls the pronunciation of a chosen word from a short spoken sample of it, encoded with the model's own speech tokenizer and bound to the word's position in the prompt. Voice conversion during training-data construction disentangles the hint speaker from the target speaker, so the hint may come from any voice while the output stays in the target voice. In a blind English listening study, human raters rank GRAFT first by a clear margin, judging its rendering of the difficult word closest to a reference recording of that word. On a five-language objective benchmark, GRAFT reduces target-word phoneme error rate by 22-39% over the identical text-only backbone and outperforms competitive open-source zero-shot systems, both phoneme- and text-conditioned, on target-word pronunciation, while preserving speaker similarity and naturalness.
I. INTRODUCTION
Text-based speech systems can mispronounce rare or ambiguous words, while phoneme-based control still lacks a direct acoustic handle for a chosen word. GRAFT uses a spoken word sample to guide pronunciation and voice-conversion training to keep the hint speaker separate from the target voice.
I. INTRODUCTION
- Text underdetermines valid pronunciations, and its prior often fails on proper nouns, brand names, loanwords, and technical terms.Phoneme-based systems offer some control, but fixed lexicons assume a canonical pronunciation and generalize poorly to unseen words.Source: §I · p. 1
- GRAFT binds a short spoken example to the chosen word, using the model’s speech tokens to preserve pronunciation details such as stress and tone.It needs neither phonetic notation nor a lexicon entry.Source: §I · p. 1
- GRAFT adds per-word pronunciation control to neural codec language-model TTS without additional parameters or architectural changes, and is demonstrated across five languages.The paper also introduces a multilingual difficult-word benchmark and releases checkpoints, code, and evaluation tooling.Source: §I · p. 2
- Voice-converted training data teaches GRAFT to transfer the hint’s pronunciation while ignoring its speaker identity, so the output can use the target voice.The hint may come from any speaker.Source: §I · pp. 1–2
II. RELATED WORK
Neural codec language-model TTS generates speech by predicting discrete codec tokens, usually extending a pretrained text-only LLM. GRAFT adds a per-word pronunciation hint while retaining that architecture.
A. Neural codec language model TTS
- Neural codec language-model TTS casts synthesis as next-token prediction over discrete codec tokens, typically extending a pretrained text-only LLM with codec indices.GRAFT retains this architecture and adds a per-word audio conditioning slot, encoded with the existing codec tokenizer, with no new parameters.Source: A. Neural codec language model TTS · p. 2
- WESCON similarly splices an acoustic prompt at a chosen word, but uses it to control emotional expression rather than pronunciation.GRAFT adds voice-conversion training so the pronunciation hint may come from a speaker different from the target.Source: A. Neural codec language model TTS · p. 2
B. Phoneme-conditioned TTS
The related work contrasts symbolic phoneme control with GRAFT’s use of spoken examples, and describes alignment and voice conversion as supporting tools in its data pipeline.
B. Phoneme-conditioned TTS
- Phoneme-conditioned TTS inserts phonemic symbols into the prompt, but fixed phonemic vocabularies restrict coverage; GRAFT instead conditions on an instance of the desired pronunciation.Per-word lexicon fixes and weight edits still require a written phonetic specification.Source: B. Phoneme-conditioned TTS · p. 2
C. Forced alignment
- GRAFT uses the Qwen3-TTS aligner only during data preparation to cut per-word reference clips; isolated inputs make alignment unnecessary at inference.Forced alignment methods locate word boundaries from transcripts, either with a dedicated model or decoder attention-derived timing.Source: C. Forced alignment · p. 2
D. Voice conversion
- GRAFT uses voice conversion to create hints in a different speaker’s voice, teaching it to retain word pronunciation and prosody while ignoring hint-speaker identity.Here voice conversion serves training-data construction rather than being the synthesis goal.Source: D. Voice conversion · p. 2
III. METHOD
GRAFT uses the base neural codec language model and binds spoken word clips into the text prompt in place of the corresponding word tokens. The model then generates the remaining codec frames autoregressively.
A. Preliminaries
- The base system autoregressively predicts N-codebook codec tokens from a serialized text prompt and uses Qwen3-TTS-12Hz-0.6B-Base with 16 codebooks.Its codec encoder maps 24 kHz waveforms to tokens, and its decoder maps tokens back to audio.Source: A. Preliminaries · p. 2
B. Per-word audio binding
- GRAFT conditions on target text, a target-voice reference clip, and isolated recordings paired with the words whose pronunciations should be grafted.The reference and word clips are encoded with the base model’s codec.Source: B. Per-word audio binding · p. 3
- Each grafted word is replaced in the prompt by its audio-codec slot, while ungrafted words retain their text tokens.The prompt also contains a separate style slot for the target-speaker reference.Source: B. Per-word audio binding · p. 3
- Codec embeddings for the grafted audio condition the talker transformer as it generates the remaining codec frames autoregressively.Embeddings are summed across the 16 residual codebooks.Source: B. Per-word audio binding · p. 3
C. Training data construction
GRAFT training pairs voice-converted per-word hints with the original utterance as the prediction target, encouraging pronunciation transfer without copying the hint speaker. Selected hints are varied to better match isolated-word inference.
C. Training data construction
- Words are eligible for grafting when they last at least 0.04 seconds and contain at least four characters; each eligible word is selected independently with probability 0.5.Source: C. Training data construction · p. 3
- Voice-converted word clips are aligned, codec-encoded as per-word grafts, and inserted in place of text while the original utterance remains the prediction target.The converted clip is input only, so output quality is not limited by the voice-conversion model.Source: Figure 2 · p. 3, C. Training data construction · p. 3
- The no-VC ablation reaches the lowest D-PER only by copying hint tokens, which collapses speaker similarity and is described as a degenerate solution.Source: Table II · p. 3
- Grafts may come from another utterance, breaking the coarticulation and prosodic consistency of same-utterance clips to match isolated-word prompts at inference.The training-data caption illustrates a converted word from a different voice replacing its text while the model predicts the original utterance.Source: C. Training data construction · pp. 3–4, Figure 2 · p. 3
- Hint augmentation perturbs decoded clips before re-encoding, varying noise, level, pitch, speed, filtering, compression, reverberation, and silence to broaden recording conditions.Variable lead and trailing silence targets loosely cut word boundaries, while recording perturbations cover noisy-room and consumer-microphone conditions.Source: C. Training data construction · p. 4
D. Training objective
GRAFT fine-tunes the base model with the standard next-token objective on talker codec tokens and an auxiliary sub-talker loss.
D. Training objective
- Training uses talker-token cross-entropy plus an auxiliary sub-talker loss weighted λsub = 0.3, with a 48-hour run on one 96 GB GPU.The full data mix uses effective batch size 192, AdamW, learning rate 5×10^-6, weight decay 0.01, cosine scheduling, and 0.02 warmup; ablations keep the schedule and data count identical.Source: D. Training objective · p. 4
E. Inference
At inference, GRAFT takes the target text, a voice reference, and already isolated word-hint audio, so forced alignment is unnecessary.
E. Inference
- Inference uses target text, a style reference, and isolated word-hint pairs; each hint is VAD-trimmed before encoding to better match training.The model then uses standard autoregressive sampling.Source: E. Inference · p. 4
IV. EXPERIMENTAL SETUP
The experiments evaluate difficult-word pronunciation across five languages and compare GRAFT with text-only, phoneme-input, and other zero-shot systems. The setup also includes ablations of voice conversion and hint augmentation.
A. Difficult-word benchmark
- The benchmark uses 2,500 rare-word recordings across English, German, French, Spanish, and Italian, placed in carrier phrases with clean isolated recordings as hints.It reports target-word D-PER and hardest-10% D-PER alongside WER, UTMOS, and speaker similarity; each phrase is synthesized once per system.Source: A. Difficult-word benchmark · pp. 4–5
B. Compared systems
- Comparisons include a text-only Qwen3-TTS baseline, four phoneme-input systems given hint-audio transcriptions, and three text-input systems given only the spelled carrier.All systems receive the same carrier and speaker reference; systems designed for per-word correction require written phonetic specifications, which are absent for the rare words.Source: B. Compared systems · p. 5
- Two ablations isolate training-data choices: no-VC removes voice conversion, cross-utterance swapping, and augmentation, while no-aug removes hint perturbations and silence padding.Source: B. Compared systems · p. 5
C. Metrics
GRAFT is evaluated on target-word pronunciation using objective and human measures, with voice-conversion ablations testing whether it transfers pronunciation without copying the hint speaker. It improves phonetic faithfulness over the text-only base while largely preserving naturalness.
C. Metrics
- The evaluation reports WER, D-PER, UTMOS, Whisper similarity, speaker preservation, and blind pairwise Bradley-Terry scores.D-PER compares phonemized hint and generated target-word audio; speaker preservation compares the output with the cloned reference.Source: C. Metrics · p. 5
- GRAFT leads the blind English listening study by 150 Bradley-Terry points, with its confidence-interval lower bound above every baseline point estimate.Twenty listeners judged 30 difficult words across 600 comparisons against clean recordings of the target words.Source: Figure 3 · p. 5
A. Difficult-word evaluation
- D-PER falls significantly versus the base in every language, and its English spread roughly halves from ±0.42 to ±0.16.The no-VC ablation reaches D-PER 0.16 only by copying the hint, with speaker similarity collapsing to 0.20.Source: A. Difficult-word evaluation · p. 5
- GRAFT leads the base on UTMOS in four of five languages, while sentence-level WER is concentrated on the target word.Removing the target word drops English WER from 0.09 to 0.03 and the five-language average from 0.09 to 0.05; the carrier remains over 95% correct.Source: A. Difficult-word evaluation · p. 6
- The no-VC ablation matches the hint speaker more than the target speaker (0.67 vs. 0.10), while voice-conversion training reverses this (0.47 vs. 0.12).This comparison tests whether the grafted word copies the hint waveform or retains the cloned target identity.Source: A. Difficult-word evaluation · p. 6
B. Multilingual evaluation
Across five languages, GRAFT reduces target-word D-PER while preserving naturalness and speaker similarity at a small WER cost. The discussion reports strong human preference, multi-hint trade-offs, and limits on the tested hint and model settings.
B. Multilingual evaluation
- GRAFT cuts target-word D-PER by 22–39% in every language, raises hint similarity, and preserves naturalness and speaker similarity at a small WER cost.The interface uses acoustic tokens rather than language-specific machinery.Source: B. Multilingual evaluation · p. 6
VI. DISCUSSION
- GRAFT leads the human study by 150 Bradley-Terry points, and its confidence-interval lower bound exceeds every other system’s point estimate.The comparison spans 30 difficult words and 600 judgments; raters assess closeness to the supplied pronunciation, not canonical correctness.Source: §VI · p. 6
- Neither audio-derived phonemes nor human-curated CMUDICT pronunciations close the gap to GRAFT on 225 English CMUDICT-covered words.This comparison rules out transcription error as the cause of the baseline deficit on those covered words.Source: Table V · p. 6
- A spoken example lets a non-expert prescribe a pronunciation for rare names, loanwords, and technical terms missing from lexicons while retaining the chosen target voice.The output re-renders the pronunciation rather than echoing the hint recording.Source: §VI · p. 6
- Several hints can be grafted into one carrier: across up to five grafted words, D-PER rises gently from 0.28 to 0.38 while WER rises from 0.05 to over 0.20.Naturalness is unchanged; the reported cost of additional grafts falls on fluency rather than pronunciation.Source: §VI · p. 6
- Reliability depends on hints matching the trimmed, neutral training distribution; the evaluation covers single-word hints on one 0.6B model and non-tonal languages.Multi-word spans, tonal-language transfer, and scaling remain future work.Source: §VI · p. 6
VII. CONCLUSION
GRAFT adds per-word spoken-audio conditioning to neural codec TTS, using voice-conversion training so a hint from any speaker guides pronunciation in the target voice. It improves target-word pronunciation across five languages and ranks first in a blind English listening study.
VII. CONCLUSION
- A short spoken example guides a chosen word’s pronunciation, while voice-conversion training renders it in the target voice even when the hint comes from another speaker.GRAFT uses the base model’s own tokenizer and adds no parameters or architectural changes.Source: §VII · p. 6
- The spoken-example interface lets non-experts prescribe pronunciations for rare proper nouns, loanwords, and technical terms that text and phoneme front-ends cannot supply.The interface adds only a few extra tokens at inference.Source: §VII · p. 6
- In a blind English listening study, human raters ranked GRAFT first.The study compared how systems rendered the difficult word against a reference recording.Source: §VII · p. 6
- 22–39%: GRAFT reduced target-word D-PER versus the text-only base across five languages, while preserving speaker similarity and naturalness.The result is reported for the five-language objective benchmark.Source: §VII · p. 7