PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech

Rikuto Kotoge, Yuichi Sasaki

arXiv:2510.05799v2cs.CLcs.AIcs.SD

TL;DR

Current TTS preference optimization commonly needs paired desirable and undesirable utterances, although such pairs are limited and utterance-level labels do not target pronunciation units. TKTO estimates token weights and optimizes token-level preferences without paired data; it reports higher Japanese pronunciation accuracy and lower CER.

  • Problem

    Preference optimization for TTS commonly requires paired desirable and undesirable outputs, which are limited, and utterance-level labels do not directly target pronunciation units.

    Source: §1 · p. 1

  • Method

    TKTO uses contrastive LLMs to estimate token-level weights and optimize token preferences without paired data or token-level annotations.

    Source: §1 · pp. 1–2

  • Results

    Japanese pronunciation accuracy improves by 39% and CER decreases by 54% with TKTO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TKTO is an off-policy approach compatible with human feedback, while extending it to an on-policy setting remains future work.

    Source: §5 · p. 5

    Evaluation covers Japanese and Chinese and one 0.5B model size; scaling and other architectures remain unexplored.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Aligning text-to-speech (TTS) system outputs with human feedback through preference optimization has been shown to effectively improve the robustness and naturalness of language model-based TTS models. Current approaches primarily require paired desirable and undesirable samples at the utterance level. However, such pairs are often limited in TTS output data, and utterance-level formulation prevents fine-grained token-level optimization needed for accurate pronunciation alignment. In this study, we propose TKTO that eliminates the need for paired data, enabling a more data-efficient training paradigm, and directly targets token-level units, automatically providing fine-grained alignment signals without token-level annotations. TKTO improves the challenging Japanese TTS accuracy by 39% and reduces CER by 54%, automatically assigning 12.8 times stronger reward to targeted tokens.

1 Introduction

Japanese TTS must resolve context-dependent readings, while existing preference alignment relies on paired, utterance-level samples. TKTO targets this gap with token-level optimization and reports improved Japanese pronunciation accuracy and CER.

1 Introduction

  • Japanese G2P may mispronounce ambiguous words because their reading and meaning can change with context.Prior G2P-free LLM TTS methods generate context-aware pronunciations directly from raw text (Wang et al., 2025a; Du et al., 2024).Source: §1 · p. 1, Figure 1 · p. 1
  • Earlier applications of Direct Preference Optimization (DPO) improved intelligibility, speaker similarity, and overall naturalness by enlarging the preference gap between pairwise samples.Source: §1 · p. 1
  • DPO-based methods require paired desirable and undesirable outputs, while utterance-level labels mismatch pronunciation’s character- or token-level units.The paper identifies these requirements as sources of data inefficiency and suboptimal alignment signals.Source: §1 · p. 1
  • TKTO estimates token-level weights with contrastive LLMs, then optimizes token preferences without paired data or token-level annotations.The framework is grounded in Kahneman-Tversky’s prospect theory.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • 39% higher Japanese pronunciation accuracy and 54% lower CER are reported for TKTO, which can use 6× more training data than DPO.TKTO also automatically assigns 12.8× stronger rewards to targeted tokens.Source: §1 · p. 2

2 LLM-based Text-to-Speech

The section formulates LLM-based TTS as conditional generation from text to speech tokens. The decoder’s generated token sequence is converted to audio, and the decoder is trained to learn token-level preferences from feedback.

2 LLM-based Text-to-Speech

  • The task is conditional generation from input text x to an output speech-token sequence y.The sequence is denoted y = (y1, . . . , yT).Source: §2 · p. 2
  • An LLM decoder autoregressively predicts acoustic tokens conditioned on text and previously generated tokens.The decoder is denoted πθ(y | x).Source: §2 · p. 2
  • A neural vocoder transforms the output token sequence into speech audio, while the decoder learns token-level preferences from feedback.The feedback may come from users or preference data.Source: §2 · p. 2

3 Proposed Method

TKTO learns token-level preferences from unpaired data through token-importance weighting and a token-level KTO objective. Contrastive models estimate the weights, which guide token values.

3 Proposed Method

  • TKTO uses a two-step approach: estimate how informative each token is for preference learning, then use those weights to guide its objective.The method targets token-level preferences from unpaired data.Source: §3 · p. 2

3.1 Targeted Token Weight Estimation

  • Token-importance weights use the log-ratio between π+ and π−; π− estimates token reward, while clamping bounds and µ control reward weighting by sample desirability.µ is positive for desirable samples and negative for undesirable samples.Source: §3.1 · p. 2
  • Two contrastive LLMs, π+ and π−, are trained on the same data with desirable and undesirable labels swapped, without sharing parameters.The construction uses KTO, which supports training on unpaired data.Source: §3.1 · p. 2

3.2 Token-level KTO

  • Each token’s reward is its log-ratio against a reference policy, with a fixed reference baseline estimated across a microbatch without gradient propagation.The token reward is denoted rθ,t(x, y), and the baseline z0,t.Source: §3.2 · p. 2
  • Table 1 compares models by accuracy, character error rate, and bad-case ratio, marking the best and second-best results.Source: Table 1 · p. 3
  • Each token value uses a logistic-shaped function, with β controlling curvature and λD and λU adjusting aversion weights; the TKTO loss weights values by wt.The loss is a weighted sum of token-level values.Source: §3.2 · p. 3

4 Experiments

TKTO is evaluated on ambiguous Japanese pronunciation and subjective speech quality, with analyses of its token-level behavior, data efficiency, and Chinese generalization. Across the Japanese evaluation, it improves pronunciation accuracy and other objective metrics, and subjective evaluations favor it over the base model and standard KTO.

4.1 Experimental Setup

  • Experiments use 5,000 ambiguous 辛い sentences, selecting pronunciation-correct and incorrect speech samples by CER from five samples per speaker.Comparisons include DPO, KTO, SFT, F5-TTS, and industry models; evaluations measure accuracy, CER, bad-case ratio, NMOS, and ABX preference.Source: §4.1 · pp. 3–4

4.2 Objective Evaluation Results

  • 0.668→0.958 accuracy; TKTO also achieves the highest Japanese accuracy and lowest CER and bad-case ratio among evaluated systems.The reported comparison says TKTO surpasses strong industry models and improves across all metrics relative to KTO.Source: §4.2 · p. 4
  • TKTO increases desirable-token log-likelihood during training, whereas SFT tends to increase undesirable-token log-likelihood.Figure 3 tracks the average log-likelihood of desirable and undesirable tokens.Source: §4.2 · p. 4, Figure 3 · p. 4
  • 0.22 versus 0.12 overall mean for desirable-token reward; undesirable tokens receive -1.54, indicating greater weights for targeted tokens.The analysis reports two reward peaks for undesirable samples when the contrastive models differ clearly.Source: §4.2 · p. 4, Figure 4 · p. 4
  • Target-character tokens receive notably higher weights than non-targeted tokens in the case study.Source: §4.2 · p. 4, Figure 5 · p. 4

4.3 Subjective Evaluation Results

  • Subjective evaluations favor TKTO over the base model and standard KTO on NMOS and ABX tests.Table 2 reports NMOS scores; Figure 6 presents ABX preference-test results.Source: §4.3 · p. 4, Table 2 · p. 5, Figure 6 · p. 5

4.4 Computational Cost and Data Efficiency

  • 9K unpaired samples versus 1.5K paired samples for DPO gives TKTO about 6× more usable data; two contrastive LLMs take approximately 10 minutes to train on 8×A100 GPUs.The added training overhead is described as negligible compared with TTS pre-training that typically takes weeks.Source: §4.4 · pp. 4–5

4.5 Additional Validation on Chinese

  • On Chinese polyphonic disambiguation, TKTO achieves the highest accuracy and lowest CER and bad-case ratio, showing generalization beyond Japanese.The evaluation uses context-dependent readings of 行; Table 3 presents the Chinese results.Source: §4.5 · p. 5, Table 3 · p. 5

5 Conclusion

TKTO removes the need for paired data and directly optimizes token-level units without token-level annotations. The authors describe it as an off-policy approach compatible with human feedback, while on-policy extension remains future work.

5 Conclusion

  • TKTO eliminates the need for paired data and directly targets token-level units to provide fine-grained alignment signals without token-level annotations.Source: §5 · p. 5
  • The framework is an off-policy approach compatible with human feedback.Source: §5 · p. 5
  • Extending TKTO to an on-policy setting remains a direction for future work.Source: §5 · p. 5

Limitations

The evaluation is limited to challenging Japanese and Chinese data and a single 0.5B model size; scaling and other architectures remain unexplored.

Limitations

  • Evaluation covers only challenging Japanese and Chinese data.Source: Limitations · p. 5
  • Only a 0.5B model is evaluated because larger versions of the backbone are unavailable and computational constraints prevent extensive scaling experiments.Source: Limitations · p. 5
  • Scaling behavior and extension to other model architectures remain unexplored.Source: Limitations · p. 5

Ethical Considerations

The section notes that the paper’s TTS models were used for research under controlled conditions, then describes KTO as a pair-free, binary-feedback objective that TKTO extends to token-level optimization.

Ethical Considerations

  • The models used in the paper were for research purposes under controlled conditions, despite potential concerns about unauthorized generation or misuse.Source: Ethical Considerations · p. 5

A Kahneman–Tversky Optimization

  • KTO maximizes a proxy of human utility using binary feedback about whether an output is desirable or undesirable, without requiring paired comparisons.Source: A Kahneman–Tversky Optimization · pp. 6–7
  • TKTO extends KTO from a single sequence-level utility to token-wise weights that estimate each token’s relative contribution to human utility.This preserves KTO’s prospect-theoretic inductive bias while enabling finer-grained credit assignment.Source: A Kahneman–Tversky Optimization · p. 7

B Implementation Details

The implementation builds on prior public code and pretrained checkpoints, with preference-optimization experiments run for one epoch; the paper also reports sensitivity and alternate-ASR evaluation tables.

B Implementation Details

  • The implementation used publicly available code from prior work released under research-permissive licenses, with hyperparameters and libraries following those studies.The paper also labels Table 4 as parameter sensitivity analysis and Table 5 as evaluation using the NVIDIA Parakeet Japanese ASR model.Source: B Implementation Details · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • Preference optimization experiments used one epoch, a 1e-6 learning rate, λD = λU = 1, and β = 0.10.The authors set µ = 1 for desirable tokens and µ = −1 for undesirable ones, following Ethayarajh et al. (2024).Source: B Implementation Details · p. 7
  • The base model and F5-TTS baseline were initialized from pretrained checkpoints and fine-tuned once; training took a few minutes on 8×A100 GPUs.Source: B Implementation Details · p. 7

C Parameter Sensitivity Analysis

TKTO’s performance is stable across moderate reward clamping ranges, while wide bounds can slightly degrade accuracy; the authors select (−2, 2).

C Parameter Sensitivity Analysis

  • The selected clamping range is (−2, 2); wide bounds such as (−3, 3) can slightly degrade accuracy, while moderate ranges maintain stable performance.The analysis identifies clamping as a control on reward-weight scaling and says reasonable constraints are beneficial for optimal results.Source: C Parameter Sensitivity Analysis · p. 7

D Robustness across ASR Backends

With the Parakeet Japanese ASR backend, TKTO has the lowest CER and Bad ratios for both speakers, consistent with the Whisper trend.

D Robustness across ASR Backends

Loading 2510.05799v2…