PaperlayerView exact revision on arXiv

Source-linked AI summary

Phoneme-Guided Initialization for LLM-based Speech Recognition

Ryo Magoshi, Shinsuke Sakai, Tatsuya Kawahara

arXiv:2610.08994v1eess.AScs.CLcs.SD

TL;DR

Paired speech-text data needed to align audio and language representations is scarce in many settings. Phoneme-guided initialization pre-trains the audio encoder on S2P and the LLM on P2G before end-to-end ASR fine-tuning. It reduces CER on Japanese and Chinese and outperforms CTC and cascaded baselines there, while gains on Tatar and Urdu are limited.

  • Problem

    Continuous speech LLMs usually need substantial paired speech-transcription data for cross-modal alignment, although such data is scarce in many conditions.

    Source: §I · p. 1

  • Method

    Phoneme-guided initialization pre-trains the audio encoder on S2P and the LLM on text-derived P2G sequences, then connects them for end-to-end target-task fine-tuning.

    Source: §I · p. 1

  • Results

    On Japanese and Chinese, CER fell 27–72% relative to the end-to-end baseline and the method outperformed CTC and cascaded S2P-P2G; gains on Tatar and Urdu were limited but not negative.

    Source: §V · p. 6

  • Takeaways & Limitations

    P2G initialization can use text-only data to improve LLM-based ASR, complementing S2P initialization by aligning both components toward phoneme-level representations before end-to-end training.

    Source: §I · p. 1

    On the low-resource languages Tatar and Urdu, Base+P2G showed no improvement on Tatar and only a modest improvement on Urdu.

    Source: D. Results · p. 5

Abstract

from arXiv · show

Speech large language models (speech LLMs) perform well on automatic speech recognition (ASR) when sufficient paired speech-text data is available, but their performance degrades in low-resource settings. A cascaded pipeline that performs speech-to-phoneme (S2P) conversion followed by phoneme-to-grapheme (P2G) conversion has been shown to outperform end-to-end speech LLMs in this regime, suggesting that phoneme-mediated processing is beneficial when paired data is scarce. We propose \textit{phoneme-guided initialization}, a simple method that uses this insight within an end-to-end framework: we pre-train the audio encoder on S2P and the LLM on P2G tasks, then connect them and fine-tune the full model end-to-end on the target ASR task. Experiments on Japanese (CSJ), Chinese (AISHELL-1), and two low-resource languages from Common Voice 25.0 (Tatar and Urdu) show that our method matches or outperforms both the cascaded S2P-P2G baseline and the end-to-end model without P2G initialization.

I. INTRODUCTION

Continuous speech LLMs require substantial paired speech-transcription data to align audio and language representations, a constraint where paired data is scarce. Phoneme-guided initialization trains the encoder on speech-to-phoneme and the LLM on phoneme-to-grapheme tasks before end-to-end fine-tuning, improving ASR particularly in low-resource, homophone-rich settings.

I. INTRODUCTION

  • Speech LLMs support ASR and other spoken-language tasks, and this work focuses on the continuous approach to feeding audio into an LLM.Continuous systems pass pretrained audio-encoder representations through a learnable projector into the LLM embedding layer.Source: §I · p. 1
  • Continuous audio input is mapped into the LLM’s embedding space, but aligning acoustic and semantic representations usually requires substantial paired speech-transcription data.Paired speech-text data is scarce in many conditions, while text-only data is more abundant and easier to obtain.Source: §I · p. 1
  • Phoneme-guided initialization trains the audio encoder on speech-to-phoneme and the LLM on phoneme-to-grapheme tasks before connecting them for end-to-end fine-tuning.The encoder’s S2P task is simpler than direct grapheme prediction and requires relatively little paired data; P2G uses phoneme sequences generated from text by G2P tools.Source: §I · p. 1
  • The method outperforms conventional approaches particularly in low-resource settings in homophone-rich languages.The authors attribute the contribution to text-only target-language or domain data improving ASR through P2G initialization that complements S2P initialization.Source: §I · p. 1

II. RELATED WORK

Prior work motivates phoneme-mediated recognition as a way to use sound units that preserve meaning distinctions while simplifying prediction. Earlier LLM approaches explored sequential or joint phoneme-and-grapheme output, while cascaded S2P-P2G systems performed well in low-resource settings.

A. Phonemes in Speech Recognition

  • Phonemes are the minimal sound units that speakers can distinguish, and phonemic transcription preserves meaning distinctions while abstracting away other speech information.Phoneme representations can be closer to linguistic information than raw speech; phonemes are typically fewer than graphemes, especially in Chinese and Japanese.Source: A. Phonemes in Speech Recognition · p. 1
  • Phonemic information has also been explored for grapheme prediction in end-to-end systems [10], –.Source: A. Phonemes in Speech Recognition · p. 2

B. Phoneme-Mediated LLM-based ASR

  • Jie and Cheng used pinyin for Chinese LLM-based ASR, with sequential phoneme-then-character output or interleaved phoneme and character output.Both approaches improved over grapheme-only prediction, but require training on both sequences; iterative output is limited to languages with one-to-one grapheme-phoneme correspondence.Source: B. Phoneme-Mediated LLM-based ASR · p. 2
  • Poncelet and Van hamme [19] reported gains from sequential phoneme-first prediction in English and Dutch, with joint phoneme-first and S2G training further improving performance, particularly with high-quality annotations in low-resource settings.Their jointly trained model could be decoded in grapheme-only or phoneme-first mode.Source: B. Phoneme-Mediated LLM-based ASR · p. 2
  • Li et al. [10] found that cascaded CTC-based S2P followed by LLM P2G outperformed end-to-end LLM ASR in low-resource settings, and that S2P encoder initialization reduced modality mismatch.They did not test additionally training the LLM on P2G, which the present work found effective for further modality alignment.Source: B. Phoneme-Mediated LLM-based ASR · p. 2

C. Differences from Prior Work

Unlike prior methods that change the LLM’s output format at inference, this approach uses phoneme-related tasks only for initialization. The ASR model then projects encoded speech into the LLM and trains grapheme prediction.

C. Differences from Prior Work

  • Phoneme-related tasks are used only for initialization: the encoder is pretrained on S2P and the LLM on P2G before projector connection and end-to-end fine-tuning.The model can use either grapheme-only or phoneme-first output.Source: C. Differences from Prior Work · p. 2

A. LLM-based ASR

  • The ASR stack comprises an audio encoder, a projector, and an LLM; the encoder converts the input waveform into frame-level representations.The encoder output has T frames and dimension denc.Source: A. LLM-based ASR · p. 2
  • The projector maps encoder representations into the LLM embedding space while downsampling temporal resolution by a factor of r.Its output dimension is dllm.Source: A. LLM-based ASR · p. 2
  • Projected audio representations are prepended to text token embeddings, and training minimizes negative log-likelihood for the target grapheme sequence.The model’s parameters are denoted by θ.Source: A. LLM-based ASR · p. 2
  • At inference, beam search approximately maximizes sequence log-likelihood; beam size and other decoding settings are specified in the training configuration.Source: A. LLM-based ASR · p. 2
  • Figure 2 depicts model configurations; the LLMs in (a), (c), and (d) use P2G initialization, while (a) freezes all components and runs as a cascade.Source: Figure 2 · p. 3
  • Table I gives language data configurations; its sentence counts are P2G phoneme-grapheme pairs, and Japanese requires no external G2P tool.Source: Table I · p. 3

B. Phoneme-First Prediction

Phoneme-first prediction trains the LLM to generate phonemes before graphemes, with both conditioned on audio. A joint variant mixes this training with standard speech-to-grapheme examples.

B. Phoneme-First Prediction

  • Phoneme-first prediction trains the LLM to generate a phoneme sequence followed by a grapheme sequence, both conditioned on audio representations.Source: B. Phoneme-First Prediction · p. 3
  • At inference, grapheme output is conditioned on both the audio representation and predicted phonemes, then decoded by beam search.Source: B. Phoneme-First Prediction · p. 3
  • Joint training mixes phoneme-first and standard S2G examples, using a text prompt to specify the expected output format.This lets the model learn from both training signals simultaneously.Source: B. Phoneme-First Prediction · p. 3

C. Proposed Method: Phoneme-Guided Initialization

Phoneme-guided initialization pre-trains the audio encoder and LLM on complementary phoneme tasks, then connects and fine-tunes them end-to-end for ASR.

Stage 1: Component Pre-training

  • The audio encoder is initialized with CTC training to predict phoneme sequences from speech.Phoneme sequences can be obtained from grapheme transcriptions using G2P tools, and S2P is simpler than S2G.Source: C. Proposed Method: Phoneme-Guided… · p. 3
  • The LLM is initialized on P2G, using phoneme sequences generated from text with G2P tools and requiring no speech data.The P2G inputs are generated from grapheme sequences; this stage uses text data only.Source: C. Proposed Method: Phoneme-Guided… · p. 3

Stage 2: Encoder and Projector Warmup

  • The S2P-initialized encoder and P2G-initialized LLM are connected through a randomly initialized projector and fine-tuned on the target ASR task.Stage 2 updates only the encoder and projector; training can use either standard S2G or phoneme-first loss.Source: C. Proposed Method: Phoneme-Guided… · p. 3

Stage 3: Full Fine-tuning

  • Full fine-tuning jointly updates all parameters on the same task, aligning the encoder and LLM toward phoneme-level representations.The LLM joins the encoder and projector in this final stage.Source: C. Proposed Method: Phoneme-Guided… · p. 4

A. Model Configuration

The model configurations compare a non-LLM CTC baseline, a cascaded S2P-P2G system, and end-to-end LLM systems with different output and initialization choices.

A. Model Configuration

  • The system uses an XLS-R 300M encoder, a downsampling-plus-MLP projector, and a Qwen3.5-2B LLM with its image components removed.Source: A. Model Configuration · p. 4
  • The non-LLM XLS-R + CTC baseline performs direct S2G recognition and isolates the LLM’s contribution.Source: A. Model Configuration · p. 4
  • The Cascade uses a CTC-trained encoder for S2P and a separately trained LLM for P2G at inference.The predicted phonemes are supplied as text input to the LLM.Source: A. Model Configuration · p. 4
  • End-to-end models use an S2P-initialized audio encoder, while Base omits P2G initialization and trains with the S2G objective.The CTC head is discarded; without S2P initialization, end-to-end models produced CER near 100%.Source: A. Model Configuration · p. 4
  • PF outputs phonemes before graphemes, PF+Joint mixes phoneme-first and standard S2G data, and +P2G adds P2G initialization before end-to-end fine-tuning.The proposed method’s P2G-initialized variants are Base+P2G, PF+P2G, and PF+Joint+P2G.Source: A. Model Configuration · p. 4

B. Data Configuration

Experiments cover simulated speech-data scarcity in Japanese and Chinese and inherently low-resource Tatar and Urdu, using language-specific text corpora for P2G initialization.

B. Data Configuration

  • The low-resource setting uses limited CSJ and AISHELL-1 training subsets, while Tatar and Urdu use all available Common Voice 25.0 speech data.The CSJ subsets contain 10 and 36 hours; AISHELL-1 subsets contain 10 hours, 50 hours, and the full set; Tatar and Urdu contain 8.3 and 8.4 hours.Source: B. Data Configuration · p. 4
  • P2G training uses the full text corpus for each language: BCCWJ, THUCNews, the Tatar Folklore Text Corpus, and Makhzan.Source: B. Data Configuration · p. 4
  • G2P tools generate phoneme-grapheme pairs, with BCCWJ phoneme transcriptions used directly and language-specific phonological rules handled by the tools or manual annotation.The selected tool varies by language according to availability and quality.Source: B. Data Configuration · p. 4

C. Training Configuration

Across the reported settings, P2G initialization improves end-to-end comparisons with the cascade and XLS-R + CTC, but its benefits differ between data-limited settings and low-resource languages.

C. Training Configuration

  • Training uses staged encoder/projector warmup followed by full-model fine-tuning; phoneme-first targets prepend phonemes, and decoding uses beam search with size 3.CTC uses a 3 × 10^-4 learning rate, P2G uses 5 × 10^-5, and end-to-end warmup and full tuning use 2 × 10^-4 and 2 × 10^-5, respectively.Source: C. Training Configuration · pp. 4–5

D. Results

  • Results use CER for CSJ and AISHELL-1 and WER for Tatar and Urdu; phoneme-first models are scored on graphemes only.Source: D. Results · p. 5
  • Base+P2G reduced CER from 20.8% to 11.7% on 10-hour CSJ and from 39.1% to 11.1% on 10-hour AISHELL-1, a 72% relative reduction on AISHELL-1.Relative reductions were 27% on 36-hour CSJ, 49% on 50-hour AISHELL-1, and 37% on full AISHELL-1; gains grew as paired speech data decreased.Source: D. Results · p. 5
  • Base+P2G outperformed the Cascade on all languages and XLS-R + CTC on all CSJ evaluation sets, reversing Base’s CSJ disadvantage against the non-LLM baseline.On 10-hour AISHELL-1, Base+P2G achieved 11.1% versus 11.4% for Cascade.Source: D. Results · p. 5
  • P2G initialization also trains on target-language text, so the effects of phoneme-level alignment and plain-text adaptation are not isolated.The authors leave a matched text-only control to future work.Source: D. Results · p. 5
  • Phoneme-first prediction helped AISHELL-1 without P2G but degraded CSJ; with P2G, PF hurt across settings, while PF+Joint+P2G remained competitive.PF+Joint+P2G phoneme-first decoding achieved 11.0% on 10-hour AISHELL-1; the authors suggest P2G already provides sufficient phoneme knowledge.Source: D. Results · p. 5
  • In Tatar and Urdu, Base+P2G showed no improvement on Tatar (41.6%) and only modestly improved Urdu from 29.3% to 28.4%.The authors suggest lower-quality G2P output and more phonemic writing systems may leave less additional information for P2G; Tatar’s best result was PF+Joint without P2G (40.6%).Source: D. Results · p. 5

E. Error Analysis

P2G initialization increases the share of sound-faithful homophone substitutions across the tested objectives and homophone-rich languages. The analysis links this shift to paired-data availability and script homophony, while identifying lexical tone as a remaining limitation.

E. Error Analysis

  • Homophone share measures the fraction of scorable substitutions whose reference and hypothesis graphemes have the same reading.The analysis aligns reference and hypothesis by minimum edit distance and uses toneless pinyin for Mandarin and kana for Japanese.Source: E. Error Analysis · p. 5
  • P2G shifts errors from fluent, phonetically unrelated semantic drift toward sound-faithful confusions where the correct sound is preserved but the wrong character is chosen.Table IV illustrates the two error types; Mandarin near-homophones can match segmentally while differing in tone.Source: E. Error Analysis · pp. 5–6, Table IV · p. 6
  • All fifteen Mandarin and Japanese comparisons show a higher homophone share with P2G, across every objective variant; Table III defines higher shares as more sound-faithful.The table compares each +P2G row with the row above it.Source: E. Error Analysis · p. 5, Table III · p. 6
  • On AISHELL-1, the P2G increment shrinks as paired speech data grows; its homophone-share effect is low and nearly unchanged for Tatar and Urdu, whose orthographies are more phonemic.The reported effect tracks script homophony, with Mandarin greater than Japanese, and Japanese greater than Urdu and Tatar.Source: E. Error Analysis · p. 6
  • Because Mandarin readings use toneless pinyin, near-homophones differing only in tone count as homophones, leaving accurate lexical-tone recovery as a limitation.至 and 之 are given as examples of this case.Source: E. Error Analysis · p. 6, Table IV · p. 6

V. CONCLUSION

The conclusion presents phoneme-guided initialization as S2P pre-training for the audio encoder and P2G pre-training for the LLM, followed by end-to-end fine-tuning. It reports lower CER on Japanese and Chinese, limited but non-negative gains on Tatar and Urdu, and identifies future directions.

V. CONCLUSION

  • CER fell by 27 to 72% relative to the end-to-end baseline on Japanese and Chinese, and the method outperformed both the CTC baseline and cascaded S2P-P2G pipeline.The largest gains were in low-resource settings.Source: §V · p. 6
  • Phoneme-guided initialization pre-trains the audio encoder on S2P and the LLM on P2G before end-to-end fine-tuning, without paired speech data for the initialization.Source: §V · p. 6
  • P2G shifts residual errors from semantic drift toward sound-faithful homophone confusions, indicating that remaining errors are primarily orthographic rather than acoustic.Source: §V · p. 6
  • Gains on Tatar and Urdu were limited but not negative; the authors point to higher-quality G2P, larger text corpora, and initialization beyond phoneme-level alignment as future work.Source: §V · p. 6
Loading 2610.08994v1…