PaperlayerView exact revision on arXiv

Source-linked AI summary

Phoneme-First Prediction for LLM-Based Speech Recognition

Jakob Poncelet, Hugo Van hamme

arXiv:2606.10864v1eess.AS

TL;DR

Speech-augmented LLMs must align semantic text embeddings with speech features related to pronunciation, a relation that labeled ASR data teaches from limited examples. This paper has the LLM predict phonemes before words and reports improved recognition, including gains over S2T on TED-LIUM.

  • Problem

    Speech-text alignment is challenging because LLM embeddings reflect semantics while speech features reflect pronunciation, and ASR data provides limited examples for learning their relation.

    Source: §I · p. 1

  • Method

    The LLM predicts phonemes from speech features before generating the transcript, with phoneme targets automatically derived from transcripts.

    Source: §I · p. 1

  • Results

    Phoneme-first prediction improves LLM-based speech recognition over the standard approach, with improvements of up to 25% over S2T on TED-LIUM.

    Source: §I · p. 1, 2) TEDLIUM-100h · p. 3

  • Takeaways & Limitations

    Phoneme-first decoding provides explainability and yields phonemically grounded transcripts.

    Source: §VI · p. 5

    Generating phonemes requires a decent pronunciation lexicon and an acoustic model for the language or accent.

    Source: §V · p. 4

Abstract

from arXiv · show

Recent research has explored integrating Large Language Models (LLMs) with speech encoders to create speech-augmented LLMs capable of contextualized speech recognition. The main challenge lies in aligning the semantic embeddings of LLMs with the acoustic representations of speech encoders. We propose a novel approach that teaches the LLM to first predict phonemes from the speech features before generating the final transcript. By integrating a phoneme prediction step directly into the LLM, the model develops a fine-grained knowledge of pronunciation, reducing acoustic confusion and improving transcription accuracy and explainability. Our method is cheap and simple, as phoneme targets can be automatically derived from existing transcripts. Through comprehensive experiments, we show that intermediate phoneme prediction can improve speech recognition, particularly in low-resource settings, and yields outputs that are acoustically more faithful to the speech.

I. INTRODUCTION

Speech-augmented LLMs connect speech encoders to text-based LLMs, but acoustic pronunciation features and semantic embeddings are difficult to align, especially with limited examples. The paper proposes predicting phonemes before words and reports improved recognition, acoustically closer transcripts, and no need for manual phoneme labels.

I. INTRODUCTION

  • Speech features emphasize pronunciation while LLM embeddings reflect semantics, and labeled ASR data provides only limited examples for learning their relationship.Homophones and similar-sounding words further challenge speech-text alignment.Source: §I · p. 1
  • The proposed method has the LLM predict phonemes from speech features before generating words, explicitly modeling pronunciation alongside text.The authors say this can help relate similar-sounding words and provide users clues about which sounds were recognized.Source: §I · p. 1
  • Phoneme-first prediction improves LLM-based speech recognition over the standard approach, and phoneme targets can be generated automatically from transcripts.The paper also reports acoustically closer transcripts across experiments on varied datasets and models.Source: §I · p. 1

II. SPEECH-AUGMENTED LLM

The speech-augmented LLM pipeline comprises a speech encoder, a projection layer, and a backbone LLM. The encoder produces speech features that the projection maps into the LLM embedding space before the LLM processes them with a textual instruction.

II. SPEECH-AUGMENTED LLM

  • A speech-augmented LLM consists of a speech encoder, projection layer, and backbone LLM, each of which can be further optimized for speech tasks.Source: §II · p. 2
  • The speech encoder converts input speech into feature vectors and is kept frozen in this work.The encoders are from pretrained Whisper or HuBERT models.Source: §II · p. 2
  • An MLP projection layer preceded by feature stacking maps speech features into the LLM embedding space.Source: §II · p. 2
  • Projected speech features precede a textual instruction in the LLM prompt, and training jointly optimizes LoRA weights and the projection layer with next-token cross-entropy loss.The LLM is adapted using 4-bit QLoRA.Source: §II · p. 2

A. Speech-to-Text (S2T)

Standard S2T training teaches a speech-augmented LLM to map speech features to text. PF-S2T adds phoneme prediction before text generation, while Joint training randomly alternates between the two instruction prompts.

A. Speech-to-Text (S2T)

B. Phoneme-First Speech-to-Text (PF-S2T)

  • PF-S2T changes the instruction and target format so the LLM generates a phonemic transcription before the text transcript.The authors propose this sequence to improve the LLM’s acoustic awareness.Source: B. Phoneme-First Speech-to-Text (PF-S2T) · p. 2
  • Attention maps show the LLM attends to speech features to predict phonemes but almost only to phonemes to generate text; Joint randomly selects S2T or PF-S2T prompts during training.The authors attribute the attention pattern to teacher-forcing training in decoder-only LLMs.Source: B. Phoneme-First Speech-to-Text (PF-S2T) · p. 2

III. SETUP

The experimental setup uses a frozen speech encoder, a projection layer, and a quantized, QLoRA-adapted Llama model, with optimization choices designed for compute constraints. Inference uses beam search with validity checks, and results are reported as normalized transcript error rates.

III. SETUP

  • The default setup stacks five Whisper-medium encoder features, projects them through a two-layer network, and fine-tunes Llama-3.1-8B with 4-bit QLoRA.The encoder features have a 10 Hz rate; the projection totals 18.6M parameters and the adaptation has 10.7M trainable parameters.Source: §III · p. 2
  • Quantized 8-bit Adam, gradient checkpointing, and Flash Attention support training the pipeline on a 24GB GPU under the stated compute constraints.The effective batch size is 128 or 256 utterances, depending on the dataset.Source: §III · p. 2
  • Inference uses deterministic beam search with beam size 10 and checks prompt-word presence and compression ratio before accepting a result.If no beam candidate is valid, generation is repeated with sampling and increasing temperature; reported error rates use normalized transcripts.Source: §III · p. 2
  • Table I reports WER (%) for models trained on 100 hours of LibriSpeech or TED-LIUM.Source: Table I · p. 3

IV. EXPERIMENTS

Experiments compare phoneme-first decoding with standard speech-to-text (S2T), across training-data sizes and types.

1) LibriSpeech-100h:

The experiments evaluate phoneme-first speech-to-text on 100-hour LibriSpeech and TED-LIUM data, then examine scaling on 960-hour LibriSpeech. Joint training improves WER for nearly all test sets, with TED-LIUM gains of up to 25% over S2T.

1) LibriSpeech-100h:

  • The 100-hour LibriSpeech experiment uses train-clean-100, Montreal Forced Aligner phonetic transcripts, and zero-shot Whisper-medium as a reference.WER is reported in Table I.Source: 1) LibriSpeech-100h · p. 3

2) TEDLIUM-100h:

  • The 100-hour TED-LIUM experiment uses the provided dictionary and forced alignments to select word pronunciations, with evaluation on the official three-hour test split.WER results are reported in Table I.Source: 2) TEDLIUM-100h · p. 3
  • Joint training with phoneme-first S2T improves WER for nearly all test sets, with TED-LIUM gains of up to 25% over S2T.For LibriSpeech, improvements are especially strong on clean test sets; the authors suggest phoneme models trained only on clean data have more difficulty on other test sets.Source: 2) TEDLIUM-100h · p. 3

B. Evaluation on 960h of LibriSpeech

  • To test scaling with more labeled data, the authors repeat the experiment on the full 960-hour LibriSpeech set.In this setting, the LLM QLoRA rank is 32, with alpha 64 and dropout 0.05.Source: B. Evaluation on 960h of LibriSpeech · p. 3

1) Whisper Encoder:

On 960-hour LibriSpeech, joint training with phoneme-first S2T improves results, although standard S2T decoding tends to do better than phoneme-first decoding in this larger-data regime. The experiments use both Whisper and self-supervised HuBERT encoders.

1) Whisper Encoder:

  • The Whisper-large-v3 encoder setup has 32 layers, 1280-dimensional outputs, and 52M trainable parameters.Zero-shot Whisper-large results are included as a reference.Source: 1) Whisper Encoder · p. 3

Discussion of Results:

2) SSL Encoder:

  • The HuBERT-large encoder was pretrained on 60k hours of LibriLight, and its layer weights are optimized jointly with the projection layer and LLM.This setup totals 50M trainable parameters; a fully finetuned HuBERT-Large CTC model is included for reference.Source: 2) SSL Encoder · p. 3

Discussion of Results:

  • With more data, standard S2T decoding tends to outperform phoneme-first decoding, although phoneme-first decoding was optimal with 100 hours.The authors attribute the larger-data pattern to S2T having seen enough training examples.Source: Discussion of Results · p. 3
  • The self-supervised HuBERT encoder with a weighted sum of layer states tends to outperform the supervised Whisper encoder on this in-domain task.The fully trained baseline remains slightly better than the LLM model for both encoder types.Source: Discussion of Results · p. 3, 2) SSL Encoder · p. 3

C. Impact of Phoneme Label Quality

The Spoken Dutch Corpus experiments compare manual, automatic, and combined phoneme-label conditions. Higher-quality manual labels benefit phoneme recognition, while phoneme-first S2T also scales to automatic labels.

  • The comparison uses 240 hours of labeled Flemish Dutch speech, including 25 hours with manually corrected phoneme labels, across manual-only, automatic-only, and combined conditions.Source: C. Impact of Phoneme Label Quality · p. 3
  • The Dutch Tweety-7B LLM and Dutch fine-tuned ASR encoder are evaluated on a test set of 4,000 utterances (3 hours), reporting WER and PER.WER is measured against transcripts, and PER against manually labeled phonemes.Source: C. Impact of Phoneme Label Quality · p. 4, Table III · p. 4
  • Manual phoneme labels benefit phoneme recognition and the phoneme-first method, which also scales to automatically generated labels.Across all label conditions, phoneme-first S2T outperforms S2T training; however, it tends to have higher WER than joint training because it overfits predicted phonemes for final words.Source: C. Impact of Phoneme Label Quality · p. 4

D. Analysis of Speech-Text Alignment

The analysis compares G2P-based phoneme error rate (PER) with word error rate (WER) and finds a larger PER reduction than WER reduction on dev-clean. Error examples also show more acoustically plausible mistakes, especially for rare and out-of-vocabulary words.

V. DISCUSSION

The discussion describes phoneme-first prediction as applicable across many labeled speech datasets and languages, while noting requirements for pronunciation resources and slower inference. The conclusion summarizes its ASR, explainability, and phonemic-grounding benefits.

V. DISCUSSION

  • Phoneme-first prediction is presented as applicable to most labeled speech datasets and languages, using a decent pronunciation lexicon and an acoustic model for phoneme generation.Source: §V · p. 4
  • Speech-LLMs may have broader applications than ASR, although dedicated ASR baselines can still achieve lower WER.Source: §V · p. 4
  • Predicted phonemes can help identify sounds recognized poorly by the LLM, speech encoder, or user, supporting speech-assessment tasks that need phonemic explanations.Source: §V · pp. 4–5
  • The two-step output is slower at inference, and the approach requires low-rank LLM adaptation.Source: §V · p. 5

VI. CONCLUSION

  • The conclusion characterizes phoneme-first decoding as improving ASR, providing explainability, and yielding phonemically grounded transcripts.Source: §VI · p. 5
Loading 2606.10864v1…