PaperlayerView exact revision on arXiv

Source-linked AI summary

Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR

Ziwei Li, Lukuang Dong, Saierdaer Yusuyin, Xianyu Zhao, Zhijian Ou

arXiv:2604.09332v2eess.AS

TL;DR

Speech–language interface choice matters in LLM-based ASR, but controlled comparisons of phoneme and projector approaches have been limited. This work compares the interfaces under shared backbones and adds BPE-phoneme and phoneme-informed hybrid variants. Phoneme interfaces are competitive on LibriSpeech, BPE-phoneme is best in the study, and phoneme-based recognition substantially outperforms the projector on low-resource Tatar.

  • Problem

    Controlled comparisons of phoneme-based and projector-based interfaces within the same LLM-ASR backbone remain limited, despite interface design affecting recognition performance and data efficiency.

    Source: §I · p. 1

  • Method

    The paper compares projector and phoneme interfaces under shared backbones, adds BPE-phoneme units for phoneme-to-grapheme generation, and tests phoneme-supervised encoders in a projector pathway.

    Source: §I · p. 1, E. Training · p. 4

  • Results

    Across tasks, phoneme interfaces are competitive with projectors on LibriSpeech, BPE-phoneme performs best there, and CV-tt test WER falls from 33.58% (projector) to 17.44% (phoneme).

    Source: §I · p. 2, A. High-Resource Evaluation on… · p. 5, C. Low-Resource Evaluation on Tatar · p. 6

  • Takeaways & Limitations

    The study offers a controlled interface-level comparison, with BPE-phoneme prompting and phoneme-supervised hybrid projector training as practical additions.

    Source: §VI · p. 6

    Evaluation covers LibriSpeech and CV-tt; broader domain and language coverage is left for future work.

    Source: §VI · p. 6

Abstract

from arXiv · show

Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector.

I. INTRODUCTION

LLM-based ASR connects a speech encoder to a language model through an interface that affects recognition performance and data efficiency. This work compares continuous projector and discrete phoneme interfaces, proposes BPE-phoneme and phoneme-informed hybrid variants, and reports results on LibriSpeech and Tatar.

I. INTRODUCTION

  • LLM-ASR connects a speech encoder and an LLM through an interface, which is a key determinant of recognition performance and data efficiency.Interfaces include continuous learned projectors and discrete tokenized units such as phonemes.Source: §I · p. 1
  • Phonemes provide a linguistically grounded, textlike intermediate representation that separates pronunciation from orthography and retains structure partially shared across languages.The paper motivates this interface especially for settings with limited paired supervision, where post-hoc alignment between separately pretrained spaces may be less robust.Source: §I · p. 1
  • The study compares projector and phoneme interfaces under controlled backbones and proposes BPE-phoneme units that merge frequent local phoneme patterns while retaining word boundaries.The paper frames its contribution as an interface-level study, not a claim that phoneme interfaces dominate all projector or speech-unit alternatives.Source: §I · pp. 1–2
  • A phoneme-informed hybrid reuses an encoder fine-tuned with phoneme supervision in a projector pathway, and the authors report substantially stronger performance.The two interface approaches are therefore not strictly orthogonal.Source: §I · p. 1
  • On LibriSpeech, projector- and phoneme-based methods are competitive, while BPE-phoneme performs best; on Tatar, phoneme-based recognition substantially outperforms the projector.Both Tatar systems use the same frozen pretrained Whistle-large encoder, with no target-data encoder adaptation.Source: §I · p. 2

II. RELATED WORK

The related work moves from conventional end-to-end and pretrained ASR toward systems that pair speech encoders with LLM decoders. It also situates learned connectors and projectors as bridges between continuous speech features and LLM representations.

A. End-to-End and Pretrained ASR

  • End-to-end ASR commonly uses CTC, RNN-T, or attention-based encoder–decoder models, while large-scale speech pretraining provides transferable acoustic representations.The passage distinguishes self-supervised pretraining from supervised or weakly labeled multilingual pretraining.Source: A. End-to-End and Pretrained ASR · p. 2

B. LLM-based ASR

  • LLM-based ASR reformulates recognition as conditional generation using a pretrained speech encoder and LLM decoder, with learned connectors, adapters, or projectors bridging their representations.The cited work includes both frozen-LLM and lightweight-PEFT approaches; recent LibriSpeech-960h systems provide comparison baselines.Source: B. LLM-based ASR · p. 2

C. Speech–Language Interfaces and Phoneme Modeling

Speech–language interfaces bridge continuous speech representations and an LLM's discrete token space; controlled comparisons of phoneme and projector interfaces remain limited. The projector pathway downsamples acoustic features, maps them into the LLM embedding space, and trains the projector and decoder in separate phases.

C. Speech–Language Interfaces and Phoneme Modeling

  • Continuous interfaces learn a mapping from speech representations to LLM embeddings, whereas discrete interfaces provide token sequences such as phonemes; phoneme sequences can also be segmented into BPE units.The paper positions its S2P-to-LLM P2G use of BPE-phoneme units as an intermediate interface, and extends the comparison with phoneme-informed projector variants.Source: C. Speech–Language Interfaces and… · pp. 2–3

A. Projector-based Interface

  • The projector pathway encodes speech into frame-level acoustic representations, downsamples them, then projects the resulting features into the LLM embedding space as speech-conditioned prefix embeddings.The projector uses a hidden layer with ReLU followed by a regression layer.Source: A. Projector-based Interface · p. 3
  • Projector training has two phases: first train the projector with the speech encoder and LLM frozen, then freeze the projector and adapt the LLM with LoRA.This lets the decoder learn to use the speech-conditioned prefix after the alignment stage.Source: A. Projector-based Interface · p. 3

B. Phoneme-based Interface

The phoneme-based interface factorizes recognition into speech-to-phoneme and phoneme-to-grapheme models, marginalizing over intermediate phoneme sequences. Training samples CTC paths to expose phoneme uncertainty to the LLM, while decoding rescores top-K phoneme hypotheses.

B. Phoneme-based Interface

  • ASR is factorized into a CTC-based speech-to-phoneme model and an autoregressive LLM for phoneme-to-grapheme generation, with recognition marginalizing over phoneme sequences.The objective combines p(h | x) and p(y | h).Source: B. Phoneme-based Interface · p. 3
  • Training uses two stages: CTC phoneme prediction, followed by LLM training on sampled phoneme sequences to approximate marginalization over phoneme uncertainty.The CTC paths are collapsed into phoneme sequences before the LLM is trained.Source: B. Phoneme-based Interface · p. 3
  • The Monte Carlo training approximation draws K CTC-path samples, collapses repeats and blanks, and uses the resulting phoneme sequences for LLM training.K denotes the number of sampled phoneme sequences used to approximate marginalization over phoneme uncertainty.Source: B. Phoneme-based Interface · p. 3
  • At inference, Top-K Marginalization rescores multiple phoneme hypotheses rather than relying on a single sequence.The S2P beam search supplies top-K phoneme hypotheses, and P2G generates text candidates for them.Source: B. Phoneme-based Interface · p. 3

C. BPE-Phoneme Modeling

The BPE-phoneme interface groups frequent local phoneme patterns while retaining word boundaries, unlike mono-phoneme prompts. Its design emphasizes boundary-aware chunks rather than reducing the LLM input length.

C. BPE-Phoneme Modeling

  • BPE-phoneme merges frequent local phoneme patterns while preserving whitespace between phoneme words, unlike mono-phoneme prompts without explicit word boundaries.The resulting sequences can be slightly longer in LLM tokens, so the interface is not designed as a prompt-shortening trick.Source: C. BPE-Phoneme Modeling · p. 3
  • The proposed gain is interpreted as coming from boundary-aware segmentation and chunked local phonotactic units, rather than fewer LLM input tokens.BPE merges are learned from training phoneme transcripts, and generated units are converted back into phoneme words for the LLM.Source: C. BPE-Phoneme Modeling · pp. 3–4
  • BPE-phoneme units serve as CTC targets in Stage 1, with whitespace-preserving sequences passed to the LLM in Stage 2.SentencePiece is used to learn the merges on the training phoneme transcripts.Source: C. BPE-Phoneme Modeling · p. 4

IV. EXPERIMENTAL SETUP

The experiments compare interface designs in a data-rich English setting and a label-scarce Tatar setting, intended to isolate interface effects rather than provide an exhaustive multilingual benchmark.

IV. EXPERIMENTAL SETUP

  • The evaluation uses LibriSpeech (960h) for high-resource English and the Tatar subset of Common Voice v11.0 (20h) for low-resource Tatar.These settings are intended to contrast data-rich English with a label-scarce target-language condition, not to serve as an exhaustive multilingual benchmark.Source: A. Datasets · p. 4
  • For English, the study validates on dev-other and reports WER on test-clean and test-other.Source: A. Datasets · p. 4
  • For Tatar, the study validates on dev and reports WER on both dev and test.Source: A. Datasets · p. 4

B. Model Backbones

The experiments use Qwen3-Base with speech encoders spanning pretrained paradigms, generate phoneme targets from pronunciation lexicons, and compare projector-based and phoneme-based interfaces.

B. Model Backbones

  • The study uses Qwen3-Base, adapting Qwen3-1.7B by default and also reporting Qwen3-8B scaling results.LoRA is applied to attention and MLP modules.Source: B. Model Backbones · p. 4
  • LibriSpeech experiments cover WavLM-base, WavLM-large, and phoneme-supervised Whistle-small; Tatar uses the same frozen Whistle-large encoder for projector and phoneme systems.The Tatar encoder is not fine-tuned on the 20-hour target data, isolating the interface comparison from additional target-task encoder adaptation.Source: B. Model Backbones · p. 4

C. Pronunciation Lexicons and Phoneme Targets

D. Speech–Language Interfaces

  • The projector pathway downsamples encoder states by k=5 and maps them to LLM embeddings with a two-layer MLP; the phoneme pathway uses CTC-based S2P and feeds phoneme sequences to the LLM.The phoneme-informed projector initializes its encoder from Ph-Stage1 phoneme fine-tuning.Source: D. Speech–Language Interfaces · p. 4

E. Training

Training uses staged projector and phoneme pathways, with a frozen-encoder comparison on CV-tt and separate optimization schedules for each interface.

E. Training

  • Phoneme-based training first fine-tunes the encoder for CTC phoneme prediction, then trains the LLM for phoneme-to-grapheme generation; projector training separates projector learning from LoRA adaptation.For CV-tt, both interfaces instead use the same frozen Whistle-large encoder without Tatar-specific fine-tuning.Source: E. Training · p. 4
  • Table II compares frozen-encoder projector baselines with phoneme-informed projectors and phoneme interfaces using phoneme-fine-tuned encoders.PER is reported only for phoneme-fine-tuned encoders.Source: Table II · p. 5
  • The LibriSpeech BPE-phoneme S2P ablation selects BPE size 100, which is used in subsequent BPE-phoneme experiments.Source: Table III · p. 5

F. Additional Optimization Details

  • For the phoneme-to-grapheme stage, all models use the same optimization strategy: linear warmup followed by cosine annealing, with a peak learning rate of 2 × 10^-4.Source: F. Additional Optimization Details · p. 5
  • Projector Stage 1 uses batch size 80 and a 1 × 10^-4 learning rate; Stage 2 uses batch size 128 and a fixed 1 × 10^-5 learning rate.Both projector stages use early stopping after five stagnant validations.Source: F. Additional Optimization Details · p. 5

V. RESULTS

On LibriSpeech, phoneme-informed projectors improve over vanilla baselines, while BPE-phoneme achieves the study's best overall result. Its gains are attributed to segmentation cues and multi-phoneme chunks, not shorter prompts.

A. High-Resource Evaluation on LibriSpeech

  • Phoneme-informed projectors are substantially stronger than vanilla projector baselines and become competitive with the phoneme-based interface.The comparison uses controlled backbones and comparable LoRA adaptation.Source: A. High-Resource Evaluation on… · p. 5
  • LoRA adaptation in projector Stage 2 consistently improves WER, while phoneme fine-tuning yields more projector-friendly representations, especially on test-other.Source: Table IV · p. 5, A. High-Resource Evaluation on… · p. 5
  • Scaling the encoder or LLM improves final WER, and the best overall result is achieved by the scaled BPE-phoneme interface.The reported PER reductions correlate with downstream WER gains; after encoder scaling reduces PER, increasing LLM size yields further WER improvements.Source: Table V · p. 5, A. High-Resource Evaluation on… · p. 6
  • WER is 2.78/6.75 for BPE-phoneme versus 2.85/6.79 for the corresponding phoneme-informed projector.Source: A. High-Resource Evaluation on… · p. 5
  • Table VI contextualizes absolute WERs against recent LLM-ASR baselines, but the comparison is not perfectly matched because backbones, model scales, and tuning recipes differ.Source: Table VI · p. 6
  • BPE-phoneme prompts are slightly longer than mono-phoneme prompts, 125 versus 113 tokens, so gains are attributed to explicit segmentation cues and multi-phoneme chunks.Source: A. High-Resource Evaluation on… · p. 6

B. Contextualizing Absolute WERs with Recent LibriSpeech-960h LLM-ASR Baselines

E8’s LibriSpeech results are positioned as competitive rather than a standalone state-of-the-art recipe, while the low-resource Tatar comparison shows a strong phoneme-interface advantage with a shared frozen encoder.

B. Contextualizing Absolute WERs with Recent LibriSpeech-960h LLM-ASR Baselines

  • E8’s WER (1.97/3.78) is stronger than early frozen-connector Q-Former [11] and SALM [12], competitive with SLAM-ASR [13], but weaker than stronger LoRA projector recipes [14].The paper frames these absolute LibriSpeech results as competitive, not as a standalone SOTA recipe.Source: B. Contextualizing Absolute WERs with… · p. 6
  • The LibriSpeech baseline comparisons are informative but not fully matched in backbones, model scale, or training recipe.The stronger projector recipes apply LoRA to both the speech encoder and the LLM.Source: B. Contextualizing Absolute WERs with… · p. 6

C. Low-Resource Evaluation on Tatar

  • 17.44% test WER for the phoneme interface versus 33.58% for the projector on CV-tt (20h), nearly halving WER with the same frozen Whistle-large encoder.The comparison supports the value of the phoneme sequence beyond simple transcript supervision.Source: Table VII · p. 6, C. Low-Resource Evaluation on Tatar · p. 6

D. Training and inference complexity

The phoneme system offers a K-controlled speed–accuracy trade-off. The conclusion situates the work as a controlled interface study with BPE-phoneme prompting and phoneme-supervised hybrid projector training.

D. Training and inference complexity

  • With K=1, E5-Ph matched E5-Pr1 inference speed (0.090/0.097 vs. 0.095/0.094 RTF on test-clean/test-other) while retaining better WER.For the representative setting, projector training took 9.25h, while the phoneme system with S-SKM (K=8) finished in 5.83h despite higher per-iteration cost; K=8 was slower at inference but gave the best E5 WER.Source: D. Training and inference complexity · p. 6

VI. CONCLUSION

  • The conclusion presents a controlled interface-level comparison, highlighting BPE-phoneme prompting and phoneme-supervised hybrid projector training.Source: §VI · p. 6

VII. GENERATIVE AI USE DISCLOSURE

  • The authors report using generative AI only for language editing, polishing, and formatting, not for core research content or results.Source: §VII · p. 7
Loading 2604.09332v2…