PaperlayerView exact revision on arXiv

Source-linked AI summary

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs

Hemant Yadav, Sunayana Sitaram, Roger Zimmermann, Rajiv Ratn Shah

arXiv:2610.08085v1cs.CL

TL;DR

Speech-LLMs trained on ASR instructions can fail to follow unseen instructions, motivating methods that preserve an LLM’s instruction-following ability with speech input. DirectSpeech2LLM aligns speech embeddings using modified CTC, and reports strong ASR results alongside zero-shot performance on unseen tasks.

  • Problem

    Speech-LLMs trained on ASR can continue transcribing when prompted for unseen tasks, and existing approaches often fail to generalize beyond seen tasks.

    Source: §1 · p. 2

  • Method

    DirectSpeech2LLM uses a modified CTC objective with vector quantization to align speech and LLM embeddings, then adds LLM cross-entropy training.

    Source: §1 · p. 3, §3.3 · p. 5

  • Results

    DirectSpeech2LLM reports IFR = 1.00 on the two unseen tasks, closely matching the cascaded system, and outperforms comparable systems on ASR after training on only 1k hours.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    The results suggest that this framework can preserve instruction following on unseen tasks when an LLM receives speech input, despite training solely on English ASR data.

    Source: §6 · p. 11

    Because training uses a single ASR task, behavior in multilingual and multi-task settings remains underexplored.

    Source: §6 · p. 11

Abstract

from arXiv · show

Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the instruction-following ability of the LLM on unseen tasks when conditioned on speech. It computes distance-based CTC loss over the frozen LLM embedding matrix and uses greedy CTC labels to derive geometrically and temporally aligned speech embeddings respectively as an input to the LLM. Trained solely on 960 hours of LibriSpeech ASR data, DirectSpeech2LLM outperforms the cascaded system on ASR (seen task) and generalizes zero-shot to speech translation and emotion recognition (two unseen tasks), closely matching the cascaded system upper bound on these two new instructions despite seeing neither during training. We also find that geometric alignment strength plays a smaller role than previously assumed, as our modified CTC loss is shown to provide sufficient implicit geometric grounding without requiring an explicit regression loss. Results are consistent across two LLM families and scale with both more training data and model capacity.

1 Introduction

DirectSpeech2LLM targets Speech-LLM prompt overfitting by aligning speech embeddings with the LLM’s embedding space and input timing. It reports strong ASR performance and zero-shot generalization to two unseen tasks.

1 Introduction

  • Speech-LLMs can overfit to ASR instructions, continuing to transcribe when prompted for translation or question answering instead.The paper focuses on improving linguistic generalization to unseen tasks, leaving non-linguistic modeling for future work.Source: §1 · pp. 1–2
  • The authors hypothesize that unconstrained speech embeddings and fixed temporal downsampling impair generalization by misaligning embedding geometry and token boundaries.Variable phoneme durations make fixed downsampling produce inconsistent boundaries, so sequence length can reflect speaking rate rather than semantic content.Source: §1 · pp. 2–3
  • The framework uses a modified CTC objective with vector quantization, then adds the LLM cross-entropy loss during end-to-end training.Its stated aim is geometric and temporal alignment between speech-derived embeddings and the LLM input embedding space.Source: §1 · pp. 1–3
  • DirectSpeech2LLM achieves strong ASR performance and generalizes zero-shot to speech translation and emotion recognition, while showing robustness to out-of-domain ASR.The paper contrasts this with simple adapter-based methods, which it describes as limited in out-of-domain robustness.Source: §1 · p. 3
  • DirectSpeech2LLM aims to preserve instruction following without large multitask instruction-tuning datasets or multiple LLM forward passes.Its SFM can operate as standalone CTC ASR, CTC plus an n-gram language model, or the full Speech-LLM, with embeddings passed directly to the LLM.Source: §1 · p. 3

2 Related works

Related work addresses Speech-LLM alignment through behavior alignment, temporal alignment, and regression-based geometric alignment. DirectSpeech2LLM combines temporal and geometric alignment through distance-based CTC logits.

2 Related works

  • Behavior-alignment methods mimic LLM responses or use cross-modal distillation, but may require two LLM forward passes or costly large-scale data.The paper describes behavior alignment on the LLM output embedding space as complementary to its own alignment of the LLM input embedding space.Source: §2 · p. 3
  • CTC-based approaches include AlignFormer’s temporal guidance for a Q-Former, Soundwave’s projection layers, LegoSLM’s CTC-posterior-weighted text embeddings, and TASU’s simulated erroneous CTC text.Source: §2 · p. 4
  • Prior speech-translation work separates temporal CTC alignment from distribution-level geometric alignment, while DirectSpeech2LLM combines them using negative squared distances to frozen LLM embeddings.The embeddings serve as fixed codebooks, providing dynamic temporal compression and geometric grounding.Source: §2 · p. 4
  • Unlike Wav2Prompt’s CIF-plus-regression design, DirectSpeech2LLM uses distance-based CTC logits and does not require explicit regression loss for geometric grounding.The authors say its distance logits pull positive embeddings closer while pushing negative embeddings farther away, and that commitment loss is optional.Source: §2 · p. 4
  • Unlike AlignFormer, DirectSpeech2LLM constrains geometry as well as using CTC for temporal alignment; the paper says its approach can also complement AlignFormer’s input-ordering trick.Source: §2 · p. 4

3 Method

DirectSpeech2LLM aligns speech embeddings to the frozen LLM embedding space with a distance-based CTC objective, then uses greedy CTC labels to form temporally compressed inputs for the LLM. An optional commitment loss further constrains nonblank embeddings, while the LLM stays frozen during training.

3.1 Preliminaries

  • Section 3.1 refers readers to the paper’s preliminary material in Appendix A.1.Source: §3.1 · p. 4

3.2 Problem formulation

  • The formulation maps acoustic input to frame-level SFM embeddings and seeks embeddings close to their equivalent LLM input embeddings in both temporal and geometric dimensions.Source: §3.2 · p. 4

3.3 Speech tokenizer

  • Negative squared distances to the frozen LLM embeddings define CTC logits, unifying temporal alignment with implicit geometric alignment while using a smaller CTC vocabulary of acoustically suitable units.The CTC vocabulary is built from transcript BPE tokens that overlap the LLM vocabulary; the LLM still generates over its full vocabulary.Source: §3.3 · pp. 4–5

3.4 CTC guided downsampling

  • Greedy frame-level CTC labels are grouped into consecutive identical-label runs, whose embeddings are mean-pooled to create dynamically downsampled segments.Blank segments are globally mean-pooled into one embedding placed first; exploring whether it captures paralinguistic information is left for future work.Source: §3.4 · p. 5

3.5 Commitment loss

  • A commitment loss uses greedy CTC labels to pull SFM outputs toward pretrained LLM embeddings, excluding blank-token indices because they lack equivalent linguistic content.The modified CTC loss already encourages geometric closeness, though less directly than commitment loss.Source: §3.5 · pp. 5–6

3.6 LLM

  • Segmented speech embeddings are concatenated with prompt embeddings and passed to the LLM, which is trained with autoregressive cross-entropy over target text.Source: §3.6 · p. 6

3.7 Overall training objective

  • Only SFM parameters are optimized; training first enables CTC alone, then enables LLM loss for contextual feedback, while clamping commitment loss so it never exceeds CTC loss.Source: §3.7 · p. 6

4 Experimental details

Experiments train on English ASR data, using LibriSpeech alone or with Common Voice, and evaluate ASR, zero-shot speech translation, and zero-shot spoken question answering. Training uses frozen LLM backbones and a two-stage procedure.

4.1 Training data

  • Training data are English ASR only: the 960-hour LibriSpeech corpus is the 1k-data setting, while adding 1,800 hours of English Common Voice yields the 2.8k-data setting.The Common Voice experiment uses the official train split without additional filtering or data cleaning.Source: §4.1 · p. 6

4.2 Model configuration

  • The LLM backbones are Qwen3-4B and Phi3.1-3.8B with all LLM parameters frozen; the SFM starts from an ASR-finetuned Data2Vec-Large checkpoint.The SFM has about 300 million parameters, and its CTC head is reinitialized for the LLM input dimension.Source: §4.2 · p. 6

4.3 Training procedure

  • Training optimizes a fixed 50k stage-1 steps and 20k stage-2 steps, using only English ASR supervision unless otherwise noted.Stage 1 is a computationally cheap temporal-alignment initialization for end-to-end stage 2.Source: §4.3 · p. 7

4.4 Evaluation procedure

  • Evaluation covers in-domain and out-of-domain ASR, zero-shot English-to-German and English-to-Chinese translation, and zero-shot four-class IEMOCAP emotion recognition as an instruction-following test.Unseen-task evaluation omits blank tokens; task-specific prompts are used, and ASR is measured with WER, translation with BLEU, and emotion recognition with accuracy.Source: §4.4 · p. 7

5 Results

DirectSpeech2LLM performs strongly on ASR and preserves zero-shot instruction following on unseen speech translation and emotion recognition. Ablations find that its modified CTC objective provides implicit geometric grounding, while blank-token handling and training scale also affect unseen-task behavior.

5.1 Main results

  • With 1k hours of ASR data, DirectSpeech2LLM outperforms comparable systems on ASR and reaches IFR = 1.00 on unseen ST and ER, closely matching the cascaded system.The cascaded system is described as an upper bound because it passes discrete text to the LLM.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 Ablation study

  • Removing commitment loss in Stage 2 improves ASR and only marginally degrades unseen-task performance, whereas Stage 1 becomes usable only with commitment loss.The authors attribute the limited Stage 2 effect to the implicit geometric constraint in CTC; whether commitment loss remains necessary in multitask training is left open.Source: §5.2 · p. 8
  • More Stage 1 training steps and added speech data improve ASR and zero-shot translation, with Stage 2 ASR performance not yet saturated.The main-results discussion also reports improved zero-shot translation with more ASR data and improved performance with a larger LLM backbone.Source: §5.2 · p. 8, §5.1 · p. 8
  • No-blank training avoids prompt overfitting; with blank-token training, removing or repositioning the blank at inference restores unseen-task instruction following.The authors connect the effect to positional overfitting of a fixed, shared blank embedding; multitask training substantially reduces it.Source: §5.2 · pp. 8–9
  • Stage 2 narrows the backbone sensitivity seen in Stage 1, while adding AlignFormer’s input-ordering trick reduces prompt overfitting but performs worse than DirectSpeech2LLM.The slight Stage 2 underperformance of Phi3.1 relative to Qwen3 may reflect a weaker backbone.Source: §5.2 · p. 9

5.3 Regression loss sensitivity of Wav2Prompt vs DirectSpeech2LLM on unseen task

  • For en-de, removing commitment loss changes DirectSpeech2LLM’s BLEU from 19.97 to 19.69, while removing Wav2Prompt’s MSE loss drops en-es BLEU from 13.8 to 5.4.The authors report stable zero-shot performance despite a 10× increase in embedding distances and describe modified CTC as providing implicit geometric alignment.Source: §5.3 · p. 10

6 Conclusion and Future work

DirectSpeech2LLM aligns speech and LLM embeddings and shows strong ASR performance alongside instruction following on unseen speech translation and emotion recognition tasks. The authors identify several areas for future work, including broader task settings and more expressive decoding.

6 Conclusion and Future work

  • Trained only on English ASR data, DirectSpeech2LLM achieves strong ASR performance and robust instruction following on unseen speech translation and emotion recognition tasks.The authors describe this as evidence that the framework can preserve generalization to unseen tasks.Source: §6 · p. 11
  • Behavior in multilingual and multi-task settings remains underexplored because training uses a single ASR task.The authors also identify the roles of blank and non-blank embeddings, particularly when jointly optimized with paralinguistic tasks, as unclear.Source: §6 · p. 11
  • Temporal alignment uses CTC greedy decoding, which may limit performance relative to beam search augmented with an n-gram language model.The authors suggest more expressive decoding strategies as a future direction.Source: §6 · p. 11
  • The framework currently uses only the student part of Data2Vec, while teacher pseudo-labels are proposed for future semi-supervised training without target transcriptions.The proposed setting has source speech and LLM targets but no source transcriptions.Source: §6 · p. 11

A.1.1 Large Language Model (LLM) Input

LLM inputs have a token-sequence dimension and an embedding-space dimension, corresponding to temporal and geometric alignment. The authors hypothesize that minimizing distributional differences along both dimensions may improve cross-modal alignment and generalization to unseen tasks.

A.1.1 Large Language Model (LLM) Input

  • LLM input alignment has two dimensions: the token sequence, or temporal alignment, and the embedding space, or geometric alignment.The authors note that changes to either dimension can significantly degrade performance and instruction following.Source: §A.1.1 · p. 15
  • Temporal alignment concerns mapping raw text into sequences of discrete tokens using fixed rules.The token sequence is one of the dimensions along which input can deviate from the seen distribution.Source: §A.1.1 · p. 15
  • Geometric alignment maps each token to a high-dimensional embedding, and the authors hypothesize that minimizing distribution along both dimensions may improve unseen-task generalization.They frame this as a hypothesis about cross-modal alignment.Source: §A.1.1 · p. 15

A.1.2 Connectionist Temporal Classification (CTC)

Connectionist Temporal Classification (CTC) learns from input and target sequences when their exact frame-to-token alignment is unknown. It accounts for possible frame-level alignments using a blank symbol and collapses repeats and blanks during inference.

A.1.2 Connectionist Temporal Classification (CTC)

  • CTC is a sequence-learning objective for tasks such as ASR where exact alignment between input frames and output tokens is unknown.It takes acoustic representations X and a target token sequence Y.Source: §A.1.2 · p. 15
  • CTC models the conditional probability by marginalizing over frame-level alignments that collapse to the target token sequence.A blank symbol allows the model to emit no valid token at a timestep.Source: §A.1.2 · p. 15
  • During inference, CTC removes repeated tokens and blanks through a collapsing operation to produce the final transcription.Source: §A.1.2 · p. 15

A.1.3 Vector Quantization (VQ)

Vector quantization commits continuous speech representations to discrete codebook vectors, with this work identifying the codebook as the LLM input embedding space. Unlike standard dynamic-codebook VQ, the method fixes that codebook and treats quantization as classification over its indices.

A.1.3 Vector Quantization (VQ)

  • Vector quantization commits a continuous SFM output representation to the nearest discrete codebook vector, identified here with the LLM input embedding space.The quantized counterpart encourages the representation to align before being passed downstream.Source: §A.1.3 · p. 15
  • Unlike standard VQ with a dynamic codebook, this method fixes the codebook to the LLM input embedding space.The authors follow approaches [45] [46].Source: §A.1.3 · p. 15
  • The method treats quantization as classification, using mean squared error scores as logits to predict codebook indices.The authors describe this as a stable interface for end-to-end integration between speech and the language model.Source: §A.1.3 · p. 15

A.2 Training Procedure

The appendix examines multitask training, longform inference, training dynamics, token-boundary preservation, and related implementation and failure cases. It reports stronger multitask performance across seen tasks, degradation at long durations, and limited gains from the TBP extension.

A.2 Training Procedure

  • Training uses a modified CTC objective in Stage 1 and adds frozen-LLM cross-entropy in Stage 2.Stage 1 trains the last six Data2Vec-Large layers for 50k steps; Stage 2 trains the last two for 20k steps. The method also derives greedy CTC labels, and the speech tokenizer vocabulary intersects ASR BPE and LLM vocabularies.Source: §A.2 · p. 15, §A.7 · p. 20

A.3 Multitask Training Setup

  • In multitask training, the WB setup outperforms the cascaded system on all seen tasks.The configuration combines LibriSpeech ASR and English-to-German CoVoST2 data without commitment loss; the authors attribute reduced positional overfitting to training on multiple tasks.Source: Table 4 · p. 16, §A.3 · p. 16

A.4 Longform Speech Inference

  • At 10 minutes, WB reaches WER 14.26 while NB reaches WER 4.22; at 20 minutes, all setups degrade significantly.Both setups match the CTC baseline at 1 minute, but evaluation sequences extend beyond the training limit of 20 seconds; the authors cite self-attention extrapolation as a limitation.Source: Table 5 · p. 16, §A.4 · p. 16

A.5 Training dynamics

  • Stage 2 WER on test-other keeps improving through 20,000 steps for both backbones, while Phi-3.1-3.8B has higher, more erratic commitment loss than Qwen3-4B.The authors say results may improve with longer training and that backbone quality is a meaningful factor in geometric alignment; the cause of the commitment-loss difference remains open.Source: §A.5.1 · p. 16, Figure 3 · p. 17, §A.5.2 · p. 17

A.6 Token Boundary Preservation

  • TBP uses tokenizer-specific spans to aggregate byte-level representations, preserving the LLM’s original token boundaries.Its reported extension yields marginal ASR improvements and no clear gains on unseen tasks; the table discussion says input tokenization has no substantial effect on task-solving ability.Source: §A.6 · p. 18, Table 6 · p. 19, §A.9 · p. 22

A.7 Pseudocode

  • The pseudocode derives distance-based logits over the frozen LLM embeddings, then uses greedy CTC labels for guided downsampling and commitment loss.The Qwen3 template provides task-specific prompts for ASR, speech translation, and emotion recognition.Source: §A.7 · p. 20, Figure 6 · p. 21

A.9 Discussion

  • Using the same WB or NB decoding strategy as training eliminates the reported Stage 1 NB formatting and instruction-following errors.The discussion proposes byte-level inputs as a direction for formatting errors but reports marginal ASR improvement and no clear unseen-task gains for TBP; inference beyond 300 seconds lags the cascaded system after training on up to 20 seconds.Source: §A.9 · p. 22

A.10 Effect of Restricted CTC Vocabulary as input to LLM

  • With ground-truth transcripts as input, restricted-vocabulary tokenization raises WER mainly through formatting errors, with Phi-3.1-3.8B more sensitive than Qwen3-4B.The discussion reports no clear scaling trend across Qwen model sizes and says these errors are largely resolved after fine-tuning.Source: §A.10 · pp. 22–23, Table 9 · p. 23
Loading 2610.08085v1…