Connecting Speech Encoder and Large Language Model for ASR

1. INTRODUCTION

• Connectors must retain useful speech information while compressing speech features to shorten LLM inputs and limit computation and storage costs.The paper identifies information retention and efficient compression as the connector's two requirements.Source: §1 · p. 1
• Q-Former connectors outperform strong Whisper baselines in-domain and achieve over 12% relative WER reduction on out-of-domain Eval2000.The paper also proposes segment-level Q-Former for long speech inputs and reports lower WER than other connector structures on those inputs.Source: §1 · p. 1
• The study connects a Whisper encoder to a Vicuna LLM and compares fully connected layers, multi-head cross-attention, and Q-Former connectors.Experiments use a training set of approximately 4,000 hours; the paper also studies connector output-token counts and model size.Source: §1 · p. 1

2. RELATED WORK

• SpeechGPT and AudioPaLM augment LLM vocabularies with discrete speech tokens, while another approach directly connects speech encoders with LLMs.The cited direct-connection work includes integrated multilingual ASR and speech translation, as well as domain adaptation and uncertainty estimation for ASR.Source: §2 · p. 1
• Other multimodal work connects audio and visual encoders to LLMs, including X-LLM's interface between ChatGLM and those encoders.The paper also notes reduced-frame-rate speech encoders connected to LLMs for multilingual ASR and speech translation.Source: §2 · p. 1
• Visual LLM studies include Q-Former and alternative connectors, while audio and music LLM studies examine audio reasoning and music understanding.The cited examples include InstructBLIP and Video-LLaMA using Q-Former as a connector, and MU-LLaMA for music understanding.Source: §2 · p. 1

3. MODULE CONNECTOR

• The system places a trainable connector between a frozen speech encoder and frozen LLM, mapping encoder features into speech representations used for transcription.The section compares fully connected layers, multi-head cross-attention, and Q-Former.Source: §3 · p. 2, Figure 1 · p. 2

3.1. Fully connected layers

• Fully connected layers compress speech features by stacking adjacent frames and applying two linear layers with an intervening ReLU.The stacking operation and first linear layer together work like a one-dimensional convolution.Source: §3.1 · p. 2

3.2. Multi-head cross-attention

• Multi-head cross-attention first reduces speech-input length with Conv1d, then uses textual embeddings in attention to convert hidden states into LLM input representations.The attention operation uses hidden states as queries and textual embeddings as keys and values.Source: §3.2 · p. 2

3.3. Q-Former

• Q-Former maps variable-length inputs to fixed-length query representations through trainable queries that interact with speech features using self-attention and cross-attention.The described module comprises two Transformer decoder blocks with causal attention masks removed.Source: §3.3 · p. 2

4. SEGMENT-LEVEL Q-FORMER

• Segment-level Q-Former transforms each encoder-output segment separately and concatenates the fixed-length output sequences before sending them to the LLM.The query embeddings and Q-Former layers are shared across segments, and the design can be initialized from a pre-trained standard Q-Former.Source: §4 · p. 2
• Unlike input-end concatenation with a fixed nq outputs, seg-QF produces N × nq output tokens, varying the output length with the number of segments.The paper describes this as more suitable for speech inputs with variable lengths in a wide range.Source: §4 · p. 2
• Segment-level position embeddings are added to each segment's encoder features to represent positions in the full input sequence.The encoder provides relative frame positions within a segment, but not absolute positions across the complete input.Source: §4 · p. 2

5.1. Data specifications

• Models in Sections 6.1–6.4 used LibriSpeech 960h; Section 6.5 used about 4,000 hours from LibriSpeech, Common Voice 7.0 English, and GigaSpeech subset M; Section 6.6 used LibriSpeech train-clean-100.The named dataset test sets served for in-domain evaluation, while Eval2000 was used for out-of-domain evaluation.Source: §5.1 · p. 3

5.2. Model specifications

• Whisper speech encoders and Vicuna LLMs were frozen; Whisper base, medium, and large-v2 encoders and Vicuna 7B and 13B models were compared.Whisper large-v2 with Vicuna 13B was selected as the best-performing setup for the other sections.Source: §5.2 · p. 3
• Sections 6.1–6.5 trained for 90k steps with batch size 24, while Section 6.6 models trained for 10k steps with batch size 8.Final checkpoints were selected by highest validation accuracy with teacher forcing.Source: §5.2 · p. 3

6.1. Random audio concatenation in training

• Randomly concatenating training utterances reduced deletion errors and improved WER, so the strategy was used in the remaining experiments.Concatenated utterances were capped at a sampled duration T drawn uniformly from 0 to 30 seconds.Source: Table 1 · p. 3, §6.1 · p. 3

6.2. Comparisons with different connectors

• Q-Former produced the lowest WER on both test sets with 80 output tokens; FC with 300 tokens achieved similar WER but used more computation and memory.FC with 75 tokens performed worse, while cross-attention had about 133.4 million parameters—around six times more than the other connectors—yet the worst WER.Source: Table 2 · p. 3, §6.2 · p. 3

6.3. Trainable queries of Q-Former

• Increasing Q-Former queries to 80 considerably reduced WER, with ∼80 tokens retaining sufficient ASR information for inputs under 30 seconds.Similar WER changes were reported for utterances close to 30 seconds in duration.Source: Table 3 · p. 3, §6.3 · p. 3

6.4. The sizes of LLMs and speech encoders

• Increasing both encoder and LLM size lowered WER, but doubling Whisper medium to large reduced WER more than doubling LLM size.The paper attributes this comparison to the greater importance of a stronger speech encoder for ASR, which requires less content understanding ability.Source: Table 4 · p. 4, §6.4 · p. 4

6.5. Results with large-scale training set

• ∼11% relative WER reduction over Whisper large-v2 averaged across three in-domain test sets, and ∼12% on the out-of-domain Eval2000 test sets.The evaluated configuration connected Vicuna 13B to Whisper large-v2 with an 80-query Q-Former and was trained on 4,000 hours of speech.Source: Table 5 · p. 4, §6.5 · p. 4

6.6. Results of segment-level Q-Former

• After fine-tuning, Seg-QF outperformed QF and FC and showed some generalization to 120-second speech, though WER degraded as input duration increased.The full test-other WER rose due to more frequent extreme cases, including repeated outputs and large deletion chunks; successfully decoded cases showed lower WER with longer contexts.Source: Table 6 · p. 4, §6.6 · p. 4

• Q-Former-connected LLMs outperformed other connector structures and the Whisper baseline on all in-domain and out-of-domain test sets.Source: §7 · p. 4
• A novel segment-level Q-Former improved performance on long-form speech inputs exceeding the pre-trained speech encoder's duration limitations.Source: §7 · p. 4
• Rich context in long-form speech can improve ASR accuracy, but overly long inputs can aggravate LLM hallucination.Source: §7 · p. 4