Source-linked AI summary
A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR
Van Tung Pham, Yist Lin, Tao Han, Wei Li, Jun Zhang, Lu Lu, Yuxuan Wang
TL;DR
LLM-based ASR still has limited fine-tuning options, no explicit mechanism for speech-text alignment, and insertion errors in low-resource or mismatched settings. This paper studies fine-tuning choices, adds a matching loss, and tests training and inference techniques. The reported findings favor parameter-efficient LoRa tuning and show improved alignment and reduced insertion errors, especially out of domain.
Problem
LLM-based ASR has limited fine-tuning options, lacks an explicit mechanism to align speech and text representations, and can produce high insertion errors with limited training data or mismatched tests.
Source: §3.1 · p. 2, §3.3 · p. 2Method
The paper studies module-specific fine-tuning choices, adds a cross-attention-based matching loss, and tests decoding constraints, acoustic augmentation, and non-speech training data.
Source: §1 · p. 1Results
Parameter-efficient LoRa tuning on both encoder and LLM is the most cost-effective strategy; matching loss improves ASR performance, and insertion errors fall substantially, especially on out-of-domain tests.
Source: §6 · p. 4Takeaways & Limitations
LoRa on both the encoder and LLM offers the paper's most cost-effective fine-tuning strategy.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Recent works have shown promising results in connecting speech encoders to large language models (LLMs) for speech recognition. However, several limitations persist, including limited fine-tuning options, a lack of mechanisms to enforce speech-text alignment, and high insertion errors especially in domain mismatch conditions. This paper presents a comprehensive solution to address these issues. We begin by investigating more thoughtful fine-tuning schemes. Next, we propose a matching loss to enhance alignment between modalities. Finally, we explore training and inference methods to mitigate high insertion errors. Experimental results on the Librispeech corpus demonstrate that partially fine-tuning the encoder and LLM using parameter-efficient methods, such as LoRA, is the most cost-effective approach. Additionally, the matching loss improves modality alignment, enhancing performance. The proposed training and inference methods significantly reduce insertion errors.
1. Introduction
The paper connects a speech foundation model to an LLM for ASR, addressing limited fine-tuning choices, missing explicit modality alignment, and insertion errors under low-resource or mismatched conditions.
1. Introduction
- The work aims to connect speech and language foundation models, motivated by their advances in speech processing, vision, and natural language processing.The stated task is speech recognition.Source: §1 · p. 1
- Prior LLM-based ASR work explored speech compression, adapters, and selective LLM tuning, but left fine-tuning configurations limited, explicit alignment absent, and insertion errors under low-resource or mismatched conditions understudied.The architecture generally combines a speech encoder, adapter, and LLM.Source: §1 · p. 1
- The paper studies more efficient fine-tuning schemes, proposes a matching loss for modality alignment, and explores training and inference strategies to reduce insertion errors.The inference strategies include n-gram non-repetition and length-penalty constraints; training uses non-speech audio with empty transcripts and speed and volume perturbations.Source: §1 · p. 1
2. Related works
Prior work connects modalities to LLMs through discrete audio tokens or continuous speech-encoder representations; this paper follows the continuous-representation approach for ASR.
2. Related works
- LLM-based audio research covers ASR, speech translation, and audio event understanding, with approaches using either discrete audio tokens or continuous encoder representations.The paper follows the continuous-representation paradigm for ASR.Source: §2 · p. 1
- Discrete-token approaches can generate speech outputs but may lose information through discretization, which may make them less suitable for speech-to-text tasks.The paper focuses on ASR and connects continuous pretrained audio representations to the LLM.Source: §2 · p. 1
- LoRa uses low-rank matrices to reduce training memory without increasing inference time, addressing costs associated with adapting large LLMs.Adapters and prefix embeddings are parameter-efficient alternatives, but increase inference costs.Source: §2 · p. 1
3. Proposed solution
The proposed solution examines fine-tuning choices, explicitly aligns acoustic and text representations with a matching loss, and combines decoding constraints with training methods to reduce insertion errors.
3.1. Fine-tuning schemes for LLM-based ASR
- The fine-tuning study compares options for the LLM, encoder, and adapter, including frozen or LoRa-tuned LLMs and frozen, partially LoRa-tuned, or fully tuned encoders.Adapter variants include Conv1dMLP, a lower-parameter depthwise-separable DwsMLP, and the higher-capacity Conv1dTransformer.Source: §3.1 · p. 2
3.2. Improving modalities alignment by matching loss
- The matching loss uses cross-attention to give acoustic representations the text sequence length, then applies MSE or cosine distance to align them with text embeddings.The model combines this loss with cross-entropy during training.Source: §3.2 · p. 2
3.3. Inference and training methods to reduce insertion errors
- Insertion errors are especially pronounced with 10h training data and out-of-domain tests, where models may repeat n-grams until reaching the output-length limit.Inference constraints include preventing repeated n-grams and penalizing long transcripts during beam search.Source: §3.3 · p. 2
- Training methods include speed and volume perturbations and short fine-tuning on non-speech audio segments paired with empty transcripts.The authors report that adding random noise or music to training utterances negatively impacts performance.Source: §3.3 · p. 2
4. Experimental setup
Experiments use LibriSpeech training subsets and in-domain and out-of-domain evaluation sets, with HuBERT and Vicuna models, several adapter configurations, and checkpoint averaging.
4.1. Dataset
- Models are trained on LibriSpeech's 960h, 100h, and 10h datasets and evaluated by WER on four LibriSpeech dev/test sets and two out-of-domain sets.The out-of-domain sets are CoVoST2 and GigaSpeech; Musan noise and music subsets supply non-speech audio for NSET training.Source: §4.1 · pp. 2–3
4.2. Model settings
- The setup pairs a 300M-parameter HuBERT-large speech encoder with a 7B-parameter Vicuna LLM and considers three adapter architectures.Conv1dMLP has 48M parameters, DwsMLP 20M, and Conv1dTransformer 320M; encoder and LLM LoRa configurations use 0.65M and 16M parameters, respectively.Source: §4.2 · p. 3
4.3. Training and inference setting
- Training runs for 50k, 20k, and 10k steps on the 960h, 100h, and 10h datasets, respectively, and averages five consecutive checkpoints selected by validation loss.Checkpoint saving intervals vary by dataset size.Source: §4.3 · p. 3
5. Experimental results and analysis
Across the experiments, inference constraints and training methods reduce insertion errors, particularly on out-of-domain data; LoRA fine-tuning and matching loss improve performance and support the selected system configurations.
5.1. Results of different fine-tuning scheme
- LoRA fine-tuning of both the encoder and LLM with the Conv1dMLP adapter (S4) offers the reported optimal balance of performance and added parameters.Partial encoder fine-tuning in S4 improves over frozen S2 on 960h dev-clean/other, reducing WER from 2.01%/4.71% to 1.70%/3.59% with 0.65M extra parameters.Source: §5.1 · p. 3, Table 2 · p. 4
5.2. Results of matching loss
- Matching loss improves performance across most conditions, with larger gains for S1 than S4 and on dev/test-other than dev/test-clean.The authors suggest LoRA may already help align acoustic and text representations, reducing the additional benefit of matching loss for S4.Source: §5.2 · p. 3, §5.3 · p. 4
5.3. Results of inference and training techniques
- NRNS = 10 reduces dev-other IER from 1.88% to 0.89% and WER from 10.35% to 9.35% in the 10h condition.Combining NRNS and LP adds no benefit; the reported optimal setting is NRNS = 10, LP = 0.Source: §5.3 · pp. 3–4, Table 3 · p. 4
- 33.06% to 14.52% IER and 46.42% to 27.83% WER on GigaSpeech with the 960h model and constraints.The authors report IER reductions on out-of-domain test sets for both 960h and 10h conditions.Source: §5.3 · p. 4, Table 4 · p. 4
- DA improves performance across conditions, while DA+NSET generally yields better WER and suppresses insertion errors despite not always minimizing IER.NSET can increase deletion errors and WER, particularly with 10h training; DA also reduces deletion and substitution errors, especially in that condition.Source: §5.3 · p. 4, Table 5 · p. 4
6. Conclusion
The paper presents a systematic speech-encoder-to-LLM approach for ASR, combining parameter-efficient fine-tuning, matching loss, and methods to mitigate insertion errors. It reports reduced insertion errors and improved ASR performance, especially on out-of-domain test sets.
6. Conclusion
- The conclusion identifies LoRA fine-tuning of both encoder and LLM as the most cost-effective strategy.The paper also presents matching loss for modality alignment and training and inference methods to mitigate insertion errors.Source: §6 · p. 4
- Matching loss enhances modality alignment and improves ASR performance.Source: §6 · p. 4
- The explored training and inference methods reduce insertion errors and improve ASR performance, especially on out-of-domain test sets.Source: §6 · p. 4