PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren, Abeer Alwan

arXiv:2609.17981v2eess.AScs.CL

TL;DR

Speech-LLMs can perform poorly on acoustically shifted speech such as child or dialectal speech, and adaptation is complicated by limited target-domain data. EAVA trains encoder adapters first and then jointly fine-tunes the model; experiments report consistent gains over baselines across evaluated datasets.

  • Problem

    Speech-LLMs often perform poorly on acoustically shifted child or dialectal speech, while limited target-domain data complicates adaptation and the encoder may be insufficiently adapted.

    Source: §I · p. 1

  • Method

    EAVA first trains lightweight adapters in each encoder layer with other components frozen, then jointly fine-tunes the encoder, adapters, projector, and LLM LoRA modules.

    Source: §I · p. 1, A. Stage 1: Encoder Awakening · p. 2

  • Results

    EAVA consistently outperforms all baselines across evaluated datasets; with 64-dim adapters, test WER is 9.05% on OGI.

    Source: A. Main Results and Analysis · p. 4

  • Takeaways & Limitations

    Different adapter architectures are effective within EAVA, while larger adapters do not yield consistent improvements.

    Source: A. Main Results and Analysis · p. 4, §VI · p. 6

Abstract

from arXiv · show

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.

I. INTRODUCTION

Speech-LLMs pair a pretrained speech encoder and modality projector with an LLM, achieving strong general-domain ASR. Their adaptation to acoustically shifted speech is challenging, motivating EAVA’s two-stage approach, which reports consistent gains across evaluated datasets and backbones.

A. Speech Large Language Models for ASR

  • A standard Speech-LLM for ASR combines a pretrained speech encoder, a modality projector, and an LLM commonly fine-tuned with LoRA.The encoder produces acoustic representations, the projector maps them into the LLM token-embedding space, and the LLM generates text autoregressively.Source: §I · p. 1, A. Speech Large Language Models for ASR · p. 2

I. INTRODUCTION

  • Domain-shifted speech, including child and dialectal speech, challenges pretrained Speech-LLMs because of acoustic mismatch; limited target-domain data further complicates adaptation.The LLM may dominate while cross-entropy is applied only at its output, leaving the encoder insufficiently adapted; aggressive encoder training can also risk losing pretrained knowledge.Source: §I · p. 1
  • EAVA reports consistent gains over baselines on child and dialectal speech datasets and across different Speech-LLM backbones.Its two stages first train encoder adapters while freezing other components, then jointly fine-tune the adapters, projector, and LLM LoRA modules.Source: §I · p. 1

A. Speech Large Language Models for ASR

  • Open-source examples include Canary-Qwen and Phi-4-Multimodal, which are pretrained on paired speech-transcription data and perform strongly on general-domain ASR.Canary-Qwen pairs a FastConformer encoder with Qwen3-1.7B; Phi-4-Multimodal integrates speech understanding into Phi-4-Mini.Source: A. Speech Large Language Models for ASR · p. 2

B. Problem Formulation

The problem is adapting a Speech-LLM pretrained on general-domain speech to a target domain with acoustic shift. The goal is improved ASR performance on that target domain.

C. Previous Fine-tuning Strategies

Prior approaches include vanilla fine-tuning and multi-stage alignment, but each has limitations for adapting a well-trained Speech-LLM to shifted speech. EAVA is proposed as a domain-adaptive alternative.

1) Vanilla Fine-tuning:

  • Vanilla fine-tuning jointly updates the encoder, projector, and LLM LoRA on target-domain data using cross-entropy, but struggles to balance adaptation and preservation under limited data.The standard Speech-LLM generates transcription autoregressively from embedded speech representations and a prompt.Source: 1) Vanilla Fine-tuning · p. 2, Figure 1 · p. 2

2) Multi-stage Alignment:

  • Multi-stage alignment seeks to align speech and LLM representations, but prior work targets building a Speech-LLM from a raw encoder rather than adapting a well-trained model to a new domain.Source: 2) Multi-stage Alignment · p. 2

III. METHOD

  • EAVA is proposed to address the underexplored problem of domain-adaptive fine-tuning for Speech-LLMs.The method consists of two stages.Source: §III · p. 2

A. Stage 1: Encoder Awakening

EAVA first trains lightweight adapters in each encoder layer while freezing pretrained parameters, then jointly fine-tunes the encoder, adapters, projector, and LLM LoRA modules. This sequence aims to incorporate target-domain acoustic knowledge while preserving pretrained knowledge before refining the model for target-domain ASR.

A. Stage 1: Encoder Awakening

  • Stage 1 inserts randomly initialized lightweight adapters into every encoder layer and trains only those adapters on target-domain data, keeping pretrained parameters frozen.The stage is intended to inject target-domain knowledge into the encoder while preserving its pretrained knowledge.Source: A. Stage 1: Encoder Awakening · p. 2
  • The default Residual Adapter uses two feed-forward layers with a residual connection, layer normalization, and a Swish activation.Its bottleneck dimension is the adapter dimension; only the down- and up-projection weights are updated in Stage 1.Source: A. Stage 1: Encoder Awakening · p. 2

B. Stage 2: Continual Fine-tuning

  • Stage 2 jointly fine-tunes the encoder adapters, pretrained encoder, projector, and LLM LoRA modules on target-domain data.The domain-aware initialization from Stage 1 is intended to support further adaptation under the same ASR objective.Source: B. Stage 2: Continual Fine-tuning · pp. 2–3, Figure 2 · p. 3

IV. EXPERIMENTAL SETTINGS

The experiments evaluate domain-shifted child and dialectal speech using three datasets with specified train, development, and test splits.

A. Datasets

  • OGI uses spontaneous classroom responses from children aged 4–15, split into 22/2/7 hours for train, development, and test.Transcriptions preserve disfluencies.Source: A. Datasets · p. 3
  • MyST contains dialogues between elementary students aged 8–10 and virtual tutors, split into 133/21/25 hours for train, development, and test.Source: A. Datasets · p. 3
  • CORAAL uses six African American Language subsets for 137 training hours, with ROC and VLD held out for development and testing.The held-out sets are speaker- and region-disjoint; utterances retain interviewee speech and are capped at 30 seconds.Source: A. Datasets · p. 3

B. Model Settings

Experiments use Canary-Qwen as the primary backbone and compare target-domain adaptation against zero-shot, vanilla fine-tuning, and multi-stage alignment baselines.

B. Model Settings

  • Canary-Qwen pairs an 810M-parameter FastConformer encoder with a Qwen3-1.7B LLM; Phi-4-Multimodal is also used to test generalizability.The default adapter dimension is 64, with 32, 128, 256, and 512 also evaluated.Source: B. Model Settings · p. 3
  • Training uses five epochs per stage, development-set checkpoint selection, and Word Error Rate (WER) evaluation.Adapters use a 1e-3 peak learning rate during Encoder Awakening and 1e-4 during Continual Fine-tuning; training runs on one NVIDIA A6000 GPU with effective batch size 16.Source: B. Model Settings · p. 3

C. Baselines

  • Zero-shot evaluates the backbone without target-domain fine-tuning, while vanilla fine-tuning jointly trains the encoder, projector, and LLM LoRA.Source: C. Baselines · p. 4
  • Multi-stage alignment sequentially tunes model components; vanilla and alignment baselines receive the same total epoch budget and use the best development-set checkpoint.Previous SOTA results use standard training data and processing, with additional-unlabeled-data results reported for reference.Source: C. Baselines · pp. 3–4

V. EXPERIMENTAL RESULTS

EAVA improves ASR across the tested domain-shifted datasets, while encoder analysis shows targeted changes concentrated in upper layers.

A. Main Results and Analysis

  • Vanilla fine-tuning improves over zero-shot on all three datasets, but gains are limited; multi-stage alignment helps OGI marginally and degrades the other two datasets.Source: A. Main Results and Analysis · p. 4
  • EAVA outperforms all baselines across evaluated adapter dimensions and datasets, achieving new state-of-the-art results even against prior work using additional unlabeled data.The stable 64-dimension configuration reports test WERs of 9.05% on OGI, 8.04% on MyST, and 8.54% on CORAAL.Source: A. Main Results and Analysis · p. 4
  • On OGI, vanilla fine-tuning has CKA values of 0.92–0.99, while EAVA preserves lower layers 0–10 at CKA > 0.97 and shifts upper layers 23–31 as low as 0.82.Lower CKA indicates greater representational change relative to the zero-shot encoder.Source: A. Main Results and Analysis · pp. 4–5, Figure 3 · p. 4

B. Ablation Study of Training Procedure

The ablations support adapter-only Encoder Awakening followed by joint fine-tuning, rather than simultaneous early tuning or full encoder tuning.

B. Ablation Study of Training Procedure

  • Skipping Stage 2 after adapter-only Stage 1 yields 9.16% WER on OGI, while joint optimization in the proposed procedure performs best.Fine-tuning only a subset of Stage 2 components also worsens results, supporting complementary refinement from joint optimization.Source: B. Ablation Study of Training Procedure · p. 5
  • Training the projector and LoRA alongside adapters in Stage 1 degrades performance, as does removing the dedicated Encoder Awakening stage.The reported explanation is that simultaneous module optimization divides training focus and is less effective at targeted knowledge injection.Source: B. Ablation Study of Training Procedure · p. 5
  • Replacing Stage 1 adapters with full encoder fine-tuning causes a substantial performance drop because it fails to preserve pre-trained encoder knowledge.Source: B. Ablation Study of Training Procedure · p. 5

C. Effect of Adapter Choice

EAVA's gains are robust to adapter architecture, and Encoder Awakening generally transfers across datasets, with matched Stage 1 data usually strongest.

C. Effect of Adapter Choice

  • All three adapter variants outperform vanilla fine-tuning on all three datasets, with small differences and the Residual Adapter strongest on OGI.The comparison uses variants with approximately equal parameter counts of 4.2–4.3M.Source: C. Effect of Adapter Choice · p. 5

D. Cross-Dataset Transfer Analysis

  • Matched target-domain data for Stage 1 generally yields the strongest results, while mismatched data outperforms vanilla fine-tuning in most cases.The analysis evaluates test WER after Stage 2 is run separately on each target dataset.Source: D. Cross-Dataset Transfer Analysis · p. 6, Table IV · p. 6

E. Effect of Loss Function in the Encoder Awakening Stage

EAVA outperforms baselines across three datasets with Phi-4-Multimodal, though its gains vary by backbone and dataset. In Encoder Awakening, default LLM CE performs better overall than CTC loss.

E. Effect of Loss Function in the Encoder Awakening Stage

  • CTC loss still beats vanilla fine-tuning on OGI and MyST, but default LLM CE achieves better overall performance and consistency.The authors relate CE's advantage to alignment with the Stage 2 fine-tuning objective.Source: E. Effect of Loss Function in the… · p. 6

F. Generalizability to Other Speech-LLMs

VI. CONCLUSIONS

EAVA combines encoder adapters trained in an initial awakening stage with subsequent whole-model target-domain fine-tuning. Across datasets and backbones, it outperforms baselines, including prior work using additional unlabeled data.

VI. CONCLUSIONS

  • EAVA first trains lightweight encoder adapters to inject target-domain knowledge while preserving pre-trained knowledge, then fine-tunes the whole model.The conclusion also reports that larger adapters do not yield consistent improvements.Source: §VI · p. 6
  • EAVA consistently outperforms baselines across datasets and Speech-LLM backbones, achieving new state-of-the-art results on all three domain-shifted datasets.The reported results surpass prior works that use additional unlabeled training data.Source: §VI · p. 6

VIII. GENERATIVE AI USE DISCLOSURE

  • The authors report using Chat-GPT (GPT-5.5) for language editing, while technical content, experimental design, results, and conclusions were produced and verified by the authors.The authors state they reviewed and edited the manuscript and take full responsibility for its content.Source: §VIII · p. 7
Loading 2609.17981v2…