PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-only adaptation in LLM-based ASR through text denoising

Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar, Kadri Hacioglu, Srikanth Madikeri, Pradeep Rangappa, Manjunath K E, Petr Motlicek, Shankar Venkatesan, Andreas Stolcke

arXiv:2601.20900v3cs.SDcs.CLcs.LGeess.AS

TL;DR

Text-only adaptation is attractive for LLM-based ASR because paired audio–text data can be scarce or expensive, yet adapting to new domains remains challenging. The paper trains the LLM to denoise target-domain text, recovering clean transcripts from noisy inputs. It reports up to 22.1% relative improvement, surpassing state-of-the-art text-only adaptation methods.

  • Problem

    Text-only adaptation is attractive because paired audio–text data can be scarce or expensive, but adapting LLM-based ASR to new domains remains challenging.

    Source: §1 · p. 1

  • Method

    The method adapts LLM-based ASR by training the LLM to denoise target-domain text and recover clean transcripts from noisy inputs.

    Source: §2 · p. 2

  • Results

    Up to 22.1% relative improvement, surpassing state-of-the-art text-only adaptation methods.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The approach preserves the model’s ability to interpret audio-conditioned inputs while acquiring target-domain knowledge.

    Source: §6 · p. 4

    Experiments use ground-truth transcripts from conversational speech, rather than arbitrary text such as web-page text.

    Source: §2.2 · p. 2

Abstract

from arXiv · show

Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on the target domain text often disrupts the critical alignment between the speech and text modality learned by the projector, degrading performance. We introduce a novel text-only adaptation method that frames this process as a text denoising task. Our approach trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.

1. INTRODUCTION

The introduction motivates text-only domain adaptation as a response to scarce paired audio–text data and proposes denoising as a way to adapt LLM-based ASR while preserving cross-modal alignment.

  • The projector maps speech representations into the LLM text-embedding space, where the LLM treats them as noisy text and reconstructs clean transcripts.Source: §1 · p. 1
  • Text-only adaptation is a practical alternative because paired audio–text data are scarce or costly, while domain mismatch can degrade recognition.Source: §1 · p. 1
  • Fang et al. monitor alignment, but degradation can persist; Ma et al. use fixed soft prompts after optimization and require hyperparameter tuning.Source: §1 · p. 1
  • Across two datasets, the proposed text-only adaptation achieves up to 22.1% relative improvement and surpasses the state-of-the-art.Source: §1 · p. 1
  • The method trains the LLM to reconstruct inputs mimicking speech-projector outputs through multi-view noise-driven batching, without additional learnable parameters.Source: §1 · p. 1

2. METHOD

The method adapts an LLM-based ASR system using target-domain transcripts alone, framing adaptation as denoising noisy text-like inputs while mixing source and target examples to preserve speech–text alignment.

2. METHOD

  • The LLM is trained to reconstruct clean transcripts from noisy, text-like sequences produced by the speech projector, motivating text-only domain adaptation through denoising.The projector’s outputs can resemble corrupted transcripts, which the LLM learns to interpret and recover.Source: §2 · p. 2

2.1. Task Formulation

  • Adaptation replaces unavailable target-domain audio with perturbed transcripts and trains the LLM on (noise(t), t) pairs to recover the clean text.The target data comprises ground-truth transcripts from conversational speech, not arbitrary text such as web pages.Source: §2.1 · p. 2, §2.2 · p. 2

2.2. Batch Construction for Text Denoising Adaptation

  • Each batch mixes source audio–transcript pairs, projector-induced noisy transcripts, synthetic character-noised source transcripts, and synthetic-noised target transcripts.Projector-induced noise comes from mapping projected source audio to nearest LLM-vocabulary tokens; synthetic noise uses random character substitutions and duplications. The four components respectively preserve alignment, bridge audio and text-noise views, and drive target-domain adaptation.Source: §2.2 · p. 2, Figure 1 · p. 2
  • A small nonzero source-audio share helps prevent catastrophic forgetting, while the target share controls adaptation strength and is set proportional to target-set size.The remaining source shares are distributed equally; mixing the three source views is intended to retain transcription ability while adapting without target-domain audio.Source: §2.2 · p. 2

3. DATASETS PREPARATION

The experiments use realistic conversational corpora and construct source audio–transcript and target text-only partitions across selected domains. SlideSpeech domains were chosen to provide enough data for training and development/test splits.

3. DATASETS PREPARATION

  • DefinedAI4 contributes 125 hours of manually transcribed customer–agent telephone calls, while SlideSpeech comprises 1,705 online conference videos across 22 domains, totaling 1,000 hours, with 473 hours transcribed.Both corpora were selected for explicit domain grouping and relevance to real-world ASR; DefinedAI4’s conversational data includes disfluencies, interruptions, and colloquial expressions.Source: §3 · p. 3
  • DefinedAI source data pairs audio with transcripts across Banking, Insurance, and Healthcare, while its Banking and Insurance target partitions are text-only.For SlideSpeech, source domains are Life, Talent, and English, while target domains are Agriculture, Animation, and Musical Instruments; target training data is text-only.Source: §3 · p. 3
  • SlideSpeech domains were ranked by perplexity, and only training-set domains with enough examples for training and development/test partitions were selected.The original development and test partitions contained approximately one hour per domain, limiting their usefulness; Table 1 presents dataset composition across training, development, and test splits.Source: §3 · p. 3

4. EXPERIMENTAL SETUP

Experiments used SLAM-ASR with a WavLM-Large speech encoder, Llama 3.2 3B Instruct decoder, and a learnable projector. The setup compared a source-trained base model, audio-based adaptation, the proposed text-only method, and two reimplemented text-only approaches, evaluating WER and relative improvement over the base model.

EXPERIMENTAL SETUP

  • Experiments used SLAM-ASR [3] with WavLM-Large, Llama 3.2 3B Instruct5, and a single-hidden-layer ReLU/regression projector.The speech encoder and decoder LLM were connected through the learnable projector.Source: §4 · p. 3
  • The shared prompt requested speech transcription, with inputs comprising audio, noise(audio), or noise(text) paired with the ground-truth transcript.The prompt output was always the ground-truth transcript.Source: §4 · p. 3
  • The base model trained its projector on source audio-text pairs for four epochs, with the encoder and LLM frozen, then was evaluated on the target test split.Training used a learning rate of 1×10−4, 1000 warm-up steps, and batch size 4.Source: §4 · p. 3
  • The proposed text-only adaptation trains on target transcripts corrupted by random character substitutions and duplications.Substitutions affected 30% of characters in 15% of words, with 1–10 edits per utterance; each character had a 10% chance of being duplicated 1–3 times. The substitution rate was reduced from the nlpaug default after preliminary experiments, and duplication was designed to emulate noise(audio).Source: §4 · p. 3
  • The audio-adapted comparison fine-tuned the LLM on target audio-text pairs using LoRA on self-attention query and value projections.Fine-tuning ran for four epochs at learning rate 1×10−4, with 1000 warm-up steps and LoRA rank 8.Source: §4 · p. 3
  • The reimplemented alternatives were Ma et al., using 30 learned audio-slot embeddings before LLM fine-tuning, and Fang et al., selecting a text-fine-tuned checkpoint before validation perplexity rose.Fang et al. monitored validation perplexity with audio every 200 steps.Source: §4 · p. 3
  • All experiments reported word error rate (WER) and relative improvement (∆) over the corresponding base model.Source: §4 · p. 3

5. EXPERIMENTAL RESULTS

The experiments evaluate text-only adaptation in in-domain, out-of-domain, and cross-domain settings. The method improves over the base model in the cross-domain setting, though it remains below audio adaptation, and ablations examine batch components and text denoising.

5. EXPERIMENTAL RESULTS

  • 10.11% vs. 9.92% in Banking and 8.71% vs. 7.92% in Insurance show text-only adaptation approaching audio-based adaptation in-domain.Source and target partitions are both drawn from DefinedAI, representing a familiar domain with similar speech and acoustic characteristics.Source: §5 · p. 4
  • Text-only adaptation improves WER in two of three out-of-domain target domains, with gains described as modest where τ values are below 0.25 in MI.The target domains are absent from source data but share its speech and acoustic characteristics; the passage suggests larger τ could yield further gains.Source: §5 · p. 4, Table 3 · p. 4
  • Text-only adaptation improves over the base model and performs comparably to Ma et al. in cross-domain adaptation, but remains well below audio adaptation.This setting combines domains with different lexical and acoustic characteristics; audio adaptation also uses target-domain audio to address the acoustic mismatch.Source: §5 · p. 4, Table 4 · p. 4

5.1. Ablation experiments

  • Using all three batch components (σa, σta, σt) yields the best performance, while omitting audio (σa = 0) sharply increases WER.The authors suggest the increase may result from catastrophic forgetting of speech-text alignment.Source: §5.1 · p. 4, Table 5 · p. 4
  • Syntactic noise as LLM input improves performance more effectively than unperturbed target-domain text, supporting the text-denoising formulation.The authors say denoising helps the model capture target-domain lexical and syntactic patterns.Source: §5.1 · p. 4, Table 6 · p. 4

6. CONCLUSIONS

The paper presents a text-only adaptation method for LLM-based ASR that mitigates catastrophic forgetting while adapting to target-domain text. It reports consistent gains over prior text-only adaptation strategies and proposes future evaluation of noise functions and τ.

  • The method consistently improves performance across multiple domains and datasets, surpassing prior state-of-the-art text-only adaptation strategies.Source: §6 · p. 4
  • Fine-tuning alternates source audio, projector-induced text noise, and synthetic noisy text to preserve audio-conditioned interpretation, learn text denoising, and acquire target-domain knowledge.Source: §6 · p. 4
  • Future work will explore noise functions that better approximate projection-layer outputs and thoroughly evaluate τ under text-rich, real-world conditions.Source: §6 · p. 4
Loading 2601.20900v3…