PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Adaptation of Encoder-Decoder Models in Speech Recognition

Yen Meng, Sharon Goldwater, Hao Tang

arXiv:2609.33865v1cs.CLcs.SDeess.AS

TL;DR

Prior ASR in-context adaptation studies have often focused on specific models or training, leaving open whether adaptation generalizes across encoder-decoder models and which demonstration properties matter. This paper evaluates collated and interleaved demonstrations across six models and finds adaptation across the tested models, with lexical and speaker similarity associated with substantial gains.

  • Problem

    Prior ASR in-context adaptation work often centers on specific models or training and does not establish which demonstration properties are most useful.

    Source: §I · p. 1

  • Method

    The paper evaluates collated and interleaved speech-text demonstrations across six encoder-decoder ASR models with varied architectures and training recipes.

    Source: §I · p. 1, §II · p. 2

  • Results

    All tested models adapt out of the box, and sharing lexical similarity and speaker identity yields 25% to 30% WER reduction across models with collated demonstration.

    Source: §I · p. 1

  • Takeaways & Limitations

    Smaller AED models adapt as effectively as larger LLM-based models, suggesting ASR in-context adaptation does not inherently depend on an LLM decoder.

    Source: §VII · p. 6

    Both demonstration approaches are constrained by model context length, although gains from using more demonstrations diminish in the experiments.

    Source: §VI · p. 6

Abstract

from arXiv · show

In-context learning offers an appealing approach to adapt automatic speech recognition (ASR) models to new speakers, accents, and domains by providing speech-text pairs as demonstrations at inference time. Recent work shows that some LLM-based speech models are capable of ASR in-context adaptation, when providing interleaved speech-text demonstrations. In this work, we ask whether in-context adaptation is an inherent ability for all encoder-decoder models. We study two forms of demonstration, collated and interleaved demonstration, across six encoder-decoder models, spanning conventional cross-attention-based and LLM-based architectures. We find that all tested models are able to perform in-context adaptation out of the box, achieving up to 30% relative improvement in the oracle experiments and up to 23% using first-pass hypotheses. Through controlled experiments on three English datasets, we show that lexical and speaker information both contribute to successful adaptation. While interleaved demonstration is effective in certain cases, collated demonstration brings consistent adaptation across the board. Our results suggest that in-context adaptation for ASR is not unique to specific architectures, training, or demonstration approaches.

I. INTRODUCTION

The paper asks whether ASR in-context adaptation extends beyond specific models and whether demonstration properties matter. Across six encoder-decoder models, it finds out-of-the-box adaptation, with collated demonstrations working across the tested models and first-pass hypotheses retaining substantial gains.

I. INTRODUCTION

  • Prior ASR adaptation studies often focus on specific models or training, and do not isolate which demonstration properties help adaptation.The paper addresses this by comparing demonstration properties across a diverse model set.Source: §I · p. 1
  • All six tested models perform in-context adaptation out of the box, without additional training.Collated demonstrations work across the models, while interleaved demonstrations work only for models inheriting in-context learning from LLMs.Source: §I · p. 1
  • The study compares collated and interleaved demonstrations for in-context ASR adaptation.Fig. 1 illustrates the two approaches for the applicable model types.Source: §I · p. 1, Figure 1 · p. 2
  • Sharing lexical similarity and speaker identity yields 25% to 30% WER reduction across models with collated demonstration.The authors report this result in controlled L2-Arctic oracle experiments.Source: §I · pp. 1–2
  • First-pass hypotheses retain sizable improvements, closing up to 60% of the gap against oracle results.The paper uses first-pass hypotheses to find demonstrations after relaxing access to the target speech’s ground-truth transcript.Source: §I · p. 2

II. COLLATED AND INTERLEAVED DEMONSTRATION

Encoder-decoder ASR models use speech-text demonstrations as decoder context, and the paper describes both collated and interleaved arrangements. Collated demonstrations place all example recordings before their transcripts and the target transcription.

II. COLLATED AND INTERLEAVED DEMONSTRATION

  • Demonstrations are paired example recordings and transcripts supplied as additional decoder context when predicting the target transcript.The models considered include cross-attention AED architectures and decoder-only architectures that still require speech encoders.Source: §II · p. 2

A. Collated demonstration

  • Collated demonstration presents all example recordings followed by their transcripts, then asks the model to complete the target transcription.For AED models, the transcripts are intended to condition the decoder to attend to the example recordings.Source: A. Collated demonstration · p. 2
  • For decoder-only models, collated demonstration uses the same speech-first ordering, with the final concatenation differing and extra text prompts sometimes required.The decoder-only sequence contains encoded recordings followed by example transcripts.Source: A. Collated demonstration · p. 2
  • Collated demonstration is framed as continued or prefix decoding: the decoder receives one long, partially transcribed recording and completes it.Prior work explored this form mostly with Whisper, while one cited LLM-based ASR application used additional instruction finetuning.Source: A. Collated demonstration · p. 2

B. Interleaved demonstration

Interleaved demonstration alternates each encoded speech example with its transcript before presenting the target speech. It can work in some LLM-based ASR models, but depends on training and prompt construction.

B. Interleaved demonstration

  • Interleaved demonstration alternates encoded speech examples with their transcripts, then adds the target speech for decoding.Text prompts are almost always needed when the model was not explicitly trained on that exact interleaved task input.Source: B. Interleaved demonstration · p. 3
  • Interleaved demonstration has been effective in certain LLM-based ASR models, but its performance depends heavily on training and text-prompt construction.Models not intended for interleaved inputs are not meant to be used this way unless explicitly trained for it.Source: B. Interleaved demonstration · p. 3

III. EXPERIMENTAL SETUP

The experiments test whether six selected models adapt in context and which demonstration properties matter, using controlled settings across three datasets. Evaluation uses WER with normalized hypotheses and references.

III. EXPERIMENTAL SETUP

  • The experimental design samples six models and controls phonetic, lexical, and speaker identity properties across three datasets to study adaptation.The stated questions are whether the selected models adapt in context and which demonstration properties most affect adaptation.Source: §III · p. 3
  • WER is evaluated after hypotheses and references are normalized with the Whisper normalizer for consistent comparison.To stabilize collated generation, the first-step EOS token is suppressed so each model produces at least one target-audio output token.Source: §III · p. 3

A. Models and Data

The study samples six open-source ASR models and uses L2-Arctic, LibriSpeech, and AMI for controlled comparisons. Demonstrations vary in phonetic, lexical, and speaker similarity, with oracle and first-pass transcript selection considered.

A. Models and Data

  • Six open-source models span attention-based encoder-decoder and decoder-only LLM-based architectures, with some models marked as intended for interleaved ICL.Canary-Qwen and Qwen3-ASR are not intended for ICL, whereas Phi-4-MM and Qwen2.5-Omni are potentially designed with ICL in mind.Source: A. Models and Data · p. 3, Table I · p. 3
  • L2-Arctic supports controlled speaker and accent comparisons, LibriSpeech supports lexical-similarity studies, and AMI tests generalization beyond read speech.L2-Arctic contains phonetically balanced sentences read by multiple non-native English speakers.Source: A. Models and Data · p. 3

B. Demonstration settings

  • Eight demonstration settings explore phonetic, lexical, and speaker similarity, including Random and Gold text extremes and similarity-ranked examples.Phonetic similarity uses TF-IDF phone trigrams, lexical similarity uses TF-IDF words excluding stopwords, and demonstrations are ordered with the closest example last.Source: B. Demonstration settings · p. 3
  • Phonetic and Lexical selection are also tested with first-pass transcripts, relaxing their ground-truth-target-transcript assumption; oracle experiments assess adaptation capability and demonstration effects.The study also reports WERs for separating demonstrations and target speech with silence in Gold text on L2-Arctic, probing potential long-form audio limits.Source: B. Demonstration settings · p. 4, Figure 2 · p. 4

IV. RESULTS

The results section first tests whether models can ignore irrelevant demonstrations and copy correct transcripts, then compares adaptation across demonstration types and datasets.

IV. RESULTS

  • The experiments begin with random and gold demonstrations as corner cases for testing whether models can ignore irrelevant examples and copy a supplied transcript.The section then compares phonetic and lexical settings and examines generalization to other datasets.Source: §IV · p. 4
  • A comprehensive comparison adds phonetic and lexical demonstration settings to the initial corner-case tests.The paper also studies whether adaptation generalizes to other datasets.Source: §IV · p. 4
  • The results are intended to address whether the selected models adapt in context and which demonstration properties matter most.Source: §IV · p. 4

A. In-context adaptation in corner cases

All tested models pass the corner-case tests, while controlled settings show that speaker identity and lexical or phonetic relevance shape adaptation. More demonstrations generally lower WER, with diminishing returns after the first two.

A. In-context adaptation in corner cases

  • All models pass both corner-case tests: random demonstrations yield WERs close to no demonstrations, while gold demonstrations produce near-perfect WERs.The tests assess ignoring irrelevant examples and copying ground truth.Source: A. In-context adaptation in corner cases · p. 4
  • Canary and Phi-4-MM adapt only within certain context lengths when silence is inserted, so later experiments fit demonstrations within those limits.Source: A. In-context adaptation in corner cases · p. 4

B. Impact of different demonstration settings

  • Except for random demonstrations from different speakers, the tested demonstration settings improve WER; Gold-text demonstrations yield near-perfect WERs.Gold text is unrealistic, and its WER is slightly higher when the demonstrated speakers differ from the target speaker.Source: B. Impact of different demonstration… · p. 4, Figure 3 · p. 5
  • Among settings without duplicate target text, same-speaker random examples give small but consistent gains, while similar lexical or phonetic examples improve most across models.Lexical similarity is slightly better, especially when examples share the target speaker.Source: B. Impact of different demonstration… · p. 4
  • Whisper and Canary-1B achieve relative WER reductions similar to larger models with pretrained LLM decoders, suggesting adaptation does not inherently require an LLM.Source: B. Impact of different demonstration… · p. 4
  • From one to four demonstrations, WER falls overall, with the first two examples bringing the largest improvement and later additions diminishing in return.The comparison covers Random and Lexical settings.Source: B. Impact of different demonstration… · p. 4, Figure 4 · p. 5

C. Comparing collated and interleaved demonstration

Collated and interleaved demonstrations both enable adaptation for some LLM-based models, but collated demonstrations work across all tested models without prompt changes.

C. Comparing collated and interleaved demonstration

D. In-context adaptation on different domains

Across LibriSpeech and AMI, same-speaker lexical demonstrations remain the strongest strategy, though adaptation is weaker than on L2-Arctic. A second pass using first-pass transcripts for selection yields further gains over random selection.

D. In-context adaptation on different domains

V. SECOND-PASS IN-CONTEXT ADAPTATION

  • The second-pass method selects same-speaker, lexically similar demonstrations using first-pass ASR transcripts, while retaining gold transcripts as demonstration text.The example transcript is excluded from TF-IDF similarity computation.Source: §V · pp. 5–6
  • 3% to 12% relative WER reduction over Random closes 25% to 60% of the oracle gap in second-pass L2-Arctic experiments.The comparison uses four same-speaker demonstrations, with first-pass transcript selection contrasted against oracle selection using ground-truth transcripts.Source: §V · p. 6, Table III · p. 6

VI. DISCUSSION

Collated demonstrations frame adaptation as a single ASR task, whereas interleaved demonstrations require interpreting text instructions and the interleaving format. Collated demonstrations are more broadly applicable and more consistent across datasets, while both approaches remain constrained by model context length.

  • Collated demonstrations treat adaptation as one ASR task; interleaved demonstrations additionally require understanding text instructions and the interleaving format.Source: §VI · p. 6
  • Interleaved demonstrations require specific prompt wording and vary more across datasets, while collated demonstrations are broadly applicable to encoder-decoder ASR systems.Source: §VI · p. 6
  • Both approaches are limited by model context length, although gains from adding demonstrations diminish in the experiments.ASR systems are also known to suffer on long-form audio or audio beyond training length.Source: §VI · p. 6

VII. CONCLUSION

The conclusion reports in-context adaptation gains across diverse models and settings without additional training, with oracle experiments showing capability across all tested models. It also highlights practical first-pass performance, differences in demonstration applicability, and future research directions.

VII. CONCLUSION

  • In-context adaptation achieves significant gains across diverse models and settings without any training.Source: §VII · p. 6
  • Oracle experiments show that all tested models can adapt, with smaller AED models adapting as effectively as larger LLM-based models.Source: §VII · p. 6
  • Collated demonstrations apply to any encoder-decoder ASR system, whereas interleaved demonstrations require specific text prompts and suit only certain models.Source: §VII · p. 6
  • First-pass hypotheses retain a large proportion of the improvements seen with oracle results; future work includes mechanism studies, demonstration retrieval, and broader ASR settings.Source: §VII · p. 6

VIII. GENERATIVE AI USE DISCLOSURE

  • Claude was used only for language and grammar polishing; the authors produced the technical content, experimental design, and results.Source: §VIII · p. 7

Related papers

Loading 2609.33865v1…