PaperlayerView exact revision on arXiv

Source-linked AI summary

In-context Language Learning for Endangered Languages in Speech Recognition

Zhaolin Li, Jan Niehues

arXiv:2505.20445v5cs.CLcs.AI

TL;DR

Language information remains an underexplored part of ASR for endangered languages, and ICL is less effective when low-resource languages are poorly represented in pretraining. This paper applies in-context language learning to unseen languages and finds that selected examples support strong language-modeling and ASR results.

  • Problem

    How to apply LLMs to ASR for endangered languages remains unclear, while language information is underexplored and low-resource ICL is weakened by sparse pretraining representation.

    Source: §1 · p. 1

  • Method

    The authors provide target-language examples through ICL and investigate sample selection for language modeling and re-ranking ASR hypotheses.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    Across four endangered languages, ICL improves language-model perplexity over strong baselines and matches or surpasses dedicated language models on ASR.

    Source: §5 · p. 4, §4.3 · p. 4

  • Takeaways & Limitations

    ICL can achieve strong language-modeling and ASR performance with substantially fewer samples than dedicated language models use for training, while preserving the LLM's original capabilities.

    Source: §1 · p. 2, §4.3 · p. 4

    Probability-based selection requires access to model parameters, and results from one LLM and four languages may not generalize to other models or languages.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

With approximately 7,000 languages spoken worldwide, current large language models (LLMs) support only a small subset. Prior research indicates LLMs can learn new languages for certain tasks without supervised data. We extend this investigation to speech recognition, investigating whether LLMs can learn unseen, low-resource languages through in-context learning (ICL). With experiments on four diverse endangered languages that LLMs have not been trained on, we find that providing more relevant text samples enhances performance in both language modelling and Automatic Speech Recognition (ASR) tasks. Furthermore, we show that the probability-based approach outperforms the traditional instruction-based approach in language learning. Lastly, we show ICL enables LLMs to achieve ASR performance that is comparable to or even surpasses dedicated language models trained specifically for these languages, while preserving the original capabilities of the LLMs. Our code is publicly available.

1. Introduction

Language information remains underexplored in ASR for endangered languages, and ICL performance is limited for low-resource languages. This work investigates whether ICL can teach LLMs unseen languages for ASR using selected text examples.

1. Introduction

  • ASR performance remains limited for endangered languages, and integrating language information is underexplored despite gains from acoustic information.The introduction also contrasts human-level performance for high-resource languages with the remaining endangered-language gap.Source: §1 · p. 1
  • Low-resource ICL effectiveness drops as target languages are poorly represented in pretraining, weakening generalization and reliability.Source: §1 · p. 1
  • ASR remains underexplored among low-resource ICL evaluations, with acoustic variability, complex phonetics and prosody, and nonstandardized orthography posing added challenges.Source: §1 · p. 1
  • A prior ASR ICL study paired target audio with audio-text examples, but long inputs restricted sample counts and required audio-capable LLMs [15].Source: §1 · p. 1
  • The authors propose in-context language learning with a few hundred target-language sentences, evaluating ASR adaptation and sample-selection strategies, including instance-specific selection.Source: §1 · pp. 1–2

2. In-context Language Learning

The method supplies target-language examples in prompts, retrieves samples for language learning, and re-ranks ASR candidates using language-model probabilities, optionally combined with acoustic probabilities.

2. In-context Language Learning

  • ICL language learning provides target-language sentences without instructions; for ASR, an initial system's n-best transcriptions are re-ranked rather than newly generated.The approach aims to provide linguistic knowledge, not teach a new task through prompt examples.Source: §2 · p. 2

2.1. Samples Retrieval Strategy

  • Retrieval compares test queries with training examples using embedding cosine similarity, enabling example-specific samples instead of one corpus-wide set.Random selection is the baseline; the test query can be an initial ASR hypothesis, audio, or a human transcript for an oracle evaluation.Source: §2.1 · p. 2
  • The top three ASR hypotheses can also serve as retrieval keys, with their similarity scores averaged when ranking training samples.Source: §2.1 · p. 2

2.2. ASR Hypotheses Re-ranking

  • Hypothesis re-ranking can use instruction-based selection or language-model probabilities, with the latter combined with acoustic probabilities.The acoustic probability is defined over N time steps and the language-model probability over M steps.Source: §2.2 · pp. 2–3

3. Experimental Setups

Experiments use four endangered languages unseen to LLMs, with ASR candidates and text and audio embeddings prepared using multilingual models.

3.1. Datasets

  • Experiments cover Khinalug, Kichwa, Mboshi, and Japhug, four endangered languages described as linguistically diverse and unseen to LLMs.Source: §3.1 · p. 3

3.2. Hypotheses Generation

  • Wav2Vec2 mms-300 generates ASR hypotheses, and beam search supplies the ten candidates with the best acoustic probabilities for language-model selection.The model was pretrained on over 1,400 languages.Source: §3.2 · p. 3

3.3. Language Models

  • The language-model comparisons use Llama-3-8B, 5-gram statistical language models, and GPT-2 transformer-based language models.Source: §3.3 · p. 3

3.4. Embedding Similarity Measurement

  • SONAR generates embeddings for hypotheses and audio, providing multilingual representations for the study's low-resource languages.Source: §3.4 · p. 3

4. Results and Analysis

ICL-based selection outperforms acoustic selection across languages and matches or exceeds trained language models in some languages. Retrieval strategy affects language-modelling performance, while the approach is limited by parameter access and evaluation scope.

4.1. Language Modelling

  • Example-specific retrieval improves language modelling over corpus-level selection, while acoustic or combined acoustic-text embeddings underperform text embeddings alone.Corpus-level retrieval is similar to n-gram-LM selection but falls short of transformer LMs.Source: §4.1 · pp. 3–4
  • A single ASR hypothesis as the retrieval key performs comparably to gold transcripts despite high ASR word error rates.Source: §4.1 · p. 4

4.2. LLM-based Hypotheses Selection

  • Probability-based hypothesis selection performs significantly better than instruction-based selection, and adding acoustic information consistently improves selection.Increasing instruction-based ICL samples from 10 to 50 degrades performance; the authors suggest sample diversity may contribute.Source: §4.2 · p. 4

4.3. ICL in Speech recognition

  • All ICL-based selection methods outperform acoustic selection across languages, and ICL beats n-gram and transformer LMs for Kichwa and Mboshi while matching them for two other languages.The comparison uses significantly fewer ICL samples than the LMs used in training and preserves the LLM’s original capabilities.Source: §4.3 · p. 4, Table 4 · p. 4

4.4. Limitations

  • Probability-based selection requires access to model parameters, and results from one LLM and four languages do not establish applicability across all languages or models.Source: §4.4 · p. 4

5. Conclusion

The conclusion presents in-context language learning as a way for LLMs to learn new languages from in-context examples. Across four endangered languages, it reports gains in language modelling and ASR reranking, with the ASR improvement observed in two languages.

  • ICLL enables large language models to learn a new language through in-context examples.Source: §5 · p. 4
  • Example-specific sample selection improves on corpus-level selection, while an initial system’s hypothesis performs nearly as well as oracle human transcriptions as a selection proxy.Source: §5 · p. 4
  • Across four endangered languages, ICLL improves language-modelling perplexity over strong baselines and improves ASR performance in two languages when used for reranking.Source: §5 · p. 4
Loading 2505.20445v5…