PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting

Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, Andreas Stolcke

arXiv:2309.15649v2cs.CLcs.AIcs.LGcs.SDeess.AS

TL;DR

The paper investigates whether frozen LLMs can improve ASR post-processing through in-context learning, despite limited evidence for its benefits in ASR. It tests prompted second-pass rescoring and error correction, and reports 31% (on ATIS) to 38% (on WSJ) WER reduction over first-pass ASR for its best methods.

  • Problem

    The benefits of in-context learning for ASR remain underexplored, while scaling ASR alone has not proven adequate for challenging conversational tasks.

    Source: §1 · p. 1

  • Method

    The paper evaluates frozen-LLM second-pass pipelines for error correction before standard rescoring and for prompt-based N-best rescoring, alongside fine-tuning.

    Source: §1 · p. 1, §3.1.4 · p. 3

  • Results

    31% (on ATIS) to 38% (on WSJ) WER reduction over first-pass ASR is reported for the best methods.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The reported reductions show that LLM-based post-processing can improve first-pass ASR output on the evaluated tasks.

    Source: §5 · p. 6

    Few-shot demonstrations were manually selected, and demonstration selection for the N-best task was left for future work.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

We explore the ability of large language models (LLMs) to act as speech recognition post-processors that perform rescoring and error correction. Our first focus is on instruction prompting to let LLMs perform these task without fine-tuning, for which we evaluate different prompting schemes, both zero- and few-shot in-context learning, and a novel task activation prompting method that combines causal instructions and demonstration to increase its context windows. Next, we show that rescoring only by in-context learning with frozen LLMs achieves results that are competitive with rescoring by domain-tuned LMs, using a pretrained first-pass recognition system and rescoring output on two out-of-domain tasks (ATIS and WSJ). By combining prompting techniques with fine-tuning we achieve error rates below the N-best oracle level, showcasing the generalization power of the LLMs.

1. INTRODUCTION

The paper asks whether frozen LLMs can improve ASR through in-context learning, given limits of ASR scaling and compact external language models. It explores two second-pass pipelines: error correction before standard rescoring and prompting an LLM to rescore N-best hypotheses directly.

1. INTRODUCTION

  • Frozen LLMs offer a potential cloud-based second-pass approach using in-context learning, while external LMs for on-device deployment are often limited to 10M–100M parameters.The paper motivates this approach by noting that scaling ASR models to 10B parameters has not by itself ensured high performance on challenging conversational tasks.Source: §1 · p. 1
  • The paper explores two ASR post-processing pipelines that use frozen LLMs and in-context learning.One inserts LLM error correction before a standard rescoring model; the other prompts the LLM to rescore N-best hypotheses directly.Source: §1 · pp. 1–2
  • The work compares these pipelines with different in-context learning schemes and standard rescoring methods.Its stated focus is a first exploration of LLM use for ASR post-processing.Source: §1 · p. 2

2. RELATED WORK

Related work covers ASR text correction and rescoring, zero-shot language modeling, and prompting methods for in-context learning. The paper applies these prompting techniques to ASR rescoring and evaluates them against a baseline.

LLM-based post-processing to improve hypotheses

  • ASR post-processing work uses pretrained LLMs to correct grammar or deletion errors, while N-best T5 uses an encoder-decoder model for discriminative rescoring.Liao et al. generate readable text from ASR transcriptions; N-best T5 [11] uses discriminative training.Source: §2 · p. 2

Zero-shot learning for acoustic and language modeling

  • Prior zero-shot and few-shot language modeling approaches often require fine-tuning and redeployment of pretrained models.The paper frames in-context learning as inference without fine-tuning, using one or a few prompts.Source: §2 · p. 2

In-context learning based on information prompting

  • The paper applies prompting methods, including chain-of-thought and zero-shot reasoning, to ASR rescoring and evaluates their performance against a baseline.The cited prior work describes chain-of-thought as prompts that guide predictions through a sequence of questions, and zero-shot reasoning from a specific prompt.Source: §2 · p. 2

3. METHOD

The method applies in-context prompting and fine-tuning to second-pass ASR rescoring. It includes domain hints, zero-shot reasoning, demonstrations, a hypotheses-to-transcription loss, and task-activating prompting.

3.1. In-Context Learning Background and Techniques

  • A task-oriented question is one way to prompt a pretrained LLM for in-context learning.The method frames prediction as conditioned on a prompt variable and a test-time sample.Source: §3.1 · p. 2

3.1.1. Zero-shot domain-hint prompting

  • Zero-shot domain-hint prompting supplies domain context, such as airline information or financial-market language, as a hard-coded input.The described domains are airline travel and financial markets.Source: §3.1.1 · pp. 2–3

3.1.2. Zero-shot reasoning

  • Zero-shot reasoning uses a reasoning-extraction prompt followed by answer extraction; the authors report that the reasoning prompt is essential to boost zero-shot rescoring.The fixed reasoning prompt is “Let’s think step by step”; the LLM first explains the rescoring task, then produces scores for the N-best hypotheses.Source: §3.1.2 · p. 3

3.1.3. Few-shot and one-shot in-context learning

  • Few-shot prompting supplies question-and-task demonstrations from training data, with one-shot learning using a single demonstration.The demonstrations are distinct from test examples.Source: §3.1.3 · p. 3

3.1.4. N-best hypotheses to transcription fine-tuning

  • The hypotheses-to-transcription loss combines the probability of the true transcription given each hypothesis with an MSE penalty on predicted probabilities against posterior scores.The MSE term uses λ = 0.01 to incorporate acoustic information; parameter-efficient fine-tuning updates only a subset of parameters to avoid potential overfitting that could hurt generalization.Source: §3.1.4 · p. 3

3.2. Task-activating Prompting (TAP) Framework

  • Task-activating prompting uses multiple rounds of leading questions, an example, and then the top-N hypotheses to elicit the task output.The authors report that LLMs produce top-N utterances with analogous pronunciations, which they take as evidence of assimilated acoustic, lattice-level information during pretraining. Demonstration selection is left for future work; their few-shot experiments manually select longer utterances.Source: §3.2 · p. 3

4. EXPERIMENTS AND RESULTS

The experiments evaluate two second-pass ASR pipelines on ATIS and WSJ: LLM correction followed by rescoring, and LLM prompting for direct rescoring. They test frozen-model prompting and in-domain fine-tuning against rescoring baselines.

4.1. Pretrained ASR and Rescoring Model Training

  • The experiments used a pretrained 760M-parameter conformer-based RNN-transducer to generate hypotheses for P1 and P2 on ATIS and WSJ.The first-pass model was pretrained on Librispeech, GigaSpeech, VoxPopuli, and Libri-Light data; WSJ adaptation used train-si284 and testing used 93dev.Source: §4.1 · p. 3, §4.3 · pp. 4–5
  • P1 trained an Albert base-v2 rescoring model with 4.8 million trainable parameters using MLM adaptation followed by MWER training on LLM-corrected hypotheses.For P2, the study tested frozen GPT-2, OpenLLaMA, BLOOM, and InstructGPT configurations.Source: §4.1 · pp. 3–4, §4.2 · p. 4

4.4. Pipeline 1 Results

  • WER decreased across P1’s three stages: LLM-corrected N-best hypotheses, RescoreBERT fine-tuned with MLM loss, and MWER training.The dashed red line in Figure 4 marks N-best WER; existing rescoring reduced WER from 11.3% to 8.7%, with frozen LLM correction providing an additional boost.Source: Figure 4 · p. 5, §4.4 · p. 5

4.5. Pipeline 2 Results

  • 19.7% relative WER reduction: InstructGPT was the best P2 rescoring setup, compared with rescoring using fine-tuned GPT-2.Frozen GPT-2 did not improve over a 4-gram baseline; for LLMs over 100B parameters, prompting information performed better than softmax scores alone.Source: §4.5 · p. 5, Table 3 · p. 6
  • 14.6% relative: standard P1 rescoring outperformed zero-shot P2, although step-by-step prompting achieved the best P2 results for both evaluated LLMs.Step-by-step prompting also outperformed one-shot prompting by 1.8% relative WER difference.Source: §4.5 · p. 5, Table 2 · p. 6
  • With frozen InstructGPT, rescoring improved as demonstrations increased from 1 to 12, and accumulating model history performed better than resetting it after each utterance.Figure 5 compares the two history strategies on ATIS and WSJ.Source: §4.5 · p. 5, Figure 5 · p. 5
  • For P2 generative error correction, LoRA and residual adapters outperformed full fine-tuning, while prefix tuning performed worse than full or adapter-based fine-tuning for larger LLMs.InstructGPT could not be fine-tuned, and prompting choice made little difference after fine-tuning.Source: §4.5 · pp. 5–6, Table 4 · p. 6
  • Best fine-tuning results improve on the WSJ WER of 3.2% reported for the Universal Speech Model, despite using a smaller underlying conformer-RNN-T ASR system.Table 4 reports results for fine-tuning and parameter-efficient adaptation of P2-TAP.Source: §4.5 · p. 6, Table 4 · p. 6

5. CONCLUSIONS

The paper applies frozen LLMs to first-pass ASR N-best output through two post-processing pipelines, then combines prompting with fine-tuning. Frozen InstructGPT reduces WER on ATIS and WSJ, while fine-tuned OpenLLaMA with LoRA achieves reductions below the N-best oracle error rate.

5. CONCLUSIONS

  • 31% on ATIS and 38% on WSJ WER reduction over first-pass ASR are achieved by frozen InstructGPT; the best methods outperform fine-tuned GPT-2.The paper reports that fine-tuned OpenLLaMA with LoRA achieves 86% and 80% WER reduction on ATIS and WSJ, respectively, below the N-best oracle error rate.Source: §5 · p. 6
  • Fine-tuning OpenLLaMA with LoRA achieves 86% WER reduction on ATIS and 80% on WSJ, below the N-best oracle error rate.The authors interpret these results as showing that LLMs can use prelearned knowledge to correct ASR output errors.Source: §5 · p. 6
  • The two pipelines use in-context learning either to correct first-pass hypotheses before standard rescoring or to prompt a frozen LLM to rescore them directly.The latter pipeline can be enhanced with chain-of-thought and example prompting, as well as task-activating prompting.Source: §5 · p. 6
  • The authors identify integrating additional acoustic representations into pretrained LLMs as future work for generative ASR error correction.Source: §5 · p. 6

Related papers

Loading 2309.15649v2…