PaperlayerView exact revision on arXiv

Source-linked AI summary

Lossless Inference Acceleration for ASR via EAGLE-Style Speculative Decoding

Sen Zhang, Xianghu Yue, Yiming Chen, Wenhuan Lu, Jianguo Wei, Haizhou Li

arXiv:2610.13506v1cs.SDeess.AS

TL;DR

Autoregressive ASR decoding incurs substantial latency, while existing speculative approaches can have low draft acceptance or rely on external draft models. EAGLE-ASR combines target-model representations and acoustic features in a trained Speech-Draft module, achieving lossless speedups of up to 1.9× on Whisper and 2.4× on Conformer AED models.

  • Problem

    Autoregressive ASR is bottlenecked by sequential decoding, while existing ASR speculative methods can produce low-quality drafts or rely on external draft models.

    Source: §I · pp. 1–2

  • Method

    EAGLE-ASR uses Speech-Draft, a lightweight autoregressive module combining encoder acoustic features and hierarchical decoder representations, trained with self-distilled data and multi-step generation.

    Source: §I · p. 2

  • Results

    Up to 1.9× on Whisper and 2.4× on Conformer AED, EAGLE-ASR achieves lossless inference speedups and outperforms existing ASR speculative-decoding baselines.

    Source: §I · p. 2

  • Takeaways & Limitations

    Experiments also demonstrate the framework's effectiveness on multilingual ASR and speech translation tasks.

    Source: §I · p. 2

Abstract

from arXiv · show

Autoregressive Transformer and Conformer models have achieved state-of-the-art performance in automatic speech recognition (ASR). However, their sequential token generation mechanism imposes a significant bottleneck during decoding, resulting in substantial inference latency. While speculative decoding has emerged as a promising lossless acceleration technique, existing ASR-oriented approaches either suffer from low draft acceptance rates due to suboptimal draft quality or rely on specially designed external draft models, which limit their generality and practicality. To overcome these limitations, we propose EAGLE-ASR, a novel framework that accelerates ASR inference via EAGLE-style speculative decoding while strictly preserving lossless output quality. Specifically, we introduce Speech-Draft, a lightweight autoregressive module that serves as a universal draft model by jointly leveraging encoder acoustic features and hierarchical decoder representations from the target ASR model. Furthermore, we adopt a multi-step generation strategy with self-distilled data to train the Speech-Draft, enabling it to better approximate the target model's output distribution and maintain causal consistency during multi-token generation. Extensive experiments on representative Whisper and Conformer AED models demonstrate that EAGLE-ASR achieves significant lossless speedups of up to $1.9\times$ and $2.4\times$, respectively, consistently outperforming existing speculative decoding baselines for ASR. Our proposed framework further generalizes effectively to multilingual ASR and speech translation tasks, establishing EAGLE-ASR as a practical and universally applicable solution for accelerating modern ASR systems without compromising recognition accuracy.

I. INTRODUCTION

Autoregressive ASR decoding is latency-bound, while existing acceleration approaches may sacrifice recognition accuracy or depend on poorly aligned draft models. EAGLE-ASR combines target-model representations with acoustic features and multi-step training to achieve lossless speedups on Whisper and Conformer AED models.

I. INTRODUCTION

  • Autoregressive token-by-token generation creates a decoding bottleneck, making latency reduction without sacrificing recognition accuracy a challenge.This matters for resource-constrained edge devices and high-throughput server environments.Source: §I · p. 1
  • Compression and non-autoregressive decoding can reduce inference cost, but may require architectural changes or sacrifice recognition accuracy.The introduction identifies lossless acceleration for autoregressive ASR as a remaining need.Source: §I · p. 1
  • Existing ASR speculative-decoding approaches either rely on aligned external draft models or produce low-quality drafts with poor acceptance rates.Directly adapting EAGLE-style drafting also must account for acoustic conditioning and decoder–encoder interactions.Source: §I · pp. 1–2
  • EAGLE-ASR introduces Speech-Draft, a lightweight autoregressive module using encoder acoustic features and hierarchical decoder representations to draft tokens.This is designed for attention-based encoder-decoder ASR models.Source: §I · p. 2
  • Self-distillation and multi-step generation training aim to improve draft quality while maintaining causal consistency during multi-token speculative decoding.The paper describes this strategy as addressing dependencies among consecutive future tokens.Source: §I · p. 2
  • Up to 1.9× on Whisper and 2.4× on Conformer AED, EAGLE-ASR achieves lossless speedups and outperforms existing ASR speculative-decoding baselines.The reported speedups preserve recognition accuracy.Source: §I · p. 2

II. RELATED WORK

Prior ASR acceleration methods reduce latency through compression or parallel token generation, while speculative decoding drafts tokens and verifies them with the target model. Integrated approaches differ in how they produce drafts and preserve dependencies between drafted tokens.

A. Traditional Inference Acceleration Techniques in ASR

  • Compression and non-autoregressive ASR reduce inference latency but typically trade off recognition performance for speed.Examples include distillation, quantization, pruning, and parallel token generation.Source: A. Traditional Inference Acceleration… · p. 2

B. Speculative Decoding

  • Speculative decoding uses a lightweight draft model to propose tokens for parallel target-model verification, preserving the target distribution under its acceptance procedure.Rejected drafts are replaced from the residual distribution, and later draft tokens are discarded.Source: B. Speculative Decoding · p. 2
  • Medusa predicts future positions with independent lightweight heads, so later tokens do not condition on their immediate predecessors.The passage links this lack of sequential dependency to less coherent drafts and diminished acceptance.Source: B. Speculative Decoding · p. 3
  • EAGLE instead attaches a single lightweight autoregressive module to the frozen target, using representations from different layers while preserving temporal dependencies.The passage reports state-of-the-art decoding speed among existing speculative-decoding approaches.Source: B. Speculative Decoding · p. 3

C. Speculative Decoding in ASR

ASR speculative-decoding approaches can depend on external draft models aligned to the target or neglect acoustic features and temporal dependencies. EAGLE-ASR uses a target-integrated Speech-Draft and parallel verification to accelerate autoregressive ASR while preserving the target output distribution.

C. Speculative Decoding in ASR

  • External draft models can lose acceptance when target and draft families or tokenizers differ, while Whisper-Medusa neglects temporal dependencies and encoder acoustic features.The passage also reports WER degradation for Whisper-Medusa because it fine-tunes the target model's final decoder layer.Source: C. Speculative Decoding in ASR · p. 3
  • EAGLE-ASR eliminates the need for an independently trained draft model while targeting faster decoding with lossless output quality.This is presented as an alternative to external-draft and Whisper-Medusa approaches.Source: C. Speculative Decoding in ASR · p. 3

A. Overview

  • Speech-Draft combines encoder acoustic features with multi-level target-decoder representations to recursively draft tokens from the target-predicted next token and prefix.At the first step it uses the encoder and decoder features; later steps condition on its preceding hidden state and drafted token.Source: A. Overview · p. 3
  • The target decoder verifies all drafted tokens in parallel and accepts them sequentially until the first mismatch, preserving the target output distribution.The figure depicts training above and inference below, with recursive drafting followed by parallel verification.Source: A. Overview · p. 3, Figure 1 · p. 4
  • Linear verification yields a better speedup than tree-based verification in the ASR setting.Source: A. Overview · p. 4

B. Speech-Draft Module

Speech-Draft is a lightweight module that combines target-decoder representations with encoder acoustic information for autoregressive drafting. Its architecture reuses the target model's embedding and LM-head parameters and incorporates acoustic features through cross-attention.

B. Speech-Draft Module

  • At the first decoding step, one FC layer fuses low-, middle-, and high-level decoder representations; the second combines that representation with prior-token embeddings.At later steps, the second layer uses the preceding hidden states instead of the fused representation.Source: B. Speech-Draft Module · p. 4
  • Speech-Draft comprises two fully connected layers, an embedding layer, a Transformer decoder block, and an LM head.The embedding layer and LM head reuse the target model's corresponding parameters and require no additional training.Source: Figure 2 · p. 4, B. Speech-Draft Module · p. 4
  • Cross-attention incorporates encoder acoustic features into drafting, combining acoustic information with decoder-side semantic representations.The module is described as integrable with autoregressive ASR systems using a Transformer-based decoder, including Whisper and Conformer AED.Source: B. Speech-Draft Module · pp. 4–5

C. Self-Distilled Training Data

EAGLE-ASR trains Speech-Draft on target-generated transcriptions to align its drafts with the target model. Multi-step training addresses the mismatch between training and inference, using exponentially decayed loss weights to prioritize near-term predictions.

C. Self-Distilled Training Data

  • Target-generated transcriptions replace original annotations as Speech-Draft training targets, matching the target model’s generation patterns and preferences.The paper identifies a distribution discrepancy between target outputs and original transcriptions that may limit target-draft alignment.Source: C. Self-Distilled Training Data · p. 5

D. Training Strategy

  • Multi-step training addresses the mismatch that arises when later inference drafts rely on Speech-Draft’s own predicted hidden states rather than target-model ground truth.The paper notes that this discrepancy can cause error accumulation and lower acceptance rates.Source: D. Training Strategy · p. 5
  • At step k=1, Speech-Draft uses target-model representations and shifted tokens; later steps use preceding draft hidden states and increasingly shifted tokens.As generation depth increases, predictions become more difficult and uncertainty accumulates.Source: D. Training Strategy · p. 5
  • Exponentially decayed weighted cross-entropy prioritizes near-term predictions while training Speech-Draft to generate multiple consecutive future tokens.The weight α_k applies to the loss at step k, and β is the decay factor.Source: D. Training Strategy · p. 5

IV. EXPERIMENTAL SETUP

Experiments use LibriSpeech for English ASR and VoxPopuli to assess multilingual ASR and speech translation. Speech-Draft is trained on target-generated transcriptions rather than the datasets’ human-annotated transcripts.

A. Datasets

  • Speech-Draft is trained on LibriSpeech’s full 960-hour training split, with evaluation on the standard test-clean and test-other subsets.LibriSpeech contains approximately 1,000 hours of read English speech.Source: A. Datasets · pp. 5–6
  • VoxPopuli experiments assess multilingual ASR in five languages and speech translation in four directions into English.The ASR subsets are English, German, Spanish, French, and Italian; translation directions are German, Spanish, French, and Italian to English.Source: A. Datasets · p. 6
  • Training uses target-model-generated transcriptions instead of the paired human-annotated transcripts available for both datasets.The self-distilled data generation procedure is described in subsection III-C.Source: A. Datasets · p. 6

B. Model Configurations

The experiments evaluate Whisper-large-v3 and Conformer AED as target models, with Speech-Draft training and evaluation configured across LibriSpeech and VoxPopuli. Metrics capture recognition or translation quality alongside decoding speed and draft acceptance.

B. Model Configurations

  • The target models are Whisper-large-v3, a 1,550M-parameter Transformer, and Conformer AED, a 220M-parameter CTC/attention encoder-decoder.The Conformer AED CTC branch is not used in inference.Source: B. Model Configurations · p. 6

C. Training Details

  • Speech-Draft uses prediction depth K=7 and decay factor β=0.8; evaluations run on a single NVIDIA RTX 4090 GPU with 24 GB memory.Training steps vary by model and dataset: Whisper uses 200K/400K on LibriSpeech/VoxPopuli, and Conformer AED uses 150K/300K.Source: C. Training Details · p. 6

D. Evaluation Metrics

  • Evaluation reports ΔWER for ASR and ΔBLEU for translation quality, plus speedup ratio and mean accepted length τ for acceleration.The speedup baseline is vanilla autoregressive decoding; τ is the average number of draft tokens accepted per verification step.Source: D. Evaluation Metrics · p. 6

V. RESULTS AND ANALYSIS

On LibriSpeech, EAGLE-ASR achieves the highest speedups across Whisper and Conformer AED while preserving lossless decoding. Its results reflect a balance between draft quality and computational cost, with strong performance across target architectures.

A. Main Results

  • Whisper reaches 1.91× on test-clean and 1.79× on test-other; Conformer AED reaches 2.43× and 2.30×, respectively, outperforming baselines losslessly.For Whisper, ∆WER = 0.00; conventional speculative decoding baselines on Conformer AED fail to accelerate and can slow decoding on test-other.Source: A. Main Results · pp. 6–7, Table I · p. 7, Figure 4 · p. 7
  • Distil-large-v3 has the highest acceptance length but lower speedup than EAGLE-ASR, as its 756M parameters add substantial drafting overhead.Other lightweight drafts have lower mean accepted lengths, limiting their speedups.Source: A. Main Results · p. 7
  • EAGLE-ASR generalizes across target architectures and parameter scales without architecture-specific changes or same-family distilled drafts.The authors attribute its acceleration to balancing draft quality against computational cost.Source: A. Main Results · p. 7

B. Performance on Multilingual Speech Recognition

On VoxPopuli multilingual ASR, EAGLE-ASR accelerates Whisper decoding across five languages while preserving transcription quality. Speedups vary relatively little across languages.

B. Performance on Multilingual Speech Recognition

C. Performance on Speech Translation

EAGLE-ASR provides lossless acceleration on speech translation and maintains acceleration across varied translation directions. Its decoding hyperparameters show task-dependent trade-offs between accepted draft length and verification cost.

C. Performance on Speech Translation

D. Impact of Decoding Hyperparameters

  • With top-k fixed at 1, mean accepted length rises from 1.82 at K=2 to 4.57 at K=8, while speedup peaks near 1.85× at K=5 before declining.The authors attribute the decline beyond K=5 to drafting and verification overhead outgrowing the smaller gains in accepted length.Source: Figure 5 · p. 8, D. Impact of Decoding Hyperparameters · pp. 8–9

C. Performance on Speech Translation

  • Speech translation achieves a 1.62× speedup versus 2.09× for multilingual ASR, with the latter reporting τ = 4.42; ST’s lower speedup is attributed to complexity and larger top-k overhead.Source: C. Performance on Speech Translation · p. 8

D. Impact of Decoding Hyperparameters

  • For ASR, increasing top-k from 1 to 4 raises mean accepted length from 3.67 to 4.56 but lowers speedup from 1.85× to 1.54×.The authors report greater acceleration with linear verification (top-k = 1) than tree-based verification (top-k > 1).Source: Figure 6 · p. 9, D. Impact of Decoding Hyperparameters · p. 9
  • For German-to-English translation, speedup rises from 1.44× at top-k=1 to 1.48× at top-k=2, then falls to 1.38× at top-k=4 as mean accepted length rises from 2.70 to 3.90.The authors report that the benefit of larger top-k quickly saturates as verification overhead grows.Source: Figure 7 · p. 9, D. Impact of Decoding Hyperparameters · pp. 9–10

E. Ablation Studies

The ablations test Speech-Draft’s architecture, training data, and training strategy. They show that acoustic grounding, fused decoder features, self-distilled data, and multi-step training contribute to the reported acceleration.

E. Ablation Studies

  • Removing encoder features causes slowdown to 0.89× and reduces mean accepted length to 0.08; using only second-top-layer decoder features yields 1.35× speedup, below the full system.The results support the role of acoustic grounding and fused multi-layer decoder features in effective drafting.Source: Table IV · p. 10, E. Ablation Studies · p. 10
  • Training on ground-truth transcripts instead of self-distilled data lowers speedup to 1.43× and mean accepted length to 2.02.The authors explain that self-distillation aligns the draft with the target model’s output distribution, including punctuation and capitalization.Source: Table IV · p. 10, E. Ablation Studies · p. 10
  • Single-step training yields 1.17× speedup and mean accepted length 1.00; removing exponential-decay weighting yields 1.66× and 3.28, respectively.The authors report that multi-step training supports multi-token generation, while exponential decay prioritizes accuracy on earlier draft tokens.Source: Table IV · p. 10, E. Ablation Studies · p. 10

VI. CONCLUSION

EAGLE-ASR accelerates autoregressive ASR with EAGLE-style speculative decoding while preserving output quality. Experiments report speedups on Whisper and Conformer AED models, with additional evidence of generalization to multilingual ASR and speech translation.

VI. CONCLUSION

  • EAGLE-ASR achieves substantial inference speedups on Whisper and Conformer-based AED models, outperforming speculative decoding baselines without output-quality degradation.The conclusion also reports robustness across multilingual ASR and speech translation.Source: §VI · p. 11
  • Speech-Draft combines hierarchical decoder representations with encoder acoustic features and uses self-distilled data and multi-step generation to support reliable multi-token prediction.It is designed to approximate the target model's output distribution while balancing draft quality and computational overhead.Source: §VI · p. 11
  • Additional experiments confirm EAGLE-ASR's generalizability to multilingual ASR and speech translation.The conclusion describes these as diverse languages and cross-lingual generation settings.Source: §VI · p. 11
Loading 2610.13506v1…