PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Lookahead for Attention-Based Streaming ASR

Yichen Jia, Bastiaan Tamm, Hugo Van hamme

arXiv:2609.37611v1eess.AS

TL;DR

Streaming attention-based ASR must make decisions from partial audio, limiting acoustic context and raising WER. The paper combines a bounded-lookahead chunk encoder with dynamic decoding that waits and re-decodes selected tokens. On test-clean, dynamic decoding matches the best static-lookahead accuracy at lower median latency.

  • Problem

    Streaming ASR must emit tokens from partial audio, limiting each decision’s acoustic context and raising WER.

    Source: §I · p. 1

  • Method

    A bounded-lookahead encoder and dynamic future-chunk decoding use a per-token trigger to commit a token or wait and re-decode it with richer context.

    Source: §I · p. 1

  • Results

    6.5% WER at 306 ms median latency matches static R=4 accuracy, versus 860 ms median latency for static R=1.

    Source: D. Dynamic Future-Chunk Results · p. 5

  • Takeaways & Limitations

    The dynamic system matches fixed-lookahead WER at substantially lower median latency, with a wait budget to bound the heavier deferral tail.

    Source: §VI · p. 6

    Deferral can undo errors from committing too early, but cannot resolve tokens that remain acoustically ambiguous because the encoder has not seen future audio.

    Source: C. Trigger Study · p. 4

Abstract

from arXiv · show

End-to-end attention-based speech recognition is accurate offline but hard to stream: outputs can depend on future audio, and a little future context per layer makes the lookahead grow with the number of layers. We address this with two mechanisms. A bounded-lookahead chunk encoder caps every chunk's future receptive field at a constant number of chunks, independent of the number of layers, via one age-selection rule shared by self-attention and the depthwise convolution. On this encoder, dynamic future-chunk decoding lets a per-token trigger commit a token or wait and re-decode it; we propose a learned trigger as the general mechanism, with a simple confidence threshold as an effective fallback. On full LibriSpeech test-clean the dynamic system matches the best static-lookahead accuracy (6.5%) at a median latency of 306 ms versus 860 ms for one-chunk static lookahead, and a wait budget bounds the deferral tail below the static baseline's 90th percentile at 0.1 points more WER.

I. INTRODUCTION

Streaming attention-based ASR must make decisions from partial audio, creating a latency–accuracy trade-off. The paper addresses this with bounded-lookahead encoding and per-token decisions about whether to wait for more context.

I. INTRODUCTION

  • Partial-audio decisions limit acoustic context and raise WER, making controlled movement along the latency–accuracy trade-off the central engineering question.The motivating applications include live captioning, voice assistants, and simultaneous translation.Source: §I · p. 1
  • The bounded-lookahead chunk encoder uses one age-selection rule for self-attention and depthwise convolution, capping future context independently of layer count.It also lets a committed chunk be re-encoded with more context at inference.Source: §I · p. 1
  • Dynamic future-chunk decoding uses a per-token trigger to commit or wait and re-decode, while fixed lookahead delays every token equally.The paper also characterizes chunk-size, lookahead, and latency trade-offs against streaming baselines.Source: §I · p. 1

II. BACKGROUND AND RELATED WORK

Streaming ASR methods control latency through encoder receptive fields or emission timing. This work’s age-versioned encoder bounds future context while retaining graded future versions that support re-encoding for dynamic decoding.

II. BACKGROUND AND RELATED WORK

  • Attention-based encoder–decoders are difficult to stream because attention is global, while CTC and neural transducers have streaming variants.Across model families, latency control primarily concerns encoder receptive field or emission timing.Source: §II · p. 1
  • Most attention-based methods use fixed lookahead; unlike prior bounded-lookahead approaches, age-versioning provides graded future versions and applies one age rule to convolution and self-attention.The graded versions retain partial future context, and permit committed chunks to be re-encoded with richer context for dynamic decoding.Source: §II · pp. 1–2
  • The receptive-field comparison shows symmetric per-layer masks accumulating future reach, while the asymmetric mask caps it at R chunks.Latency-control alternatives include emission regularization within fixed acoustic context and methods that adapt reading or revision timing.Source: Figure 2 · p. 2, §II · p. 2

III. BOUNDED-LOOKAHEAD CHUNK ENCODER

With symmetric per-layer attention, future context compounds across layers, undermining a fixed streaming bound. The asymmetric age-versioned mask prevents that growth by limiting how far future chunks can contribute.

A. The Growing-Receptive-Field Problem

  • Symmetric per-layer attention lets a chunk’s effective lookahead grow linearly with encoder depth, and symmetric Conformer convolution can compound future context too.A streaming bound therefore must cover every cross-chunk path.Source: A. The Growing-Receptive-Field Problem · p. 2

B. Attention: Asymmetric Age-Versioned Mask

  • The asymmetric mask lets the current chunk attend to L past chunks and R future chunks, but future chunks attend only to their past and themselves.Because a future chunk cannot import still-later context, the current chunk’s future receptive field stays capped at R chunks across layers.Source: B. Attention: Asymmetric Age-Versioned… · p. 2
  • The encoder builds R+1 age versions per chunk, from a preliminary view with no future context to newer versions that absorb one additional future chunk.The age rule selects a readable key version; sufficiently old keys use the final version, and keys outside the L-chunk past window are inadmissible. Figure 3 illustrates the R=1 case, where the current chunk reaches one future chunk but that future chunk remains causal.Source: B. Attention: Asymmetric Age-Versioned… · p. 2, Figure 3 · p. 2

C. Convolution: Cross-Age Depthwise Convolution

Cross-age depthwise convolution uses the attention path’s age-selection rule, so its future horizon remains bounded while still mixing future audio. The implementation trades additional computation and memory for that bound.

C. Convolution: Cross-Age Depthwise Convolution

  • Cross-age convolution routes each tap to the version selected by the shared age rule and zeros taps beyond the version’s future horizon.Every reachable version stays within chunk i+k, preserving the attention bound while mixing real future audio; attention and convolution are the network’s only cross-chunk paths.Source: C. Convolution: Cross-Age Depthwise… · p. 2

D. Efficient Masking and Cost

  • Age-versioning builds up to R+1 versions per chunk in training and re-encodes each chunk R+1 times as future chunks arrive during streaming inference.For R=1, inference uses a preliminary causal pass and then a final pass, with past key/value states cached.Source: D. Efficient Masking and Cost · p. 2
  • The block-structured mask uses FlexAttention’s block-sparse kernel, while FlashAttention is used only for the dense fallback.Age-versioning’s re-encoding is in-distribution because all R+1 versions are seen during training.Source: D. Efficient Masking and Cost · p. 2

A. Chunk-Synchronous Beam Search with a Trigger

The decoder processes audio chunk by chunk and uses a trigger to commit tokens or wait for more context and re-decode. The controller and latency analysis focus on when waiting helps and how to limit its delay.

A. Chunk-Synchronous Beam Search with a Trigger

B. Latency Definition

  • Per-word emission latency measures time from a word’s acoustic end to its commitment, using forced alignment for the acoustic end.Wrong words are excluded because they have no reference-word anchor; the measure is emission delay, not compute time.Source: B. Latency Definition · p. 3

C. When Does Waiting Help?

  • Waiting can help when word pieces cross a chunk boundary or when later audio resolves ambiguity, such as “write” versus “right.”An early short BPE piece may be irreversible, while another chunk can let the decoder recover a single in-vocabulary word piece.Source: C. When Does Waiting Help? · p. 3

D. Controller

  • Commit-stable-prefix commits the longest prefix shared by consecutive passes, leaving only the uncertain suffix to wait for more context.WER stayed at 6.5% despite 191 of 52.6k hypothesis words being locked before they would have flipped under deeper context.Source: D. Controller · p. 3

E. Trigger Signals

The paper compares confidence-based triggers with a learned trigger to decide when future audio is worth waiting for.

E. Trigger Signals

  • A trigger reads a per-token signal and defers below a threshold for confidence scores or above a threshold for learned defer probability.The trigger is intended to fire when an early commitment would be wrong.Source: E. Trigger Signals · p. 3
  • Confidence triggers use token probabilities, margins, entropy, or top-k mass, read at a decoder step or aggregated over non-blank CTC frames.The paper compares raw softmax top-1 probability with temperature scaling and SR-CEM, a calibrated score-rank confidence estimator adapted for streaming ASR.Source: E. Trigger Signals · p. 3
  • The learned trigger predicts directly from score-rank features whether deferring would fix a token, skipping a separate calibrated-confidence step.Its comparison against confidence thresholds is evaluated in Section V.Source: E. Trigger Signals · p. 3

V. EXPERIMENTS

The experiments characterize chunk size and lookahead, then compare streaming results with an un-fine-tuned initialization. Chunk size shapes the WER–latency trade-off, and chunked fine-tuning is essential for the reported streaming performance.

A. Setup

  • The evaluation uses LibriSpeech train-clean-360 and reports main accuracy results on full test-clean, with generalization checks on test-other and dev-other.SMALL is a 95-utterance subset of test-clean, and WER differences within the stated confidence intervals are treated as ties.Source: A. Setup · p. 4

B. Chunk Size, Lookahead, and the Static Baseline

  • Chunk size is the dominant factor in the static WER–lookahead trend: larger chunks lower WER, with most gains realized by S ≈32.Additional lookahead helps most for small chunks and flattens for large chunks; variations of 0.1–0.3 points across R are within noise.Source: B. Chunk Size, Lookahead, and the… · p. 4, Figure 4 · p. 4
  • At S = 16 on SMALL, static streaming WER is 9.0%, 7.3%, and 7.0% for R=0, 1, and 2, respectively, versus 5.5% offline.The passage identifies S = 16 as the low-latency end; larger chunks lower WER but proportionally raise latency.Source: B. Chunk Size, Lookahead, and the… · p. 4
  • At S = 16, the low-latency end, static streaming WERs on SMALL are 9.0%, 7.3%, and 7.0% for R=0, 1, and 2, versus un-fine-tuned offline initialization at 23.8% (R=0) and ∼12.5% (R≥1).Chunked fine-tuning is the essential step.Source: B. Chunk Size, Lookahead, and the… · p. 4

C. Trigger Study

The trigger study tests when deferral can correct errors and compares confidence-based and learned signals. It also examines the latency–WER trade-off across chunk sizes.

C. Trigger Study

  • Deferral can correct errors caused by committing too early, but cannot resolve tokens that remain acoustically ambiguous without future audio.Such errors set a gap between the streaming ceiling and offline accuracy that no trigger can close.Source: C. Trigger Study · p. 4
  • The tested confidence scores follow essentially one WER–latency curve and reach nearly the same accuracy floor, set by the model and chunk size.The scores are raw softmax top-1 probability, temperature-scaled confidence, and SR-CEM confidence.Source: C. Trigger Study · p. 4
  • On full test-clean, larger chunks reach a lower WER floor at higher latency, while S = 16 occupies the low-latency region and S = 20 lies between.With whole-chunk rollback, S = 24 reaches 5.8%; commit-stable-prefix lowers the three plotted curves.Source: Figure 5 · p. 5
  • At 6.5% WER on full test-clean, the learned trigger reaches 630/306 ms mean/median latency, versus 644/312 ms for SR-CEM and 688/320 ms for raw top-1.The learned trigger reaches ROC AUC 0.92 versus 0.90 for confidence alone; the confidence threshold is a practical fallback requiring no extra training.Source: C. Trigger Study · p. 5

D. Dynamic Future-Chunk Results

Dynamic future-chunk decoding matches the best static-lookahead WER with lower median latency, while a wait budget limits the deferral tail. Its benefits also transfer across several out-of-domain datasets, although not uniformly to WER.

D. Dynamic Future-Chunk Results

  • On the 95-utterance subset, strategy D reaches 7.1% WER, matching the same decoder’s static R=1 level; token-only strategy A reaches 8.0%.Strategy A cannot re-encode lookahead, a structural limitation; strategies B and C are within 0.1 points of D on this subset.Source: D. Dynamic Future-Chunk Results · p. 5
  • 6.5% WER at 306 ms median latency matches static R=4 accuracy and is a third of static R=1’s 860 ms median latency.The learned trigger’s 0.2-point WER edge over static R=1 is statistically distinguishable (p=0.02), though small in absolute WER.Source: D. Dynamic Future-Chunk Results · p. 5
  • With whole-chunk rollback, the learned trigger reaches 6.5% WER at 442 ms median latency, while the oracle reaches 6.3% at 292 ms.The oracle reads the reference transcript and defers chunks whose first decode contains an error; it is an upper bound, not a deployable trigger.Source: D. Dynamic Future-Chunk Results · p. 5
  • At θ=0.70, wait budget B=1 yields 964 ms p90 latency and 6.8% WER, 0.1 points higher than static R=1’s 6.7%.B=1 is below static R=1 on mean, median, and p90 latency; the WER difference is inside the test-set confidence interval.Source: D. Dynamic Future-Chunk Results · p. 5
  • On LibriAdapt-US, dynamic decoding ties static WER at 16.3 vs 16.7% while reducing median latency from 890 to 426 ms; TED-LIUM 3 reaches 22.0% WER at 506 ms.WSJCAM0 matches static R=2’s 26.0% WER at about a third of its median latency, while Common Voice WER is slightly worse (38.3 vs 37.5%).Source: D. Dynamic Future-Chunk Results · pp. 5–6

E. Streaming Baselines

On full test-clean, the dynamic trigger outperforms the evaluated blockwise Conformer on WER and latency, while the FastEmit transducer is a reference from a different model family. Out-of-domain comparisons show that latency gains can persist where WER gains do not.

E. Streaming Baselines

  • The dynamic trigger reaches 6.5% WER at 630/306 ms mean/median latency, below the blockwise Conformer’s 7.4% WER at 1366 ms mean latency.The blockwise system’s look-ahead variations do not improve on its reported best point.Source: E. Streaming Baselines · p. 6, Table V · p. 6
  • For out-of-domain transfer with S=16 on 500-utterance subsets, deferral helps up to WSJCAM0’s 26%; on Common Voice, R=1 and R=2 lookahead is equal and deferral no longer lowers WER.Source: Table IV · p. 6
  • At comparable WER, the FastEmit transducer reaches lower mean latency (327 vs 630 ms at 6.7 vs 6.5%), but it is a different model family and not a comparable baseline.It never defers; its frame-synchronous emission timing is fixed at training.Source: E. Streaming Baselines · p. 6, Table V · p. 6

VI. CONCLUSION AND FURTHER RESEARCH

The paper combines bounded-lookahead encoding with per-token future-chunk decoding to match fixed-lookahead accuracy at lower median latency. The gains transfer across several datasets, but lookahead does not resolve all domain-shift errors, and the approach has computational costs and acoustic-signal limits.

VI. CONCLUSION AND FURTHER RESEARCH

  • At matched WER, dynamic decoding reduces median latency from 860 to 306 ms, with a wait budget to bound the heavier deferral tail.The conclusion frames this as a contribution for attention-based models rather than frame-synchronous transducers.Source: §VI · p. 6
  • Latency reductions hold on LibriSpeech splits and LibriAdapt, TED-LIUM 3, and WSJCAM0 up to 26% WER, but Common Voice deferral does not lower WER.The authors attribute the residual Common Voice error to domain shift that lookahead does not fix.Source: §VI · p. 6
  • Age-versioning adds training memory and R+1 encoder passes per chunk, while commit-time acoustic signals cannot flag some linguistic errors.The cited examples include homophones, function-word swaps, and tokens later overturned by a downstream word.Source: §VI · p. 6

Related papers

Loading 2609.37611v1…