Source-linked AI summary
ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs
Xingchen Song, Di Wu, Binbin Zhang, Zhendong Peng, Bo Dang, Fuping Pan, Zhiyong Wu
TL;DR
Chunk-based streaming ASR needs low latency, but smaller chunks often raise WER and token-emission timestamps omit time spent waiting to form a chunk. ZeroPrompt appends zeroed future content to prompt future-token predictions, with Prompt-and-Refine revising them as real chunks arrive. The paper reports lower TDT-F and TDT-L with unchanged WER on Aishell-1, and supporting generalization results on Librispeech.
Problem
Chunk-based streaming ASR needs low-latency output, but reducing chunk size usually raises WER and token-emission latency omits chunk-formation wait.
Source: §1 · p. 1Method
Results
ZeroPrompt reduces TDT-F by 350 ∼700ms and TDT-L by 100 ∼400ms while keeping WER unchanged; Librispeech results support generalization without extra effort.
Source: §3.2 · p. 4Takeaways & Limitations
ZeroPrompt requires no model retraining and nearly zero engineering cost to add to chunk-based streaming decoding.
Source: §1 · p. 2Early prompts may be inaccurate from limited context and often produce homophone errors, which Prompt-and-Refine can quickly correct.
Source: §2 · p. 2, §3.4 · p. 4
Abstract
from arXiv · showhide
In this paper, we present ZeroPrompt (Figure 1-(a)) and the corresponding Prompt-and-Refine strategy (Figure 3), two simple but effective \textbf{training-free} methods to decrease the Token Display Time (TDT) of streaming ASR models \textbf{without any accuracy loss}. The core idea of ZeroPrompt is to append zeroed content to each chunk during inference, which acts like a prompt to encourage the model to predict future tokens even before they were spoken. We argue that streaming acoustic encoders naturally have the modeling ability of Masked Language Models and our experiments demonstrate that ZeroPrompt is engineering cheap and can be applied to streaming acoustic encoders on any dataset without any accuracy loss. Specifically, compared with our baseline models, we achieve 350 $\sim$ 700ms reduction on First Token Display Time (TDT-F) and 100 $\sim$ 400ms reduction on Last Token Display Time (TDT-L), with theoretically and experimentally equal WER on both Aishell-1 and Librispeech datasets.
1. Introduction
Chunk-based streaming ASR faces a latency–accuracy trade-off, and token-emission timestamps can miss the wait needed to form a chunk. ZeroPrompt uses zeroed future content to prompt future-token predictions, with TDT-F and TDT-L measuring the first and last character display times.
1. Introduction
- Reducing chunk size can lower latency, but usually raises WER; token-emission latency also omits the wait to form a valid chunk.The paper illustrates that a token timestamp moved to 800ms may still require waiting until 1200ms for a chunk to form.Source: §1 · p. 1
- TDT-F and TDT-L measure the minimum chunk cumulative time needed to output the first and last character, respectively.The metrics target user-perceived display time; the paper ignores chunk computation time because it is usually much smaller than chunk cumulative time.Source: §1 · p. 1
- ZeroPrompt decreases TDT-F and TDT-L for partial recognition while keeping WER unaffected for final recognition, achieving a latency-WER trade-off without requiring model retraining and with nearly zero engineering cost.Source: §1 · p. 2
- ZeroPrompt is training-free and nearly costless to integrate into chunk-based streaming decoding.It appends zeroed content to each chunk to prompt future-token prediction using the encoder’s implicitly learned Masked-LM ability.Source: §1 · pp. 1–2
2. Proposed Methods & Related Works
ZeroPrompt appends zeroed future frames to each chunk so the streaming encoder can predict future tokens before those frames arrive. Prompt-and-Refine revises those predictions when real future content arrives, while the attention mask protects current-chunk predictions.
2. Proposed Methods & Related Works
- The encoder receives the history cache, current chunk, and appended zeroed future frames, producing predictions for both the current and ZeroPrompt chunks.The paper relates this ability to zero-shot Masked-LM prediction and MaskPredict.Source: §2 · p. 2
- Real-context LookAhead methods wait for future content, while a two-pass method has baseline-equivalent TDT at equal chunk size.CUSIDE instead simulates future frames with an extra jointly trained encoder and an APC self-supervised loss.Source: §2 · p. 2
- ZeroPrompt concatenates current-chunk and prompted-chunk decoding results, whereas CUSIDE uses simulated futures only to enhance current-chunk recognition.CUSIDE’s TDT-F and TDT-L remain identical to the causal baseline at equal chunk size, while ZeroPrompt requires no extra parameters or retraining.Source: §2 · p. 2
- First ZeroPrompt predictions may be inaccurate when context is lacking, with many errors being homophones that Prompt-and-Refine quickly corrects.The paper’s example contrasts “剩” (sheng) and “甚” (shen).Source: §2 · p. 2
3. Experiments
Experiments on Aishell-1 and Librispeech evaluate ZeroPrompt across prompt lengths and chunk sizes, reporting lower display times with unchanged final WER while examining latency–RTF trade-offs and prompt errors.
3. Experiments
- Experiments used Aishell-1 and Librispeech with official WeNet checkpoints and the datasets’ open-source recipe settings.ZeroPrompt was applied to well-trained chunk-based ASR models without retraining.Source: §3 · p. 2
3.1. Metrics
- PER divides prompt errors by the number of prompts, while PPC measures average future characters prompted per chunk.The number of prompts is the number of future characters decoded from the ZeroPrompt chunk.Source: §3.1 · p. 4
3.2. Main Results
- Longer ZeroPrompt lengths generally lower TDT but raise PER; beyond 640ms, chunk size and leading silence impose a latency ceiling.The ceiling reflects the time needed to collect a chunk and the lack of promptable content when both chunks contain only silence or zeros.Source: §3.2 · p. 4
- Final WER remained unchanged with autoregressive masking and Prompt-and-Refine, while longer input slightly increased RTF; PPC tracked ZeroPrompt length, not chunk size.PPC values were similar across chunk sizes, according to the reported results.Source: §3.2 · p. 4
- 350 ∼700ms reduction in TDT-F and 100 ∼400ms reduction in TDT-L, with WER unchanged; results generalize to Librispeech without extra effort.The paper reports this overall trade-off from Aishell-1 and says Librispeech results further support generalization.Source: §3.2 · p. 4
3.3. Solution to balance latency-RTF Trade-off
- Intermediate ZeroPrompt varies the encoder layer where the ZeroPrompt chunk begins, offering a way to adjust computation cost for latency and RTF needs.Table 2 identifies layer 0 as the default and -1 as the baseline without ZeroPrompt.Source: §3.3 · p. 4, Table 2 · p. 4
3.4. Error Analysis
- Onset homophone errors stem from partial pronunciations, whereas trailing long-prompt errors rely on history without acoustic hints; Prompt-and-Refine quickly corrects early mistakes.The paper notes that errors commonly occur at prompt beginnings or at the ends of very long prompts.Source: §3.4 · pp. 4–5, Figure 3 · p. 4