Source-linked AI summary
X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR
Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
TL;DR
Streaming ASR must produce accurate partial transcripts with low commit latency, yet existing approaches struggle to jointly optimize recognition accuracy and emission latency. X2Streaming-ASR uses position-dependent commit decisions and three-stage training; across five test sets, it reports 27–84 ms mean latency, a 79–95% reduction versus streaming baselines, and the best CER on AISHELL-1 and AISHELL-3.
Problem
Accurate, low-latency character output matters for real-time captioning and voice interaction, while existing approaches struggle to jointly optimize recognition accuracy and emission latency.
Source: §1 · p. 1Method
X2Streaming-ASR makes position-dependent character commit decisions and trains through supervised ASR, commit-time probing, and character-level GRPO with commit-segment credit assignment.
Source: §1 · p. 2Results
27–84 ms mean latency across five test sets, a 79–95% reduction versus streaming baselines, with the best CER on AISHELL-1 and AISHELL-3.
Source: §1 · p. 2Takeaways & Limitations
The policy waits when uncertain and commits when certain rather than choosing a fixed-delay compromise.
Source: §3.3 · p. 4The authors leave evaluation in more languages and real-world scenarios, and improvements to robustness and recognition performance, for future work.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across AISHELL-1/2/3 and WenetSpeech, X2Streaming-ASR achieves a mean character-level commit latency of 27-84 ms relative to forced-aligned character endpoints, compared with 409-585 ms for the evaluated streaming baselines. It achieves the best streaming CER among the evaluated systems on AISHELL-1 and AISHELL-3 with substantially lower latency.
1. INTRODUCTION
X2Streaming-ASR makes character-level commit timing position-dependent and trains the policy in three stages to balance recognition errors and latency. Across five test sets, it reports 27–84 ms mean latency and the best CER on AISHELL-1 and AISHELL-3.
- Streaming ASR is evaluated by recognition accuracy and latency, measured here as character-level time from acoustic boundary to output.Low-latency, accurate transcripts can help downstream services and turn-taking start early and correctly [1] [2, 3].Source: §1 · p. 1
- Because needed context varies by character, global chunking, look-ahead, or delay and boundary-shifting methods cannot adapt waiting by position, while revision-based methods violate single-pass hard commitment [4, 5, 6, 7, 8, 9] [12] [14, 15, 16, 17].Source: §1 · p. 1
- X2Streaming-ASR first trains streaming ASR, warms a commit policy from probed character times, then uses character-level GRPO with commit-segment credit assignment to optimize recognition accuracy and latency.The probed times provide a warm start, not ground truth; the reward prioritizes errors, so the model waits at ambiguities and commits once evidence is sufficient.Source: §1 · pp. 1–2
- 27–84 ms mean latency across five test sets, reducing mean latency by 79–95% versus streaming baselines while achieving the best CER on AISHELL-1 and AISHELL-3.Source: §1 · p. 2
2. METHOD
X2Streaming-ASR separates streaming recognition into deciding when to commit and what text to emit, using an adaptive wait/emit policy and a three-stage training strategy.
2.1. Architecture
- X2Streaming-ASR replaces Voxtral Realtime’s global-delay conditioning with adaptive wait/emit decisions, alternating between listening to audio tokens and autoregressively decoding text after an emit.A wait continues listening to the next audio token; an emit stops audio input for that segment, after which decoding returns to listening.Source: §2.1 · p. 2
2.2. Training Strategy
- Stage 1 trains ASR alone on incremental audio, emitting only characters whose acoustics have ended, using aligned blocks of 1–6 characters with 0-, 80-, or 160-ms endpoint offsets.The loss is masked at wait and emit positions so this stage trains recognition capability rather than the commit policy.Source: §2.2 · pp. 2–3
- Stage 2 jointly trains recognition and an initial commit policy from first-stage weights, using automatically probed character emit-time labels.Training mixes in offline full-text recognition at a 0.20 ratio; utterances are discarded if full-text commit time is unavailable or any character waits over 640 ms past its acoustic endpoint.Source: §2.2 · p. 3
- Stage 3 uses GRPO to compare trajectories on the same reference character and assigns each character’s advantage to the commit segment that produced or missed it.This avoids broadcasting sentence-level rewards that would reinforce useful and superfluous waits equally; insertion penalties are assigned to the segment that produced them.Source: §2.2 · p. 3
3. EXPERIMENTS
X2Streaming-ASR combines low character-level commit latency with strong recognition results on selected test sets; global-delay and ablation experiments indicate that its learned commit policy waits adaptively rather than following a fixed delay.
3.1. Setup
- Stage 1 trains on five corpora, while commit-time labels for Stage 2 are probed only on AISHELL-1/2/3 and WenetSpeech L/S; Stage 3 uses that same labeled pool.Stage 3 rewards use reference characters and their aligned end times without extra emit-time labels; evaluation uses AISHELL-1/2/3 and WenetSpeech meeting and net tests, with latency measured relative to forced-aligned character endpoints.Source: §3.1 · p. 3
3.2. Comparison with Baseline
- 27–84 ms mean character-level latency across test sets is approximately one-tenth of Zipformer and Paraformer-online, while X2Streaming-ASR leads both in streaming and offline recognition on AISHELL-1 and AISHELL-3.P95 results indicate acceptable latency in extreme cases, while the two baseline latencies approach one second. Uni-ASR and other baselines lead on some other test sets, but X2Streaming-ASR retains an order-of-magnitude latency advantage.Source: §3.2 · pp. 3–4, Table 1 · p. 4
- On AISHELL-2 and WenetSpeech net, Uni-ASR leads, while Zipformer leads on WenetSpeech meeting; X2Streaming-ASR still has an order-of-magnitude latency advantage.The authors attribute similar streaming results on these sets to X2Streaming-ASR's weaker upper limit of recognition ability.Source: §3.2 · p. 4
3.3. Global Post-Boundary Delay
- 5.5 ms latency for immediate commitment accompanies 6.90 CER; an 80 ms global wait reduces CER to 3.40 but has worse latency and WER than natural online inference.A 160 ms wait yields 3.84 CER and 147 ms mean latency; the authors conclude the model waits when uncertain and commits when certain rather than choosing a point on a fixed-delay curve.Source: §3.3 · p. 4, Figure 2 · p. 4
- Stage 2 tends toward globally early commitments; content-based KL prevents recognition drift, commit-policy optimization drives CER and latency gains, and sentence-level rewards lead to global waiting.The KL-anchored Stage 3 model keeps recognition ability near Stage 2, while sentence-level reward optimization sacrifices waiting latency for improved recognition performance.Source: §3.3 · pp. 4–5
4. CONCLUSION
X2Streaming-ASR separates when to commit from what to submit, using multi-stage training to enable waiting under uncertainty and committing when certain. Experiments reduce average latency from hundreds of milliseconds to tens of milliseconds.
4. CONCLUSION
- X2Streaming-ASR reduces average latency from hundreds of milliseconds to tens of milliseconds.The conclusion reports this average-latency comparison without specifying exact values.Source: §4 · p. 5
- X2Streaming-ASR separates when to commit from what to submit and uses multi-stage training to optimize recognition capability and commit policy separately.This training enables the model to wait when uncertain and commit when certain.Source: §4 · p. 5
- Future work will test X2Streaming-ASR in more languages and real-world scenarios to improve robustness and recognition performance.Source: §4 · p. 5