Source-linked AI summary
X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR
Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
TL;DR
Streaming ASR must produce accurate partial transcripts with low commit latency, but common fixed-context and boundary-based methods do not directly optimize context waiting at each output position. X2Streaming-ASR separates when to commit from what to commit through three-stage training, achieving 27–84 ms mean character-level latency and the best streaming CER among evaluated systems on AISHELL-1 and AISHELL-3.
Problem
Existing streaming ASR approaches do not directly optimize how much additional context to use at each output position while jointly addressing accuracy and commit latency.
Method
X2Streaming-ASR separates when to commit from what to commit using staged training with recognition pretraining, probed commit-time warm-starting, and character-level GRPO rewards.
Results
27–84 ms mean character-level latency is reported across five test sets, with 79–95% lower mean latency than streaming baselines and the best CER on AISHELL-1 and AISHELL-3.
Takeaways & Limitations
The method enables position-dependent commitment that waits at ambiguities and commits when evidence is sufficient rather than using a single global delay.
Abstract
from arXiv · showhide
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across AISHELL-1/2/3 and WenetSpeech, X2Streaming-ASR achieves a mean character-level commit latency of 27-84 ms relative to forced-aligned character endpoints, compared with 409-585 ms for the evaluated streaming baselines. It achieves the best streaming CER among the evaluated systems on AISHELL-1 and AISHELL-3 with substantially lower latency.
X Square Robot
The passage identifies the authors’ contact address and situates the work in streaming ASR, low-latency ASR, and reinforcement learning.
- The listed contact address is linzhiwei,wanghao@x2robot.com.
- The paper concerns streaming automatic speech recognition.
- The listed index terms include low-latency ASR and reinforcement learning.
1. INTRODUCTION
The paper addresses the difficulty of jointly optimizing streaming ASR accuracy and emission latency by separating when to commit from what to commit. X2Streaming-ASR uses adaptive commit decisions and staged training, achieving low latency and strong CER on selected test sets.
- Motivation: Streaming ASR must balance recognition accuracy with low-latency text output for real-time captioning, voice interaction, and downstream services.
- Problem: Existing fixed chunk, look-ahead, and delay methods do not adapt future context to historical and acoustic information.
- Problem: The system must decide both when to commit and what to commit, but supervised emit-time targets are imperfect and cross-entropy does not directly optimize latency and accuracy.
- Approach: X2Streaming-ASR replaces global fixed delay with position-dependent commit decisions for adaptive character commitment.
- Approach: Its three-stage framework establishes ASR capability, warm-starts the commit policy through probing, and uses character-level GRPO to optimize accuracy and emission latency.
- Results: 27–84 ms mean latency is reported across five test sets, with 79–95% reductions versus streaming baselines and the best CER on AISHELL-1 and AISHELL-3.
2. METHOD
X2Streaming-ASR separates streaming recognition into adaptive wait-or-emit decisions and text generation. Its three-stage strategy trains recognition, initializes commit timing from probed trajectories, and refines decisions with character-level rewards assigned to commit segments.
- Architecture: X2Streaming-ASR uses a causal audio encoder, adapter, and decoder-only language model that alternates between listening and decoding states.Unlike the underlying Voxtral Realtime setup, it removes the global delay τ and lets the model decide whether to wait or commit.
- Architecture: The model makes a binary wait-or-emit decision for each audio token; waiting reads the next token, while emitting generates text until an end-of-sequence token.After generation, the end-of-sequence token is not written back and the model returns to listening.
- Three-stage training: The first training stage masks wait and emit positions to train streaming ASR, recognizing only content whose acoustics have ended.For example, an input containing two and a half characters yields only the first two recognized characters.
- Three-stage training: Synthetic trajectories use blocks of 1-6 characters, with each block emitted at its final character endpoint plus 0, 80, or 160 ms.When another character follows, the cut point is restricted to no later than the first half of that character’s duration.
- Three-stage training: Commit-time labels are generated by probing incremental audio with greedy decoding, assigning each matched reference prefix the current probe time.Probing advances from character endpoints or in 80 ms steps, discards incomplete utterances, and excludes waits exceeding 640 ms after an acoustic endpoint.
- Three-stage training: Training trajectories divide into commit segments, and character-level group-relative advantages are assigned to the segment that produced or missed each character.The reward scores correct alignment against substitution or deletion penalties and latency, then policy optimization updates the corresponding wait-or-emit decisions.
3. EXPERIMENTS
Experiments evaluate X2Streaming-ASR across multiple corpora, baselines, and listen-policy settings, showing low latency and strong recognition performance. Ablations indicate that adaptive waiting, rather than a fixed global delay, drives the accuracy–latency trade-off.
- 3.1. Experimental Setup: Stage 3 uses reference characters and boundary times for reinforcement-learning rewards, while Stage 2 probing labels only selected training subsets with commit times.Stage 1 trains on five corpora; evaluation covers AISHELL-1/2/3 and WenetSpeech meeting and net tests.
- 3.2. Comparison with Baseline: 27–84 ms mean character-level latency is reported across the evaluated test sets, approximately one-tenth of Zipformer and Paraformer.P95 results indicate acceptable latency in extreme cases, whereas the two baselines approach 1 second.
- 3.2. Comparison with Baseline: X2Streaming-ASR outperforms Zipformer and Paraformer in offline and streaming recognition on AISHELL-1 and AISHELL-3.On AISHELL-2 and WenetSpeech, other baselines perform best on some test sets, but X2Streaming-ASR retains an order-of-magnitude latency advantage.
- 3.3. Global Post-Boundary Delay: 5.5 ms is the fastest forced post-boundary setting, but its CER reaches 6.90.An 80 ms global wait reduces CER to 3.40, while 160 ms raises mean latency to 147 ms without helping CER.
- 3.3. Global Post-Boundary Delay: X2Streaming-ASR waits when uncertain and commits when certain rather than selecting a compromise point on a fixed-delay curve.The comparison forces emission at tend+∆t for ∆t values of 0, 80, and 160 ms.
- 3.4. Ablation Study: Content-based KL divergence prevents recognition ability from drifting, while commit-policy optimization supplies the main CER and latency gains.Sentence-level rewards instead produce global waiting that sacrifices latency for improved recognition performance.
4. CONCLUSION
X2Streaming-ASR separates when to commit from what to submit and uses multi-stage training to optimize them separately. The model reduces average latency from hundreds of milliseconds to tens of milliseconds, while future work will examine more languages and real-world scenarios.
- X2Streaming-ASR decomposes streaming speech recognition into when to commit and what to submit.
- Multi-stage training separately optimizes recognition capability and commit policy, enabling the model to wait when uncertain and commit when certain.
- Average latency is reduced from hundreds of milliseconds to tens of milliseconds.
- Future work will test more languages and real-world scenarios to improve robustness and recognition performance.