Source-linked AI summary
Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR
Xiuwen Zheng
TL;DR
Streaming ASR must balance transcription accuracy with when words appear, yet DSM’s structural delay and fixed alignment schedules do not measure or optimize actual word-emission delay. The paper defines AWED and post-trains a DSM recognizer with GRPO under a joint accuracy–delay reward; the resulting policy advances the accuracy–latency frontier across evaluated lookahead budgets.
Problem
DSM’s structural delay is only a nominal lookahead bound, and alignment-supervised training reuses a fixed emission schedule across τ values rather than measuring actual word delay.
Source: §1 · p. 1Method
The paper defines AWED relative to each word’s acoustic end and post-trains a DSM recognizer with GRPO using a joint transcription-accuracy and measured-delay reward.
Source: §1 · pp. 1–2Results
Across budgets, the policy dominates both baselines; at 480 ms, Exp 4c lowers WER to 7.84% (vs. 8.31% in Exp 0) and AWED p50 to 1.04 s (from 1.17 s).
Source: §1 · p. 2, §4.1 · p. 4Takeaways & Limitations
Latency-rewarded post-training advances the accuracy–latency Pareto frontier, generalizing from one training τ to all evaluated lookahead budgets.
Source: §1 · p. 2Reported numbers are internally consistent across systems but are not directly comparable to published leaderboard entries.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $\tau$ that bounds the decoder's lookahead. We show that $\tau$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $\tau$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($\tau=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.
1. INTRODUCTION
DSM makes large audio-language models streamable by conditioning on structural lookahead τ, but streaming ASR must balance transcription accuracy with when words appear. The paper introduces AWED and latency-rewarded GRPO to address unmeasured delay and fixed alignment schedules, advancing the accuracy–latency frontier from one training point.
1. INTRODUCTION
- Streaming ASR serves live captioning, voice assistants, meeting transcription, and simultaneous interpretation, so user-perceived quality depends on both correctness and timing.The introduction frames progress as movement along the accuracy–latency Pareto frontier rather than WER alone.Source: §1 · p. 1
- DSM’s structural delay τ bounds decoder lookahead but poorly reflects user-perceived word latency; at τ = 1, median AWED is 0.77 s despite a nominal 80 ms delay.A word is committed after its aligned end plus τ, decoder hesitation, and enough subsequent tokens to disambiguate it.Source: §1 · p. 1
- Alignment-supervised DSM reuses one schedule across τ values, requiring words to wait τ frames after their aligned ends even when they are already unambiguous.SFT imitates a predetermined schedule, while a hand-designed heuristic would require knowing each word’s optimal emission time in advance.Source: §1 · p. 1
- AWED measures word-level emission delay relative to each word’s acoustic end, while GRPO post-training jointly rewards transcription accuracy and measured delay.The latency reward is gated by a match rate that withholds credit when rollout WER is too high for early emission to be meaningful.Source: §1 · pp. 1–2
- Trained at τ = 6, the policy dominates its SFT initialization and Voxtral Realtime across the latency spectrum, reducing WER 30.8% relative at 80 ms with no latency cost.The single operating point generalizes across evaluated lookahead budgets.Source: §1 · p. 2
2. METHOD
DSM trains a streaming recognizer to follow a fixed emission schedule offset from aligned word endings, while AWED measures word-level delay against those endings. The method then uses GRPO to reward transcription quality and emission delay jointly, rather than prescribing a wait schedule.
2.1. Delayed Streams Modeling and Supervised Baseline
- DSM places audio and text on a shared time axis and trains each word to emit at its forced-aligned end plus a fixed τ-frame offset.The target fills remaining frames with [P] and shifts the same schedule across words and operating points.Source: §2.1 · p. 2
2.2. Aligned Word Emission Delay
- AWED measures matched words’ emission delay from their acoustic end and reports corpus quantiles, including p50, p90, and p95.Anchoring to word endings distinguishes it from measures based on alignment lattices, input length, or stream onset.Source: §2.2 · p. 2
2.3. Latency-Rewarded GRPO
- Because emission timing is part of DSM’s token action space, the method optimizes it with policy gradients and a clipped GRPO objective, training at one τ for transfer across operating points.Group-relative advantages remove the need for a learned critic; the policy is trained at a fixed conditioning delay.Source: §2.3 · pp. 2–3
- GRPO scores rollout accuracy and group-normalized mean AWED, withholding latency credit when the matched-word rate falls below a threshold.The gate addresses the fact that AWED is defined only for matched words.Source: §2.3 · pp. 2–3, §2.2 · p. 2
3. EXPERIMENTAL SETUP
The experiments use English speech data for training and evaluate across eight English test sets. Implementation uses a two-stage procedure, applying GRPO to an SFT checkpoint at a fixed delay.
3.1. Data Preparation
- Training uses approximately 175k hours from twelve public English corpora, retaining 1–30-second utterances and obtaining required word timestamps by forced alignment.Evaluation averages WER across eight English test sets spanning read speech, meetings, earnings calls, parliamentary speech, and lectures; the same hypotheses are used for AWED.Source: §3.1 · p. 3
- All systems use the same in-house text normalization, making reported comparisons internally consistent but not directly comparable to published leaderboard entries.The language decoder is updated with LoRA while the causal audio encoder remains frozen; Stage 1 SFT precedes Stage 2 GRPO at τ = 6 frames.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Implementation Details
- The implementation updates the language decoder with LoRA while freezing the causal audio encoder, then applies GRPO at τ = 6 frames to the Stage 1 SFT checkpoint.Stage 1 samples τ uniformly from 1 to 30 frames; six padding tokens place EOS six frames beyond the reference ending.Source: §3.2 · p. 3
4. RESULTS AND ANALYSIS
Stage 2 GRPO improves the accuracy–latency trade-off across the evaluated lookahead budgets, while ablations identify effective training-duration and latency-penalty settings. The results use mean Micro-WER and AWED delay to assess performance.
4.1. Main Results
- At 480 ms, Exp 4c lowers mean Micro-WER to 7.84% from 8.31% in Exp 0 and AWED p50 to 1.04 s from 1.17 s, dominating the accuracy–latency frontiers.At 80 ms, Exp 4c reaches 8.76% WER, a 30.8% relative reduction versus Exp 0 at comparable AWED p50 latency to Exp 1d.Source: Figure 1 · p. 3, §4.1 · p. 4
- At 80 ms, Stage 1 SFT lowers WER from 12.67% to 10.91% versus Exp 0 and reduces AWED p90 from 1.06 s to 0.89 s and p95 from 1.32 s to 0.95 s.The passage describes Stage 1 as a well-behaved checkpoint for Stage 2.Source: §4.1 · p. 4
4.2. Ablation Study
- Under λ = 0.025, 3k steps outperform 1k steps, and 3k–5k steps are sufficient for full convergence without over-fitting.The 3k-step setup has a comparable trade-off to 2k steps across the Pareto curve, with subtle additional WER gains at higher delay budgets.Source: §4.2 · p. 4
- Compared with λ values of 0.025, 0.05, and 0.075, λ = 0.01 yields noticeable accuracy gains as delay budgets expand, despite lagging slightly at 80 ms.The higher penalty settings yield near-identical trade-off profiles.Source: §4.2 · pp. 4–5