Acoustic Progress Propagation for Long-Horizon Speculative Decoding in ASR

1. INTRODUCTION

• Alignment-aware drafters can saturate at longer horizons because each step must re-identify the next token’s acoustic region, without continuously tracking audio position.Tokens span different amounts of audio, so advancing text by one word or subword does not directly indicate where to look next.Source: §1 · p. 1
• The proposed drafter recurrently propagates acoustic progress from a target-attention anchor and feeds that state back into audio cross-attention.The recurrent position updates guide successive speculative draft steps.Source: §1 · p. 1
• Joint training of the drafter and progress predictor over variable draft horizons combines token prediction with alignment supervision across rollouts of different lengths.This training is intended to improve stability and use of longer horizons.Source: §1 · pp. 1–2
• Across five ASR test sets, the method achieved longer acceptance lengths and higher lossless end-to-end speedups than evaluated baselines.The result is attributed to more effective use of long draft horizons.Source: §1 · p. 2

2.1. Speculative Drafter Architecture

• The acoustic-conditioned drafter fuses the last committed-token embedding with selected target-decoder features, then uses a Transformer block and the frozen target LM head to propose tokens.Draft states and the self-attention cache are carried across steps; target verification accepts the longest prefix consistent with greedy predictions.Source: §2.1 · p. 2

2.2. Acoustic Progress Propagation

• Each round initializes acoustic progress from target layer-21 attention at the last committed token, then predicts recurrent position updates from the prior progress and pre-cross-attention state.Later draft steps use predicted progress without additional target-attention queries; positions and increments are measured in seconds, with only the predictor’s position input normalized by utterance duration.Source: §2.2 · p. 2
• A Gaussian log-bias centered on clipped acoustic progress modifies audio cross-attention scores, and the retrieved context feeds into the next progress prediction.The bias favors positions near the predicted location while retaining access to the full encoded audio context.Source: §2.2 · pp. 2–3

2.3. Training

• The drafter and progress predictor are jointly trained on target-generated greedy trajectories, using teacher-forced next-token embeddings while progress follows its recurrence from a target-attention anchor.Training applies token cross-entropy and feature matching, and uses forced-alignment token-span midpoints for progress supervision when alignments are valid.Source: §2.3 · p. 3
• The training objective combines depth-decayed token losses with weighted feature-matching and progress-supervision terms.The horizon range is specified in Sec. 3.1.Source: §2.3 · p. 3

3.1. Experimental Setup

• Experiments used frozen Qwen3-ASR-0.6B and 1.7B targets, with 200 utterances per test split across five sets and greedy decoding.End-to-end timing used one H100 GPU at batch size 1; the figure reports acceptance length and speedup versus inference horizon for the 0.6B model.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Main Results

• The method achieved the highest end-to-end speedup and average acceptance length across five test sets and both target sizes, improving macro-averaged speedup over AnchorDraft by 34.3% and 9.0%.The respective improvements were 46.6% and 16.3% for average acceptance length.Source: §3.2 · p. 4, Table 2 · p. 4
• Acceptance length continued growing beyond baseline saturation, while the method had the highest speedup at every tested horizon and peaked at K = 8.AnchorDraft with variable-horizon training plateaued by K = 6; beyond K = 8, further acceptance gains did not offset longer draft-and-verify rounds’ increased cost.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Ablation Study

• With supervision and feedback retained, recurrent propagation outperformed absolute-position prediction in both metrics; supervision without feedback performed comparably to removing progress modeling.The ablation table covers Qwen3-ASR-0.6B, 1,000 test utterances, and inference horizon K = 8.Source: §3.3 · p. 4, Table 2 · p. 4
• Variable-horizon training outperformed fixed-K = 8 training in both metrics, while the fixed-horizon model also improved both metrics over K = 3.The frozen-drafter variant underperformed joint training, but differing schedules prevent isolating joint optimization’s effect.Source: §3.3 · p. 4, Table 2 · p. 4

• The proposed speculative ASR drafter propagates acoustic progress to guide audio cross-attention, with joint variable-horizon training improving use of longer drafts.Across five ASR test sets and two target sizes, higher acceptance lengths and lossless end-to-end speedups over evaluated baselines support progress propagation beyond local alignment alone.Source: §4 · p. 4