RVQ Position Aware Speculative Decoding for On Device Text to Speech

• RVQ position-aware speculation extends Medusa with a position-aware hidden-state offset and sparse tree attention, fusing drafting and verification into one compiled MultiCodeDecoder call.The approach reuses the model’s per-position heads rather than adding new heads.Source: §1 · pp. 1–2
• Real-time Qwen3-TTS generation requires at least 200 sequential calls per second, mainly because the MultiCodeDecoder emits 15 residual codes autoregressively for each 80 ms frame.The audio-token-to-waveform decoder adds negligible throughput demand relative to token generation.Source: §1 · p. 1, Figure 1 · p. 1
• 2–2.2× measured RVQ token generation speedup reduces real-time synthesis from 200 to approximately 88 sequential model calls per second.The method reaches 2.47 accepted tokens per MultiCodeDecoder call, with 10–20% per-round speculation and verification overhead.Source: §1 · p. 1

2.1. Medusa as the natural baseline

• The MultiCodeDecoder’s existing per-position input embeddings and LM heads make it a natural basis for Medusa-style drafting, avoiding new heads and leaving the output distribution unchanged.A direct Medusa adaptation would add heads even though the existing heads already cover the target code positions.Source: §2.1 · p. 2

2.2. Achieving higher tokens per step

• Tree attention drafts top-m candidates at each of K positions as candidate paths, then verifies the packed tree and committed prefix in one masked MultiCodeDecoder call.The sparse mask lets each token attend only to its ancestors; the longest verified path is committed.Source: §2.2 · p. 2, Figure 2 · p. 2
• Position-aware offsets let frozen LMH_k+j heads draft from h_k+b_j, while the unoffset LMH_k path emits the accepted code and remains identical to baseline autoregressive decoding.The K=3 learned residual vectors are shared across starting positions; only these offset vectors are trained, with the backbone and heads frozen.Source: §2.2 · p. 2, Figure 3 · p. 3
• The offsets are trained on English VoxPopuli data, and evaluation reports no loss in acceptance across six test languages.Training uses teacher-forced hidden states and subsequent ground-truth codes; the verifier stream never consumes offset states.Source: §2.2 · pp. 2–3
• Sparse-tree size trades a longer expected accepted prefix against higher verification cost, so the node budget is chosen per device.The tree is calibrated offline from per-depth and per-rank acceptance statistics and pruned to keep query length manageable.Source: §2.2 · p. 2, Figure 2 · p. 2

3.1. Dataset

• Offsets are trained and validated on disjoint English VoxPopuli subsets, tree configuration is selected on LibriSpeech, and acceptance and WER are evaluated on six Fleurs languages.Only English overlaps with training and calibration; experiments use the 0.6B CustomVoice variant and original sampling configuration.Source: §3.1 · p. 3, Table 1 · p. 3

3.2. Accuracy: WER parity

• The speculative decoder's WER is no higher than the autoregressive baseline in any language, with no audible differences reported in manual review.WER is measured through WhisperKit transcription; the authors report that the systems sample from the same top-k distribution.Source: §3.2 · p. 3

3.3. Theoretical acceleration: tokens/step

• 2.47 tokens/step with 3,072 added parameters matches Medusa heads at 2.60 tokens/step; acceptance remains 2.44–2.46 tokens/step across six languages.The offsets avoid Medusa's approximately 16% parameter increase and associated asset-size and memory costs.Source: §3.3 · p. 3, Table 2 · p. 4

3.4. Practical acceleration: per device wall time latency

• 2.34× and 2.13× speedup at N=32 on iPhone 17 Pro Max and MacBook M3 Pro, respectively; iPhone 12 peaks at 1.82× at N=16.Larger trees erode the iPhone 12 gain to 1.21× at N=64.Source: §3.4 · p. 4, Figure 4 · p. 4

4. CONCLUSION

• Across six Fleurs evaluation languages, the method sustains 2.44–2.46 accepted tokens per model call with WER parity against the autoregressive baseline, and delivers 2–2.2× RVQ token-generation speedup on recent iPhone and Apple Silicon Mac devices in the single-stream regime.Source: §4 · p. 4
• 2–2.2× RVQ token generation speedup on recent iPhone and Apple Silicon Mac devices in the single-stream regime.On iPhone 12, where autoregressive decoding nearly exhausts the real-time budget, the method unlocks comfortable real-time synthesis.Source: §4 · p. 4
• 3,072 trained residual offset parameters let frozen per-position LM heads draft ahead, while sparse tree attention fuses drafting and verification into one compiled call.The verifier stream remains untouched, and the method is distributionally lossless under top-k sampling.Source: §4 · p. 4
• The approach assumes an autoregressive codebook-depth decoder with reusable per-position heads; transfer to systems such as CaT-TTS remains future work.The unaccelerated 28-layer CodeDecoder is identified as a next target, alongside dynamic per-device selection of sparse tree size N.Source: §4 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

• The study uses only the publicly released VoxPopuli, LibriSpeech, and Fleurs datasets.Source: §6 · p. 5
• Each dataset is used under its respective license.Source: §6 · p. 5
• No new human-subjects data were collected, so no ethical approval was required.Source: §6 · p. 5