StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs

1 INTRODUCTION

• Single-path quantization can magnify small perturbations near quantization boundaries, while standard ASR loss does not supervise intermediate token stability.The resulting token shifts can disrupt speech–text alignment and leave LLMs with inconsistent inputs.Source: §1 · pp. 1–2
• Offline model ensembles raise inference cost and alignment challenges, while token-level consistency objectives can produce unstable gradients for discrete codes.These shortcomings motivate an integrated architecture and training strategy.Source: §1 · p. 2
• Voting-LFQ uses parallel branches and differentiable bit-level majority voting to produce a token that can withstand sparse branch errors.The bit-level vote can recover a token even when a majority of branches fail at the token level, provided bit-level errors remain sparse.Source: §1 · p. 2, Figure 2 · p. 3
• Noise-Aware Consensus Training gives a random minority of branches perturbed inputs and uses a consensus loss to align them with the clean-input majority.The training strategy supplies explicit intermediate supervision through multiple views of the input.Source: §1 · p. 2, Figure 2 · p. 3
• Over 60% relative UED reduction, from 26.17% to 10.17%, while maintaining top-tier reconstruction fidelity.The introduction reports this as StableToken’s tokenizer-level noise-robustness result.Source: §1 · p. 2
• StableToken improves SpeechLLM robustness in understanding, especially under severe noise, and substantially improves synthesis quality in speech generation.The reported performance gap over baselines widens as noise increases.Source: §1 · p. 3

2.1 OVERALL STRUCTURE

• StableToken embeds semantic tokenization in end-to-end ASR, pairing Voting-LFQ’s multi-branch fault tolerance with Noise-Aware Consensus Training to enforce perturbation invariance.Source: §2.1 · p. 3

2.2 THE VOTING-LFQ MODULE

• Parallel branches binarize projections and bit-wise average their votes; sign-thresholded consensus yields a robust token, tolerating sparse bit errors with negligible inference overhead.Source: §2.2 · p. 4

2.3 NOISE-AWARE CONSENSUS TRAINING

• A clean majority and perturbed minority of branches share each input; consensus pulls continuous noisy-branch representations toward a clean-anchored mean, providing smoother gradients than binarized-code training.Source: §2.3 · pp. 4–5

2.4 FINAL TRAINING OBJECTIVE

• The final loss weights transcript ASR cross-entropy, consensus consistency, commitment to quantized representations, and codebook-entropy regularization.Source: §2.4 · p. 5

• The evaluation plan covers tokenizer-level robustness and reconstruction fidelity, downstream ASR, SER, and TTS, then ablations, qualitative analysis, and a case study.Source: §3 · p. 5
• Training uses a whisper-large-v3-initialized encoder-decoder, 150k hours of speech, an 8192-token vocabulary, a 25Hz frame rate, and five voters.Source: §3 · p. 5
• Tokenizer robustness is measured by UED on FLEURS under synthetic and real, including OOD, noise; fidelity uses WER and MOS on LibriSpeech and SEED.Source: §3 · p. 5, Table 1 · p. 6
• Downstream comparisons use an isogenic Qwen2.5-3B SpeechLLM setup; tokenizer baselines span SSL-based, distilled, and supervised models.Source: §3 · p. 5

4.1.1 SUPERIOR NOISE ROBUSTNESS

• 10.17% average UED beats S3 Tokenizer (26.17%) and R-Spin (16.48%), including on out-of-distribution real-world noise.The comparison is achieved with a larger vocabulary, which the authors note makes token-level invariance more challenging.Source: §4.1.1 · p. 6

4.1.2 EXCELLENT RECONSTRUCTION QUALITY

• StableToken delivers state-of-the-art reconstruction performance on WER and MOS without compromising tokenizer quality.The evaluation uses synthesized audio from speech tokens on LibriSpeech and SEED.Source: §4.1.2 · p. 6, Table 2 · p. 6

4.2.1 ROBUST ASR PERFORMANCE

• At 0 dB SNR under severe OOD real-world noise, StableToken yields 20.34% WER versus the baseline’s 29.94%, a relative reduction of over 30%.On CHiME-4, WERs are 35.90% on test-real and 30.61% on test-simulated, approximately 30% lower than the next-best baseline.Source: §4.2.1 · p. 7

4.2.2 ROBUST SER PERFORMANCE

• StableToken maintains higher SER accuracy across noise types and levels, with accuracy degrading more slowly as noise increases.Performance is similar across tokenizers on clean audio.Source: §4.2.2 · p. 7, Figure 3 · p. 7

4.2.3 SUPERIOR TTS PERFORMANCE

• StableToken improves downstream TTS WER on both SEED-TTS subsets, improves MOS on SEED-TTSZH, and remains competitive on SEED-TTSEN.The paper also reports stronger downstream performance across understanding and generation tasks.Source: §4.2.3 · pp. 7–8, Table 3 · p. 8

4.3.1 COMPONENT ABLATION STUDY

• Removing Consensus Loss raises Real OOD UED from 10.96% to 17.43%, removing Noise-Aware Training raises LibriSpeech-Other WER from 4.88% to 5.52%, and the single-branch baseline performs worst overall.The authors describe the multi-branch architecture as both the structure supporting the training strategy and an inference-time ensemble that mitigates quantization errors.Source: §4.3.1 · p. 8, Table 4 · p. 8

4.3.2 ANALYSIS OF VOTER COUNT (N)

• Increasing voters from N=3 to N=5 substantially improves robustness and semantic preservation, while N=7 offers only marginal gains for added computational overhead.The authors select N=5 for all experiments.Source: §4.3.2 · p. 8, Table 5 · p. 9

4.3.3 CASE STUDY

• Bit-wise majority voting can recover the original token despite errors in most voters when the underlying bit-level errors remain sparse.At position 80, bit #5 is recovered by a 3-to-2 majority and bit #7 by a 4-to-1 majority, reconstructing token 3485.Source: §4.3.3 · p. 9, Table 6 · p. 9

Semantic Speech Tokenizers

• Early SSL tokenizers primarily encode phonetic information and can produce high Gross Pitch Error with vocoders, limiting their suitability for end-to-end SpeechLLMs (Sicherman & Adi, 2023; Polyak et al., 2021; Mousavi et al., 2024; Guo et al., 2025).These approaches derive discrete units from unlabeled data (Hsu et al., 2021; Baevski et al., 2020).Source: §5 · p. 9
• Hybrid tokenizers combine acoustic representations with semantic distillation, but high bitrate, longer sequences, and hierarchical RVQ tokens impede LLM integration (Zhang et al., 2023; Ye et al., 2025; Défossez et al., 2024; Siahkoohi et al., 2022; Yang et al., 2024b).Their acoustic detail can increase computational load and reduce training efficiency.Source: §5 · p. 10
• Fully supervised tokenizers quantize ASR encoder representations and optimize ASR loss, producing linguistic units that can retain prosody useful for expressive synthesis (Zeng et al., 2025; Du et al., 2024a;b).A downstream vocoder converts the discrete tokens into mel-spectrograms.Source: §5 · p. 10

Noise Robustness

• Noise robustness of discrete speech tokens has received less attention than ASR-model robustness; R-SPIN and NAST target SSL token stability, while Liu et al. (2025) address acoustic rather than semantic tokenizers.R-SPIN uses noise-perturbed views and acoustic-piece prediction (Chang & Glass, 2024); NAST penalizes changes in token distributions between clean and noisy inputs (Messica & Adi, 2024).Source: §5 · p. 10

6 CONCLUSION

• StableToken combines multi-branch processing with bitwise voting to address semantic-token instability in noise and improve downstream SpeechLLM robustness.Source: §6 · p. 10

A LARGE LANGUAGE MODEL (LLM) USAGE STATEMENT

• The authors report using an LLM only to polish grammar after the initial draft, with no research ideas, experimental designs, or scientific content generated by it.Source: A LARGE LANGUAGE MODEL (LLM) USAGE… · p. 17

B.1 TRAINING DATASETS FOR STABLETOKEN

• StableToken training uses hundreds of thousands of hours of open-source and in-house data, with the open-source datasets listed in Table 7.Source: §B.1 · p. 17, Table 7 · p. 17

B.2 TRAINING HYPERPARAMETERS FOR STABLETOKEN

• Table 8 summarizes the main hyperparameters used to train StableToken.Source: §B.2 · p. 17, Table 8 · p. 17

B.3 DETAILS OF STOCHASTIC PERTURBATIONS DURING TRAINING

• Training perturbations randomly select among Gaussian, pink, or brown noise, bit-crush distortion, and real-world noise, with ESC-10 reserved for OOD evaluation.Source: §B.3 · p. 18, Table 9 · p. 18

B.4 DISCUSSION OF CONSENSUS OBJECTIVE LOSS CHOICES

• The consensus objective uses L2 loss to reduce differences among branches, align with quantization commitment loss, and encourage similarity across clean and noisy views.The authors also note that cosine similarity can provide diminished gradients for localized bit flips in high-dimensional binary codes.Source: §B.4 · p. 18

B.5 AUDIO RECONSTRUCTION DETAILS

• The reconstruction pipeline maps speech-token representations to mel-spectrograms with flow matching, then uses HiFi-GAN to produce waveforms.Tables 10 and 11 summarize reconstruction datasets and training hyperparameters.Source: §B.5 · p. 18, Table 10 · p. 18, Table 11 · p. 19

B.6.1 COMPUTATIONAL OVERHEAD OF DIFFERENT VOTER COUNTS

• As voter count increases, parameters grow by about 0.033M between adjacent settings while FLOPs remain very close.The comparison covers N=1, 3, 5, and 7 on 30-second audio inputs.Source: §B.6.1 · p. 19, Table 12 · p. 19

B.6.2 EMPIRICAL INFERENCE EFFICIENCY BENCHMARK

• Parallel branches are confined to lightweight quantization, leaving the Transformer Encoder single-path and adding almost no end-to-end latency.The quantization branches can run concurrently on modern hardware.Source: §B.6.2 · p. 19
• Up to 13.5% GPU and 14.0% CPU memory savings accompany latency and throughput comparable to GLM-4-Voice-Tokenizer.The benchmark compares both systems under identical hardware conditions.Source: Table 13 · p. 20, Table 14 · p. 20, §B.6.2 · p. 20

B.7.1 VOCABULARY UTILIZATION AND DISTRIBUTION ENTROPY

• High entropy and substantial vocabulary use on LibriSpeech-test and AISHELL indicate rich, dispersed token distributions rather than collapse into a small subset.The authors interpret this as maintaining representational capacity for both English and Chinese.Source: §B.7.1 · p. 21, Table 15 · p. 21

B.7.2 CROSS-LINGUAL TOKEN SPECIFICITY

• High-frequency tokens are extremely rare or absent in the other language, indicating specialized phonetic units occupying distinct codebook sub-spaces.Tables 16 and 17 compare the top-frequency tokens in English and Chinese.Source: §B.7.2 · p. 21, Table 16 · p. 21, Table 17 · p. 21

B.7.3 QUANTITATIVE DISTRIBUTIONAL DIVERGENCE

• Non-zero KL Divergence values confirm distinct Chinese and English token probability distributions; the authors conclude voting allocates language-specific codebook regions while retaining efficient vocabulary use.The distributional comparison is reported in Table 18.Source: §B.7.3 · p. 22, Table 18 · p. 22

B.8 LONG AUDIO PROCESSING AND BOUNDARY STABILITY

• The boundary analysis is motivated by possible concatenation defects when independently processed chunks meet.The section frames boundary stability as relevant to streaming and long-audio chunking.Source: §B.8 · p. 22

B.8.1 MODEL CONFIGURATION AND SEGMENT LENGTH

• StableToken processes up to 30 seconds per inference pass; longer audio is split into chunks processed independently, then their token sequences are concatenated.Training audio was also limited to 30 seconds, with longer files truncated to the first 30 seconds.Source: §B.8.1 · p. 22

B.8.2 ANALYSIS OF BOUNDARY STABILITY

• The analysis measures UED separately in the first 15%, central 70%, and final 15% of clips under Gaussian noise.The results are presented in Table 19.Source: §B.8.2 · p. 22, Table 19 · p. 22
• 12.60% middle-region UED is only marginally below 13.85% at the start and 13.69% at the end, indicating no significant boundary degradation.Start and end regions each account for 15% of audio duration; their shares of total inconsistencies are 16.29% and 15.60%, respectively.Source: Table 19 · p. 22, §B.8.2 · pp. 22–23

C.1 EFFECT OF QUANTIZER PLACEMENT

• Layer 24 yields the lowest UED and strong WenetSpeech ASR, but SER falls to 59.51% versus 67.38% at Layer 16; Layer 8 has poorer robustness and ASR.The authors attribute the Layer 24 trade-off to abstraction of fine-grained acoustic information such as prosody and timbre.Source: §C.1 · p. 23, Table 20 · p. 23

C.2 EFFECT OF PERTURBATION RATIO

• 18.68% versus 20.94% UED favors the 3:2 over the 4:1 clean-to-noisy ratio; the authors hypothesize that more diverse training signals benefit ASR and SER.They propose that a higher proportion of perturbed branches makes the consensus objective more challenging and encourages perturbation-invariant representations.Source: §C.2 · p. 23, Table 20 · p. 23

C ABLATION STUDY ON ARCHITECTURAL HYPERPARAMETERS

• Layer 16 with a 3:2 ratio is the authors’ selected balance between semantic robustness and acoustic-feature preservation.The ablations compare Layer 8, Layer 16, and Layer 24 placements, plus a 4:1 clean-to-perturbed ratio.Source: C ABLATION STUDY ON ARCHITECTURAL… · p. 23, Table 20 · p. 23, §C.2 · p. 24

D BASELINE MODELS

• The study’s baselines span SSL-based, distilled, and supervised tokenizers, using officially released models.The listed models include HuBERT-500, NAST, R-Spin, SpeechTokenizer, X-Codec, Mimi, CosyVoice, CosyVoice2, and GLM-4-Voice.Source: D BASELINE MODELS · p. 24

E NOISE PROFILES

• Evaluation perturbations include Gaussian, Pink, and Brown Noise, Bit Crush Distortion, and real-world ESC-50 noise samples.ESC-10 is treated as out-of-domain and excluded from StableToken training; the remaining 40 ESC-50 noise categories are included in training.Source: E NOISE PROFILES · p. 24, Table 21 · p. 24
• Noise levels were adjusted so added perturbations do not obscure semantic content or affect human perception of the original speech.Table 21 summarizes perturbation types and intensities.Source: E NOISE PROFILES · p. 24, Table 21 · p. 24

F.1 TRAINING DATASETS FOR SPEECHLLMS

• Table 22 summarizes datasets used to train SpeechLLMs across ASR, SER, TTS, and SNTP tasks.The section describes these datasets as covering the four task types.Source: §F.1 · p. 25, Table 22 · p. 25

F.2 TRAINING HYPERPARAMETERS FOR SPEECHLLMS

• Table 23 summarizes downstream SpeechLLM training hyperparameters, which are shared across tasks and models unless otherwise specified.Source: §F.2 · p. 25, Table 23 · p. 26

F.3 PROMPTS USED IN DOWNSTREAM TASKS

• For each sample, a prompt is randomly selected from the relevant task’s set during fine-tuning and inference.Source: §F.3 · p. 25

F.3.1 ASR TASK PROMPTS

• The ASR prompt set contains varied instructions to transcribe or convert recorded speech into text.Source: §F.3.1 · p. 26

F.3.2 SER TASK PROMPTS

• The SER prompt set asks for emotion or sentiment identification from text, audio, or speech.Source: §F.3.2 · p. 26

F.3.3 TTS TASK PROMPTS

• The TTS prompt set contains varied instructions to synthesize or generate speech from text.Source: §F.3.3 · p. 26

G FULL RECONSTRUCTION RESULTS

• The comparison omits SSL-based semantic tokenizers because publicly available decoders for reconstructing audio from their tokens are unavailable.Source: G FULL RECONSTRUCTION RESULTS · p. 27
• Table 24 reports WER (↓) and MOS (↑) on LibriSpeech and SEED, and describes StableToken as combining strong noise robustness with competitive reconstruction quality.Source: Table 24 · p. 27
• The table notes that comparisons are most meaningful between tokenizers of the same type.Source: Table 24 · p. 27