SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

1 INTRODUCTION

• Semantic tokens capture content but lose acoustic detail, whereas acoustic tokens preserve speech information but can model content inaccurately.Hierarchical models improve both dimensions but introduce error accumulation, slower processing, and redundant information.Source: §1 · pp. 1–2
• SLMTokBench evaluates whether speech tokens provide strong text alignment and effective speech-information preservation for speech language modeling.The benchmark addresses the lack of prior evaluation of token suitability for this purpose.Source: §1 · p. 2
• SpeechTokenizer uses an encoder-decoder with residual vector quantization to disentangle content and paralinguistic information across RVQ layers.A semantic teacher guides the first quantizer toward content, while later quantizers complement the remaining paralinguistic information.Source: §1 · p. 2
• SpeechTokenizer reconstructs speech comparably to EnCodec, performs strongly on SLMTokBench, and its USLM outperforms VALL-E in zero-shot TTS.The unified model includes autoregressive and non-autoregressive components.Source: §1 · p. 3

2 SLMTOKBENCH: SPEECH LANGUAGE MODEL TOKEN BENCHMARK

• SLMTokBench evaluates speech tokens along two requirements: strong text alignment and effective preservation of speech information.The benchmark is intended to assess suitability for constructing speech language models.Source: §2 · p. 3

2.1 TEXT ALIGNMENT EVALUATION

• Mutual information between discrete speech representations and text estimates text alignment, with a downstream model also evaluated by WER.A two-layer BLSTM is trained on LibriSpeech and evaluated on held-out data using mutual information estimation and test-set WER.Source: §2.1 · p. 3

2.2 INFORMATION PRESERVATION EVALUATION

• Speech-information preservation is evaluated by resynthesizing tokens and measuring content with WER and timbre with speaker similarity.The evaluation uses unit-HiFi-GAN or an EnCodec decoder and samples 300 LibriSpeech test utterances.Source: §2.2 · p. 3

2.3 COMPARING SEMANTIC & ACOUSTIC TOKENS

• Semantic tokens achieve high text mutual information but low resynthesized speaker similarity, whereas acoustic tokens show low WER and high speaker similarity but low text mutual information.HuBERT L9 units represent semantic tokens, while EnCodec codes represent acoustic tokens.Source: §2.3 · p. 4

3.1 MODEL STRUCTURE

• SpeechTokenizer uses an EnCodec-style convolutional encoder-decoder with BiLSTM semantic modeling and residual vector quantization.A semantic teacher guides the residual quantization process.Source: §3.1 · p. 4

3.2 SEMANTIC DISTILLATION

• Semantic distillation guides the first RVQ layer toward HuBERT representations so it captures content information, while later layers retain complementary paralinguistic information.The paper uses continuous representation distillation and pseudo-label prediction with HuBERT teachers.Source: §3.2 · pp. 4–5

3.3 TRAINING OBJECTIVE

• Training jointly optimizes reconstruction and semantic distillation through reconstruction, adversarial, feature-matching, RVQ commitment, and distillation losses.The reconstruction objective includes time-domain, multiscale frequency-domain, discriminative, and feature-matching components.Source: §3.3 · p. 5

3.4 UNIFIED SPEECH LANGUAGE MODEL

• The unified speech language model uses an autoregressive model for first-layer content tokens and a non-autoregressive model for subsequent tokens conditioned on phonemes, acoustic prompts, and previous quantizers.The generated token matrix is decoded into waveform speech by SpeechTokenizer.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUPS

• Experiments train SpeechTokenizer on LibriSpeech and USLM models on 44K hours of English Multilingual LibriSpeech speech, using EnCodec and VALL-E as baselines.Zero-shot TTS evaluation uses VCTK without speaker overlap between training and test data.Source: §4.1 · p. 6, §4.3 · p. 7

4.4 MAIN RESULTS

• SpeechTokenizer achieves lower WER and higher MUSHRA than EnCodec, while obtaining a comparable ViSQOL score in speech reconstruction.Speech reconstruction uses 300 LibriSpeech test samples with objective and subjective evaluation.Source: §4.4 · p. 7, Table 2 · p. 7, §4.2 · p. 7
• SpeechTokenizer’s first RVQ layer aligns more strongly with text while discarding timbre, whereas layers 1:8 preserve both content and speaker information.Subsequent layers compensate for timbre information discarded by the first layer.Source: §4.4 · p. 7, Table 3 · p. 8
• The USLM yields lower WER and higher speaker similarity than VALL-E in zero-shot TTS, supporting more precise content and speaker-information modeling.Evaluation uses unseen VCTK speakers with separate speech prompts and text inputs.Source: §4.3 · p. 7, §4.4 · p. 8, Table 4 · p. 8

5.1 CHOICES OF SEMANTIC TEACHERS

• HuBERT L9 representations outperform HuBERT units in Text Alignment and Information Preservation for both RVQ-1 and RVQ-1:8.Source: §5.1 · p. 8
• HuBERT L9 yields higher Text Alignment, whereas HuBERT average representations achieve better Information Preservation, reflected in lower WER.The authors attribute this trade-off to HuBERT average retaining some timbre information.Source: §5.1 · p. 8

5.2 EFFECTIVENESS OF INFORMATION DISENTANGLEMENT

• Speaker similarity increases as more reference-token layers are used, indicating speaker information is embedded from the second through the last RVQ layers.Using reference tokens from layers two through four achieves low WER and high speaker similarity.Source: §5.2 · p. 9, Table 5 · p. 9
• RVQ-1 representations scatter across speakers, while RVQ-2:8 representations cluster by speaker and remain distinct across speakers.Figure 3 visualizes mean-pooled layer outputs with t-SNE, coloring samples by speaker.Source: §5.2 · p. 9, Figure 3 · p. 9

• The paper states that its related work is provided in Appendix E.Source: §6 · p. 9

7 CONCLUSION

• SLMTokBench evaluates different speech token kinds, while SpeechTokenizer unifies semantic and acoustic discretization instead of using separate extraction models.Source: §7 · p. 10
• The unified speech language model based on SpeechTokenizer reports better generated-speech content accuracy and quality.Source: §7 · p. 10

A MUTUAL INFORMATION ESTIMATION

• Mutual information I(X; Y) is estimated with a variational distribution qθ(Y|X), and the test dataset provides an unbiased estimation.X denotes discrete speech representations and Y denotes text.Source: A MUTUAL INFORMATION ESTIMATION · p. 13

B MODEL STRUCTURE ABLATIONS

• BiLSTM achieves better text alignment than LSTM in the SLMTokBench ablation.The experiment uses average representations across all HuBERT layers as semantic teachers.Source: B MODEL STRUCTURE ABLATIONS · p. 14, Table 6 · p. 14

C CONTINUOUS DISTILLATION LOSS ANALYSIS

• The T-axis loss adapts continuous distillation by matching student and teacher representations at each timestep through L1 distance and cosine similarity.The loss uses the quantized first RVQ-layer output and a D-dimensional semantic-teacher representation.Source: C CONTINUOUS DISTILLATION LOSS ANALYSIS · p. 15
• The D-axis loss outperforms the T-axis loss on SLMTokBench for both text alignment and Information Preservation.It computes cosine similarity across each feature dimension, providing richer supervision than overall timestep-wise similarity.Source: C CONTINUOUS DISTILLATION LOSS ANALYSIS · p. 15

D DETAILS OF MODEL STRUCTURE AND DISCRIMINATORS

• The encoder uses residual convolutional blocks, downsampling, and BiLSTM layers, while the decoder mirrors this structure with transposed convolutions and LSTM layers.The encoder uses strides (2, 4, 5, 8), and the decoder reconstructs the final audio signal.Source: D DETAILS OF MODEL STRUCTURE AND… · p. 16
• RVQ training uses exponential-moving-average code updates, code replacement for unused entries, straight-through gradients, and commitment loss.The commitment loss is the MSE between quantizer input and output, with gradients computed only with respect to the input.Source: D DETAILS OF MODEL STRUCTURE AND… · p. 16

E RELATED WORK

• SpeechTokenizer unifies semantic and acoustic tokens by hierarchically disentangling speech information across residual quantization layers.Its first layer targets content, while later layers model remaining paralinguistic information.Source: E RELATED WORK · p. 17

F CODEBOOK ANALYSIS

• SpeechTokenizer RVQ-1 codes show clearer phoneme specialization than EnCodec RVQ-1 codes in conditional-probability analysis.A stronger diagonal contrast indicates greater phoneme purity and more accurate code-to-phoneme mapping.Source: F CODEBOOK ANALYSIS · p. 18, Figure 4 · p. 18
• Over 600 EnCodec RVQ-1 codes are unused, implying more RVQ layers and greater generation cost may be needed for speech quality.The stated consequences are increased space, time, and computation requirements in spoken language modeling.Source: F CODEBOOK ANALYSIS · p. 18
• SpeechTokenizer RVQ-1 achieves higher PNMI than HuBERT units and significantly outperforms EnCodec-RVQ-1.The authors attribute this result to effective semantic distillation and improved text alignment.Source: F CODEBOOK ANALYSIS · p. 18, Table 8 · p. 18

G EXTENSION TO UNSEEN LANGUAGE

• On unseen German and Chinese speech, RVQ-1 preserves clear content while losing timbre and prosody, whereas RVQ-1:8 reconstruction is close to the ground truth.These results support hierarchical disentanglement across languages despite training solely on English data.Source: G EXTENSION TO UNSEEN LANGUAGE · p. 19

H MELSPECTORGRAM ANALYSIS

• SpeechTokenizer RVQ-1 loses the horizontal stripes retained by EnCodec RVQ-1, indicating diminished timbre and prosody information.The comparison is made against raw-speech and EnCodec-RVQ-1 melspectrograms.Source: H MELSPECTORGRAM ANALYSIS · p. 20, Figure 5 · p. 20
• German speech exhibits the same melspectrogram pattern as English speech when resynthesized from SpeechTokenizer RVQ-1 tokens.The result is shown by comparing raw German speech with its RVQ-1 reconstruction.Source: H MELSPECTORGRAM ANALYSIS · p. 21, Figure 6 · p. 21