PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Discrete Speech Tokens: A Review

Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, Kai Yu

arXiv:2502.06490v4eess.AScs.AIcs.MMcs.SDeess.SP

TL;DR

Speech LLMs require compact discrete representations, but existing work lacked a comprehensive account of acoustic and semantic token methodologies. This review builds a taxonomy and conducts unified analyses across token types, finding different strengths in reconstruction and voice conversion while identifying persistent limitations and research directions.

  • Problem

    Existing reviews did not comprehensively cover the diverse methodologies of both acoustic and semantic discrete speech tokens.

  • Method

    The review taxonomizes discrete speech tokens, synthesizes their motivations and limitations, and evaluates token types across reconstruction, voice conversion, and semantic modeling.

  • Results

    Semantic tokens often achieve much higher voice-conversion similarity, whereas acoustic tokens generally provide stronger acoustic reconstruction and prosody preservation.

  • Takeaways & Limitations

    Discrete speech tokens offer compact representations compatible with speech LLMs, but token design requires balancing acoustic fidelity, semantic content, prosody, and speaker information.

  • Takeaways & Limitations

    Discrete tokens still underperform continuous features in many spoken-language-understanding and speaker-related tasks because quantization inevitably loses information.

Abstract

from arXiv · show

The rapid advancement of speech generation technologies in the era of large language models (LLMs) has established discrete speech tokens as a foundational paradigm for speech representation. These tokens, characterized by their discrete, compact, and concise nature, are not only advantageous for efficient transmission and storage, but also inherently compatible with the language modeling framework, enabling seamless integration of speech into text-dominated LLM architectures. Current research categorizes discrete speech tokens into two principal classes: acoustic tokens and semantic tokens, each of which has evolved into a rich research domain characterized by unique design philosophies and methodological approaches. This survey systematically synthesizes the existing taxonomy and recent innovations in discrete speech tokenization, conducts a critical examination of the strengths and limitations of each paradigm, and presents systematic experimental comparisons across token types. Furthermore, we identify persistent challenges in the field and propose potential research directions, aiming to offer actionable insights to inspire future advancements in the development and application of discrete speech tokens.

I. INTRODUCTION

Discrete speech tokenization converts continuous, boundary-less speech into compact sequences compatible with language modeling. This review organizes the field around acoustic and semantic tokens, surveys their methods and challenges, and compares their properties across applications.

  • Motivation: Speech tokenization is necessary because LLMs typically process discrete data, whereas raw speech is continuous and boundary-less.The goal is to transform long speech waveforms into compact discrete sequences compatible with textual representations.
  • Taxonomy: Acoustic tokens derive from low-bitrate neural codecs, while semantic tokens originate from speech self-supervised learning.The two categories reflect different design principles: information-preserving acoustic encoding versus more phonetic or semantic representation learning.
  • Representation trade-offs: Continuous representations generally provide higher reconstruction fidelity or understanding performance, but discrete tokens offer compactness, symbolic abstraction, and language-model compatibility.The comparison highlights a representation-learning trade-off rather than a universally superior representation.
  • Research gap: Existing reviews had not clearly constructed a comprehensive taxonomy covering diverse methodologies in both acoustic and semantic tokens.Earlier reviews emphasized spoken language modeling or neural audio codecs while overlooking semantic tokens or broader methodological coverage.
  • Review scope: The review provides a taxonomy, examines motivations and challenges, and compares acoustic and semantic tokens for reconstruction and voice conversion.It also summarizes challenges and future directions, including decoupled and variable-frame-rate tokens.
  • Preliminaries: Discrete speech tokens are obtained by quantizing continuous representations through offline clustering or online vector quantization.The review introduces quantization methods as preliminary knowledge before discussing token categories and cross-cutting techniques.
  • Quantization methods: K-means assigns each sample to its nearest centroid and iteratively updates centroids until convergence, with the centroids forming the token codebook.It is frequently applied to self-supervised speech features, including HuBERT features in GSLM.

B. Vector Quantization

Vector quantization converts continuous representations into discrete codebook entries and supports end-to-end learning through differentiable approximations. The review covers k-means VQ, Gumbel VQ, and finite scalar quantization, alongside codebook-utilization challenges.

  • K-means VQ: Vector quantization enables a learnable module that produces discrete representations while allowing gradients to pass through the network.Unlike isolated clustering, VQ can be optimized jointly with other neural-network modules.
  • K-means VQ: K-means VQ selects the code-vector closest to the continuous input, then feeds that code-vector to subsequent networks.Because the minimum operation is nondifferentiable, straight-through estimators graft gradients to the input during backpropagation.
  • K-means VQ: Commitment and codebook losses respectively pull the continuous input toward its selected code-vector and update the code-vector toward the input.Different weighting factors control the optimization strength applied to the input and codebook.
  • VQ challenges: Codebook collapse remains a challenge in high-dimensional VQ because codebook usage can become highly imbalanced.Proposed remedies include random replacement, entropy penalties, and factorized codebook lookup.
  • Gumbel VQ: Gumbel VQ replaces Euclidean assignment with differentiable sampling from logits, using Gumbel-Softmax during training and maximum-logit selection at inference.The temperature controls the trade-off between approximation accuracy and gradient variance.
  • Finite Scalar Quantization (FSQ): Finite scalar quantization rounds each vector dimension to one of L scalar levels, yielding V = L^d possible outcomes without an explicit codebook.Straight-through estimation is used to pass gradients through the rounding operation.

4) Other VQ Tricks:

Multiple quantization strategies expand representation capacity beyond a single VQ module. GVQ partitions inputs independently, whereas RVQ quantizes residuals sequentially, offering bitrate–performance flexibility at the cost of parallelism and optimization complexity.

  • A single VQ module can have a highly limited representation space, motivating multiple-quantizer designs.
  • GVQ partitions the input vector by dimensions and independently quantizes each group before concatenating the resulting code-vectors.
  • RVQ sequentially quantizes each residual, with the next input defined as the previous input minus its quantized code-vector.
  • FSQ avoids additional training loss terms and has been reported to achieve better codebook utilization than k-means VQ at large vocabulary sizes.FSQ quantizes vector dimensions into fixed scalar levels rather than using an explicit learned codebook.
  • FSQ's vocabulary size follows V = L^d, so its low-dimensional bottleneck can hurt performance at small vocabulary sizes compared with fully utilized VQ.Unlike VQ, FSQ fixes the quantization space and ties vocabulary size to dimensionality and level count.
  • RVQ provides more flexible bitrate–performance tradeoffs than GVQ, but sequential computation prevents efficient parallelization and may complicate optimization.The optimization difficulty is associated with nested STE operations.

A. Model Architectures

Acoustic-token codecs commonly use encoder–quantizer–decoder architectures with CNN, Transformer, or U-Net-based generators, alongside VQ-GAN, diffusion, and flow-matching objectives. Architectural and training choices trade computational efficiency, scalability, quantizer flexibility, reconstruction quality, and downstream token modeling complexity.

  • Acoustic codecs commonly transform inputs through an encoder, quantizer, and decoder, with inputs and outputs spanning waveforms, frequency-domain features, and SSL features.The encoder typically downsamples temporal inputs before quantization.
  • VQ-GAN combines VQ-VAE reconstruction and quantization objectives with adversarial discriminators that train the codec generator.Acoustic codecs often use multi-resolution and multi-scale STFT discriminators.
  • U-Net-based codecs place multiple quantizers at different network layers, enabling flexible control over the resolution of each VQ stream.Their cross-quantizer token correlations may be more complex for downstream modeling than those of adjacent quantizers in a single RVQ module.
  • A separate vocoder can improve audio quality over original decoded outputs, motivating two-stage VQ-VAE-plus-vocoder training.This approach separates discrete representation learning and waveform reconstruction.
  • CNN-based generators are widely used and lightweight, while Transformer-based generators are more scalable and are believed to provide greater compression capacity.SoundStream and EnCodec exemplify end-to-end CNN-oriented waveform codecs.
  • Diffusion-based codecs avoid adversarial training and have simpler objectives, but their inference latency is a major concern unless sampling is limited.Diffusion and flow-matching codecs use discretized tokens to condition transformations from Gaussian distributions to continuous acoustic representations.
  • Temporal-redundancy reduction methods encode global speaker, channel, or prosodic information separately from time-varying VQ codes.Examples use sequential or time-invariant global tokens and low-frame-rate prosody sequences.

3) Challenges:

Mixed-objective acoustic tokens seek to add semantic information or disentangle global speaker traits while retaining acoustic reconstruction. These strategies can improve semantic utility or compactness, but may introduce reconstruction, disentanglement, or optimization trade-offs.

  • Mixed-objective acoustic tokens: Acoustic tokens preserve waveform-relevant information but can become redundant and overly complex for downstream language modeling.Their reconstruction-centered design contrasts with representations intended primarily for semantic content.
  • Semantic feature guidance: Semantic feature guidance uses SSL features in early RVQ layers to establish a semantic-to-acoustic information hierarchy.SpeechTokenizer guides the first EnCodec RVQ layer with HuBERT features.
  • Semantic feature guidance: SSL-guided acoustic tokens add no inference cost because guidance occurs during training, and language-modeling TTS has reported better robustness with them.The reported robustness comparison is between guided and unguided acoustic tokens.
  • Fixed semantic codebook: Fixed semantic codebooks initialize RVQ codebooks from LLaMA-2 token embeddings and keep them frozen to reduce bitrate and improve semantic representation ability.This approach is described for LLM-Codec.
  • Semantic features as inputs or outputs: Joint acoustic-semantic quantization compresses both information sources into a shared latent space, with some approaches using semantic reconstruction or ASR objectives.Examples combine SSL features with acoustic features before RVQ and optimize acoustic and semantic objectives.
  • Semantic features as inputs or outputs: Aligned phoneme sequences supplied to the quantizer have also been reported to reduce bitrates.The passage reports this benefit without specifying a numerical reduction.

3) Challenges:

Speaker disentanglement makes acoustic tokens more compact by removing globally repeated timbre information, using adversarial, perturbation-based, or source-separation methods. However, optimization complexity, supervision requirements, and weaker voice-conversion performance remain challenges.

  • Disentanglement motivation: Removing global speaker timbre can reduce redundancy in acoustic tokens and lower the bitrate needed for downstream modeling.Speaker information repeated at every token timestep is treated as unnecessary when timbre is represented separately.
  • Disentanglement motivation: Existing codecs may implicitly decouple global speaker timbre and local content through the VQ bottleneck, while explicit methods add specialized training criteria.The review distinguishes implicit bottleneck effects from explicit disentanglement techniques.
  • Gradient reversal layer: A gradient reversal layer negates the classifier gradient, forcing latent features to fool a speaker classifier while the classifier improves.This adversarial mechanism is used to disentangle speaker information.
  • Gradient reversal layer: SSVC separates content and speaker representations with RVQ, contrastive speaker embeddings, and gradient-reversal-based disentanglement.PromptCodec similarly minimizes an SSIM loss between content and speaker representations.
  • Gradient reversal layer: FACodec factorizes speech into speaker timbre, content, prosody, and acoustic detail using supervised decoupling and separate RVQ modules.Each component receives a supervision signal tailored to the desired information factor.
  • Perturbation: Perturbing speaker timbre during training can prevent that information from being encoded through the discrete VQ bottleneck.LSCodec adopts time-stretching-based speaker perturbation for this purpose.
  • Source separation: Source-separation codecs assign different audio sources to separate RVQ modules, with DeCodec additionally enforcing orthogonality.These designs target more efficient and targeted processing of individual audio components.
  • Challenges: Disentanglement methods risk complex optimization, may require supervised data, and remain suboptimal to semantic tokens for voice-conversion performance.The review presents these as current limitations of acoustic-token disentanglement.

A. Semantic Tokens from SSL Models

Semantic tokens from SSL models are extracted through external or internal quantization and are organized by the learning objectives of their underlying models. Their advantages are offset by substantial data, computational, acoustic-fidelity, streaming, and generalization constraints.

  • Overview: Discretized SSL tokens are motivated by reduced computation costs while leveraging SSL models’ strong performance in speech understanding tasks.The review cites ASR, speaker verification, emotion recognition, and speech translation as major SSL application areas.
  • Token extraction: SSL models learn contextual representations from data-driven pretext tasks, after which their outputs can be discretized by external or internal quantization.External quantization operates on pretrained embeddings, whereas internal quantization is trained within the SSL model.
  • Token extraction: External quantization commonly applies clustering or VQ-VAE to embeddings from one or more pretrained SSL layers.HuBERT+kmeans units are a representative example.
  • Token extraction: Internal quantization uses a quantizer trained with the SSL network, providing an efficient way to extract semantic tokens.SSL models may use such quantizers to produce targets for their training objectives.
  • SSL model designs: Contrastive, predictive, and perturbation-invariant SSL models define distinct routes to semantic or phonetic tokenization.Contrastive models distinguish positives from negatives; predictive models predict external targets; perturbation-invariant models seek robustness to content-preserving changes.
  • Challenges: SSL models require large training datasets, while k-means extraction can be costly and unreliable in high-dimensional spaces such as 768 or 1024 dimensions.The review also reports sensitivity to clustering settings and loss of acoustic detail after quantization.
  • Challenges: Perturbation-invariant SSL methods rely on handcrafted content-preserving augmentations, have mostly been tested on small-scale data and models, and lack clear generative-task benefits.The review specifically identifies uncertainty for speech generation and spoken language modeling.
  • Supervised semantic tokens: Supervised semantic tokenizers place quantization within ASR-trained networks and preserve more paralinguistic information than direct speech transcription.S3 Tokenizer uses a single-codebook VQ layer between Transformer encoders and an ASR cross-entropy objective.

C. Speech Token Vocoders

Speech token vocoders reconstruct acoustic detail from semantic tokens, while length-reduction and variable-frame-rate methods address the sequence-length mismatch between speech and text. Acoustic BPE is highlighted as a lossless and duration-preserving alternative to deduplication, and VFR tokens allocate representation more flexibly across time.

  • Speech token vocoders: Semantic tokens used for speech generation require a speech token vocoder to compensate for acoustic detail lost during tokenization.Acoustic tokens instead pair naturally with decoders that output waveforms or spectrograms.
  • Speech token vocoders: Speech resynthesis systems combine discrete units with pitch, speaker, contextual, diffusion, or flow-matching components before waveform generation.Token-to-mel training avoids adversarial training compared with end-to-end token-to-waveform training.
  • Length reduction: Speech tokens commonly run at 25–100Hz, creating a major length mismatch with text that complicates decoder-only TTS and other language-model-based generation.Longer sequences are associated with harder training and less stable inference in the reviewed setting.
  • Length reduction: Deduplication removes consecutive repeated tokens, while acoustic BPE iteratively merges frequent adjacent token pairs into a new vocabulary.BPE establishes a deterministic mapping that permits lossless reconstruction.
  • Length reduction: Acoustic BPE identifies non-repetitive redundancy, preserves duration information, and offers flexible target vocabulary sizes unlike deduplication.Deduplication removes exact duplicates and discards token-duration information.
  • Length reduction: BPE produces distinct length-reduction patterns across token types, with semantic tokens generally showing significant reduction and LSCodec exceeding WavTokenizer and BigCodec among single-codebook acoustic tokens.The passage relates stronger BPE reduction to lower information density.
  • Variable frame-rate tokens: Variable frame-rate tokens allocate more bits to temporally dense, high-variance regions and fewer bits to regions with lower uncertainty.Silence and long vowels are described as low-density segments, unlike explosive consonants.
  • Variable frame-rate tokens: VFR tokens connect to acoustic unit discovery and may reduce bitrate while supporting boundary segmentation, including approaches producing approximately 5Hz tokens.SSL-based methods heuristically locate boundaries and pool segment-level representations before clustering.

VI. ANALYSIS OF DISCRETE SPEECH TOKENS

The review evaluates discrete speech tokens across reconstruction, perceptual quality, downstream tasks, semantic relevance, and robustness. It combines existing benchmarks with unified reconstruction and semantic-modeling experiments to compare acoustic and semantic tokens.

  • Evaluation dimensions: Discrete speech tokens can be evaluated through signal-level, perceptual, downstream-task, semantic/phonetic, and robustness metrics.These include PESQ, STOI, WER, speaker similarity, ASR, intent classification, phone purity, and unit edit distance.
  • Evaluation dimensions: Existing benchmarks differ in scope, covering codec reconstruction, downstream probing, generative tasks, or invariance and vocabulary utilization.Codec-SUPERB, ESPnet-Codec, DASB, and STAB emphasize different evaluation dimensions.
  • Interpretation: The studies support a trade-off between emphasizing semantic information and comprehensively preserving speech information.Prior analyses also report incomplete speaker–phonetic disentanglement, emotion degradation, non-idempotent codecs, and instability in complex environments.
  • Experimental design: The review addresses a gap by comparing semantic-token reconstruction more thoroughly alongside acoustic-token reconstruction and downstream semantic modeling.Semantic tokens are resynthesized with a timbre-controllable vocoder, while probing tasks follow the DASB setup.
  • Experimental design: The evaluation uses LibriTTS test-B with 500 utterances from unseen speakers, and all metrics are computed on 16kHz waveforms.Original utterances provide timbre information when required, and higher-rate reconstructions are downsampled.
  • Reconstruction findings: Acoustic tokens generally reconstruct speech better, while semantic tokens retain comparable intelligibility at much lower bitrates but lose prosodic and acoustic detail.Low frame rates make low WER difficult for acoustic tokens; semantic tokens show lower GPE, PESQ, and STOI but comparable WER.

D. Voice Conversion Analysis

Voice-conversion analysis compares timbre similarity, intelligibility, and prosody preservation across token types. Semantic tokens usually improve speaker similarity but sacrifice prosody, while explicit acoustic disentanglement performs better among acoustic VC-capable tokens.

  • Voice conversion: Semantic tokens often achieve higher voice-conversion speaker similarity than acoustic tokens.VC uses source utterances as content and different target speakers as prompts, with WER, SECS, and P.Corr measured.
  • Voice conversion: Semantic tokens generally obtain lower P.Corr scores because substantial prosody information is lost.P.Corr measures Pearson correlation between converted and source pitch contours.
  • Voice conversion: Among acoustic tokens capable of VC, explicitly disentangled FACodec and LSCodec outperform TiCodec’s implicit criterion.The comparison concerns voice-conversion outcomes under the same experimental setup.
  • Voice conversion: wav2vec 2.0 tokenization settings produce sharply different speaker–prosody trade-offs.Inner-quantizer tokens preserve prosody but retain more speaker information, whereas Transformer-layer clusters show the opposite pattern.
  • Voice conversion: S3 Tokenizer provides good intelligibility and VC ability while preserving prosody better than HuBERT-style SSL models.Its supervised VQ module may encode some prosodic information relevant to ASR.
  • Connection to understanding: Discrete tokens support unified spoken-language functionality, but their semantic emphasis creates a trade-off in preserved information.Semantic tokens are more prevalent in SLU and can provide invariance to speaker and noise information.

3) Unified Speech Understanding:

Discrete tokens support unified speech understanding and generation by connecting language-modeling components across tasks. Their benefits include simpler objectives and decoder-only generation, but quantization and representation choices still limit performance relative to continuous features.

  • Unified framework: Discrete tokens can unify speech understanding and generation through task-conditioned language models.Applications include ASR, translation, intent classification, TTS, speech editing, singing synthesis, and voice conversion.
  • Limitations: Discrete tokens remain inferior to continuous SSL features in many SLU tasks, especially speaker-related tasks.Quantization causes information loss, while adding codebooks may reduce accessibility of semantic information.
  • Text-to-speech: In TTS, discrete tokens bridge text-to-token acoustic modeling and token-to-waveform vocoding.They replace spectrogram regression with classification and fit decoder-only autoregressive language modeling.
  • Generation strategies: Autoregressive codec-based TTS predicts discrete codebook indices, while alternative systems use hierarchical, interleaved, masked, or diffusion-based token modeling.VALL-E combines autoregressive and non-autoregressive prediction over EnCodec RVQ tokens.
  • Comparison with continuous generation: Diffusion and flow-matching systems generate continuous features and can avoid quantization errors, offering higher potential fidelity, diversity, and controllability.The review contrasts this potential with weaker robustness in discrete-token generation models.

C. Text-Free Spoken Language Models

Text-free spoken language modeling seeks semantically coherent speech without transcription guidance, but speech-token sequences remain inefficient for language modeling. Lower frame rates and text alignment help address this challenge, while text-guided systems trade richer context for more stable coherence.

  • Motivation: Text-free spoken language models aim to generate semantically coherent speech directly from speech tokens without text transcription guidance.Their development remains difficult because speech sequences have lower semantic density, longer lengths, and paralinguistic information.
  • Existing efforts: Current TF-SLM efforts reduce token frame rates or align speech representations with text.Frame-rate reduction can reach approximately 5Hz, while alignment uses text-model initialization, feedback, interleaving, or related architectures.
  • Existing efforts: Lower frame rates mitigate sequence-length problems but can compromise reconstruction quality and remain subject to scalability limitations.The review identifies this as a limitation of the frame-rate strategy rather than a complete solution.
  • Limitations: TF-SLMs still struggle to generate semantically reliable long speech during inference because explicit transcription guidance is absent.Semantic coherence therefore remains an open problem despite substantial modeling efforts.
  • Text-guided alternatives: Text-guided spoken language models decompose speech dialogue into LLM-based understanding and streaming TTS connected through text.They typically use continuous features for understanding and discrete tokens for TTS output.
  • Text-guided alternatives: Single-layer discrete tokens simplify TG-SLM training and inference, but text mediation can omit emotion, prosody, and environmental context.This may reduce coherence and naturalness relative to the preceding speech context.

1) Low-Bitrate Tokens:

The review identifies bitrate, streaming, disentanglement, and preservation of speech information as continuing design challenges for discrete speech tokens. It also highlights efforts to combine semantic understanding with acoustic reconstruction and to use variable frame rates.

  • The lower bounds of bitrate and frame rate remain open problems, with stronger architectures, advanced VQ, and global-information disentanglement proposed as possible solutions.
  • Real-time tokenization is straightforward for fixed-receptive-field CNN codecs but remains challenging for semantic tokens derived from non-causal SSL Transformers.
  • Disentanglement can reduce bitrate, support anonymity, lower downstream modeling complexity, and enable independent control of voice properties, but current decoupling remains limited or harms reconstruction quality.
  • Variable-rate acoustic tokens may reduce bitrate and narrow the gap between speech tokens and natural-language units, but their practical benefits require further study.
  • Combining rich speech understanding with decent acoustic reconstruction is a central direction, with SSL information and explicit text supervision improving acoustic-token reconstruction and downstream modeling.
  • Semantic tokens from simple clustering may discard speaker information and prosody, weakening modeling of emotions, tones, singing, and meaningful non-verbal vocalizations.

APPENDIX I. LIST OF WIDELY-USED DISCRETE SPEECH TOKENS

The appendix summarizes widely used acoustic and semantic tokens, then describes reconstruction, voice-conversion, and semantic-modeling evaluations. It specifies token-vocoder procedures, probing approaches, datasets, metrics, and training considerations, including frame-rate adjustment and non-convergence cases.

  • Token inventories: Tables II and III summarize widely used acoustic and semantic speech tokens, respectively, using codec and semantic-token notation.
  • Reconstruction and voice conversion: Semantic-token reconstruction and voice-conversion comparisons use a separate timbre-controllable CTX-vec2wavα vocoder for each token.
  • Reconstruction and voice conversion: The CTX-vec2wavα vocoders follow a GAN-based paradigm, use approximately 32M generator parameters, and are trained on about 585 hours of LibriTTS data.
  • Semantic modeling evaluation: Semantic modeling is probed through index-based inputs or vector-based code-vectors, with the review selecting the index-based approach because speech-generation models typically use token indices.
  • Semantic modeling evaluation: ASR and intent classification are evaluated with LSTM-based networks, using LibriSpeech for ASR and token embeddings aggregated across codebooks before recurrent modeling.
  • Evaluation assumptions: Tokens below 50Hz are repeated to at least 50Hz for character-based CTC ASR, while no repetition is applied for intent classification.
  • Evaluation limitations: Some large-vocabulary tokens fail to converge during ASR training, producing repetitive and meaningless decoded characters.
Loading 2502.06490v4…