PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrete Audio Tokens: More Than a Survey!

Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, Bhuvana Ramabhadran, Benjamin Elizalde, Loren Lugosch, Jinyu Li, Cem Subakan, Phil Woodland, Minje Kim, Hung-yi Lee, Shinji Watanabe, Yossi Adi, Mirco Ravanelli

arXiv:2506.10274v3cs.SDcs.AIcs.CLeess.AS

TL;DR

Existing benchmarks lack a holistic, aligned comparison of discrete audio tokenizers across domains and tasks. This paper provides a systematic review and benchmark, finding that performance depends strongly on bitrate, domain alignment, and sampling rate.

  • Problem

    Existing benchmarks differ in domains, tokenizer coverage, and evaluation protocols, preventing holistic and comparable conclusions across audio tokenizers.

    Source: §3 · p. 15

  • Method

    The paper systematically reviews and benchmarks publicly available tokenizers across speech, music, and general audio, including controlled ablation studies.

    Source: §3 · p. 15, §5 · p. 37

  • Results

    Performance varies with bitrate, training-domain alignment, and sampling rate, with higher bitrates and matched domains generally improving reconstruction quality.

    Source: Results and Discussion · p. 17, §4.2 · p. 36

  • Takeaways & Limitations

    The analysis identifies open challenges in designing, evaluating, and applying discrete audio tokenizers.

    Source: §5 · p. 37

    Tokenizer differences in training data, sampling rate, and domain scope complicate fair benchmarking and motivate standardized evaluation pipelines.

    Source: §5 · p. 37

Abstract

from arXiv · show

Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and inference, as well as competitive performance across diverse downstream tasks. They provide a practical alternative to continuous features, enabling the integration of speech and audio into modern large language models (LLMs). As interest in token-based audio processing grows, various tokenization methods have emerged, and several surveys have reviewed the latest progress in the field. However, existing studies often focus on specific domains or tasks and lack a unified comparison across various benchmarks. This paper presents a systematic review and benchmark of discrete audio tokenizers, covering three domains: speech, music, and general audio. We propose a taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamability, and application domains. We evaluate tokenizers on multiple benchmarks for reconstruction, downstream performance, and acoustic language modeling, and analyze trade-offs through controlled ablation studies. Our findings highlight key limitations, practical considerations, and open challenges, providing insight and guidance for future research in this rapidly evolving area. For more information, including our main results and tokenizer database, please refer to our website: https://poonehmousavi.github.io/dates-website/.

1 Introduction

The paper argues that discrete audio tokens offer compact, multimodal representations but that existing acoustic-versus-semantic classifications and domain-specific evaluations are insufficient. It therefore introduces a broader taxonomy and unified benchmark spanning speech, music, and general audio, alongside controlled ablations of tokenizer design choices.

Neural codecs

  • Neural codecs replace continuous representations with discrete tokens using differentiable quantization and jointly trained reconstruction, perceptual, adversarial, generative, and auxiliary losses.Residual vector quantization enables end-to-end training by allowing gradients through discretization.Source: §1 · p. 2

Taxonomy

  • The acoustic-semantic distinction is insufficient because acoustic tokenizers can capture semantic information and semantic tokenizers can support generation, motivating a taxonomy based on architecture, quantization, training, streamability, and application domain.In speech, “semantic” tokens are more accurately phonetic units and typically do not carry conventional linguistic semantic content.Source: §1 · p. 2

Benchmark

  • The benchmark jointly evaluates tokenizers across speech, music, and general audio for reconstruction, downstream performance, and acoustic language modeling, addressing fragmented prior evaluations.Figure 1 identifies Codec-SUPERB and Versa, DASB, SALMon, and the Zero-Resource benchmark as the evaluation components.Source: §1 · p. 3, Figure 1 · p. 3

Ablation studies

  • Controlled ablations examine how sampling rate and single-domain versus multi-domain training affect tokenizer behavior under consistent conditions.The ablation study uses ESPnet-Codec to isolate specific training-design choices and improve fairness across comparisons.Source: §1 · pp. 3–4

2 Literature Review and Proposed Taxonomy · 2.1 Overall architecture · 2.2 Quantization Method

The paper organizes audio tokenizers around encoder–quantizer–decoder pipelines and a taxonomy spanning architecture, quantization, training, streamability, and target domain. Quantization methods differ in algorithm, representation structure, and bitrate allocation, with trade-offs between simplicity, information preservation, scalability, and compression efficiency.

2.1 Overall architecture

  • Audio tokenizers typically encode waveforms into frame-wise continuous embeddings, quantize them into discrete indices and vectors, and decode the quantized sequence into reconstructed waveforms.The encoder computes Z = fe(x), quantization produces (Ẑ, qt) = Q(zt), and the decoder reconstructs ˆx = fd(ˆz).Source: §2.1 · pp. 4–5
  • The proposed taxonomy classifies tokenizers by quantization method, encoder–decoder architecture, training paradigm, streamability, and target application domain.Figure 3 organizes these dimensions, while Table 1 surveys existing tokenizers across application, architecture, quantization, training, and streaming capability.Source: §2.1 · p. 5, Figure 3 · p. 5, Table 1 · p. 6

2.2 Quantization Method

  • Quantization maps each continuous embedding zt ∈ R^D to M discrete codebook indices and reconstructs a quantized vector by retrieving and combining the selected codewords.The quantizer uses an encoder Eq and decoder Dq over M codebooks, each containing K learnable D-dimensional vectors.Source: §2.2 · p. 5
  • Quantization algorithms vary by conversion strategy—including k-means, product quantization, and RVQ—and by whether bitrate allocation is fixed or adaptive.These two axes define the main organization of the subsequent quantization-method discussion.Source: §2.2 · p. 7

2.2.1 Quantization Algorithm

  • K-means performs post-training clustering of frozen self-supervised representations, typically without a built-in decoder, whereas RVQ sequentially quantizes residuals across codebooks and can support variable bandwidth through codebook selection.K-means remains prevalent for discriminative SSL tasks, while neural RVQ originated in SoundStream and is widely adopted in codec models.Source: §2.2.1 · p. 7
  • SVQ simplifies RVQ to one codebook for easier acoustic language-model training, while GVQ groups latent dimensions and independently quantizes each group to improve performance with fewer codebooks.SVQ may use larger codebooks to offset information loss; GVQ addresses RVQ’s concentration of information in its first codebook.Source: §2.2.1 · p. 8
  • FSQ quantizes each feature dimension into uniformly spaced scalar values, while MSRVQ and CSRVQ use temporal or hierarchical multi-scale residuals to preserve detail with fewer tokens or progressively improve reconstruction.FSQ is used in several audio-generation systems, MSRVQ combines coarse and fine temporal resolutions, and CSRVQ conditions finer decoding levels on coarser quantized residuals.Source: §2.2.1 · pp. 8–9

2.2.2 Fixed vs. Adaptive Bitrate

  • Fixed bitrate uses a predetermined number of bits per code index, whereas adaptive bitrate uses entropy coding and scalable bitrate varies the number of active codebooks across fixed operating points.Huffman or arithmetic coding can reduce effective bitrate post hoc, while Encodec provides scalable bitrate through codebook dropout during training.Source: §2.2.2 · p. 9

2.3 Encoder-Decoder · 2.4 Training Paradigm

The review organizes audio tokenizers by encoder-decoder architecture, input/output representations, training strategy, objectives, and auxiliary components. It highlights trade-offs between efficiency, long-range modeling, reconstruction and perceptual quality, codebook utilization, and phonetic or disentangled representations.

2.3.1 Architecture

  • CNN tokenizers are compact and resource-efficient but cannot capture long-range dependencies, while CNN+RNN and CNN+Transformer designs extend sequential modeling with added complexity or computational demands.Fully Transformer-based tokenizers may achieve strong compression and reconstruction but require substantial training data and computation, limiting practicality.Source: §2.3.1 · pp. 9–10

2.3.2 Input and Output Representations

  • Encoders operate on raw waveforms or spectral features, while decoders output waveforms directly or time-frequency features subsequently upsampled by ISTFT.Assigning upsampling to ISTFT can reduce decoder burden and improve performance (Siuzdak, 2024).Source: §2.3.2 · p. 10

2.4.1 Training Strategies

  • Audio tokenizer training is either separate, optimizing encoder-decoder components independently from quantization, or joint, optimizing encoder, quantizer, and decoder end to end.Separate training commonly freezes pretrained SSL encoders and quantizes latent features offline, whereas joint training combines reconstruction and often adversarial losses and uses gradient approximations for nondifferentiable quantization.Source: §2.4.1 · pp. 10–11

2.4.2 Main Training Objectives

  • Main objectives include reconstruction, vector-quantization alignment, adversarial perceptual optimization, feature matching, diffusion denoising, and masked prediction for phonetic representation learning.Reconstruction uses regression losses such as MSE or MAE; masked prediction is typically used with separately trained encoders and decoders to capture phonetic information.Source: §2.4.2 · pp. 11–13
  • Codebook collapse reduces effective codebook size, bitrate, and reconstruction quality, motivating EMA updates, codebook expiration, factorized quantization, normalization, entropy penalties, and code-balancing methods.ERVQ combines intra-codebook online clustering and balancing with inter-codebook redundancy reduction to improve utilization and expressiveness.Source: §2.4.2 · pp. 11–12
  • Adversarial and feature-matching objectives improve perceptual quality and training stability by comparing real and reconstructed signals through discriminator outputs and intermediate activations.Multi-scale discriminators can process complex-valued STFT representations at multiple window scales to capture different audio structures.Source: §2.4.2 · p. 12

2.4.3 Auxiliary Components

  • Disentanglement separates speech attributes such as content, prosody, timbre, background audio, and acoustic details, enabling bitrate or entropy control, speaker decoupling, enhancement, and source separation.Approaches use separate encoders, quantizers, codebooks, global embeddings, or supervised decomposition to isolate time-invariant and time-varying information.Source: §2.4.3 · p. 13
  • Semantic distillation and supervised semantic tokenization inject phonetic information through SSL guidance, semantic codebook initialization, dual encoders, phonetic classification, or ASR objectives, improving suitability for speech-LM understanding.These methods typically concentrate phonetic information in early codebooks while later layers preserve acoustic detail; the terminology reflects source papers, although SSL features predominantly encode phonetic information.Source: §2.4.3 · pp. 13–14

2.5 Streamability and Domain Categorization

This section categorizes audio tokenizers by streamability and target domain. Streamability depends on latency from future context and computational complexity, while tokenizers target speech, music, general audio, or multiple domains.

Streamability

  • Streamability enables real-time audio processing with minimal latency and little or no future context, which is critical for communication and streaming applications.Latency has two dimensions: algorithmic latency from look-ahead requirements and computational complexity for deployment on constrained devices.Source: §2.5 · p. 14
  • CNN-based models support streaming through causal convolutions, whereas Transformer-based models require causal attention mechanisms.These mechanisms control the future context needed to compute each current frame.Source: §2.5 · p. 14
  • Real-time feasibility creates a complexity trade-off: LPCNet (Valin & Skoglund, 2019b) uses fewer than 2M parameters at 1.6 kbps, while Encodec uses ∼14M at 1.5 kbps and BigCodec uses 159M at 1.04 kbps.The larger models increase parameter counts to improve quality at low bitrates.Source: §2.5 · p. 14
  • Many SSL-based tokenizers use non-causal encoders, making high-quality, efficient causal architectures for real-time use an open research challenge.This limitation restricts their applicability in real-time settings.Source: §2.5 · p. 14

Target Domain

3 Benchmark Evaluation

This section introduces a holistic benchmark of diverse pre-trained audio tokenizers across speech, music, and general audio, addressing the lack of aligned comparisons across existing studies. It evaluates reconstruction and complexity, downstream performance, and acoustic language modeling using standardized benchmarks.

Benchmark Scope

  • The benchmark jointly compares publicly available pre-trained tokenizers across speech, music, and general audio because existing studies lack aligned, holistic evaluation protocols.Existing benchmarks differ in domains, tokenizer sets, and model configurations, limiting unified conclusions.Source: §3 · p. 15

Reconstruction Evaluation and Complexity Analysis

  • Reconstruction evaluation measures resynthesized-audio quality and computational efficiency through model size, frame rate, token rate, and MACs.Each tokenizer is evaluated with its original decoder following CodecSUPERB and VERSA protocols.Source: §3 · p. 15

Downstream Evaluation

  • Downstream evaluation tests tokenized representations as inputs to lightweight models for discriminative and generative tasks using DASB.Source: §3 · p. 15

Acoustic Language Modeling

  • Acoustic language modeling evaluates tokenizer effectiveness with the SALMon and Zero-resource benchmarks.Source: §3 · p. 15

Benchmark Scope

  • Tokenizers are selected for reproducibility, quantization diversity, and domain coverage, using checkpoints released by their original authors.The evaluation includes RVQ, SVQ, FSQ, KMeans, and semantically distilled approaches, with preference for multi-domain tokenizers.Source: §3 · p. 16

3.1 Evaluation for Reconstructed Audio Quality and Complexity

This section evaluates reconstructed audio across speech, music, and general audio while jointly examining signal/application quality, computational complexity, bitrate, and tokenization properties.

Background

  • Reconstruction fidelity matters for transmission and may predict downstream performance in decoder-dependent tasks such as speech enhancement and source separation.Tokenizer reconstruction can affect overall task performance when downstream systems rely directly on generated audio.Source: §3.1 · p. 16

Experimental Setup

  • Reconstructed audio is evaluated across speech, music, and general audio using signal-level and application-level measures based on Codec-SUPERB and VERSA scripts.Inputs are compressed by each tokenizer and resynthesized through its decoder; datasets include LibriSpeech test-clean, MUSDB, and AudioSet.Source: §3.1 · p. 16

Evaluation Setup

  • The evaluation jointly considers reconstruction quality, model parameters, MACs, bitrate, frame rate, and token rate to characterize quality–efficiency trade-offs.MACs are measured for one-second audio, while bitrate and token rate quantify compression and acoustic-language-modeling demands.Source: §3.1 · p. 17

Results and Discussion.

Across speech, general audio, and music, reconstruction quality usually falls at lower bitrates, while perceptual or semantic objectives can preserve intelligibility and subjective quality despite weak waveform-fidelity metrics.

Speech

  • Higher bitrates generally improve reconstruction, but semantic distillation preserves linguistic content as SpeechTokenizer and Mimi degrade in objective quality at lower rates.For EnCodec and DAC, all reconstruction metrics decline from higher to lower bitrates; SpeechTokenizer and Mimi show less drastic WER degradation than objective-metric declines.Source: Results and Discussion · p. 17
  • SQ-SMA-16 matches or exceeds larger-bitrate codecs, while Discrete WavLM preserves speech quality despite significantly weaker reference-based reconstruction metrics.SQ-SMA-16 performs comparably to Mimi-S-24 at 4.4 kbps and DAC-SMA-24 at 6 kbps; Discrete WavLM retains reasonable UTMOS, DNSMOS, and PLCMOS scores.Source: Results and Discussion · p. 17

General Audio and Music

  • EnCodec achieves the best overall general-audio and music reconstruction, whereas DAC trades poor time-domain fidelity for strong perceptual scores.EnCodec leads across SDR, SI-SNR, VISQOL, and SingMOS, particularly at higher bitrates; DAC has mostly negative SI-SNR but maintains strong VISQOL and SingMOS.Source: Results and Discussion · p. 18, Table 5 · p. 18
  • WavTokenizer likewise performs poorly on time-domain metrics but relatively strongly on perceptual metrics, while out-of-domain low-bitrate WT-S-24 degrades across both metric types.The degradation of WT-S-24 reflects domain mismatch, the lowest bitrate, and no explicit waveform-reconstruction loss.Source: Results and Discussion · p. 18

Summary

  • Overall, waveform-fidelity metrics are insufficient because tokenizers optimized for perception or downstream tasks can produce subjectively high-quality audio despite weak signal reconstruction.The speech results also identify SDR and SI-SNR as less reliable under over-compression, nonlinear shifts, or amplitude variations.Source: Results and Discussion · pp. 17–18

3.2 Downstream Evaluation

Downstream evaluation shows that discrete tokenizers preserve task-relevant information unevenly: semantic tokenizers are more robust, but continuous features generally outperform discrete representations. Performance depends strongly on domain, bitrate, codebook size, data scale, and downstream architecture.

Experimental Setup

  • The DASB evaluation freezes encoders, trains lightweight task-specific heads, standardizes embeddings to 1024 dimensions, and averages results across three random seeds.It spans speech, music, and general-audio discriminative and generative tasks, using shared architectures for discrete and continuous representations except stronger separation baselines in music and general audio.Source: §3.2 · pp. 18–20

Speech Tasks

  • Discrete WavLM leads speech discriminative tasks, DAC leads speaker recognition, and SSL-based tokenizers show weak speaker similarity in enhancement and separation.The reconstructed DNSMOS upper bound often does not exceed the raw-mixture lower bound, indicating reconstruction limits can constrain high-fidelity downstream tasks.Source: §3.2 · p. 20

Audio and Music Tasks

  • EnCodec consistently outperforms other tokenizers on general-audio and music tasks, while higher bitrate can worsen music separation rather than improve it.At medium bitrate, EnCodec reaches about -7 dB SI-SDR for general audio and -5.7 dB for music; general audio improves by 9.53 dB over the -16.5 dB mixture.Source: §3.2 · p. 20

Impact of Codebook Size

  • Increasing codebook count improves reconstruction but often reduces downstream performance, making medium bitrate the typical balance between fidelity and task effectiveness.Earlier RVQ codebooks tend to capture phonetic information, whereas later codebooks may add redundancy and modeling complexity.Source: §3.2 · p. 20

Summary

  • Continuous features generally outperform discrete tokens, while semantic tokenizers such as Discrete WavLM remain more stable and robust, especially in low-resource settings.Acoustic tokenizers often require larger datasets, careful model scaling, and more hyperparameter tuning; bridging this performance gap remains important for multimodal language models.Source: §3.2 · p. 21

3.3 Acoustic Language Models Evaluation

This section evaluates discrete audio tokenizers and acoustic language models across speech, text-to-speech, general audio, and music, examining reconstruction, semantic and acoustic modeling, and conditional and unconditional generation. It highlights tokenizer-dependent trade-offs and the importance of domain-specific training, bitrate, and separating language-modeling from vocoder quality.

3.3 Acoustic Language Models Evaluation

  • The evaluation covers unconditional and text-conditioned generation across speech, general audio, and music, with music generation identified as structurally challenging.The study focuses on autoregressive language-model approaches because discrete tokens are prevalent there, despite slower inference than non-autoregressive methods.Source: §3.3 · p. 22, §3.3.3 · p. 27, §3.3.4 · p. 29

3.3.1 Speech Language Modeling

  • 92.00% gender, 70.00% sentiment, and 86.50% speaker consistency make WavLM the strongest average acoustic performer.Mimi outperforms HuBERT on each acoustic task, while no method achieves substantial semantic-acoustic alignment.Source: §3.3.1 · p. 25
  • HuBERT leads semantic SLM tasks, WavLM leads acoustic consistency, and no tokenizer excels across all spoken and acoustic evaluations.Semantic stream overweighting narrows the gap to HuBERT, while semantic-acoustic alignment remains weak across methods.Source: §3.3.1 · pp. 24–25

3.3.2 Text-to-Speech

  • 3.77 UTMOS makes ESPNet EnCodec (Enc-S-24) the highest-quality TTS tokenizer, while WavLM achieves the best text adherence with 4.32 dWER.Discrete WavLM6 reaches 3.42 UTMOS and 0.90 speaker similarity, and all models obtain comparable speaker similarity from 0.88 to 0.91.Source: §3.3.2 · pp. 26–27, Table 11 · p. 27
  • Semantic tokenizers generally provide more robust TTS, but speech-domain acoustic EnCodec can become competitive or superior in high-data, deep-model settings.Enc-S-24 benefits from training on the same LibriTTS domain as the TTS model, whereas general-purpose acoustic tokenizers reduce quality or pronunciation fidelity.Source: §3.3.2 · p. 27

3.3.3 Audio Generation

  • General-audio-trained EnCodec achieves the best FAD and consistently strong audio-generation performance, whereas music-only EnCodec performs weakly, particularly on FAD.The 44kHz DAC variant has reconstruction quality comparable to audio-trained EnCodec, while WavTokenizer has the weakest reconstruction apart from music-only EnCodec.Source: §3.3.3 · pp. 28–29, Table 12 · p. 29
  • WavTokenizer performs strongly in text-to-audio generation despite poor reconstruction, while DAC and SQCodec generate weakly despite strong reconstruction.Single-stream modeling may simplify WavTokenizer generation, whereas SQCodec’s large vocabulary and DAC’s longer or structurally complex sequences can hinder autoregressive prediction.Source: §3.3.3 · p. 29

3.3.4 Music Generation

  • Evaluation uses MusicCaps and a genre-balanced FMA-test subset for text-conditioned synthesis, continuation from 2-second prompts, and tokenizer reconstruction.Metrics are FAD for audio quality, KLD for semantic consistency, and CLAP for textual alignment; Table 13 compares text-to-music LMs across tokenizers.Source: §3.3.4 · p. 30, Table 13 · p. 31
  • DAC tokenizers provide the strongest reconstruction overall: WavTokenizer has best FAD, DAC-44k and DAC-24k lowest KLD, and EnCodec-24k and DAC-24k best CLAP.EnCodec-32k does not consistently achieve the highest reconstruction quality despite being trained exclusively on music.Source: §3.3.4 · p. 31
  • EnCodec-32k leads text-conditioned music generation across all metrics and datasets, while DAC-44k outperforms other multi-domain tokenizers.The comparison is reported on MusicCaps and FMA-test in Table 13; DAC-44k’s advantage may reflect its higher bitrate and richer representation.Source: §3.3.4 · p. 31, Table 13 · p. 31
  • Unconditional generation consistently outperforms text-conditioned generation, with EnCodec-32k and DAC-44k generally strongest and SQ-Codec best on FMA by FAD.The authors attribute this pattern to higher classifier-free-guidance dropout and note that poor FMA prompts hinder text-conditioned performance.Source: §3.3.4 · p. 31
  • Higher-sample-rate, multi-codebook tokenizers tend to perform better for music language modeling, contrasting with audio and speech generation where higher bitrates are harder to model.The authors hypothesize that music’s complex harmonic and temporal structure benefits from more detailed representations; MusicCaps scores also exceed FMA-test scores because FMA has limited prompts and compromised quality.Source: §3.3.4 · pp. 30–32

3.4 General Trend

Figure 4 summarizes tokenizer rankings across speech, general audio, and music, revealing task- and domain-dependent strengths rather than a universally best tokenizer.

General Trend

  • Figure 4 averages per-metric task ranks within each domain, with higher radar values indicating better performance across reconstruction, downstream, and language-modeling categories.Speech distinguishes signal versus application reconstruction, discriminative content- versus acoustic-level tasks, text-conditioned versus unconditional modeling, and phonetic versus acoustic SpeechLM metrics.Source: §3.4 · pp. 32–33
  • No tokenizer consistently outperforms others across all axes, as models trade off reconstruction quality against semantic or downstream performance.The radar charts provide a high-level overview rather than strict recommendations; task-specific benchmark tables should guide tokenizer selection for real-world applications.Source: §3.4 · p. 33

4 Ablation Studies

The ablations control training data, sampling rate, quantization, and distillation within a standardized codec framework. Results show that domain alignment and quantization strongly affect reconstruction, while sampling-rate benefits depend on the quantizer and distillation can trade specialization for generalization.

4.1 Experimental Setups

  • The study uses ESPnet-Codec (Shi et al., 2024c) to isolate training data, codebook, quantization, and pretrained-model distillation effects under controlled implementation protocols.Models use standardized core parameters with targeted variant-specific adjustments, and include RVQ, single-layer VQ, FSQ, and semantic-distillation variants.Source: §4 · p. 33, §4.1 · pp. 33–34
  • Experiments balance speech, general audio, and music by sampling 1k hours per domain at 16 kHz and 44.1 kHz, with domain-specific test sets and rate-aware evaluation.Speech is evaluated at 16 kHz, music at 16 or 24 kHz depending on codec capability, and general audio is evaluated at the codec’s rate.Source: §4.1 · pp. 33–35

4.2 Results and Discussion

  • Reconstruction quality peaks when training and testing domains match, while balanced multi-domain training still degrades on individual domains relative to domain-specific training.The findings motivate improved domain balancing and cross-domain generalization methods.Source: §4.2 · p. 36
  • 44.1 kHz training improves RVQ performance across multiple metrics even after 16 kHz evaluation downsampling, whereas FSQ degrades on several metrics at higher rates.Sampling-rate effects therefore depend on quantization method and target domain.Source: §4.2 · p. 36
  • Pretrained speech-representation distillation improves selected reconstruction metrics, but its domain specificity can limit generalization to general audio and music.The comparison includes models with and without distillation, such as A-S versus A-S+.Source: §4.2 · p. 36
  • RVQ generally outperforms other quantization methods, while SVQ performs worst; FSQ at 16 kHz exceeds RVQ on speech UTMOS and DNSMOS.Reconstruction quality may not predict downstream performance, so these codec metrics do not establish broader application superiority.Source: §4.2 · p. 36

5 Conclusion and Future Directions

Discrete audio tokens offer compact, scalable representations that simplify generative modeling, improve storage and streaming efficiency, and integrate naturally with LLMs, although continuous features remain stronger for some fine-grained discriminative tasks. The review identifies open challenges in jointly optimizing fidelity and semantic utility, standardizing evaluation, extending semantic distillation, building unified tokenizers, and ensuring trustworthy use.

Key observations

  • Discrete tokens provide compact, modular, scalable abstractions that convert waveform regression into classification and support efficient storage, streaming, and LLM integration.They suit speech synthesis, music generation, and general audio modeling, while recent large-scale text-to-speech results show they can match or outperform continuous representations in some applications.Source: §5 · pp. 36–37

Evaluation and limitations

  • Controlled studies used moderately sized models and datasets for fair comparisons, yet semantic tokenizers’ advantage on semantic tasks aligns with larger-scale trends.Benchmarking remains difficult because tokenizers differ in training data, sampling rate, and domain scope, while reproduced results often trail reported numbers; standardized, reproducible scaling studies are needed.Source: §5 · p. 37

Key observations

  • Reconstruction quality trades off with downstream utility because reconstruction-optimized tokenizers may discard phonetic and semantic information, especially when decoders are unused.Future work should jointly optimize signal fidelity and semantic utility through multi-task or adversarial training for tasks such as ASR and SLU.Source: §5 · p. 37

Future directions

  • Semantic distillation should expand beyond speech to music and general audio, where it remains underexplored and could improve token quality across diverse tasks.Unified tokenizers could support generative and discriminative tasks across domains by balancing streamability, semantic alignment, reconstruction quality, and domain generalization through modular or hierarchical designs.Source: §5 · p. 37

Representation choices

  • Continuous representations often remain superior for speech-language understanding requiring fine-grained prosody, emotion, and speaker cues, whereas discrete tokens benefit autoregressive or masked generative modeling.Diffusion and score-matching systems may instead favor continuous conditioning, leaving the gap between representation types unresolved for multimodal LLMs.Source: §5 · p. 37

Trustworthiness and conclusion

  • Trustworthy deployment must address bias and deepfake risks (Wu et al., 2024e; Du et al., 2025) because realistic voice imitation can enable misuse.The review concludes that its comprehensive evaluation offers trends, challenges, and tokenizer-selection guidance for future progress.Source: §5 · pp. 37–38

A Additional Evaluation for Reconstructed Audio Quality

This section reports UTMOS V2 scores for different audio tokenizers across multiple bitrates.

A Additional Evaluation for Reconstructed Audio Quality

B Computational Setup

The experiments use computational settings summarized in Table 18, with runtimes varying by tokenizer characteristics and downstream-task complexity.

Computational Setup

  • Downstream-task runtimes range from 2 hours for Keyword Spotting to 48 hours for ASR, depending on task complexity.Approximate per-run times may also vary with token rate, number of token streams, and other tokenizer-specific factors.Source: B Computational Setup · p. 53

C Dataset

The survey reports each experiment’s dataset, source, and approximate duration, with a consolidated summary of dataset specifications and hours in Table 19.

C Dataset

  • Table 19 summarizes the datasets, their specifications, and approximate hours used across all experiments.Experiment-specific dataset and source details are reported in the corresponding benchmark sections.Source: C Dataset · p. 53, Table 19 · p. 54

Related papers

Loading 2506.10274v3…