PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrete Audio Tokens: More Than a Survey!

Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, Bhuvana Ramabhadran, Benjamin Elizalde, Loren Lugosch, Jinyu Li, Cem Subakan, Phil Woodland, Minje Kim, Hung-yi Lee, Shinji Watanabe, Yossi Adi, Mirco Ravanelli

arXiv:2506.10274v3cs.SDcs.AIcs.CLeess.AS

TL;DR

Existing discrete-audio-token research lacks a unified, cross-domain comparison and a taxonomy that captures evolving architectural and practical differences. This paper proposes a refined taxonomy, benchmarks tokenizers across speech, music, and general audio, and uses controlled ablations to examine design trade-offs. The study identifies domain alignment and evaluation consistency as important considerations while reporting tokenizer-dependent effects of sampling rate.

  • Problem

    Existing benchmarks and surveys lack holistic, aligned comparisons across domains, tokenizers, and evaluation criteria, while the acoustic-semantic division does not capture overlapping capabilities.

  • Method

    The paper combines a refined taxonomy with consistent multi-benchmark evaluation and controlled ablations of tokenizer design choices.

  • Results

    The study finds that performance depends on domain alignment and tokenizer design: matching training and evaluation domains improves reconstruction, while sampling-rate effects differ across quantization approaches.

  • Takeaways & Limitations

    Discrete audio tokenizer comparisons require standardized evaluation pipelines and careful consideration of training domain, sampling rate, and quantization method.

  • Takeaways & Limitations

    Training data, sampling rate, and domain scope vary substantially across tokenizers, complicating fair and unified benchmarking.

Abstract

from arXiv · show

Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and inference, as well as competitive performance across diverse downstream tasks. They provide a practical alternative to continuous features, enabling the integration of speech and audio into modern large language models (LLMs). As interest in token-based audio processing grows, various tokenization methods have emerged, and several surveys have reviewed the latest progress in the field. However, existing studies often focus on specific domains or tasks and lack a unified comparison across various benchmarks. This paper presents a systematic review and benchmark of discrete audio tokenizers, covering three domains: speech, music, and general audio. We propose a taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamability, and application domains. We evaluate tokenizers on multiple benchmarks for reconstruction, downstream performance, and acoustic language modeling, and analyze trade-offs through controlled ablation studies. Our findings highlight key limitations, practical considerations, and open challenges, providing insight and guidance for future research in this rapidly evolving area. For more information, including our main results and tokenizer database, please refer to our website: https://poonehmousavi.github.io/dates-website/.

1 Introduction

Discrete audio tokens replace continuous representations with compact, trainable codes that support efficient storage, generation, and multimodal audio processing. This paper addresses fragmented evaluation and taxonomy by proposing a unified framework spanning speech, music, and general audio.

  • Motivation: Neural codecs use encoder-decoder architectures and differentiable quantization to produce discrete audio tokens instead of continuous representations.Training commonly combines reconstruction, adversarial, generative, and auxiliary losses.
  • Motivation: Discrete tokens compact audio representations for efficient storage and transmission, while converting continuous-distribution modeling into discrete classification.They also help connect text and audio processing in multimodal models.
  • Research gap: The acoustic-semantic distinction is insufficient because both categories can support overlapping semantic and generative functions.The paper therefore refines classification around architecture, quantization, training, streamability, and application domain.
  • Research gap: Existing literature lacks a unified study covering music and general audio alongside speech and comparing diverse evaluation criteria.Prior surveys and benchmarks often focus on speech or a single aspect such as reconstruction quality.
  • Contributions: The paper proposes a comprehensive taxonomy of discrete audio tokenization methods based on architectural and functional criteria.The study is organized around taxonomy, benchmark evaluation, and controlled ablations.
  • Contributions: Tokenizers are evaluated for reconstruction, downstream tasks, and acoustic language modeling under consistent conditions across multiple benchmarks.The evaluation uses Codec-SUPERB and VERSA, DASB, SALMon, and the Zero-resource speech benchmark.
  • Contributions: Controlled ablations isolate the effects of sampling rate and single-domain versus multi-domain training using ESPnet-Codec.These experiments complement the cross-domain benchmark evaluation.
  • Contributions: The survey aims to clarify design trade-offs, highlight current limitations, and guide future research in discrete audio tokenization.Its empirical study spans speech, music, and general audio.

2 Literature Review and Proposed Taxonomy

The paper organizes audio tokenizers through a taxonomy spanning architecture, quantization, training, target domain, and streamability. It describes major quantization choices and their effects on representation capacity, token count, and bitrate efficiency.

  • Overall architecture: Audio tokenizers typically combine an encoder, quantization module, and decoder to convert waveforms into discrete representations and reconstruct audio.The encoder produces frame-wise embeddings, quantization maps them to discrete indices and quantized vectors, and the decoder reconstructs the waveform.
  • Proposed taxonomy: The proposed taxonomy classifies tokenizers by quantization method, encoder-decoder architecture, training paradigm, target domain, and streamability.Table 1 organizes these design choices across speech, music, and general audio applications, including bitrate strategy, objectives, auxiliary losses, joint optimization, and streaming capability.
  • Quantization algorithms: Residual Vector Quantization sequentially quantizes residuals with multiple codebooks, while Single Vector Quantization uses one codebook without iterative residual refinement.RVQ was introduced in SoundStream and became widely adopted; SVQ simplifies the architecture to M=1, partly addressing the complexity of multiple codebooks in acoustic language models.
  • Quantization algorithms: Group Vector Quantization divides each latent feature into non-overlapping groups, increasing first-stage capacity while reducing the number of required codebooks.The grouped structure improves performance while reducing codebook requirements.
  • Quantization algorithms: Finite Scalar Quantization maps each feature dimension to fixed scalar levels, producing 2S + 1 values per dimension and uniform latent-space coverage.FSQ first squashes features into [−1, 1], then applies scalar rounding controlled by the resolution parameter S.
  • Quantization algorithms: Multi-Scale RVQ uses quantizers at different temporal resolutions to capture coarse and fine details with fewer tokens, whereas Cross-Scale RVQ progressively encodes coarse-to-fine information.MSRVQ downsamples residuals at each stage, quantizes them, and upsamples them back to the full sequence length.
  • Bitrate strategy: Fixed bitrate assigns a predetermined number of bits per code index, whereas adaptive bitrate uses token frequencies to assign variable-length codes and improve compression efficiency.Huffman or arithmetic coding can provide post-hoc entropy coding for any quantization method.

2.3 Encoder-Decoder

Audio tokenizers use varied encoder-decoder architectures and training objectives to balance compression, reconstruction, perceptual quality, and representation learning. Their designs also differ in quantization, optimization strategy, and semantic distillation.

  • Encoder architectures: CNN tokenizers are compact and suitable for resource-constrained environments, but they cannot capture long-range dependencies.
  • Encoder architectures: CNN+RNN and CNN+Transformer designs add sequential or attention-based modeling to capture longer-range dependencies, with added complexity.
  • Encoder architectures: Transformer-only tokenizers may provide strong compression and reconstruction, but require substantial training data and computational resources.
  • Training strategy: Joint training optimizes the encoder, quantizer, and decoder together, commonly combining reconstruction and adversarial losses.
  • Quantization and objectives: Quantization objectives align encoder outputs with codebook embeddings, while EMA updates, codebook expiration, and balancing methods address codebook collapse.
  • Auxiliary components: Semantic distillation guides selected codebooks with SSL features or tokens so earlier layers capture phonetic information while later layers retain acoustic detail.

2.5 Streamability and Domain Categorization

Streamability depends on future context and computational complexity, while tokenizer domains range from speech and music to general audio and multi-domain systems.

  • Streamability: Streamability means processing and generating audio in real time with minimal latency and little or no future context.
  • Streamability: Algorithmic latency depends on look-ahead, with causal convolutions or causal attention enabling streaming in CNN- and Transformer-based models.
  • Streamability: 159M parameters at 1.04 kbps, BigCodec illustrates the complexity-quality trade-off at low bitrates compared with smaller codecs.
  • Streamability: Many SSL-based tokenizers use non-causal encoders, limiting real-time use and leaving efficient, high-quality causal architectures as an open challenge.
  • Target domain: Tokenizers target speech, music, general audio, or multiple domains, reflecting different application requirements and acoustic structures.

3 Benchmark Evaluation

The benchmark addresses the difficulty of comparing audio tokenizers across tasks and domains by evaluating reconstruction, downstream use, and acoustic language modeling under consistent conditions.

  • Motivation: Existing benchmarks emphasize individual aspects or domains, making holistic comparison and unified conclusions difficult.
  • Evaluation axes: Reconstruction evaluation uses Codec-SUPERB2 and VERSA3 across the included tokenizers.
  • Evaluation axes: Downstream effectiveness is tested with lightweight discriminative and generative models using DASB.
  • Evaluation axes: Acoustic language-modeling effectiveness is evaluated with SALMon and the Zero-resource speech benchmark.
  • Tokenizer selection: The tokenizer set prioritizes reproducible open-source models spanning quantization strategies and speech, music, and general-audio domains.

3.1 Evaluation for Reconstructed Audio Quality and Complexity

The reconstructed-audio evaluation measures fidelity across speech, music, and general audio while jointly considering signal quality, application-level performance, and computational efficiency.

  • Metrics: Reconstruction quality is evaluated alongside model parameters, frame rate, token rate, and multiply-accumulate operations.
  • Evaluation setup: The evaluation resynthesizes compressed audio with each tokenizer’s original decoder and assesses it from signal-level and application-level perspectives.
  • Datasets: Speech, music, and general-audio tests use LibriSpeech test-clean, MUSDB, and AudioSet test data, respectively.
  • Speech results: Table 4 reports reconstruction performance for speech tokenizers.

Results and Discussion.

Reconstruction quality generally declines at lower bitrates, but tokenizer objectives shape which metrics remain strong. Perceptual or downstream-oriented models can produce subjectively high-quality audio despite weak waveform-fidelity scores.

  • Speech: Higher bitrates preserve better reconstruction quality for EnCodec and DAC, while SpeechTokenizer and Mimi retain linguistic content more robustly at lower bitrates.For SpeechTokenizer and Mimi, WER declines less sharply than objective reconstruction metrics after bitrate reduction.
  • General Audio and Music: EnCodec achieves the strongest overall reconstruction for general audio and music, whereas DAC preserves perceptual quality despite poor time-domain fidelity.DAC has strong VISQOL and SingMOS scores despite negative SI-SNR values in most settings.
  • General Audio and Music: WavTokenizer similarly combines poor SDR and SI-SNR with relatively strong perceptual metrics, especially when not explicitly trained for waveform reconstruction.The out-of-domain WT-S-24 performs poorly across both objective and perceptual metrics because of domain mismatch, low bitrate, and absent waveform-reconstruction losses.
  • Summary: Reconstruction metrics alone can undervalue tokenizers optimized for perceptual or downstream objectives, which may still generate subjectively high-quality audio.The results therefore support evaluating audio tokenizers beyond traditional waveform-fidelity measures.

3.2 Downstream Evaluation

The downstream evaluation isolates token quality with frozen encoders and lightweight heads across speech, music, and general-audio tasks. Results show trade-offs among bitrate, codebook size, tokenizer type, data scale, and downstream performance.

  • Experimental Setup: Frozen encoders, lightweight task heads, and shared dimensionality isolate how much task-relevant information the tokens preserve.Generative tasks additionally use frozen decoders, while multi-codebook embeddings are combined with learned weights.
  • Speech Tasks: Discrete WavLM performs best on speech discriminative tasks, while DAC leads speaker recognition, suggesting different objectives preserve phonetic content and speaker identity.SpeechTokenizer ranks second in discriminative tasks, and WavLM performs well for separation and enhancement at low and medium bitrates.
  • Audio and Music Tasks: EnCodec consistently outperforms other tokenizers on general-audio and music tasks, while DAC lags despite strong perceptual quality.The reported unprocessed-mixture SI-SDR is approximately -16 dB for general audio and -7.7 dB for music.
  • Trade-offs: Increasing codebooks improves signal reconstruction but often reduces downstream performance by increasing output dimensionality and modeling complexity.In RVQ models, earlier codebooks capture more phonetic information, whereas later codebooks often add redundancy.
  • Discrete vs Continuous: Continuous features generally outperform discrete tokens in complex or low-resource settings, although performance improves with more data and careful model scaling.Discrete WavLM reaches 6.0% WER on LibriSpeech, 22.0% on Basque, and 58.9% on Welsh at low bitrate using a BiLSTM head.

3.3 Acoustic Language Models Evaluation

Acoustic language-modeling results vary sharply by tokenizer and objective: semantic-oriented representations support linguistic tasks, while acoustic representations can better preserve acoustic attributes. No tokenizer dominates every evaluation, and vocoder and domain effects complicate generation results.

  • Speech Language Modeling: HuBERT is strongest on semantic tasks, while WavLM leads acoustic consistency and semantic distillation narrows the semantic gap with careful tuning.Only Mimi-S-24 approaches HuBERT’s baseline on semantic tasks.
  • Speech Language Modeling: Overweighting semantic streams raises Mimi’s sBLIMP accuracy from 52.25% to 60.17%, with an average gain of 6.91 accuracy points across semantic evaluations.The same pattern appears for the overweighted ST-S-16* variant.
  • Speech Language Modeling: WavLM records 92.00%, 70.00%, and 86.50% accuracy on gender, sentiment, and speaker consistency, respectively, while no method performs substantially on semantic-acoustic alignment.Mimi also shows strong acoustic performance, and several purely acoustic tokenizers outperform HuBERT on acoustic evaluations.
  • Text-to-Speech: ESPNet EnCodec achieves the highest TTS audio quality with a UTMOS of 3.77, whereas WavLM achieves the best text adherence with a dWER of 4.32.Discrete WavLM provides the second-best audio quality with UTMOS 3.42 and speaker similarity 0.90.
  • Text-to-Speech: TTS performance is more robust with semantic tokenizers under constrained training, while acoustic tokenizers can become competitive in high-data regimes with domain-matched training.The results also show that tokenizer and vocoder quality influence generative evaluation, complicating interpretation of language-model performance.
  • Text-to-Audio Generation: WavTokenizer achieves strong text-to-audio generation despite poor reconstruction quality, whereas DAC and SQCodec show the opposite pattern.This indicates reconstruction quality does not directly determine modeling performance, and single-stream generation may simplify language-modeling.

3.4 General Trend

Across speech, general audio, and music, tokenizer strengths are strongly task- and domain-dependent. The rankings provide a high-level overview rather than a universal recommendation.

  • Ranking Method: Figure 4 ranks tokenizers by averaging per-metric task ranks within each domain, with rank 1 worst and rank N best.Multi-metric tasks are ranked separately for each metric before averaging.
  • Overall Trends: No tokenizer consistently outperforms all others across the radar-chart axes; reconstruction, semantic modeling, and downstream performance favor different models.The plots are intended to summarize performance trends rather than provide strict recommendations.

4 Ablation Studies

The ablation study controls tokenizer training and evaluation variables to examine domain alignment, sampling rate, distillation, and quantization. Results show strong interactions between these design choices and reconstruction performance.

  • Experimental Setup: Controlled experiments standardize tokenizer training and evaluation to compare data domains, sampling rates, codebook setups, quantization methods, and distillation.The study uses ESPnet-Codec and evaluates models across speech, general audio, and music datasets.
  • Data Domains: Matching training and evaluation domains consistently produces the best reconstruction quality, while balanced multi-domain training still degrades on individual domains.The result motivates better domain balancing and cross-domain generalization methods.
  • Sampling Rate: 44.1 kHz training improves RVQ models across multiple metrics, whereas FSQ models degrade on several metrics at higher sampling rates.The effect persists even when reconstructed audio is downsampled to 16 kHz for evaluation, indicating that sampling-rate impact depends on quantization method.
  • Distillation Effect: Pretrained speech-representation distillation improves some reconstruction metrics but may limit generalization to general audio and music.The comparison between distilled and non-distilled models indicates a trade-off between specialized performance and broader generalization.
  • Quantization Methods: RVQ outperforms other quantization approaches across most reconstruction metrics, while SVQ generally performs worst and FSQ exceeds RVQ on some 16 kHz speech-quality metrics.Reconstruction quality does not necessarily predict downstream performance, so the observed metric rankings should not be treated as universal application rankings.

5 Conclusion and Future Directions

The conclusion identifies trade-offs between reconstruction fidelity, semantic utility, generalization, scale, and evaluation fairness. It proposes unified, trustworthy tokenizers and more reproducible benchmarks as future directions.

  • Representational Trade-offs: Discrete tokens offer compact, modular representations suited to generative audio tasks, but continuous features can remain better for fine-grained discriminative speech understanding.Discrete representations may be advantageous for autoregressive or masked generation, whereas diffusion or score-matching systems may benefit from continuous conditioning.
  • Representational Trade-offs: Reconstruction-focused optimization can fail to preserve phonetic or semantic features needed by downstream tasks, especially when the decoder is not involved.The paper recommends jointly optimizing signal fidelity and semantic utility.
  • Evaluation and Generalizability: Fair comparison remains difficult because tokenizers differ in training data, sampling rate, and domain scope, motivating standardized evaluation pipelines and unified benchmarks.Controlled reproductions can also underperform originally reported results, supporting reproducibility-focused evaluations and scaling studies.
  • Future Directions: Semantic distillation beyond speech is underexplored, while trustworthy development must address bias and deepfake misuse enabled by realistic voice imitation.The paper also notes that moderately sized models and datasets were chosen to ensure fair controlled comparisons, limiting direct scaling evidence.
  • Future Directions: Future tokenizers may need modular or hierarchical designs that jointly balance streamability, semantic alignment, reconstruction quality, and domain generalization.The proposed direction targets both generative and discriminative tasks across multiple audio domains.

A Additional Evaluation for Reconstructed Audio Quality

Additional evaluation reports UTMOS V2 scores for different audio tokenizers and bitrate conditions.

  • UTMOS V2 Evaluation: Table 17 reports UTMOS V2 scores for audio tokenizers across different bitrates.The section provides an additional reconstructed-audio-quality evaluation.

B Computational Setup

The computational setup reports approximate runtimes while noting variation across tokenizers and downstream tasks. Runtime depends on token rate, stream count, and task complexity.

  • Computational Settings: Approximate runtimes vary with token rate, the number of token streams, tokenizer, and task complexity.Reported downstream runtimes range from 2 hours for Keyword Spotting to 48 hours for ASR.

C Dataset

The survey reports each experiment’s dataset, source, and approximate duration, with a consolidated summary in Table 19.

  • Each experiment reports its dataset, source, and approximate dataset duration.
  • Table 19 summarizes the datasets, their specifications, and approximate hours used across all experiments.
Loading 2506.10274v3…