WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

1 INTRODUCTION

• Acoustic codecs compress audio through an encoder, residual vector quantization, and decoder, with quantizer count determining bitrate.The encoder downsamples audio frames, quantizers operate successively on residuals, and the decoder upsamples the quantized representation.Source: §1 · pp. 1–2
• Current codecs still leave higher compression and semantic richness as important directions for improvement.The introduction identifies both compression level and semantic content as areas beyond existing acoustic codec capabilities.Source: §1 · p. 2
• WavTokenizer reconstructs 24kHz speech, music, and audio using one quantizer with 40 or 75 tokens per second.Its design expands the VQ space and uses clustering initialization and random awakening to preserve codebook utilization under single-quantizer compression.Source: §1 · p. 3
• The model combines extended contextual windows and decoder attention with an inverse Fourier transform structure and multi-scale discriminator.These components are presented as balancing compression, semantic information, and reconstruction quality.Source: §1 · p. 3
• WavTokenizer is reported to surpass state-of-the-art subjective reconstruction performance on speech, music, and audio at 75 tokens per second.The contribution summary also reports comparable audio quality at 40 tokens per second, though the supplied passage truncates the corresponding comparison.Source: §1 · p. 3

2 RELATED WORK

• Neural acoustic codecs typically encode audio into latent features, quantize them, and decode the quantized representation into reconstructed audio.The related work describes acoustic tokens as supporting speech, music, and audio domains while retaining acoustic detail.Source: §2 · p. 3
• Recent methods analyze or factorize codec representations into time-dependent, time-independent, content, style, and acoustic-detail information.TiCodec distinguishes temporal information types, while FACodec separates content, style, and acoustic detail.Source: §2 · p. 4
• WavTokenizer uses one quantizer and 40 or 75 tokens, compared with DAC’s 900 tokens and 9 quantizers for one second of speech.The paper positions this reduction alongside an effort to enhance semantic information within the acoustic codec itself.Source: §2 · p. 4

3 WAVTOKENIZER

• WavTokenizer encodes raw audio into latent features, discretizes them with a single quantizer, and reconstructs audio through an improved decoder.The model is trained end-to-end with reconstruction-oriented objectives.Source: §3 · p. 4

3.1 SINGLE QUANTIZER

• A single quantizer enables seamless speech-token serialization and avoids hierarchical downstream designs across channel dimensions.The method specifically targets compression of speech representations into one quantizer codebook.Source: §3.2 · p. 4

3.2 RETHINKING THE VECTOR QUANTIZATION SPACE

• The codebook analysis motivates a larger VQ space because speech-code probabilities concentrate in the lower-index region of the available codebook.The concentration persists even with 4000 hours of multilingual training data, indicating that more data alone does not increase utilization.Source: §3.2 · p. 5, Figure 2 · p. 5
• K-means initialization, exponential-moving-average updates, and random replacement of inactive codes mitigate the utilization loss caused by expanding the codebook.The codebook uses 200 cluster centers aligned with the larger codebook space and an EMA decay of 0.99.Source: §3.2 · p. 5

3.3 IMPROVED DECODER

• The decoder combines convolutional processing, an attention block, ConvNeXt blocks, and Fourier-based processing to reconstruct highly compressed audio.The discriminator setup adds MPD, MRD, and complex STFT discrimination across multiple time scales for perceptual-quality training.Source: §3.3 · pp. 5–6, §3.4 · p. 6

3.4 THE ADVANCED DISCRIMINATOR AND THE LOSS FUNCTIONS

• The generator objective combines quantizer, mel-spectrum reconstruction, adversarial, and feature-matching losses.The adversarial component uses hinge-loss discrimination to promote perceptual quality.Source: §3.4 · pp. 6–7

Experimental setup

• Approximately 8K hours of speech, music, and audio data are used to train WavTokenizer, with additional ablations on LibriTTS test-clean.Source: §4.1 · p. 7, §4.3 · p. 10

Reconstruction

• WavTokenizer maintains excellent reconstruction quality under extreme compression and outperforms single-quantizer DAC with 40 or 75 tokens across all reported metrics.It is also comparable to four-quantizer Vocos and eight-quantizer SpeechTokenizer on STOI, PESQ, and F1, while remaining robust in noisy and out-of-domain settings.Source: §4.2 · pp. 7–8
• At 0.9 kbps, WavTokenizer outperforms DAC at 9 kbps in subjective reconstruction quality across speech, music, and audio.The comparison uses MUSHRA scores.Source: §4.2 · p. 8, Table 2 · p. 9

Semantic representation

• On ARCH classification accuracy, WavTokenizer substantially outperforms single- and two-quantizer DAC and Encodec, and surpasses higher-quantizer configurations on several datasets.The stronger results span speech, music, and audio datasets, including AM, SLURP, MTT, IRMAS, FSD50K, and VIVAE.Source: §4.2 · pp. 8–9, Table 3 · p. 9

Downstream generative tasks

• A speech synthesis model using WavTokenizer’s single-layer representations outperforms one using DAC’s nine-layer representations on audio quality and prosody.The result supports the potential of single-layer and large-codebook acoustic representations for autoregressive audio generation.Source: §4.2 · p. 9, Table 4 · p. 9

Ablation study

• Removing attention or replacing the improved inverse-Fourier decoder with a mirror decoder degrades performance, while attention and extended context improve semantic preservation.The ablations use WavTokenizer with a single quantizer at 0.9 kbps; codebook-scale effects are also evaluated.Source: §4.3 · p. 10, Table 6 · p. 10, Table 5 · p. 10

• WavTokenizer quantizes one second of speech, music, and general audio into 40 or 75 tokens with a single quantizer.Source: §5 · p. 10
• Compared to SOTA acoustic codec models, WavTokenizer maintains high subjective reconstruction quality under extreme compression.Source: §5 · p. 10
• WavTokenizer preserves rich semantic information despite using a single quantizer and only 40 or 75 tokens per second.Source: §5 · p. 10

Training settings

• Training uses 2 million iterations on 8 NVIDIA A800 80G GPUs after resampling all inputs to 24 kHz.One million iterations are allocated to generator training and one million to discriminator training.Source: A TRAINING AND INFERENCE SETTINGS · p. 16
• Training samples are uniformly truncated to 10 seconds and randomly cropped to obtain 3-second waveform snippets.Source: A TRAINING AND INFERENCE SETTINGS · p. 16

Evaluation benchmark

• ARCH evaluates performance across twelve datasets spanning speech, music, and audio domains.The supplied passage lists speech datasets and begins listing acoustic-audio datasets within the benchmark.Source: B THE ARCH BENCHMARK · p. 16

MUSHRA

• MUSHRA evaluation uses a hidden reference and low anchor, with perceptual quality rated from 1 to 100.Fifty test samples are selected from each category, with at least ten annotations per sample.Source: C SUBJECTIVE EVALUATIONS · p. 16

CMOS

• CMOS-Q evaluates quality, clarity, and high-frequency details, whereas CMOS-P evaluates speech rate, pauses, and pitch.The evaluation uses 40 LibriTTS test-set samples, each heard by at least 10 testers.Source: C SUBJECTIVE EVALUATIONS · p. 16

Dataset size and codebook utilization

• Increasing the training dataset from 585 hours to 4000 hours does not lead to higher codebook space utilization.The comparison is reported on the LibriTTS test-clean dataset.Source: D ABLATION EXPERIMENTS ON MORE… · p. 16
• Codebook probability distributions trained with larger datasets remain concentrated on the left side of the distribution.Source: D ABLATION EXPERIMENTS ON MORE… · p. 16
• Table 8 presents an ablation study of dataset size and codebook utilization.Source: Table 8 · p. 16

• Adding decoder attention improves semantic-information preservation in WavTokenizer on ARCH speech-domain datasets.Source: E ABLATION ON THE ATTENTION MODULE AND… · p. 17
• Extending the codec model’s context window improves semantic-information preservation on the ARCH benchmark.Source: E ABLATION ON THE ATTENTION MODULE AND… · p. 17
• Table 9 evaluates the attention module and contextual window size as factors affecting semantic information.Source: Table 9 · p. 17

Limitation

• Current acoustic codec models lack the ASR understanding capabilities found in semantic models, limiting unified multimodal understanding and generation.Source: F LIMITATION AND FUTURE WORK · p. 17

Future Work

• Future work will explore a more robust encoder to improve compression, reconstruction, and semantic-information retention.Source: F LIMITATION AND FUTURE WORK · p. 17
• Future experiments will train WavTokenizer on hundreds of thousands of speech hours to test alignment between speech and text spaces.Source: F LIMITATION AND FUTURE WORK · p. 17

Additional Evaluations

• WavTokenizer provides high-quality, comparable reconstruction at 16kHz, 24kHz, and 48kHz, including high-fidelity reconstruction at 16kHz with 50 tokens.Source: G WAVTOKENIZER AT 16KHZ AND 48KHZ… · p. 17
• WavTokenizer’s reconstruction remains fast despite decoder attention, aided by fast inverse Fourier transform and low bitrate.Source: H RECONSTRUCTION SPEED · p. 18

• 5.1% WER versus 6.9% for DAC demonstrates lower synthesis error for WavTokenizer in zero-shot TTS.Source: I MORE TTS RESULTS · p. 18
• 0.61 speaker similarity versus DAC’s 0.59 shows better speaker similarity for WavTokenizer.Source: I MORE TTS RESULTS · p. 18
• WavTokenizer uses 0.9kbps compared with DAC’s 9.0kbps in the evaluated downstream TTS models.Source: I MORE TTS RESULTS · p. 18
• Table 12 reports WER and speaker similarity for acoustic codec models used by downstream speech synthesis systems.Source: Table 12 · p. 18