Source-linked AI summary
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, Zhou Zhao
TL;DR
Acoustic language models need compact discrete codecs that retain reconstruction quality and semantic information across speech, music, and audio. WavTokenizer uses a single-quantizer codec with expanded representational and decoder components, achieving strong reconstruction and semantic results under extreme compression while current codecs remain limited in understanding capabilities.
Problem
Acoustic codec models must improve compression and semantic richness while supporting high-quality modeling across speech, music, and audio.
Method
WavTokenizer uses a convolutional encoder, a single quantizer, and an improved decoder trained with time-frequency reconstruction and multi-resolution discriminator losses.
Results
WavTokenizer reconstructs one second of 24kHz speech, music, or audio with 40 or 75 tokens and maintains high subjective quality and rich semantic information under extreme compression.
Takeaways & Limitations
A single-layer, large-codebook acoustic representation shows potential for downstream autoregressive audio generation and for modeling speech as a special form of text.
Takeaways & Limitations
Current acoustic codecs lack the ASR understanding capabilities of semantic models, limiting their development within unified multimodal understanding-and-generation frameworks.
Abstract
from arXiv · showhide
Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer.
1 INTRODUCTION
WavTokenizer addresses two open directions in acoustic codecs: stronger compression and richer semantic information. It uses a single quantizer and expanded modeling components to target high-quality reconstruction across speech, music, and audio.
- 1 INTRODUCTION: Acoustic codecs compress audio through an encoder, residual vector quantization, and decoder, with quantizer count determining bitrate.The encoder downsamples audio frames, quantizers operate successively on residuals, and the decoder upsamples the quantized representation.
- 1 INTRODUCTION: Current codecs still leave higher compression and semantic richness as important directions for improvement.The introduction identifies both compression level and semantic content as areas beyond existing acoustic codec capabilities.
- 1 INTRODUCTION: WavTokenizer reconstructs 24kHz speech, music, and audio using one quantizer with 40 or 75 tokens per second.Its design expands the VQ space and uses clustering initialization and random awakening to preserve codebook utilization under single-quantizer compression.
- 1 INTRODUCTION: The model combines extended contextual windows and decoder attention with an inverse Fourier transform structure and multi-scale discriminator.These components are presented as balancing compression, semantic information, and reconstruction quality.
- 1 INTRODUCTION: WavTokenizer is reported to surpass state-of-the-art subjective reconstruction performance on speech, music, and audio at 75 tokens per second.The contribution summary also reports comparable audio quality at 40 tokens per second, though the supplied passage truncates the corresponding comparison.
2 RELATED WORK
Related acoustic codec work focuses on reconstructing audio from quantized latent features and increasingly examines the internal codec space and semantic information. WavTokenizer differs by targeting single-quantizer reconstruction with substantially fewer tokens while strengthening codec semantics.
- 2 RELATED WORK: Neural acoustic codecs typically encode audio into latent features, quantize them, and decode the quantized representation into reconstructed audio.The related work describes acoustic tokens as supporting speech, music, and audio domains while retaining acoustic detail.
- 2 RELATED WORK: Recent methods analyze or factorize codec representations into time-dependent, time-independent, content, style, and acoustic-detail information.TiCodec distinguishes temporal information types, while FACodec separates content, style, and acoustic detail.
- 2 RELATED WORK: WavTokenizer uses one quantizer and 40 or 75 tokens, compared with DAC’s 900 tokens and 9 quantizers for one second of speech.The paper positions this reduction alongside an effort to enhance semantic information within the acoustic codec itself.
3 WAVTOKENIZER
WavTokenizer uses a single-quantizer VQ-GAN-style codec with an improved decoder, broader codebook space, attention, and multi-scale adversarial components. These choices target extreme compression while preserving reconstruction quality and semantic information.
- 3 WAVTOKENIZER: WavTokenizer encodes raw audio into latent features, discretizes them with a single quantizer, and reconstructs audio through an improved decoder.The model is trained end-to-end with reconstruction-oriented objectives.
- 3.1 SINGLE QUANTIZER: A single quantizer enables seamless speech-token serialization and avoids hierarchical downstream designs across channel dimensions.The method specifically targets compression of speech representations into one quantizer codebook.
- 3.2 RETHINKING THE VECTOR QUANTIZATION SPACE: The codebook analysis motivates a larger VQ space because speech-code probabilities concentrate in the lower-index region of the available codebook.The concentration persists even with 4000 hours of multilingual training data, indicating that more data alone does not increase utilization.
- 3.2 RETHINKING THE VECTOR QUANTIZATION SPACE: K-means initialization, exponential-moving-average updates, and random replacement of inactive codes mitigate the utilization loss caused by expanding the codebook.The codebook uses 200 cluster centers aligned with the larger codebook space and an EMA decay of 0.99.
- 3.3 IMPROVED DECODER: The decoder combines convolutional processing, an attention block, ConvNeXt blocks, and Fourier-based processing to reconstruct highly compressed audio.The discriminator setup adds MPD, MRD, and complex STFT discrimination across multiple time scales for perceptual-quality training.
- 3.4 THE ADVANCED DISCRIMINATOR AND THE LOSS FUNCTIONS: The generator objective combines quantizer, mel-spectrum reconstruction, adversarial, and feature-matching losses.The adversarial component uses hinge-loss discrimination to promote perceptual quality.
4 EXPERIMENTS
WavTokenizer is evaluated across reconstruction, semantic representation, downstream speech synthesis, and ablation settings. The experiments report strong performance under extreme compression and examine codebook scale, context, and decoder design.
- Experimental setup: Approximately 8K hours of speech, music, and audio data are used to train WavTokenizer, with additional ablations on LibriTTS test-clean.
- Reconstruction: WavTokenizer maintains excellent reconstruction quality under extreme compression and outperforms single-quantizer DAC with 40 or 75 tokens across all reported metrics.It is also comparable to four-quantizer Vocos and eight-quantizer SpeechTokenizer on STOI, PESQ, and F1, while remaining robust in noisy and out-of-domain settings.
- Reconstruction: At 0.9 kbps, WavTokenizer outperforms DAC at 9 kbps in subjective reconstruction quality across speech, music, and audio.The comparison uses MUSHRA scores.
- Semantic representation: On ARCH classification accuracy, WavTokenizer substantially outperforms single- and two-quantizer DAC and Encodec, and surpasses higher-quantizer configurations on several datasets.The stronger results span speech, music, and audio datasets, including AM, SLURP, MTT, IRMAS, FSD50K, and VIVAE.
- Downstream generative tasks: A speech synthesis model using WavTokenizer’s single-layer representations outperforms one using DAC’s nine-layer representations on audio quality and prosody.The result supports the potential of single-layer and large-codebook acoustic representations for autoregressive audio generation.
- Ablation study: Removing attention or replacing the improved inverse-Fourier decoder with a mirror decoder degrades performance, while attention and extended context improve semantic preservation.The ablations use WavTokenizer with a single quantizer at 0.9 kbps; codebook-scale effects are also evaluated.
5 CONCLUSION
WavTokenizer quantizes one second of speech, music, and general audio into 40 or 75 tokens using a single quantizer while preserving high subjective reconstruction quality and rich semantic information under extreme compression.
- WavTokenizer quantizes one second of speech, music, and general audio into 40 or 75 tokens with a single quantizer.
- Compared to SOTA acoustic codec models, WavTokenizer maintains high subjective reconstruction quality under extreme compression.
- WavTokenizer preserves rich semantic information despite using a single quantizer and only 40 or 75 tokens per second.
A TRAINING AND INFERENCE SETTINGS
WavTokenizer is trained on resampled 24 kHz speech, music, and audio using long iterative training on eight NVIDIA A800 GPUs, while evaluation spans twelve speech, music, and audio datasets in ARCH.
- Training settings: Training uses 2 million iterations on 8 NVIDIA A800 80G GPUs after resampling all inputs to 24 kHz.One million iterations are allocated to generator training and one million to discriminator training.
- Training settings: Training samples are uniformly truncated to 10 seconds and randomly cropped to obtain 3-second waveform snippets.
- Evaluation benchmark: ARCH evaluates performance across twelve datasets spanning speech, music, and audio domains.The supplied passage lists speech datasets and begins listing acoustic-audio datasets within the benchmark.
C SUBJECTIVE EVALUATIONS
Subjective evaluation uses MUSHRA ratings with references and low anchors, while CMOS-Q and CMOS-P separately assess audio quality and prosodic properties on LibriTTS.
- MUSHRA: MUSHRA evaluation uses a hidden reference and low anchor, with perceptual quality rated from 1 to 100.Fifty test samples are selected from each category, with at least ten annotations per sample.
- CMOS: CMOS-Q evaluates quality, clarity, and high-frequency details, whereas CMOS-P evaluates speech rate, pauses, and pitch.The evaluation uses 40 LibriTTS test-set samples, each heard by at least 10 testers.
D ABLATION EXPERIMENTS ON MORE TRAINING DATA AND CODEBOOK
The codebook-utilization ablation finds that expanding the training dataset from 585 to 4000 hours does not increase utilization, with larger-data codebooks remaining concentrated on the left side of the distribution.
- Dataset size and codebook utilization: Increasing the training dataset from 585 hours to 4000 hours does not lead to higher codebook space utilization.The comparison is reported on the LibriTTS test-clean dataset.
- Dataset size and codebook utilization: Codebook probability distributions trained with larger datasets remain concentrated on the left side of the distribution.
- Dataset size and codebook utilization: Table 8 presents an ablation study of dataset size and codebook utilization.
E ABLATION ON THE ATTENTION MODULE AND THE CONTEXTUAL MODEL SIZE ON THE ARCH BENCHMARK
On ARCH speech-domain datasets, WavTokenizer’s decoder attention module and extended contextual window improve preservation of semantic information.
- Adding decoder attention improves semantic-information preservation in WavTokenizer on ARCH speech-domain datasets.
- Extending the codec model’s context window improves semantic-information preservation on the ARCH benchmark.
- Table 9 evaluates the attention module and contextual window size as factors affecting semantic information.
F LIMITATION AND FUTURE WORK
The paper identifies limited ASR-like understanding in current acoustic codecs and proposes stronger encoders and larger-scale speech training as future directions. Additional evaluations report reconstruction across sampling rates, fast reconstruction, and stable high-quality performance at low token rates.
- Limitation: Current acoustic codec models lack the ASR understanding capabilities found in semantic models, limiting unified multimodal understanding and generation.
- Future Work: Future work will explore a more robust encoder to improve compression, reconstruction, and semantic-information retention.
- Future Work: Future experiments will train WavTokenizer on hundreds of thousands of speech hours to test alignment between speech and text spaces.
- Additional Evaluations: WavTokenizer provides high-quality, comparable reconstruction at 16kHz, 24kHz, and 48kHz, including high-fidelity reconstruction at 16kHz with 50 tokens.
- Additional Evaluations: WavTokenizer’s reconstruction remains fast despite decoder attention, aided by fast inverse Fourier transform and low bitrate.
I MORE TTS RESULTS
Downstream zero-shot TTS experiments compare WavTokenizer with DAC under the same generative architecture. WavTokenizer achieves lower WER and higher speaker similarity while using a substantially lower bitrate.
- 5.1% WER versus 6.9% for DAC demonstrates lower synthesis error for WavTokenizer in zero-shot TTS.
- 0.61 speaker similarity versus DAC’s 0.59 shows better speaker similarity for WavTokenizer.
- WavTokenizer uses 0.9kbps compared with DAC’s 9.0kbps in the evaluated downstream TTS models.
- Table 12 reports WER and speaker similarity for acoustic codec models used by downstream speech synthesis systems.