Source-linked AI summary
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, Yike Guo, Wei Xue
TL;DR
Audio codecs used by audio LLMs were designed for compression, leaving semantic representation as a design gap that can matter for generated speech content. X-Codec integrates semantic and acoustic information in its codec, and evaluations report effectiveness across speech, music continuation, and sound-generation tasks. The paper also notes trade-offs in compression-related acoustic metrics.
Problem
Audio LLM codecs were designed for compression rather than language modeling, and semantic misinterpretation of acoustic tokens is associated with content inaccuracies and elevated WER.
Source: §1 · p. 2Method
X-Codec integrates semantic and acoustic features in a unified codec and adds semantic reconstruction to its training objective.
Source: §1 · p. 2Results
Evaluations across text-to-speech, music continuation, and text-to-sound report consistent effectiveness, with X-Codec outperforming existing disentanglement techniques in VALL-E-based TTS comparisons.
Source: §1 · p. 2Takeaways & Limitations
The conclusion reports improved semantic understanding and audio-generation quality across a variety of domains.
Source: §5 · p. 10Adding semantic layers can slightly impair Mel and STFT distances, metrics relevant to compressed-audio fidelity, while UTMOS scores show improved human auditory perception.
Source: §4.3 · pp. 9–10
Abstract
from arXiv · showhide
Recent advancements in audio generation have been significantly propelled by the capabilities of Large Language Models (LLMs). The existing research on audio LLM has primarily focused on enhancing the architecture and scale of audio language models, as well as leveraging larger datasets, and generally, acoustic codecs, such as EnCodec, are used for audio tokenization. However, these codecs were originally designed for audio compression, which may lead to suboptimal performance in the context of audio LLM. Our research aims to address the shortcomings of current audio LLM codecs, particularly their challenges in maintaining semantic integrity in generated audio. For instance, existing methods like VALL-E, which condition acoustic token generation on text transcriptions, often suffer from content inaccuracies and elevated word error rates (WER) due to semantic misinterpretations of acoustic tokens, resulting in word skipping and errors. To overcome these issues, we propose a straightforward yet effective approach called X-Codec. X-Codec incorporates semantic features from a pre-trained semantic encoder before the Residual Vector Quantization (RVQ) stage and introduces a semantic reconstruction loss after RVQ. By enhancing the semantic ability of the codec, X-Codec significantly reduces WER in speech synthesis tasks and extends these benefits to non-speech applications, including music and sound generation. Our experiments in text-to-speech, music continuation, and text-to-sound tasks demonstrate that integrating semantic information substantially improves the overall performance of language models in audio generation. Our code and demo are available (Demo: https://x-codec-audio.github.io Code: https://github.com/zhenye234/xcodec)
1 Introduction
Audio LLM research has emphasized model scale and architecture, while its codecs were designed for compression rather than semantic tokenization. X-Codec combines semantic and acoustic features, and the paper reports gains across several audio-generation tasks.
1 Introduction
- Audio codecs designed for compression may be suboptimal for audio LLMs, whose research has largely emphasized architecture, scale, and larger datasets.The paper identifies this mismatch as a design gap for audio tokenization.Source: §1 · p. 2
- VALL-E can produce content inaccuracies and elevated WER when acoustic tokens are semantically misinterpreted, including skipped or erroneous words.The paper relates this problem to codecs that prioritize acoustic reconstruction over semantic information.Source: §1 · p. 2
- X-Codec unifies semantic and acoustic features in an X-shaped tokenization framework with two inputs and two outputs.Its RVQ structure supports simultaneous embedding learning for semantic richness and acoustic fidelity.Source: §1 · p. 2
- Evaluations cover text-to-speech, music continuation, and text-to-sound, with X-Codec outperforming existing disentanglement techniques in comparative VALL-E-based TTS evaluation.The paper reports consistent effectiveness across the evaluated applications.Source: §1 · p. 2
2 Related Works
Prior audio language models span multi-stage semantic-to-acoustic pipelines and single-stage token modeling, while codec research has largely prioritized acoustic reconstruction. X-Codec is presented as a uniform semantic enhancement compatible with existing audio LLMs.
2.1 Audio Language Model
- Audio language models moved from multi-stage semantic-to-acoustic generation, which can accumulate errors, toward single-stage approaches such as VALL-E and MusicGen.The cited examples use autoregressive/non-autoregressive token prediction or a single-stage transformer with delayed token interleaving.Source: §2.1 · pp. 2–3
2.2 Audio Codec
- Conventional codecs prioritize acoustic reconstruction and compression, while SpeechTokenizer and SemantiCodec add semantic representations with compatibility or modeling-complexity trade-offs.X-Codec instead applies uniform semantic enhancement across tokens without requiring structural modifications to existing audio LLMs.Source: §2.2 · pp. 3–4
3 Methods
X-Codec builds on acoustic codec components but targets semantic shortcomings associated with codec tokens. It combines semantic and acoustic features before RVQ and adds semantic reconstruction to the training objective.
3 Methods
- The method section introduces X-Codec as a straightforward approach to address the semantic shortcomings of acoustic codecs.Its design integrates semantic information into the codec rather than relying only on acoustic reconstruction.Source: §3 · p. 4
3.1 Acoustic Audio codec
- The baseline acoustic codec encodes waveform samples into frame-level features, quantizes them into discrete RVQ tokens, and decodes quantized features back into a waveform.Its training uses mel, STFT, and GAN losses for acoustic reconstruction.Source: §3.1 · p. 4
3.2 Analysing Semantic Shortcoming
- VALL-E trained with Encodec exhibits high WER and content inaccuracies, while phonetic-discriminability tests find poor discriminability in Encodec representations.The analysis identifies semantic shortcomings as a significant contributing factor to audio-generation inaccuracies and motivates adding semantic features.Source: §3.2 · p. 4
3.3 Designing Auxiliary Semantic Module
- X-Codec extracts pretrained semantic features, refines them with a semantic encoder, concatenates them with acoustic features, and quantizes the combined representation with RVQ.Projectors support reconstruction of semantic features and waveform; an MSE semantic reconstruction loss is weighted alongside the other losses.Source: §3.3 · p. 5
4 Experiments
The experiments use matched baseline acoustic codecs to isolate the contribution of X-Codec’s auxiliary semantic module.
4 Experiments
- Each experiment compares X-Codec with a baseline acoustic codec matched on training data, training steps, and other hyperparameters.The matched setup is intended to support a fair, unbiased comparison.Source: §4 · p. 5
- The baseline configuration excludes X-Codec’s auxiliary semantic module, making that module the primary experimental distinction.This controlled contrast is designed to isolate the semantic enhancements’ contribution.Source: §4 · p. 5
- The experiments are designed to evaluate how the semantic enhancements affect overall audio LLM performance.The passage describes the experimental design, not specific measured outcomes.Source: §4 · p. 5
4.1 Text-to-Speech
In VALL-E zero-shot TTS, X-Codec improves WER, Sim-O, and UTMOS over SpeechTokenizer, while both semantic codecs improve WER over acoustic-codec baselines. Phonetic-discriminability analysis also favors semantic codecs, with X-Codec improving as quantization depth increases.
4.1 Text-to-Speech
- The evaluation asks whether X-Codec improves VALL-E TTS and how it compares with SpeechTokenizer’s disentanglement strategy.Source: §4.1 · p. 5
4.1.1 Baselines
- The codec baselines are EnCodec, DAC, and SpeechTokenizer, whose distinct VQ layers separate speech content from timbre.Source: §4.1.1 · pp. 5–6
4.1.2 Training Details of X-Codec
- X-Codec and SpeechTokenizer use LibriSpeech and the same HuBERT-base-ls960 representations, averaged across layers as the semantic training objective.Source: §4.1.2 · p. 6
4.1.3 Training Details of VALL-E
- VALL-E is trained on LibriTTS for 100 AR epochs and 200 NAR epochs, with the AR learning rate set to 0.01 for stability.Source: §4.1.3 · p. 6
4.1.4 Evaluation Metrics
- Zero-shot TTS evaluation uses WER for intelligibility, Sim-O for speaker similarity, and UTMOS for predicted naturalness.Source: §4.1.4 · p. 6
4.1.5 Zero-shot TTS Results
- X-Codec improves WER, Sim-O, and UTMOS over SpeechTokenizer, while X-Codec and SpeechTokenizer both improve WER over acoustic-codec baselines.The comparison uses VALL-E for continuation zero-shot speech synthesis.Source: Table 1 · p. 6, §4.1.5 · p. 7
4.1.6 Analysing the Effect of Codec
- X-Codec and SpeechTokenizer outperform EnCodec and DAC on phonetic discriminability; X-Codec’s ABX error rates fall as nq rises from 1 to 8, unlike SpeechTokenizer.The authors also report that their concatenate method surpasses SpeechTokenizer’s disentangle method.Source: Table 2 · p. 7, §4.1.6 · p. 7
4.2 Music and Sound Generation
The section evaluates semantic information in X-Codec across music continuation, text-to-sound, and general-audio semantic understanding. X-Codec outperforms the baseline acoustic codec across the reported music-continuation and text-to-sound metrics.
4.2 Music and Sound Generation
- X-Codec is presented as the first exploration of adding semantic information to audio codecs for music and general sound generation through audio LLMs.The section contrasts sequential, frame-by-frame codecs with general-audio representation methods based on 2D mel-spectrograms, which pose integration challenges for existing generation frameworks.Source: §4.2 · pp. 7–8
4.2.1 Training Details of Self-Supervised General Audio Representation
- HuBERT-General-Audio is trained on about 200,000 hours with an AudioSet-like distribution, and its acoustic-unit targets use Encodec’s first VQ layer instead of MFCCs.Using the Encodec target removes the K-means discretization step; X-Codec is trained for 400,000 steps with HuBERT-General-Audio in its semantic module.Source: §4.2.1 · p. 8, §4.2 · p. 8
4.2.2 Music Continuation
- Music continuation uses 600 MUSDB18 samples, with 10-second prompts followed by 30-second generated segments.The model is trained on approximately 100,000 hours of music-only data, including about one million songs.Source: §4.2.2 · p. 8
- X-Codec significantly outperforms the baseline acoustic codec across all reported music-continuation metrics.Generated and ground-truth segments are both reconstructed using the codec’s first VQ layer to isolate generative-model differences from reconstruction fidelity.Source: §4.2.2 · p. 8, Table 3 · p. 8
4.2.3 Text-to-Sound
- Text-to-sound evaluation uses FD, IS, FAD, and CLAP score to assess generation and correspondence with the text prompt.The section reports X-Codec outperforming the baseline acoustic codec across all metrics.Source: §4.2.3 · p. 9
- X-Codec significantly outperforms the baseline acoustic codec across all reported text-to-sound metrics.Table 4 compares the baseline and X-Codec on text-to-sound tasks.Source: §4.2.3 · p. 9, Table 4 · p. 9
4.2.4 Analysing the Effect of Codec
- HuBERT-General-Audio outperforms DAC, Encodec, and the baseline acoustic codec across all ARCH benchmark metrics, while X-Codec is comparable or superior to HuBERT-General-Audio.The benchmark covers acoustic-event, music, and speech domains.Source: §4.2.4 · p. 9, Table 5 · p. 9
4.3 Limitation
X-Codec’s semantic layer trades off some codec efficiency and acoustic reconstruction fidelity. These costs are balanced by reported gains in human auditory perception, including UTMOS scores.
4.3 Limitation
- The codec’s compression-for-transmission functionality may be affected by adding semantic information, though the supplied passages give no quantitative estimate.The authors characterize this as a potential impact and describe the method as involving trade-offs.Source: §4.3 · p. 9
- The semantic extraction layer adds computational overhead and may increase processing time, reducing efficiency where rapid compression and transmission matter.The authors identify high-speed data compression as a setting where the approach may be less effective.Source: §4.3 · pp. 9–10
- Semantic layers can slightly impair Mel and STFT distance metrics, which are used to assess compressed-audio fidelity.The cited table covers reconstruction metrics on 1,000 LibriTTS speech samples.Source: §4.3 · p. 10, Table 6 · p. 10
5 Conclusion
X-Codec integrates semantic information from self-supervised learning models into an audio codec. Evaluations report improved semantic understanding and audio-generation quality across several domains.
5 Conclusion
- X-Codec integrates semantic information through self-supervised learning models to improve audio LLM performance.The conclusion names text-to-speech synthesis, music continuation, and general audio classification as application areas.Source: §5 · p. 10
- The reported application scope includes text-to-speech synthesis, music continuation, and general audio classification.These are the tasks named in the conclusion.Source: §5 · p. 10
- Evaluations report improved semantic understanding and audio-generation quality across a variety of domains.Source: §5 · p. 10
6 Model Details
X-Codec’s model details describe acoustic and semantic encoder-decoder components around an RVQ. The implementation also specifies the quantizer’s codebook updates and training choices.
6 Model Details
- The acoustic encoder uses four convolutional blocks with downsampling rates [2] [4], a hidden size of 256, and a model size of 12.18MB.Source: §6 · p. 14
- The RVQ uses exponential-moving-average codebook updates, a straight-through estimator, commitment loss, and randomly selected layer counts during training.The selected RVQ layer counts are [2] [4].Source: §6 · p. 14
- The acoustic decoder mirrors the encoder with four upsampling layers at inverse rates [4] [2] and is approximately 19.27 MB.Source: §6 · p. 14
- The semantic encoder and decoder each add two convolutional blocks with hidden size 768 to process and integrate semantic information.Source: §6 · p. 14
7 Music Continue with 4 VQ Flattern
The four-VQ music-continuation experiments use shorter segments and five-second prompts and generations. The reported results indicate improved FAD and perceptual quality with X-Codec.
7 Music Continue with 4 VQ Flattern
- With four flattened VQs, experiments use approximately 20-second segments, prompting with 5 seconds and generating 5 seconds.The shorter segment length is attributed to increased data density.Source: §7 · p. 14
- Table 7 compares the baseline acoustic codec and X-Codec on music continuation.Source: Table 7 · p. 14
- X-Codec shows noticeable performance improvement, particularly in FAD and perceptual quality metrics, in the four-VQ music-continuation setting.Source: §7 · p. 14