Source-linked AI summary
Rethinking Speech Codecs: From Compression to Autoregressive Generative Modeling
Yazheng Yang, Yao Qiu, Hui Su, Qi Liu
TL;DR
Existing speech codecs are optimized primarily for compression, leaving their discrete outputs misaligned with the autoregressive dependencies used by language models. The paper introduces autoregressive regularization and heterogeneous downsampling to produce more predictable, text-aligned tokens, and reports consistent improvements across codecs and downstream speech-language-model tasks. Overall, the results support codec design that jointly considers compression and generative modeling.
Problem
Existing codecs do not explicitly align speech-token sequences with the conditional dependencies and statistical structure assumed by autoregressive language models.
Method
The framework combines an auxiliary next-token prediction objective with heterogeneous downsampling of semantic first-layer tokens during codec training.
Results
The method consistently improves speech-language-model performance across multiple codecs and downstream tasks while maintaining compression quality.
Takeaways & Limitations
Speech codec tokenization can be made more suitable for generative modeling by jointly shaping token predictability and temporal granularity.
Abstract
from arXiv · showhide
Recent advances in speech language models leverage discrete speech representations from pretrained codecs to enable scalable training and generation. However, existing codecs are primarily optimized for compression without accounting for the autoregressive nature of language model training, resulting in suboptimal performance when modeling compressed speech tokens. In this work, we revisit speech discretization from a generative modeling perspective and propose a novel framework that explicitly aligns speech tokenization with autoregressive training. Our approach introduces autoregressive-compatible constraints during codec training, encouraging token sequences that exhibit temporal consistency and predictability. In addition, we propose a heterogeneous downsampling strategy for different layers of speech tokens, distinguishing semantic from acoustic layers, to improve the alignment between semantic tokens and corresponding textual content. Extensive experiments across multiple benchmarks demonstrate that our method bridges the gap between speech compression and generative modeling, enabling more effective continued pretraining of existing language models on speech data. The approach consistently improves performance across multiple codecs, validating its generality and applicability to diverse speech modeling scenarios.
1 Introduction
Existing speech codecs produce discrete tokens optimized for compression, but their sequences and distributions are poorly aligned with autoregressive language-modeling assumptions. ARDDS addresses this mismatch through autoregressive regularization and heterogeneous downsampling, improving speech-language-model performance while preserving compression quality.
- Motivation: Compression-oriented codecs do not explicitly model the conditional dependencies required by autoregressive LLMs.Their token streams can therefore lack temporal consistency and next-token predictability.
- Motivation: Existing audio-token distributions differ substantially from the Zipfian frequency-rank pattern observed in textual tokens.This statistical divergence creates challenges for autoregressive speech modeling and generation.
- Method: ARDS introduces an auxiliary next-token prediction objective during codec training to encourage autoregressively coherent speech-token sequences.The framework explicitly accounts for downstream generative-modeling requirements beyond reconstruction fidelity.
- Method: Heterogeneous downsampling reduces the frame rate of semantic first-layer tokens to better match textual-token granularity.This narrows the distributional gap between semantically rich speech tokens and text while reducing sequence length.
- Results: Across multiple representative codecs, the method improves speech-generation and language-modeling performance without degrading compression efficiency.The authors present this as evidence that the framework is codec-agnostic and general.
2 Related Work
Prior audio codecs efficiently compress waveforms into discrete representations, while speech-language models increasingly reuse language-modeling methods for audio. Existing approaches generally overlook autoregressive compatibility, motivating a codec-training framework that directly incorporates next-token predictability.
- Audio Codecs and Discrete Representations: Neural audio codecs use vector quantization and multi-level encoders to compress waveforms into discrete token sequences.They are typically optimized for rate-distortion trade-offs and perceptual quality rather than autoregressive sequential dependencies.
- Speech-Language Modeling with LLMs: Speech-language models increasingly represent audio with discrete codec tokens to reuse the autoregressive training paradigm of LLMs.Related systems also use continuous features, pretrained encoders, or speech-text alignment methods.
- Proposed Direction: The method combines standard hierarchical codec representations with autoregressive regularization and heterogeneous downsampling of first-layer tokens.These design choices target both language-like token statistics and alignment between speech and text tokenization.
- Tokenizer Alignment: Prior speech tokenization methods explore expressive token spaces but largely overlook compatibility with autoregressive modeling.The proposed framework differs by introducing an autoregressive decoder into codec training to enforce next-token predictability.
3 Methodology
The methodology adapts neural audio codecs for autoregressive language-model training by regularizing token predictability and using heterogeneous downsampling to align semantic speech tokens with text while preserving acoustic detail.
- Autoregressive regularization: Conventional codec training emphasizes reconstruction and perceptual fidelity but does not constrain the sequential structure needed by autoregressive models.Consequently, token sequences may lack predictable conditional dependencies suitable for next-token prediction.
- Autoregressive regularization: The framework combines conventional codec components with autoregressive regularization so discrete speech tokens become more predictable for language-model training.An auxiliary decoder predicts subsequent tokens during codec training, while the original codec objectives preserve reconstruction and perceptual quality.
- Autoregressive regularization: Soft token assignments enable gradients to pass through quantization and the encoder into the auxiliary autoregressive decoder.A temperature-controlled softmax approximates hard codeword selection, and the combined objective balances codec fidelity with autoregressive predictability.
- Heterogeneous downsampling: Heterogeneous downsampling reduces the semantic layer's token rate more aggressively than acoustic layers to better match textual token frequency.The first stream uses r0 < r1 = r2 = . . . rL−1, while acoustic layers retain a shared finer sampling rate.
- Heterogeneous downsampling: Preserving higher acoustic-layer rates while compressing semantic tokens improves speech-text alignment and supports autoregressive modeling in LLMs.The design targets the mismatch between speech-token and transcript lengths without discarding finer acoustic detail.
4 Experiments & Analysis
Experiments evaluate ARDDS across codec architectures, token statistics, SpeechLM training efficiency, downstream tasks, and codec quality. The method improves autoregressive speech modeling while preserving compression performance.
- Experimental scope: Experiments span multiple codec architectures, token statistics, SpeechLM training, and downstream speech understanding, recognition, and generation tasks.The evaluation includes standardized codec comparisons and end-to-end SpeechLM assessment.
- Token statistics: ARDDS-augmented tokens produce more Zipf-like distributions across 1-gram to 7-gram statistics, indicating greater predictability for autoregressive modeling.The observed shift is attributed to autoregressive regularization and heterogeneous downsampling jointly reshaping token statistics.
- Training efficiency: SpeechLMs trained with ARDDS-augmented XCodec converge faster and reach a lower final loss than models trained with vanilla XCodec.The loss decreases more rapidly early in training and stabilizes at a lower value.
- Ablation study: Removing either autoregressive regularization or heterogeneous downsampling degrades downstream performance, while removing both returns results to the original XCodec baseline.The full model performs best across the evaluated downstream tasks.
- Codec quality: Augmented codecs maintain comparable codec quality without significant degradation across reconstruction, perceptual, and speech-quality evaluations.The reported evaluations include WER, SPKSIM, UTMOS, PESQ, and STOI.
- Downstream performance: SpeechLMs using augmented codecs consistently outperform vanilla-codec counterparts across StoryCloze, TopicStoryCloze, AISHELL-I, and SeedTTS.The improvements cover reasoning, speech understanding, recognition, and generation tasks.
5 Conclusion
The paper reframes speech codec design around autoregressive generative modeling, combining autoregressive regularization with heterogeneous downsampling. Across multiple codecs, the framework improves speech-token quality while retaining compression quality and produces more Zipf-like token distributions.
- The framework combines autoregressive regularization with heterogeneous downsampling to produce compact, autoregressively coherent speech tokens.It is designed as a codec-agnostic training framework applicable to a wide range of neural audio tokenizers.
- Extensive experiments across multiple codecs show consistent improvements in audio token sequence quality for SpeechLM training.
- The framework induces more Zipf-like speech-token distributions, bringing them closer to natural-language patterns.
- The authors present the framework as a practical pathway toward unifying compression and generative modeling in future audio systems.
A Minimum Final Token Rate Analysis
The analysis identifies the lowest final token rate that remains trainable and semantically effective under the proposed autoregressive framework. Training is stable through 6.25 Hz but becomes severely unstable at 3 Hz, establishing 6.25 Hz as the practical lower bound.
- The controlled analysis evaluates final frame rates of 50 Hz, 25 Hz, 12.5 Hz, 6.25 Hz, and 3 Hz.The study seeks the minimal rate that preserves semantic content while remaining trainable under autoregressive learning.
- Commitment loss remains stable down to 6.25 Hz but becomes substantially larger and more variable at 3 Hz.Commitment loss reflects how encoder outputs commit to discrete codebook entries, so instability indicates unstable quantization and training dynamics.
- 6.25 Hz is the practical lower bound for the final token rate, whereas reducing it to 3 Hz causes severe training instability.The 3 Hz setting produces commitment loss values with large magnitude and high variance.
B Spectrogram Analysis of Reconstruction Quality
Spectrogram comparisons across English and Chinese test samples show reconstructed audio closely matching reference speech. The visual results support preservation of spectral detail, temporal coherence, and perceptual quality across languages.
- Reconstructed spectrograms closely align with reference spectrograms across English LibriSpeech and Chinese Common Voice samples.The qualitative evaluation covers six test samples: three English utterances and three Chinese utterances.
- The visual consistency indicates that the framework preserves both spectral detail and temporal coherence.
- The results indicate that enforcing autoregressive compatibility does not compromise perceptual quality or reconstruction fidelity across languages.This conclusion is stated together with the objective metrics reported earlier.