PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models

Yitian Gong, Kuangwei Chen, Zhaoye Fei, Xiaogui Yang, Ke Chen, Yang Wang, Kexin Huang, Mingshu Chen, Ruixiao Li, Qingyuan Cheng, Shimin Li, Xipeng Qiu

arXiv:2602.10934v2cs.SDeess.AS

TL;DR

Existing audio tokenizers rely on pretrained components, staged training, or specialized architectural biases that limit unified scaling and representation flexibility. The paper introduces CAT and the 1.6-billion-parameter MOSS-Audio-Tokenizer, trained end-to-end with causal Transformers. It reports state-of-the-art reconstruction across speech, sound, and music, plus strong autoregressive TTS and competitive ASR without an auxiliary encoder.

  • Problem

    Existing audio tokenizers rely on pretrained encoders, multi-stage pipelines, or architectural biases that complicate unified scaling while audio requires both acoustic detail and long-range structure.

  • Method

    The paper develops CAT, a homogeneous causal Transformer architecture, and MOSS-Audio-Tokenizer, which jointly optimizes tokenizer components end-to-end from scratch.

  • Results

    MOSS-Audio-Tokenizer achieves state-of-the-art reconstruction across speech, sound, and music, while its discrete representations support strong downstream speech generation and understanding.

  • Takeaways & Limitations

    CAT provides a unified, scalable interface for autoregressive audio foundation models spanning compression, generation, and understanding.

  • Takeaways & Limitations

    The proposed design assumes that unified audio representations must preserve both fine-grained acoustic information and semantic structure, while remaining simple and homogeneous for scaling.

Abstract

from arXiv · show

Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.

1 Introduction

The paper proposes a fully end-to-end, homogeneous Transformer audio tokenizer designed as a unified interface for autoregressive audio modeling. MOSS-Audio-Tokenizer scales this approach to diverse audio and supports strong reconstruction, generation, and understanding results.

  • Motivation: Existing audio tokenizers use pretrained encoders, multi-stage pipelines, or architectural biases that complicate unified scaling.
  • Results: Across speech, sound, and music, reconstruction quality is state-of-the-art at all evaluated bitrates and improves with increased model capacity and computation.
  • Results: The discrete tokens support a purely autoregressive TTS system that outperforms prior non-autoregressive and cascaded approaches, alongside competitive ASR without an auxiliary encoder.
  • Method: CAT is a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, decoder, causal language model, and discriminator end-to-end.
  • Method: 1.6 billion parameters and 3 million hours of diverse audio underpin MOSS-Audio-Tokenizer’s scalable general-audio tokenization.
  • Capabilities: 12.5 Hz tokenization supports high-fidelity reconstruction across speech, sound, and music, with variable bitrates from 0.125 kbps to 4 kbps.

2 Rethinking Discrete Audio Tokenization for Future Audio Foundation Models

The paper frames audio tokenization as a discrete interface for autoregressive audio foundation models, requiring unified representations, scalability, causality, and bitrate flexibility. CAT addresses these requirements with a homogeneous causal Transformer design.

  • Unified Audio Representation: Audio tokenizers must preserve fine-grained acoustic information and semantic structure across speech, sound, and music for unified autoregressive modeling.
  • Simplicity and Scalability: A simple, homogeneous architecture is intended to scale jointly with model capacity, data, and computation without fixed bottlenecks from specialized components.
  • Causality: CAT uses causal Transformers so each token is computed without future audio context, aligning tokenization with autoregressive generation and low-latency inference.
  • Low Frame Rate and Bitrate Robustness: An effective tokenizer should combine a low frame rate with robustness across bitrates to support reconstruction, understanding, and generation without task-specific redesign.

3 Causal Audio Tokenizer with Transformer (CAT)

CAT is a CNN-free, causal Transformer architecture that jointly learns audio encoding, quantization, decoding, and auxiliary semantic alignment. Its RVQ-based design supports streaming, variable-bitrate autoregressive audio generation through quantizer dropout and prefix-conditioned prediction.

  • Architecture: CAT uses causal Transformer blocks for both encoder and decoder, operating directly on raw waveforms while enabling streaming encoding and decoding.The architecture avoids intermediate mel-spectrogram representations and progressively compresses waveform sequences through patchification.
  • Quantization: 32-layer residual vector quantization with quantizer dropout supports robust tokenization across variable bitrates.Quantizer dropout exposes the decoder to differing effective bitrate depths during training.
  • Training objectives: An auxiliary audio-to-text objective uses a decoder-only language model to encourage CAT representations that align with text-based language modeling.The language model receives quantizer hidden states and autoregressively predicts text for tasks including ASR and audio captioning.
  • Training objectives: CAT jointly optimizes encoder, quantizer, decoder, and discriminators end-to-end without pretrained encoders or external semantic teachers.The generator combines semantic, reconstruction, commitment, codebook, adversarial, and feature-matching losses with scalar weights.
  • Autoregressive generation: CAT-TTS predicts CAT’s RVQ tokens autoregressively from text and a speaker prompt, without semantic disentanglement or cascaded generative models.Temporal and depth Transformers model long-range temporal dependencies and coarse-to-fine RVQ structure, respectively.
  • Autoregressive generation: Progressive Sequence Dropout trains on randomly truncated RVQ prefixes, enabling one autoregressive model to generate speech at controllable bitrates.At inference, the selected RVQ depth determines which token streams are predicted and decoded into waveforms.

4 Experiments

The experiments evaluate MOSS-Audio-Tokenizer on reconstruction and speech generation, including robustness to bitrate reduction and comparisons with open-source systems.

  • Reconstruction Evaluation: MOSS-Audio-Tokenizer is evaluated across low, medium, and high bitrates on speech, general audio, and music benchmarks.Objective and subjective metrics are used for comparison with open-source audio tokenizers.
  • Reconstruction Evaluation: MOSS-Audio-Tokenizer outperforms prior methods on speech reconstruction at low bitrates and achieves state-of-the-art results at medium and high bitrates.On audio and music benchmarks, it remains competitive across evaluated bitrates, with quality improving as bitrate increases.
  • Speech Generation: At full bitrate, all Progressive Sequence Dropout variants achieve comparable TTS performance, with low word error rate and high speaker similarity.Without dropout, performance degrades more steeply as bitrate decreases because training uses full RVQ depth while inference uses reduced depth.
  • Speech Generation: Progressive Sequence Dropout makes fully autoregressive TTS substantially more robust under reduced-bitrate settings.Performance remains highly consistent across dropout probabilities p = 0.25, 0.5, and 1.0.
  • Speech Generation: Increasing dropout probability significantly reduces GPU memory consumption while maintaining comparable synthesis quality, motivating the adoption of p = 1.0.The selected setting is used in subsequent experiments to maximize training efficiency.
  • Speech Generation: CAT-TTS significantly outperforms previous discrete fully autoregressive models in speaker similarity and achieves competitive performance against recent state-of-the-art systems.All compared systems maintain very low word error rates, typically below 2%, and CAT-TTS obtains the highest speaker similarity on Seed-TTS-Eval in English and Chinese.

5 Analysis Of Scalling Behavior

The analysis finds that CAT scales with end-to-end optimization, model size, bitrate, and training batch size, while parameter and quantization capacity must be scaled together.

  • End-to-End Optimization: End-to-end training yields sustained improvements across all metrics without early saturation, unlike partial optimization protocols that freeze the encoder and quantizer.The comparison supports end-to-end optimization as important for scaling CAT with increased computation and capacity.
  • Co-Scaling of Model Parameters and Quantization Capacity: Increasing parameter count improves reconstruction quality across 0.5–4 kbps, while smaller models saturate earlier and the 1169M model benefits most from high bitrates.The evaluated combined encoder–decoder sizes are 319M, 505M, 710M, and 1169M parameters.
  • Co-Scaling of Model Parameters and Quantization Capacity: At low bitrates, the 1169M model can underperform smaller models at higher bitrates, making bitrate rather than parameter count the primary bottleneck.The results indicate that parameter scaling and quantization depth are co-dependent and should be expanded together.
  • Training-Scale Scaling: Increasing training batch size yields strictly better reconstruction quality across SIM, STOI, and PESQ at any given training point.Larger batch-size curves continue rising at 250k steps and achieve higher quality within the same step budget.
  • Training-Scale Scaling: CAT exhibits stable and predictable scaling with batch size, allowing increased data throughput to translate into systematically higher-fidelity representations.This supports trading computational resources directly for reconstruction quality in large-scale tokenizer training.

6 Related Works

Related work spans neural audio codecs, semantic audio tokenizers, and end-to-end audio generation, motivating a unified tokenizer with fewer handcrafted constraints and predictable scaling.

  • Discrete Audio Tokenization: Most discrete audio tokenizers use an encoder–quantizer–decoder with adversarial training, while systems such as SoundStream, Encodec, and DAC add different quantization or discriminator designs.These approaches target high-fidelity reconstruction through quantizer dropout, multi-scale STFT discrimination, or factorized vector codes.
  • Semantic Audio Tokenization: Semantic audio tokenizers commonly use knowledge distillation from pretrained teachers or large-scale audio–text supervision to enrich representations.These strategies are represented by SpeechTokenizer, Mimi, Qwen3 TTS Tokenizer, Baichuan Audio Tokenizer, XY-Tokenizer, and MiMo Audio Tokenizer.
  • Motivation: The paper argues that a suitable tokenizer should minimize handcrafted priors and architectural constraints while scaling computation and data through end-to-end training.The goal is a simple, scalable design aligned with native audio language-modeling needs.
  • Audio Generation: Audio generation has progressed from hierarchical and cascaded pipelines toward end-to-end architectures that directly generate audio tokens.End-to-end designs can reduce system complexity and error propagation while improving training stability and inference efficiency.
  • Audio Generation: Discrete-generation systems use temporal and depth modeling, while continuous-representation methods use local diffusion transformers to capture fine-grained acoustic detail.Examples include MusicGen, Moshi, and LocDiT-based approaches.
  • End-to-End Audio Language Models: Recent end-to-end audio language models increasingly scale data and use information-preserving representations, reinforcing the central role of audio tokenizers as native discrete interfaces.This motivates tokenizers that combine reconstruction fidelity with compatibility with autoregressive modeling and predictable scaling.

7 Conclusion

The paper introduces CAT as a fully end-to-end Transformer-based discrete interface and develops MOSS-Audio-Tokenizer as a large-scale model with strong reconstruction and downstream speech performance.

  • Conclusion: CAT is a fully end-to-end Transformer-based architecture designed as a unified discrete interface for autoregressive audio language modeling.Its architecture jointly supports representation learning and audio-token modeling.
  • Conclusion: MOSS-Audio-Tokenizer uses 1.6 billion parameters and 3 million hours of diverse audio to learn general representations across multiple audio domains.Joint end-to-end optimization covers the encoder, quantizer, decoder, discriminators, and decoder-only language model for semantic alignment.
  • Conclusion: MOSS-Audio-Tokenizer achieves state-of-the-art reconstruction among open-source audio tokenizers and strong downstream speech generation and understanding performance.The conclusion positions CAT as a unified, scalable interface for future native audio foundation models.

A.1 Architecture

MOSS-Audio-Tokenizer combines causal Transformer encoder and decoder stacks with residual vector quantization and jointly trained semantic and adversarial objectives. Its evaluation spans reconstruction across speech, sound, and music using objective and subjective measures at variable bitrates.

  • Architecture: The encoder and decoder each use 68 causal Transformer blocks with 10 s sliding-window attention for efficient streaming inference.The encoder progressively reduces sequence length through patchify operations, followed by linear projections.
  • Quantization: A 32-layer residual vector quantizer with 1,024-entry codebooks supports variable-bitrate tokenization through quantizer dropout.Each layer uses factorized vector quantization with latent dimension 8 and L2-normalized codes.
  • Semantic supervision: A 0.5B decoder-only causal language model provides audio-to-text supervision through ASR, multi-speaker ASR, and audio captioning tasks.The language model autoregressively predicts text conditioned on quantizer outputs.
  • Training objectives: Multi-period and complex STFT discriminators are optimized jointly with the encoder, quantizer, decoder, and semantic head.This forms a single end-to-end optimization procedure for the tokenizer components.
  • Evaluation: Evaluation covers speech, sound, and music reconstruction using objective metrics, MUSHRA listening scores, and multiple bitrates when supported.Speech metrics include SIM, STOI, and PESQ; sound and music use mel-spectrogram and STFT distances.

B.2 Results Of Subjective Evaluation

Subjective evaluation shows that MOSS-Audio-Tokenizer maintains strong perceptual quality across bitrates, whereas several baselines degrade away from their preferred operating points. Target-bitrate tokenizers remain competitive at the bitrates for which they were trained.

  • Subjective results: MOSS-Audio-Tokenizer achieves strong and consistent subjective performance across a wide range of bitrates in reconstructed speech.The evaluation compares MOSS-Audio-Tokenizer with open-source tokenizers using crowd-sourced MUSHRA scores.
  • Subjective results: Encodec, DAC, and SpeechTokenizer have competitive subjective scores at higher bitrates but degrade noticeably at lower bitrates.This comparison is reported in the speech listening evaluation.
  • Baseline comparison: BigCodec, XCodec 2.0, XY-Tokenizer, and Qwen3 TTS Tokenizer perform well at their respective training bitrates.At comparable bitrates, their perceptual quality is competitive with MOSS-Audio-Tokenizer.

C.2 Training Details

CAT-TTS is a purely autoregressive, discrete-token TTS system that generates speech from text without predefined intermediate representations. It supports variable-bitrate generation and is compared with cascaded, non-autoregressive, and prior autoregressive systems.

  • Inference: Inference concatenates prompt transcription, target text, and prompt audio tokens before autoregressively predicting target speech tokens.The predicted speech tokens are decoded into waveforms with the CAT decoder.
  • Baselines: The comparison includes cascaded TTS systems, purely non-autoregressive systems, and prior fully autoregressive models.The three groups include systems such as MaskGCT, F5-TTS, and Llasa.
  • CAT-TTS design: CAT-TTS performs zero-shot TTS end-to-end by directly generating discrete speech tokens from text.It does not rely on predefined intermediate representations such as semantic tokens.
  • CAT-TTS design: Progressive Sequence Dropout enables CAT-TTS to support variable-bitrate speech generation.The strategy is presented as part of the CAT-TTS architecture.

E Speech Understanding On CAT

CAT-ASR tests whether CAT speech tokens can be directly consumed by a large language model for speech understanding. It achieves competitive performance on English and Chinese benchmarks without additional alignment or auxiliary supervision.

  • Motivation: CAT-ASR directly feeds discretized CAT speech tokens into an LLM to evaluate linguistic content and text alignment.The study uses ASR as the speech-understanding task.
  • Model: Each speech frame’s RVQ tokens are summed into one input embedding for a Qwen3-1.7B backbone.The LLM is trained autoregressively to predict the corresponding text sequence.
  • Training: CAT-ASR is trained on approximately 2 million hours of paired audio-text data without additional alignment or auxiliary supervision.Training uses the standard ASR objective.
  • Results: CAT-ASR achieves competitive performance on both English and Chinese benchmarks.Evaluation reports WER on LibriSpeech test-clean and CER on the AIShell-2 iOS subset.
Loading 2602.10934v2…