PaperlayerView exact revision on arXiv

Source-linked AI summary

UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization

Dongchao Yang, Yuanyuan Wang, Dading Chong, Songxiang Liu, Xixin Wu, Helen Meng

arXiv:2602.04683v3cs.SD

TL;DR

Audio language models need an intermediate representation that supports both understanding and generation, while also generalizing across unseen tasks. The paper introduces ReasoningCodec and a unified, specialized autoregressive model trained with large-scale multi-task data, achieving competitive seen-task performance and strong few-shot and zero-shot generalization. The approach is intended as a unified foundation model for speech, sound, and music processing.

  • Problem

    Audio language models lack a representation that simultaneously supports scalable generation and strong understanding, and they lag text models in generalization and task versatility.

  • Method

    The paper factorizes audio into text-aligned reasoning and reconstruction tokens, then trains a unified autoregressive architecture with specialized layers and multi-stage multi-task training.

  • Results

    UniAudio 2.0 performs competitively on seen speech, sound, and music tasks and shows strong few-shot and zero-shot generalization to unseen tasks.

  • Takeaways & Limitations

    The framework supports unified audio understanding and generation across speech, sound, and music within a foundation-model setting.

  • Takeaways & Limitations

    Audio generation and voice conversion can be misused for impersonation, fraud, harassment, or unauthorized content creation.

Abstract

from arXiv · show

We study two foundational problems in audio language models: (1) how to design an audio tokenizer that can serve as an intermediate representation for both understanding and generation; and (2) how to build an audio foundation model that generalizes in few-shot and zero-shot settings, analogous to large language models. To this end, we make the following two contributions. First, we propose ReasoningCodec, a discrete audio codec that factorizes audio into (i) reasoning tokens, which encode text-aligned, high-level analysis and planning representations for audio understanding and hierarchical generation, and (ii) reconstruction tokens, which encode semantic-rich acoustic cues for high-fidelity waveform reconstruction. This design achieves understanding performance comparable to strong continuous representations while improving generation quality and reconstruction fidelity over prior discrete tokenizers. Second, we introduce a unified autoregressive architecture for text and audio, together with multi-stage training and multi-task data construction. Using this framework, we train UniAudio 2.0 on 100B text tokens and 60B audio tokens. Across a wide range of speech, sound, and music tasks, UniAudio 2.0 performs competitively on in-domain evaluations and demonstrates strong few-shot and zero-shot generalization to unseen tasks. Demo, code, and checkpoints will be available at \href{https://dongchaoyang.top/UniAudio2Demo/}{https://dongchaoyang.top/UniAudio2Demo/}.

1. Introduction

UniAudio 2.0 addresses limitations in audio representations, unified architectures, and scalable multi-task data by combining factorized audio tokens, specialized transformer layers, and auditory-sentence training. Trained on 100B text tokens and 60B audio tokens, it performs competitively on seen tasks and generalizes strongly to unseen tasks.

  • Current audio language models lag text models in generalization, scalability, and task versatility because of representation, architecture, and data challenges.
  • Continuous representations support understanding but are difficult to generate autoregressively, whereas discrete codecs scale generation but lack text-aligned high-level abstractions.
  • Auditory sentences organize related audio and text segments into long-context sequences that induce multiple understanding and generation tasks.
  • ReasoningCodec factorizes audio into text-aligned reasoning tokens and reconstruction tokens that preserve high-fidelity waveform information.
  • Functional layer specialization assigns transformer layers to audio understanding, cross-modal alignment, and audio generation experts.
  • 100B text tokens and 60B audio tokens support competitive seen-task performance and strong few-shot and zero-shot generalization on unseen tasks.

2. Related Works

Prior audio language models typically choose between continuous representations optimized for understanding and discrete tokens optimized for scalable generation. ReasoningCodec is introduced to bridge this trade-off with factorized reasoning and reconstruction token streams.

  • Audio tokenization determines the intermediate representation used by both audio understanding and generation systems.
  • Continuous representations: Continuous audio features preserve rich perceptual information and typically perform strongly on understanding tasks but are difficult to generate directly.
  • Discrete tokens: Discrete acoustic tokenizers support scalable generation but mainly encode low-level details and provide limited semantic abstraction for understanding.
  • Towards unified representations: ReasoningCodec factorizes audio into reasoning and reconstruction tokens to retain language-aligned information while supporting faithful reconstruction and autoregressive generation.

3. ReasoningCodec

ReasoningCodec separates text-aligned reasoning from multi-level reconstruction information, using distinct branches and conditioning to support both audio understanding and waveform reconstruction. Its reasoning branch combines pretrained audio encoders, text-LLM decoding, quantization, and staged optimization.

  • ReasoningCodec targets an audio representation that is both language-model-friendly for autoregression and information-preserving for understanding.
  • Reasoning tokens encode text-aligned high-level perceptual analyses, while reconstruction tokens capture semantic content and fine-grained acoustics.
  • Reasoning tokens intentionally capture only text-level semantics and therefore contain less information than conventional semantic tokens.
  • The codec produces reasoning tokens r = Tr(x) and reconstruction tokens s = Ts(x | r), with waveform reconstruction performed from D(s).
  • Reasoning branch: The reasoning branch fuses frozen Whisper and music encoders, uses a query-based transformer, and decodes through a pretrained text LLM.
  • Reasoning branch: Reasoning training combines supervised audio-understanding fine-tuning with GRPO-style reinforcement learning for detailed, grounded analyses.
  • Reconstruction branch: The reconstruction branch uses separate quantization resources for phone semantics, music structure, environmental sound semantics, and remaining acoustic information.
  • Reconstruction branch: FiLM conditions reconstruction semantic features on quantized reasoning features, while a flow-based decoder is trained with reconstruction and semantic feature losses.

4. UniAudio 2.0

UniAudio 2.0 combines factorized audio tokenization with a unified autoregressive model whose specialized components support audio understanding, generation, and cross-modal learning. Multi-task, multi-stage training on large-scale text and audio data yields competitive seen-task performance and strong generalization to unseen tasks.

  • Representation: ReasoningCodec separates audio into reasoning tokens for understanding and reconstruction tokens for high-fidelity generation.The tokenizer provides semantic abstraction while preserving fine-grained acoustic information.
  • Unified representation: UniAudio 2.0 uses a unified vocabulary and multi-stream representation for audio and text within one autoregressive backbone.Audio is represented with reasoning and reconstruction streams, while text uses a single stream.
  • Architecture: The architecture specializes transformer layers into audio understanding, cross-modal, and audio generation experts.Audio-specific experts operate on audio positions, while cross-modal experts retain initialization from the pretrained text language model.
  • Autoregressive modeling: Training combines text and audio autoregressive objectives, with modality-specific prediction heads and balanced loss contributions.Text tokens are predicted at token level, whereas audio frames use a lightweight local decoder to predict parallel audio tokens.
  • Training: UniAudio 2.0 is trained through four stages that progressively integrate audio understanding, audio generation, and audio–text capabilities.The stages include understanding warm-up, generation warm-up, audio–text pre-training, and audio–text mid-training.
  • Capabilities: UniAudio 2.0 achieves competitive performance on seen tasks and strong few-shot and zero-shot generalization across unseen tasks and domains.The reported capabilities include more robust complex understanding, more controllable generation, and improved adaptability to unseen scenarios.

5. Experiments

Experiments evaluate ReasoningCodec for reconstruction, token modeling, and understanding, then assess UniAudio 2.0 across seen and few-shot tasks. Ablations examine reasoning tokens, multi-stage training, layer specialization, and model size.

  • Audio Tokenizer Evaluation: ReasoningCodec achieves consistently strong reconstruction quality across speech, sound, and music modalities.It is evaluated against universal audio codecs, with subjective MUSHRA results also reported.
  • Audio Tokenizer Evaluation: ReasoningCodec improves token modeling, with reasoning tokens easier for language models to capture and improving reconstruction-token prediction accuracy.Combining reasoning and reconstruction tokens also reduces reconstruction-token modeling difficulty.
  • Audio Tokenizer Evaluation: ReasoningCodec achieves the best understanding performance among discrete audio tokenizers and approaches continuous Whisper-tokenizer performance on multiple benchmarks.The evaluation covers ASR, emotion recognition, audio classification, and music classification.
  • UniAudio 2.0 Evaluation: UniAudio 2.0 performs strongly across speech, general-audio, and music benchmarks, including competitive seen-task results and multilingual speech capabilities.The model uses 3B parameters and is reported to outperform MiMo-Audio 7B on ASR and TTS while outperforming Qwen2.5-Omni 7B on audio captioning.
  • UniAudio 2.0 Evaluation: UniAudio 2.0 shows strong few-shot generalization on most evaluated benchmarks, especially in the 1-shot setting.Few-shot evaluations include speech denoising, voice conversion, sound classification, and emotion classification.
  • Ablation Study: Removing reasoning tokens, mid-training, or functional layer specialization degrades performance, while model-size reductions weaken multi-task and few-shot or zero-shot generalization.The ablations link mid-training to text capability and generalization, and identify model size as important for model capacity.

6. Conclusion

The paper presents UniAudio 2.0 as a unified audio foundation model supporting understanding and generation, built around ReasoningCodec and a unified autoregressive training framework. Experiments show strong performance on seen tasks and promising few-shot and zero-shot generalization.

  • ReasoningCodec factorizes audio into reasoning and reconstruction tokens, while UniAudio 2.0 combines unified autoregressive modeling with multi-stage, multi-task training.
  • Extensive experiments demonstrate strong and consistent performance across seen speech, sound, and music tasks, with promising few-shot and zero-shot generalization to unseen scenarios.
  • Scaling data, task diversity, and model capacity is identified as important for improving generalization.

7. Impact Statement

The paper’s broader impact includes both research contributions and risks from unified audio understanding and generation. Its methods span tokenizer training, reconstruction, evaluation, and multi-stage model development.

  • Audio generation and voice conversion may be misused for impersonation, fraud, harassment, or unauthorized content.
  • ReasoningCodec training uses supervised fine-tuning, GRPO-style reinforcement learning, and reconstruction-branch training with a flow-based decoder.
  • FiLM connects reasoning and reconstruction branches effectively without influencing reconstruction performance.
  • Subjective MUSHRA evaluation reports the best reconstruction performance for speech, sound, and music among the compared tokenizers.

A.3.5. WHY REASONING TOKENS ARE IMPORTANT FOR BOTH UNDERSTANDING AND GENERATION TASKS?

Reasoning tokens are designed as a text-aligned intermediate representation that supports audio understanding while simplifying reconstruction-token prediction for generation. Experiments and analysis connect this design to faster learning, lower uncertainty, and improved tokenizer performance.

  • Reasoning tokens consistently improve understanding performance and reduce reconstruction-token modeling difficulty, including lower perplexity.
  • As a language-aligned bottleneck, reasoning tokens retain language- and reasoning-relevant content while filtering task-irrelevant acoustic details.
  • For understanding, reasoning tokens provide a compact semantic abstraction aligned with the text language model for ASR, classification, and reasoning evaluations.
  • For generation, reasoning tokens summarize high-level intent and semantics, reducing long-range uncertainty and stabilizing autoregressive decoding.
  • Conditioning reconstruction tokens on reasoning tokens lowers conditional entropy by the amount of their conditional mutual information.
  • Reasoning-token designs outperform the CosyVoice 3 tokenizer on ASR and emotion recognition, while ReasoningCodec achieves the best subjective reconstruction performance across speech, sound, and music.

B.3. The details of multiple stage training

UniAudio 2.0 is trained progressively through audio-specialized warm-up stages, joint audio-text pre-training, and longer-context mid-training. The recipe gradually aligns modalities, expands context, and targets generalization across tasks.

  • The four-stage training process begins with audio understanding warm-up and audio generation warm-up before joint audio-text pre-training and mid-training.
  • Stage 1: Audio understanding warm-up: Understanding warm-up trains audio understanding experts while freezing other components and adds semantic distillation from frozen speech and music encoders.
  • Stage 2: Audio generation warm-up: Generation warm-up updates the audio generation expert and local decoder while keeping understanding and cross-modal experts fixed.
  • Stage 3: Audio-text pre-training: Audio-text pre-training jointly updates model parameters across understanding, generation, text-only, and audio-only tasks under a unified autoregressive objective.
  • Stage 4: Audio-text mid-training: Audio-text mid-training uses auditory sentences and a maximum context length of 2048 tokens to model longer, more complex sequences and improve robustness across task settings.

B.4.1. SEEN TASKS

Seen-task evaluation covers speech, sound, music, and lyric tasks using task-specific benchmarks and metrics. The section specifies the datasets and measures used to assess recognition, synthesis, captioning, and generation.

  • ASR evaluation uses LibriSpeech, SEED-TTS, and WSYue benchmarks, with WER for English and CER for Chinese and Cantonese.
  • TTS evaluation uses LibriSpeech, SEED-TTS, and WSYue benchmarks, combining speech-recognition scores with DNS-MOS quality assessment.
  • Instruct TTS is evaluated with WER, style accuracy, and UTMOSv2, including Chinese evaluation despite no Chinese Instruct TTS training.
  • Audio and music captioning use CIDER and GPT-score on their respective captioning benchmarks.
  • Audio and music generation use FD, KL, and CLAP-based metrics, while song generation uses WER and AudioBox Score.
  • Lyric recognition is evaluated using the benchmark from SongGen.

B.4.2. FEW-SHOT TASKS

Few-shot evaluation tests speech denoising, voice conversion, emotion classification, and sound-event classification on datasets constructed from established audio corpora. The tasks use task-specific quality, recognition, similarity, and classification measures.

  • Few-shot speech denoising uses LibriTTS-test-clean mixed with WHAM noise and evaluates PESQ, STOI, WER, and DNS-MOS.
  • Few-shot voice conversion is evaluated on VCTK using WER, Speaker Similarity (SIM), and DNS-MOS.
  • Few-shot emotion classification uses English and Chinese splits from ESD to construct its evaluation set.
  • Few-shot sound-event classification uses the TUT acoustic scenes 2017 dataset.

B.5. Zero-shot tasks

Zero-shot evaluation covers text understanding, speech interaction, dysarthric speech recognition, guided TTS, and speech–sound generation, alongside broader benchmark comparisons. The section defines zero-shot evaluation as task novelty without inference demonstrations.

  • Zero-shot tasks are never seen during training and provide no demonstrations at inference time.
  • Text understanding uses MMLU under a direct zero-shot evaluation setting.
  • Speech-to-speech/text question answering is treated as zero-shot because speech conversation data are excluded from training, and is evaluated on InstructS2S-Eval with GPT-score.
  • Dysarthric speech recognition is evaluated zero-shot because dysarthric speech is absent from training data.
  • The section introduces audio-prompt-and-caption-guided TTS, which combines timbre conditioning with speaking-style control and measures similarity, style accuracy, WER, and UTMOSv2.
  • Speech–sound generation asks the model to generate speech and a corresponding sound event from sound-event tags and speech content, evaluated with WER, CLAP score, and UTMOSv2.

C. Limitation

The paper reports limitations involving generation latency, incomplete unseen-task coverage, unexplored scaling, domain imbalance, limited post-training analysis, and non-exhaustive comparisons. These constraints bound the interpretation of the model’s generality and efficiency.

  • Flow-based waveform decoding improves reconstruction quality for sound and music but increases inference latency through multi-step decoding.
  • Unseen-task coverage remains incomplete, depends on training data, and currently excludes speech diarization without diarization- or duration-related supervision.
  • Scaling behavior is underexplored because experiments cover only 1B- and 3B-parameter variants.
  • The model performs better on speech-related tasks because sound and music data are relatively limited compared with speech data.
  • The study focuses on pre-training design choices and does not extensively investigate post-training methods such as multi-task SFT or reinforcement learning.
  • The compared-model set is not exhaustive, and the paper does not claim universal outperformance across architectures and specialized systems.
Loading 2602.04683v3…