UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization

• Current audio language models lag text models in generalization, scalability, and task versatility because of representation, architecture, and data challenges.Source: §1 · p. 1
• Continuous representations support understanding but are difficult to generate autoregressively, whereas discrete codecs scale generation but lack text-aligned high-level abstractions.Source: §1 · p. 1
• Auditory sentences organize related audio and text segments into long-context sequences that induce multiple understanding and generation tasks.Source: §1 · p. 2
• ReasoningCodec factorizes audio into text-aligned reasoning tokens and reconstruction tokens that preserve high-fidelity waveform information.Source: §1 · p. 2
• Functional layer specialization assigns transformer layers to audio understanding, cross-modal alignment, and audio generation experts.Source: §1 · p. 2
• 100B text tokens and 60B audio tokens support competitive seen-task performance and strong few-shot and zero-shot generalization on unseen tasks.Source: §1 · pp. 2–3

• Audio tokenization determines the intermediate representation used by both audio understanding and generation systems.Source: §2.2 · p. 3

Continuous representations

• Continuous audio features preserve rich perceptual information and typically perform strongly on understanding tasks but are difficult to generate directly.Source: §2.2 · p. 3

Discrete tokens

• Discrete acoustic tokenizers support scalable generation but mainly encode low-level details and provide limited semantic abstraction for understanding.Source: §2.2 · p. 3

Towards unified representations

• ReasoningCodec factorizes audio into reasoning and reconstruction tokens to retain language-aligned information while supporting faithful reconstruction and autoregressive generation.Source: §2.2 · p. 4

• ReasoningCodec targets an audio representation that is both language-model-friendly for autoregression and information-preserving for understanding.Source: §3 · p. 4
• Reasoning tokens encode text-aligned high-level perceptual analyses, while reconstruction tokens capture semantic content and fine-grained acoustics.Source: §3 · p. 4
• Reasoning tokens intentionally capture only text-level semantics and therefore contain less information than conventional semantic tokens.Source: §3 · p. 4
• The codec produces reasoning tokens r = Tr(x) and reconstruction tokens s = Ts(x | r), with waveform reconstruction performed from D(s).Source: §3 · p. 4

Reasoning branch

• The reasoning branch fuses frozen Whisper and music encoders, uses a query-based transformer, and decodes through a pretrained text LLM.Source: §3.1 · p. 4
• Reasoning training combines supervised audio-understanding fine-tuning with GRPO-style reinforcement learning for detailed, grounded analyses.Source: §3.1 · p. 5

Reconstruction branch

• The reconstruction branch uses separate quantization resources for phone semantics, music structure, environmental sound semantics, and remaining acoustic information.Source: §3.2 · p. 5
• FiLM conditions reconstruction semantic features on quantized reasoning features, while a flow-based decoder is trained with reconstruction and semantic feature losses.Source: §3.2 · p. 5

Representation

• ReasoningCodec separates audio into reasoning tokens for understanding and reconstruction tokens for high-fidelity generation.The tokenizer provides semantic abstraction while preserving fine-grained acoustic information.Source: §4 · p. 5, §4.5 · p. 7

Unified representation

• UniAudio 2.0 uses a unified vocabulary and multi-stream representation for audio and text within one autoregressive backbone.Audio is represented with reasoning and reconstruction streams, while text uses a single stream.Source: §4.1 · p. 5, §4.2 · p. 6

Architecture

• The architecture specializes transformer layers into audio understanding, cross-modal, and audio generation experts.Audio-specific experts operate on audio positions, while cross-modal experts retain initialization from the pretrained text language model.Source: §4.3 · p. 6, §4.5 · p. 7

Autoregressive modeling

• Training combines text and audio autoregressive objectives, with modality-specific prediction heads and balanced loss contributions.Text tokens are predicted at token level, whereas audio frames use a lightweight local decoder to predict parallel audio tokens.Source: §4.3 · p. 6

Training

• UniAudio 2.0 is trained through four stages that progressively integrate audio understanding, audio generation, and audio–text capabilities.The stages include understanding warm-up, generation warm-up, audio–text pre-training, and audio–text mid-training.Source: §4.4 · pp. 6–7

Capabilities

• UniAudio 2.0 achieves competitive performance on seen tasks and strong few-shot and zero-shot generalization across unseen tasks and domains.The reported capabilities include more robust complex understanding, more controllable generation, and improved adaptability to unseen scenarios.Source: §4.5 · p. 7

Audio Tokenizer Evaluation

• ReasoningCodec achieves consistently strong reconstruction quality across speech, sound, and music modalities.It is evaluated against universal audio codecs, with subjective MUSHRA results also reported.Source: §5.3 · p. 10
• ReasoningCodec improves token modeling, with reasoning tokens easier for language models to capture and improving reconstruction-token prediction accuracy.Combining reasoning and reconstruction tokens also reduces reconstruction-token modeling difficulty.Source: §5.3 · p. 10
• ReasoningCodec achieves the best understanding performance among discrete audio tokenizers and approaches continuous Whisper-tokenizer performance on multiple benchmarks.The evaluation covers ASR, emotion recognition, audio classification, and music classification.Source: §5.2 · p. 8, §5.3 · p. 10

UniAudio 2.0 Evaluation

• UniAudio 2.0 performs strongly across speech, general-audio, and music benchmarks, including competitive seen-task results and multilingual speech capabilities.The model uses 3B parameters and is reported to outperform MiMo-Audio 7B on ASR and TTS while outperforming Qwen2.5-Omni 7B on audio captioning.Source: §5.4 · p. 11
• UniAudio 2.0 shows strong few-shot generalization on most evaluated benchmarks, especially in the 1-shot setting.Few-shot evaluations include speech denoising, voice conversion, sound classification, and emotion classification.Source: §5.5 · p. 11

Ablation Study

• Removing reasoning tokens, mid-training, or functional layer specialization degrades performance, while model-size reductions weaken multi-task and few-shot or zero-shot generalization.The ablations link mid-training to text capability and generalization, and identify model size as important for model capacity.Source: §5.7 · pp. 11–12

• ReasoningCodec factorizes audio into reasoning and reconstruction tokens, while UniAudio 2.0 combines unified autoregressive modeling with multi-stage, multi-task training.Source: §6 · p. 12
• Extensive experiments demonstrate strong and consistent performance across seen speech, sound, and music tasks, with promising few-shot and zero-shot generalization to unseen scenarios.Source: §6 · p. 12
• Scaling data, task diversity, and model capacity is identified as important for improving generalization.Source: §6 · p. 12

• Audio generation and voice conversion may be misused for impersonation, fraud, harassment, or unauthorized content.Source: §7 · p. 12
• ReasoningCodec training uses supervised fine-tuning, GRPO-style reinforcement learning, and reconstruction-branch training with a flow-based decoder.Source: §A.2 · p. 18
• FiLM connects reasoning and reconstruction branches effectively without influencing reconstruction performance.Source: §A.3.3 · p. 20
• Subjective MUSHRA evaluation reports the best reconstruction performance for speech, sound, and music among the compared tokenizers.Source: §A.3.4 · p. 20

• Reasoning tokens consistently improve understanding performance and reduce reconstruction-token modeling difficulty, including lower perplexity.Source: §A.3.5 · p. 21
• As a language-aligned bottleneck, reasoning tokens retain language- and reasoning-relevant content while filtering task-irrelevant acoustic details.Source: §A.3.5 · p. 21
• For understanding, reasoning tokens provide a compact semantic abstraction aligned with the text language model for ASR, classification, and reasoning evaluations.Source: §A.3.5 · p. 21
• For generation, reasoning tokens summarize high-level intent and semantics, reducing long-range uncertainty and stabilizing autoregressive decoding.Source: §A.3.5 · p. 21
• Conditioning reconstruction tokens on reasoning tokens lowers conditional entropy by the amount of their conditional mutual information.Source: §A.3.5 · p. 22
• Reasoning-token designs outperform the CosyVoice 3 tokenizer on ASR and emotion recognition, while ReasoningCodec achieves the best subjective reconstruction performance across speech, sound, and music.Source: §A.3.6 · p. 23

• The four-stage training process begins with audio understanding warm-up and audio generation warm-up before joint audio-text pre-training and mid-training.Source: §B.3 · pp. 26–28

Stage 1: Audio understanding warm-up

• Understanding warm-up trains audio understanding experts while freezing other components and adds semantic distillation from frozen speech and music encoders.Source: §B.3 · pp. 26–27

Stage 2: Audio generation warm-up

• Generation warm-up updates the audio generation expert and local decoder while keeping understanding and cross-modal experts fixed.Source: §B.3 · p. 27

Stage 3: Audio-text pre-training

• Audio-text pre-training jointly updates model parameters across understanding, generation, text-only, and audio-only tasks under a unified autoregressive objective.Source: §B.3 · p. 28

Stage 4: Audio-text mid-training

• Audio-text mid-training uses auditory sentences and a maximum context length of 2048 tokens to model longer, more complex sequences and improve robustness across task settings.Source: §B.3 · p. 28

• ASR evaluation uses LibriSpeech, SEED-TTS, and WSYue benchmarks, with WER for English and CER for Chinese and Cantonese.Source: §B.4.1 · p. 28
• TTS evaluation uses LibriSpeech, SEED-TTS, and WSYue benchmarks, combining speech-recognition scores with DNS-MOS quality assessment.Source: §B.4.1 · p. 28
• Instruct TTS is evaluated with WER, style accuracy, and UTMOSv2, including Chinese evaluation despite no Chinese Instruct TTS training.Source: §B.4.1 · p. 28
• Audio and music captioning use CIDER and GPT-score on their respective captioning benchmarks.Source: §B.4.1 · p. 29
• Audio and music generation use FD, KL, and CLAP-based metrics, while song generation uses WER and AudioBox Score.Source: §B.4.1 · p. 29
• Lyric recognition is evaluated using the benchmark from SongGen.Source: §B.4.1 · p. 29

• Few-shot speech denoising uses LibriTTS-test-clean mixed with WHAM noise and evaluates PESQ, STOI, WER, and DNS-MOS.Source: §B.4.2 · p. 29
• Few-shot voice conversion is evaluated on VCTK using WER, Speaker Similarity (SIM), and DNS-MOS.Source: §B.4.2 · p. 29
• Few-shot emotion classification uses English and Chinese splits from ESD to construct its evaluation set.Source: §B.4.2 · p. 29
• Few-shot sound-event classification uses the TUT acoustic scenes 2017 dataset.Source: §B.4.2 · p. 29

• Zero-shot tasks are never seen during training and provide no demonstrations at inference time.Source: §B.5 · p. 29
• Text understanding uses MMLU under a direct zero-shot evaluation setting.Source: §B.5 · p. 29
• Speech-to-speech/text question answering is treated as zero-shot because speech conversation data are excluded from training, and is evaluated on InstructS2S-Eval with GPT-score.Source: §B.5 · p. 29
• Dysarthric speech recognition is evaluated zero-shot because dysarthric speech is absent from training data.Source: §B.5 · p. 29
• The section introduces audio-prompt-and-caption-guided TTS, which combines timbre conditioning with speaking-style control and measures similarity, style accuracy, WER, and UTMOSv2.Source: §B.5 · p. 29
• Speech–sound generation asks the model to generate speech and a corresponding sound event from sound-event tags and speech content, evaluated with WER, CLAP score, and UTMOSv2.Source: §B.5 · p. 30

• Flow-based waveform decoding improves reconstruction quality for sound and music but increases inference latency through multi-step decoding.Source: C. Limitation · p. 31
• Unseen-task coverage remains incomplete, depends on training data, and currently excludes speech diarization without diarization- or duration-related supervision.Source: C. Limitation · p. 31
• Scaling behavior is underexplored because experiments cover only 1B- and 3B-parameter variants.Source: C. Limitation · p. 31
• The model performs better on speech-related tasks because sound and music data are relatively limited compared with speech data.Source: C. Limitation · p. 31
• The study focuses on pre-training design choices and does not extensively investigate post-training methods such as multi-task SFT or reinforcement learning.Source: C. Limitation · p. 32
• The compared-model set is not exhaustive, and the paper does not claim universal outperformance across architectures and specialized systems.Source: C. Limitation · p. 32