PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Audio Intelligence Without Regressing on Text Intelligence

Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

arXiv:2607.05196v2cs.CLcs.AIcs.LGcs.SDeess.AS

TL;DR

Combining audio understanding and generation while preserving strong text reasoning remains challenging. Audex addresses this with a unified audio-text decoder and achieves strong audio capabilities with marginal or no regression in text intelligence.

  • Problem

    Combining audio understanding and generation while preserving strong text reasoning remains an open challenge because the tasks require different kinds of representations.

  • Method

    Audex uses a single Transformer decoder that projects encoded audio into text-embedding space and treats text and quantized audio tokens uniformly during generation.

  • Results

    Audex achieves state-of-the-art speech recognition and strong audio understanding, generation, translation, and speech-to-speech performance while preserving text benchmark performance with marginal or no regression.

  • Takeaways & Limitations

    Audex demonstrates that one unified model can combine broad audio capabilities with compelling reasoning, knowledge, long-context, and agentic text capabilities.

  • Takeaways & Limitations

    Audio generation may remain limited by duration and consistency issues, partly because much training data uses fixed 10-second examples.

Abstract

from arXiv · show

Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.

1. Introduction

Audio intelligence is presented as essential to AGI, but multimodal understanding and generation can regress text intelligence. Audex addresses this challenge as a unified audio-text model built on a strong text-only backbone, targeting broad audio capabilities while preserving text intelligence.

  • Motivation: Audio intelligence spans understanding, reasoning about, and generating audio and speech, making it an indispensable modality for AGI.The introduction situates audio across speech, music, and environmental sound, including perception, communication, emotional expression, and appreciation.
  • Challenge: Multimodal LLMs often exhibit noticeable regressions on text benchmarks, threatening reasoning, knowledge, and agentic capabilities encoded in text.The introduction identifies this regression as especially challenging for models with multimodal output capabilities.
  • Challenge: Building native multimodal LLMs without compromising text intelligence is framed as a critical step toward AGI, while open frontier models remain a contrasting context.The introduction contrasts this challenge with efforts integrating audio and vision into frontier proprietary LLMs such as GPT-4o and Gemini 2.5.
  • Contribution: Audex is built on Nemotron-Cascade-2-30B-A3B, a text-only 30B Mamba2-Transformer hybrid MoE LLM with 3B activated parameters.The model is described as a strong general-purpose text-only backbone.
  • Architecture: Audex uses a single unified MoE Transformer decoder that projects encoded audio into the text embedding space and treats text and quantized audio tokens uniformly.This design supports instruct and thinking modes across text-only and audio-to-text tasks, unlike cascaded or thinker–talker systems.

2. Main Results

Audex is evaluated against its text-only backbone on both text capabilities and a broad range of audio tasks. It retains highly comparable text-only accuracy while delivering strong audio understanding and state-of-the-art speech recognition.

  • Evaluation scope: Audex is evaluated on text-only reasoning, knowledge, alignment, long-context, and agentic capabilities, alongside audio understanding, recognition, translation, synthesis, generation, and speech-to-speech generation.The main results appear in Table 1, with detailed results in Section 5 and Appendix A.
  • Text-only capabilities: Audex achieves accuracies highly comparable to Nemotron-Cascade-2 on text-only benchmarks, with slightly higher accuracies on IMO AnswerBench, AIME 2026, and HMMT Feb2025 using tool-integrated reasoning.These results indicate minimal text-only regression relative to its backbone.
  • Text-only capabilities: Qwen3-Omni-30B-A3B-Thinking shows substantial degradation on reasoning benchmarks relative to its text-only backbone, while Qwen3.5-Omni-Flash performs substantially worse than its LLM counterpart.The cited comparison references noticeable reasoning regressions for Qwen3-Omni in its technical report.
  • Audio capabilities: On audio understanding, Audex performs comparably to Step-Audio-R1.1-33B and Qwen3-Omni-30B-A3B-Thinking, lagging only behind proprietary Qwen3.5-Omni-Flash.The comparison covers the strongest open models named in the passage.
  • Audio capabilities: On speech recognition, Audex achieves state-of-the-art performance, outperforming Step-Audio-R1.1-33B and Qwen3-Omni-30B-A3B-Thinking while performing comparably to Qwen3.5-Omni-Flash.This result places Audex ahead of the named open-model baselines and comparable to the proprietary model.

3. Model Architecture

Audex uses Nemotron-Cascade-2-30B-A3B as its MoE backbone and integrates audio through an AF-Whisper encoder plus codec tokens predicted alongside text tokens. Separate speech and non-speech codecs support specialized audio generation and decoding strategies.

  • Backbone: Audex-30B-A3B uses a 52-layer hybrid Mamba-Transformer MoE backbone with dimension 2688, 128 routable experts, and 6 activated experts.The backbone is initialized from the post-SFT Nemotron-Cascade-2-30B-A3B checkpoint.
  • Audio encoder: AF-Whisper encodes 16kHz audio into 1280-dimensional features at 25Hz, producing length 750 for each 30-second window.AF-Whisper shares Whisper Large-v3’s architecture and extends audio understanding beyond speech recognition.
  • Unified token modeling: Audex expands the text vocabulary with discrete audio codec indices and predicts audio and text tokens using the same cross-entropy loss.This directly supports audio generation within the language-modeling objective.
  • Audio codecs: Speech uses X-Codec2 at 50 token/s with a 65536-entry codebook, while non-speech audio uses X-Codec at 50Hz with the first 4 layers of an 8-layer RVQ.Separate codec vocabularies enable specialized decoding strategies for speech and non-speech sound.
  • Audio decoding: Non-speech decoding uses an enhancement VAE that raises sampling to 48kHz, while a causal ConvNeXt streaming decoder supports faster X-Codec2 inference.The streaming decoder uses a small look-ahead context window to maintain semantic accuracy and fast inference.

4. Model Training

Audex is trained by combining text, speech, and general-audio datasets with multi-stage supervised training, followed by text-only reinforcement learning and distillation. The multi-stage strategy preserves long-context capability better than consolidated single-stage SFT, while text-only RL yields marginal or no regression on audio and speech tasks.

  • Training Data: Audex training combines text, speech, and general non-speech audio data spanning ASR, AST, TTS, audio understanding, and text-to-audio generation.Text data cover reasoning, math, coding, alignment, long-context, agentic, and multilingual tasks, while audio data include speech and general sound.
  • Multi-stage SFT: The multi-stage SFT pipeline starts from the Nemotron-Cascade 2 SFT checkpoint, then adds audio and speech tokens, an audio encoder, and MLP adapters during audio warmup.During warmup, only the MLP adapters and audio token embeddings are trained, while text token embeddings are frozen.
  • SFT Comparison: Single-stage and multi-stage SFT achieve comparable overall performance, but single-stage SFT fails almost completely on NIAH long-context tasks.The authors attribute this to simultaneous learning of long text and long audio samples, which drastically changes the attention mechanism.
  • SFT Comparison: Although single-stage SFT uses a lower total training budget, Audex favors multi-stage SFT because it preserves long-context attention and is easier to tune and stabilize.The multi-stage strategy requires more training budget when epochs across stages are summed.
  • Post-training: After supervised training, Audex applies text-only Cascade RL, multi-domain on-policy distillation, RLHF, and long-context RL, improving text tasks without materially harming audio or speech tasks.The reported audio- and speech-related regression after text-only RL is marginal or absent.

5. Detailed Results and Analysis

Audex delivers strong results across text, speech, and audio benchmarks while largely preserving the capabilities of its text-only backbone. Its strongest reported areas include reasoning and alignment, public-data text-to-audio generation, multilingual speech tasks, and speech interaction, with specific gaps and fixed-duration limitations in some settings.

  • Text Benchmark Results: Audex-30B-A3B achieves best-in-class reasoning and alignment results, while remaining comparable to leading baselines on knowledge, long-context, and agentic benchmarks.Performance on knowledge, long-context, and agentic tasks is sometimes higher and sometimes lower than the strongest baselines.
  • Text-to-Audio Results: 60.9 on AudioCaps and 58.1 on SongDescriber make the Audex-30B-A3B SFT checkpoint state of the art among models trained on public data.After text RL, the scores experience a marginal drop (+6.0/4.6, respectively), described as negligible for FDopenl3.
  • Speech and Audio Generation: For TTA, CFG values λ∈[3, 5] provide the best overall quality, whereas TTS achieves its best WER near λ≈1.5.The experiments select λ=3 for TTA and λ=1.5 for TTS; CFG is required for good TTA quality but is not necessary for good TTS quality.
  • ASR and AST Results: Audex matches strong audio-language-model baselines and outperforms several ASR-specific baselines, especially for noisy English ASR, while also showing strong multilingual ASR and AST abilities.Multilingual evaluation covers German, Spanish, French, Italian, Portuguese, Russian, and Korean, using WER for ASR and BLEU and COMET for AST.
  • Audio Understanding and Speech Interaction: Audex matches strong MMAU baselines, has gaps on MMAR and MMSU, and shows strong audio-entailment and CMM-hallucination results indicating better trustworthiness.In speech interaction, Audex achieves a score of 90 and is competitive in instruction following, reasoning, multiple-choice QA, and safety.

6. Related Work · Appendix

The related work frames unified audio-text modeling as difficult because understanding and generation impose different requirements, while preserving text reasoning remains unresolved. It situates Audex among unified multimodal, speech, audio-understanding, and audio-generation models, emphasizing its systematic improvements over UALM and scaling to a 30B MoE backbone.

  • 6.1. Unified Understanding and Generation Models: Unified models must reconcile fine-grained generation with succinct semantic understanding while avoiding regressions in text reasoning.The literature reports that optimizing multimodal capabilities simultaneously can improve some tasks while degrading others.
  • 6.1. Unified Understanding and Generation Models: Prior work finds that early multimodal fusion and MoE architectures help unified modeling, whereas diffusion heads provide an orthogonal generation strategy.These approaches are presented as complementary design choices rather than a single unified solution.
  • 6.1. Unified Understanding and Generation Models: UALM-7B achieved state-of-the-art text-to-audio and audio understanding at its size, but its text evaluation used saturated benchmarks and omitted text RL.The omitted text RL potentially limited performance on complex multi-step reasoning tasks.
  • 6.1. Unified Understanding and Generation Models: Audex extends UALM by improving text-to-audio and audio understanding, adding speech generation and recognition, and strengthening text reasoning through text SFT+RL.Audex is also reported as comparable with Nemotron-Cascade 2 across a wide range of text reasoning tasks.
  • 6.2. Speech LLMs: Speech LLMs cover ASR, AST, speech understanding, TTS, S2S, and combinations thereof, typically producing discrete speech tokens from codec models.Codec models encode waveforms into discrete tokens and reconstruct waveforms from them.
  • 6.2. Speech LLMs: Speech LLM inputs may use discrete codec tokens, continuous audio representations, or hybrid formats, trading recognition quality against input-output space mismatch.Duplex models add interaction channels for simultaneous speaking and listening, which is outside this paper’s scope.
  • 6.3. Audio Understanding Models: Audio understanding models extend speech LLMs to non-speech sound and music, supporting captioning, question-answering, classification, and sometimes speech outputs.Most use continuous representations from one or more task-specific audio encoders.
  • 6.4. Audio Generation Models: Text-to-audio models generate semantically consistent audio clips, with diffusion models recently leading on sample quality, controllability, and duration.UALM improved LLM-based audio generation using data scaling, classifier-free guidance, and post-training, which Audex scales to a 30B MoE backbone.

A. Benchmarks, Evaluation Setups, and Baselines … A.4. Long Context

The appendix standardizes evaluation across text, audio, reasoning, knowledge, alignment, and long-context tasks, reusing established setups and specifying task-specific datasets, inference budgets, and reporting protocols. Long-context evaluation combines multi-document reasoning over approximately 100k-token inputs with retrieval-style tests at a 1M-token context length.

  • A. Benchmarks, Evaluation Setups, and Baselines: Text benchmarks reuse the Nemotron-Cascade 2 evaluation setups, while audio understanding follows Audio Flamingo 3 and text-to-audio evaluation follows ETTA.These protocols are documented in Appendix A.1–A.5 for text and A.6–A.10 for audio tasks.
  • A.1. Reasoning: Reasoning evaluation covers AIME 2025, AIME 2026, HMMT February 2025, IMO-AnswerBench, and LiveCodeBench.The benchmark suite spans competition mathematics, altered Olympiad problems, and diverse algorithmic coding problems.
  • A.1. Reasoning: 131K tokens is the thinking budget for Audex and Nemotron-Cascade-2 on most reasoning tasks, with up to 100 tool calls in the with-tool setting.Sampling uses temperature 1.0 and top-p 1.0; IMO-AnswerBench instead uses a 256K-token budget.
  • A.1. Reasoning: Baseline reasoning results use official reported numbers when available; otherwise, models are evaluated with recommended settings and at least a 128K-token thinking budget.This establishes a consistent fallback protocol for baseline comparisons.
  • A.2. Knowledge: Knowledge evaluation uses MMLU-Redux, MMLU-Pro, and GPQA-Diamond, reporting exact-match or pass@1 accuracy under their specified generation protocols.MMLU-Redux uses one generation per question, while GPQA-Diamond averages eight generations per question.
  • A.2. Knowledge: Knowledge tasks use thinking mode with temperature 1.0, top-p 0.95, and a 128K-token maximum response length.These settings apply to Audex and Nemotron-Cascade-2 on MMLU-Redux, MMLU-Pro, and GPQA-Diamond.
  • A.3. Alignment: Alignment evaluation includes IFBench and ArenaHard, using thinking mode with temperature 0.6, top-p 0.95, and a 32K-token maximum response length.ArenaHard contains 750 prompts spanning software engineering, mathematics, and creative writing, with automated LLM-as-Judge evaluation.
  • A.4. Long Context: Long-context evaluation tests AA-LCR over approximately 100k-token document sets and four needle-in-a-haystack categories at a 1M-token context length.AA-LCR contains 100 multi-document reasoning questions; each NIAH category uses 100 instances, reasoning-off mode, and single-generation pass@1 reporting.

A.5. Agentic Tasks … A.10. Text-to-Audio

The appendix specifies evaluation protocols for agentic, speech, audio-understanding, and text-to-audio tasks, including datasets, prompting, interaction-state retention, and generation metrics. These protocols cover both specialized benchmarks and unified Audex-based pipelines.

  • A.5. Agentic Tasks: τ2-Bench evaluates multi-turn customer-service agents across airline, retail, and telecom subsets, using avg@16 for airline and avg@8 for retail and telecom.The subsets contain 50, 114, and 114 examples, respectively.
  • A.5. Agentic Tasks: Agentic evaluation retains the latest-turn reasoning for τ2-Bench and complete tool-interaction histories for SWE-bench Verified through the OpenHands scaffold.The τ2-Bench setup avoids earlier-turn reasoning carry-over, while SWE-bench preserves file views, search results, commands, and patches.
  • A.6. Text-to-Speech: Text-to-speech evaluation uses 1000 Common Voice samples from the Seed-TTS-Eval English subset with a transcription-to-speech prompt.The evaluation compares Audex with state-of-the-art TTS models and general-purpose audio language models.
  • A.7. Speech Recognition and Speech-to-Text Translation: ASR and AST evaluation spans open, noisy, multilingual, and FLEURS speech-to-text translation benchmarks, reporting WER, BLEU, and COMET.Noisy ASR mixes LibriSpeech test-clean utterances with noise across SNR levels from −5 to 100 dB.
  • A.7. Speech Recognition and Speech-to-Text Translation: Audex uses language-free prompts for English ASR and AST, adds language IDs for multilingual ASR, and omits source-language prompts for AST.These prompting choices differ between Audex and baseline models in the reported evaluation setups.
  • A.8. Speech Interaction: Speech interaction is evaluated on VoiceBench and BigBenchAudio using one Audex checkpoint sequentially for ASR, text reasoning, and TTS.The pipeline extracts answers from boxed outputs before generating speech for BigBenchAudio.
  • A.9. Audio Understanding: Audio understanding covers MMAU, MMAR, MMSU, and Audio Entailment on Clotho-v2 and AudioCaps using multiple-choice and deductive-reasoning prompts.The benchmark suite targets general sound, speech, music understanding, and deductive reasoning.
  • A.10. Text-to-Audio: Text-to-audio evaluation uses 10-second AudioCaps and SongDescriber clips and reports FDopenl3, where lower values indicate more statistically realistic generated audio.The evaluation follows ETTA’s setup and measures Fréchet Distance in OpenL3 feature space.

B. Detailed Results Across Training Stages

Audex retains strong text capabilities across intermediate supervised-training stages while progressively improving audio generation, TTS, SongDescriber, and audio-understanding abilities. Text-focused RL improves targeted text benchmarks with limited multimodal degradation, and temporary ASR/MMSU losses after MOPD recover in later RL stages.

  • Supervised training stages: Audio Gen. SFT achieves the best AudioCaps results while also producing very good SongDescriber and Seed-TTS-Eval results.AudioCaps and SongDescriber use OpenL3 Fréchet Distance, while Seed-TTS-Eval uses WER; parenthesized values indicate fixed-voice TTS.
  • Supervised training stages: After Audio Gen. + Audio Und. SFT, TTS and SongDescriber improve, alongside strong audio understanding, ASR, and AST abilities.Fleurs (xx→en)4 reports averaged BLEU|COMET AST scores across German, French, Italian, and Spanish.
  • Reinforcement learning: Text benchmarks improve during domain-specific RL stages, while most audio benchmarks remain stable and multimodal abilities avoid major degradation.MOPD is the exception: it degrades ASR and MMSU, but both recover after two additional RL stages.

C. Ablation Studies · C.1. Audio Warmup

The audio warmup ablation compares freezing text embeddings while updating audio embeddings against unfreezing both. Freezing text embeddings preserves text capability while supporting audio generation, whereas unfreezing provides no benefit and creates a large gap.

  • C.1. Audio Warmup: Audex freezes text token embeddings and updates only audio token embeddings during multi-stage SFT audio warmup.This is implemented by gradient masking of text token embeddings while the LLM remains frozen.
  • C.1. Audio Warmup: The alternative unfreezes both text and audio token embeddings and trains across text and audio tasks.The approach was motivated by the hypothesis that both embedding types could be jointly optimized.
  • C.1. Audio Warmup: Unfreezing text token embeddings provides no benefit in any tested situation after audio warmup.The comparison results are reported in Table 16.
  • C.1. Audio Warmup: The performance gap between unfreezing and freezing text embeddings is described as huge.This finding is also summarized by the Audio Warmup comparison in Table 16.
  • C.1. Audio Warmup: Freezing text embeddings keeps the model’s initial text scores high during the subsequent Audio Gen. SFT stage.The text evaluation results after different warmup methods are plotted in Figure 5.
  • C.1. Audio Warmup: Freezing text embeddings causes only small text degradation at the beginning of Audio Gen. SFT while compensating with audio generation performance.The passage frames this as the observed trade-off for the freezing strategy.
  • C.1. Audio Warmup: Figure 5 compares Audio Gen. SFT results between freezing and unfreezing text embeddings during audio warmup.The figure specifically presents the text-embedding warmup comparison.

C.2. Text Data Blending Ratios · C.3. CFG Values and Hyper-parameters for TTS and TTA

The ablations show that preserving text quality requires higher text-data blending ratios during multimodal SFT, especially in consolidated training. Inference tuning identifies task-specific CFG, Top-k, and temperature settings for TTA and TTS.

  • C.2. Text Data Blending Ratios: Text blending ratios are ablated to assess their effect on text quality during SFT training.The study compares ratios in the Audio Gen. + Audio Und. SFT stage and the single-stage consolidated SFT stage.
  • C.2. Text Data Blending Ratios: 0.56 causes considerable degradation on text evaluation compared with 0.69 in the Audio Gen. + Audio Und. SFT stage.The passage attributes this difference potentially to the many audio-text pairs with text outputs in the combined stage.
  • C.2. Text Data Blending Ratios: 0.75 produces significant text-evaluation cliffs around 50%-60% of training in the single-stage consolidated SFT model.Restarting before the cliff with different random seeds led to another cliff around 60%-80% of training.
  • C.2. Text Data Blending Ratios: 0.88 yields mild or no text degradations after restarting from the checkpoint before the cliff, motivating a significantly higher blending ratio for consolidated SFT.The conclusion follows from the observed instability at lower blending ratios.
  • C.3. CFG Values and Hyper-parameters for TTS and TTA: For TTA, the best CFG value is λ∈[3, 5], with λ= 3 used in experiments, alongside Top-k 80 and temperature 1.0.These settings are selected from inference hyper-parameter studies for text-to-audio generation.
  • C.3. CFG Values and Hyper-parameters for TTS and TTA: For TTS, the best CFG value is λ≈1.5, with λ= 1.5 used in experiments, alongside Top-k 80 and temperature 0.1.WER is close to optimal even without CFG.

D. Additional Details for ASR and AST Evaluations

Audex’s ASR and AST evaluations use structured chat-style outputs, with language identification and transcription or translation depending on the task. Baseline comparisons follow public OpenASR results, model-specific inference recipes, and task-specific prompting for speech/audio LLMs.

  • Evaluation format: ASR outputs language identification followed by transcription, while AST outputs source-language identification, source transcription, and English translation.For language-conditioned multilingual ASR, Audex is teacher-forced with the known source language before continuing generation.
  • ASR baselines: ASR baselines comprise Whisper Large v3, Canary-1B-v2, Parakeet-TDT-0.6B-v3, and Canary-Qwen-2.5B using public OpenASR leaderboard results.Noisy and multilingual evaluations reproduce each model’s OpenASR inference recipe, with source-language information supplied for multilingual ASR.
  • Speech/audio LLM baselines: Speech/audio LLM baselines use task-specific prompts, while Qwen3-Omni follows official ASR and AST prompts and scores only final answers for Thinking.Step-Audio R1.1 is evaluated in thinking mode with reasoning-budget control, capping its thinking segment at 13000 tokens.
Loading 2607.05196v2…