PaperlayerView exact revision on arXiv

Source-linked AI summary

DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

Jiaming Zhou, Xuxin Cheng, Shiwan Zhao, Yuhang Jia, Cao Liu, Ke Zeng, Xunliang Cai, Yong Qin

arXiv:2601.23161v1cs.SDcs.CL

TL;DR

Existing audio language models are predominantly autoregressive, creating data, computation, and sequential-decoding challenges, while prior diffusion audio modeling remained limited in practical scale and training. DIFFA-2 addresses this gap with upgraded audio representations, dual adapters, a four-stage training curriculum, and practical diffusion decoding. Across MMSU, MMAU, and MMAR, it consistently improves over DIFFA and remains competitive with strong autoregressive models under practical budgets.

  • Problem

    Predominantly autoregressive LALMs face costly scaling and sequential decoding, while prior diffusion audio models lacked large-scale instruction tuning, preference alignment, and practical decoding schemes.

  • Method

    DIFFA-2 uses semantic and acoustic adapters with a four-stage curriculum combining alignment, supervised fine-tuning, variance-reduced preference optimization, and factor-based parallel decoding.

  • Results

    DIFFA-2 consistently outperforms DIFFA across MMSU, MMAU, and MMAR and is competitive with strong open autoregressive LALMs.

  • Takeaways & Limitations

    The results support diffusion-based modeling as a competitive backbone for general audio understanding under practical training budgets.

  • Takeaways & Limitations

    DIFFA-2 is optimized for fine-grained audio understanding rather than open-domain spoken dialogue and is evaluated only for offline speech-in/text-out interaction.

Abstract

from arXiv · show

Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.

1 Introduction

DIFFA-2 develops a practical diffusion-based audio language model to address the cost and sequential-decoding limits of autoregressive LALMs. It combines scalable training and parallel decoding to make diffusion backbones competitive for audio understanding.

  • 1 Introduction: DIFFA-2 strengthens diffusion-based audio modeling through comprehensive semantic–acoustic alignment and scalable training.The framework is designed to move diffusion audio models beyond proof-of-concept backbones.
  • 1 Introduction: Its four-stage curriculum progressively aligns audio representations, applies large-scale supervised fine-tuning, and uses variance-reduced preference optimization.Factor-based parallel decoding is added at inference to avoid relying on autoregressive decoding.
  • 1 Introduction: DIFFA-2 unifies understanding of speech, sound, and music while using fully open-source data and updating about 1.1% of parameters.The reported training resources include 11,000 hours of ASR data and 3,767 hours of supervised fine-tuning data.

2 Preliminaries

LLaDA replaces left-to-right likelihood factorization with masked-sequence denoising. Its training masks tokens and reconstructs them, while inference iteratively refines low-confidence predictions.

  • 2 Preliminaries: LLaDA uses a discrete random masking process and trains a mask predictor to approximate reverse denoising.This avoids autoregressive factorization while leveraging bidirectional context.
  • 2 Preliminaries: The pre-training objective reconstructs original tokens at masked positions after independently replacing tokens with a mask symbol.The masking probability is t ∈(0, 1], and the corrupted sequence is denoted x_t.
  • 2 Preliminaries: The resulting objective provides a tractable upper bound on negative log-likelihood while avoiding autoregressive factorization.L denotes the sequence length.
  • 2 Preliminaries: During supervised fine-tuning, only the response is randomly masked, while the prompt remains fully observed.The model learns from prompt–response pairs under the same corruption–reconstruction principle.
  • 2 Preliminaries: At inference, LLaDA starts from a fully masked response and repeatedly remasks low-confidence tokens to refine the sequence.The denoising process runs for a fixed number of steps before producing the final output.

3 Methods

DIFFA-2 combines a dual-adapter audio interface with a four-stage curriculum and diffusion-based training and inference. Its preference optimization and parallel decoding target faithful audio understanding and practical deployment.

  • 3.1 Model Architecture: DIFFA-2 uses a frozen Whisper-Large-V3 encoder, dual semantic–acoustic adapters, and a diffusion language-model backbone.The semantic adapter aligns temporally aggregated audio with textual semantics, while the acoustic adapter captures paralinguistic and non-linguistic cues.
  • 3.3 Training Curriculum: The four-stage curriculum progresses from ASR semantic alignment to joint semantic–acoustic alignment, LoRA-based backbone tuning, and VRPO preference optimization.The stages use transcription data, audio-centric supervised data, and chosen–rejected preference pairs.
  • 3.4 Supervised Fine-Tuning: DIFFA-2 masks only response tokens during training while keeping audio embeddings and prompt tokens visible for multimodal masked reconstruction.The objective reconstructs masked response tokens from audio and textual context, allowing bidirectional context use.
  • 3.5 Variance-Reduced Preference Optimization: VRPO reduces variance in diffusion-model preference optimization through Monte Carlo ELBO estimates, optimal budget allocation, antithetic sampling, and shared masking patterns.Preferred and rejected responses are compared using DPO-style log-ratio optimization.
  • 3.6 Inference Procedure: Inference begins with a fully masked response and iteratively denoises it, using left-to-right blocks with parallel within-block prediction and confidence-based re-masking.Factor-based parallel decoding increases parallelism for confident tokens and reduces it in uncertain regions.

4 Experimental Setup

The experiments use open-source training components and compare DIFFA-2 with proprietary and open-source audio language models. Evaluation centers on three audio understanding benchmarks, with VoiceBench as an auxiliary dialogue test.

  • Training Configuration: DIFFA-2 uses about 11,000 hours of ASR data, 3,767 hours of supervised fine-tuning data, and roughly 3,000 preference pairs.The model uses LLaDA-8B-Instruct and updates lightweight adapters and LoRA parameters, totaling roughly 1.1% trainable parameters.
  • Compared Models: Baselines include GPT-4o-Audio and Gemini 2.0 Flash, alongside Qwen3-Omni, Qwen2.5-Omni, Kimi-Audio, and first-generation DIFFA.The comparison covers both proprietary and open-source audio language models.
  • Evaluation Benchmarks: MMSU, MMAU, and MMAR form the primary evaluation benchmarks, while VoiceBench provides an auxiliary assessment of semantic dialogue ability.The primary benchmarks target audio understanding; VoiceBench is not part of the main evaluation focus.

5 Experiments

Across MMSU, MMAU, and MMAR, DIFFA-2 consistently improves over DIFFA and is competitive with strong open autoregressive audio models. Ablations show stronger holistic audio understanding, while transcription retains a small autoregressive advantage and dialogue remains behind heavily tuned omni models.

  • MMSU: 60.45 overall accuracy makes DIFFA-2 the strongest comparable-size open model on MMSU, ahead of Kimi-Audio at 59.28 and Qwen2.5-Omni at 59.09.It also leads open models in perception average at 45.58 and reasoning average at 76.40.
  • MMAU: 69.60 and 67.00 average accuracy on MMAU Test-mini and Test make DIFFA-2 the best open model on both splits.It is particularly strong on sound and speech, while music remains competitive without music-specialized design.
  • MMAR: 50.80% average accuracy on MMAR improves over DIFFA’s 37.20% by 13.6 points, but three-way sound–music–speech mixtures remain difficult.DIFFA-2 outperforms several 8B open baselines and narrows the gap to Qwen2.5-Omni on single-modality tasks.
  • VoiceBench: DIFFA-2 improves over DIFFA and several open baselines on VoiceBench but trails heavily instruction-tuned omni models.This matches the stated design focus on audio understanding rather than extensive conversational tuning.
  • Ablation Study: Stage 4 VRPO yields DIFFA-2’s best overall ablation performance, while AR decoding retains a small WER advantage for token-level ASR.For holistic audio QA, DIFFA-2 performs more strongly under the same data and multi-stage training, although backbone pre-training effects are not fully disentangled.

6 Related Work

Prior audio language models mainly use autoregressive backbones, while diffusion language models offer bidirectional context and parallel token updates. DIFFA-2 extends diffusion modeling to audio with audio-specific encoders, curricula, and inference methods absent from text-focused predecessors.

  • Autoregressive Audio Language Models: Recent LALMs predominantly couple speech encoders with autoregressive LLMs, with omni models additionally supporting streaming and multimodality.Other approaches tokenize audio into discrete sequences, while Kimi-Audio combines discrete and continuous representations.
  • Diffusion Language Models: Diffusion language models iteratively denoise corrupted tokens, enabling bidirectional context modeling and parallel token prediction.LLaDA established large-scale diffusion language modeling for understanding and reasoning tasks.
  • Efficient Diffusion Inference: Inference work improves diffusion efficiency through confidence-aware parallel decoding, KV-cache-like reuse, and adaptive length prediction.LLaDA and related variants focus on text generation and do not address audio encoders or audio-specific training curricula.

7 Conclusions

DIFFA-2 is an enhanced diffusion-based audio language model that achieves competitive performance with leading autoregressive models using modest open-source training resources. The results position diffusion modeling as a competitive alternative for universal audio understanding.

  • Conclusion: DIFFA-2 uses only 1.1% trainable parameters, 99M parameters, and 14.8k hours of open-source data while substantially improving over its predecessor.Evaluations on MMSU, MMAU, and MMAR demonstrate competitiveness with leading autoregressive models.
  • Conclusion: The benchmark results establish dLLM-based modeling as a highly competitive alternative for universal audio understanding tasks.The conclusion specifically supports this claim across MMSU, MMAU, and MMAR.

Limitations

DIFFA-2 remains primarily optimized for fine-grained, offline text-based audio understanding rather than open-domain spoken dialogue or speech-to-speech interaction.

  • Limited conversational and alignment-style supervision constrains DIFFA-2’s spoken-dialogue performance relative to heavily instruction-tuned AR omnimodels.The authors identify balanced training for audio understanding and spoken dialogue as future work.
  • DIFFA-2 supports offline speech-in/text-out evaluation but does not address speech generation, streaming, or full-duplex interaction.The authors propose end-to-end speech-in/speech-out integration and user-centric latency and interaction metrics as next steps.
  • Factor-based parallel decoding reduces diffusion steps with negligible accuracy loss, but DIFFA-2 is not uniformly faster than strong AR audio LLMs.

A Data Details and Prompt Templates

The appendix documents broad, open-source data construction spanning ASR, caption-grounded and direct audio QA, preference data, benchmark coverage, and baseline evaluation setup.

  • A.1 ASR Data: LibriSpeech and GigaSpeech transcripts are converted into instruction-style ASR samples with 25 Qwen-generated templates.Examples include requests to transcribe audio or state its exact content.
  • A.2 SFT Data: Stage 2 caption-grounded AQA combines speech, environmental-sound, music, and paralinguistic datasets to create diverse audio-question-answer examples.
  • A.2 SFT Data: Direct audio QA includes simple, complex, and empathetic categories, using text QA sources synthesized into speech with CosyVoice2.
  • A.2 SFT Data: Five percent of Stage-1 ASR data is retained in Stage 2 to preserve ASR-related acoustic grounding.
  • A.2 SFT Data: AudioMCQ supplies multi-choice AQA across speech, music, and environmental sounds and is combined with Stage-2 adapter-training data without chain-of-thought.
  • A.3 Preference Data and Prompts: Preference data begins with high-quality SFT audio QA and uses language-model-generated fluent answers containing subtle audio-related errors as rejected responses.
  • A.3 Preference Data and Prompts: Dataset statistics are organized by Sound, Music, and Speech domains, with duration defined as unique audio hours.
  • B Additional Experimental Details: The appendix provides prompt examples for Stage-2 audio QA, preference-data generation, and benchmark evaluation, alongside the complete baseline-model list.

B.3 Training Details

The appendix specifies DIFFA-2’s multi-stage training and iterative diffusion inference, including semi-autoregressive and confidence-adaptive parallel decoding configurations.

  • Training Configuration: Training uses four stages with progressively reduced learning rates, LoRA in Stage 3, preference optimization in Stage 4, and approximately five days of computation.The first three stages use 64 NVIDIA A100 GPUs, while the final stage uses 4.
  • Training Configuration: The evaluation appendix lists baseline models and stage-wise training configurations used to contextualize DIFFA-2’s results.
  • Diffusion Inference: Inference initializes a fully masked response and iteratively denoises it over T steps conditioned on padded audio and prompt inputs.
  • Diffusion Inference: Low-confidence tokens are selectively re-masked during denoising, enabling iterative refinement with full bidirectional context.
  • Diffusion Inference: Semi-autoregressive decoding generates response blocks left-to-right while decoding tokens in each block in parallel.
  • Factor-Based Parallel Decoding: Factor-based parallel decoding adaptively determines the number of parallel token updates from confidence estimates rather than a fixed threshold.
  • Factor-Based Parallel Decoding: Confidence-adaptive parallel decoding becomes more aggressive in confident regions and more conservative in uncertain regions.
  • Factor-Based Parallel Decoding: The factor-based strategy improves inference efficiency while preserving generation quality, with benchmark-specific hyperparameters reported in Table B.4.Inference experiments use one NVIDIA A100 GPU, and factor-based decoding sets f to 1.0.

C Additional Experiments

DIFFA-2 performs strongly on audio-understanding benchmarks but is weaker on spoken-dialogue evaluation, reflecting its training emphasis on fine-grained audio understanding.

  • DIFFA-2 scores 59.63 on VoiceBench, below heavily instruction-tuned AR omnimodels but above or competitive with several open-source baselines.It improves markedly over DIFFA’s 48.22 across most VoiceBench metrics.
Loading 2601.23161v1…