PaperlayerView exact revision on arXiv

Source-linked AI summary

DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

Jiaming Zhou, Xuxin Cheng, Shiwan Zhao, Yuhang Jia, Cao Liu, Ke Zeng, Xunliang Cai, Yong Qin

arXiv:2601.23161v1cs.SDcs.CL

TL;DR

Audio language models are predominantly autoregressive, raising whether diffusion models can serve as competitive, practical audio backbones under realistic data and latency budgets. DIFFA-2 uses staged training and semantic–acoustic alignment, and across three benchmarks it outperforms DIFFA while often matching or surpassing similar-sized open autoregressive models.

  • Problem

    Whether diffusion language models can become competitive, practical audio backbones under realistic data and latency budgets remains an open question.

    Source: §1 · p. 1

  • Method

    DIFFA-2 combines semantic–acoustic alignment, a four-stage training strategy, and factor-based parallel decoding to build a practical diffusion-based audio language model.

    Source: §1 · p. 2

  • Results

    Across MMSU, MMAU, and MMAR, DIFFA-2 consistently outperforms DIFFA and often matches or surpasses similar-sized open AR models; on MMAR it scores 50.80% versus DIFFA’s 37.20% (+13.6 points).

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    The results establish diffusion-based modeling as a highly competitive alternative for universal audio understanding tasks.

    Source: §7 · p. 9

    DIFFA-2 targets fine-grained text-based audio understanding, with limited conversational and alignment-style supervision and no speech generation, streaming, or full-duplex interaction.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.

1 Introduction

DIFFA-2 addresses whether diffusion models can become competitive, practical audio backbones by combining semantic–acoustic alignment, scalable training, and parallel decoding. The paper reports strong benchmark performance with limited parameter updates and open-source data.

Introduction

  • Diffusion models iteratively denoise partially masked sequences, enabling any-order modeling and parallel token updates; data-limited text results suggest they can be strong data learners (Zhu et al., 2025; Ni et al., 2025).These properties motivate their use for audio, where high-quality audio–text supervision is expensive and sequential decoding can bottleneck long-form or interactive applications.Source: §1 · p. 1
  • DIFFA showed gains over a matched AR backbone on MMAU and MMSU, but remained a proof of concept without large-scale instruction tuning, preference objectives, or practical inference acceleration (Zhou et al., 2025).It used mainly speech-centric supervision, a relatively small Whisper encoder, and a frozen diffusion backbone.Source: §1 · p. 1
  • DIFFA-2 uses a progressive four-stage curriculum for semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization with fully open-source data.Factor-based parallel decoding supports practical inference without AR decoding.Source: §1 · p. 2
  • With 11,000 hours of ASR data and 3,767 hours of supervised fine-tuning data, DIFFA-2 updates about 1.1% of parameters and remains highly competitive with strong AR LALMs on MMSU, MMAU, and MMAR.The paper also reports open-sourcing its training and inference pipeline.Source: §1 · p. 2
  • The framework strengthens acoustic modeling and semantic–acoustic alignment to support unified understanding of speech, sound, and music.It combines improved acoustic modeling with factor-based parallel decoding for practical audio interaction.Source: §1 · p. 2

2 Preliminaries

LLaDA replaces left-to-right autoregressive modeling with discrete corruption and reverse denoising, enabling bidirectional context and parallel token prediction.

2 Preliminaries

  • LLaDA independently masks tokens and trains a Transformer predictor to reconstruct them, yielding a tractable negative-log-likelihood upper bound without autoregressive factorization (Shi et al., 2024; Ou et al., 2025).Each token is masked with probability t; the predictor reconstructs original tokens at masked positions.Source: §2 · p. 2
  • For LLaDA supervised fine-tuning, the prompt remains fully observed while the response is randomly masked and reconstructed.The method applies the same corruption–reconstruction principle to prompt–response pairs.Source: §2 · p. 2
  • At inference, LLaDA starts with a fully masked response and iteratively predicts masked tokens, re-masking low-confidence tokens until a fixed number of denoising steps is complete.The repeated refinement produces the final output sequence.Source: §2 · p. 2

3 Methods

DIFFA-2 pairs semantic and acoustic audio representations with a diffusion language-model backbone, then trains and decodes it through a staged pipeline for general audio understanding.

3.1 Model Architecture

  • A frozen Whisper-Large-V3 encoder feeds a semantic adapter and a 64-query Q-former acoustic adapter, giving the dLLM complementary content-oriented and acoustic views.The semantic adapter subsamples audio features from 50 Hz to 12.5 Hz and projects them toward textual semantics; the acoustic adapter attends to intermediate encoder states to capture prosody, emotion-related patterns, environmental sounds, and music (Li et al., 2023).Source: §3.1 · pp. 2–3

3.2 Training Data Overview

  • Training data combines instruction-formatted LibriSpeech and GigaSpeech transcriptions with four AQA types: caption-grounded, TTS-generated, multiple-choice, and ASR.Qwen3-32B generates 25 instruction templates for Stage 1; Stage 4 preference triplets pair audio and questions with reference answers and plausible but subtly audio-inaccurate rejected responses, retaining only unambiguous preferences.Source: §3.2 · pp. 3–4

3.3 Progressive Four-Stage Training

  • The four-stage curriculum progresses from semantic adapter alignment on ASR, to joint semantic–acoustic alignment, LoRA backbone fine-tuning, and VRPO preference optimization.The Whisper encoder remains frozen throughout; the diffusion backbone is frozen in Stages 1–2 and adapted with LoRA in Stage 3 (Hu et al., 2022), before Stage 4 uses VRPO to favor responses faithful to subtle audio cues (Zhu et al., 2025).Source: §3.3 · p. 4

3.4 Supervised Fine-Tuning of DIFFA-2

  • SFT masks only response tokens while keeping audio embeddings and prompt tokens visible, training the model to reconstruct masked response tokens from multimodal context.At each iteration, response tokens are independently masked with probability t ∈ (0, 1]; the model predicts the end-of-sequence token as well as response content.Source: §3.4 · p. 4

3.5 Variance-Reduced Preference Optimization

  • VRPO applies a DPO-style objective to Monte Carlo ELBO log-likelihood estimates, using optimal budget allocation and antithetic sampling to reduce variance.It takes four independent sampling steps and shares identical masking patterns between policy and reference models when estimating their log probabilities, stabilizing preference learning on long, acoustically rich inputs (Zhu et al., 2025).Source: §3.5 · pp. 4–5

3.6 Inference Procedure

  • Inference iteratively denoises a fully masked response and decodes left-to-right blocks with parallel token prediction and selective re-masking.Factor-based parallel decoding adaptively updates more tokens when confidence is high and fewer in uncertain regions (Wu et al., 2025).Source: §3.6 · p. 5

4 Experimental Setup

DIFFA-2 is trained on open-source data with lightweight updates to an LLaDA-8B-Instruct backbone and evaluated against audio LLM baselines on three primary audio-understanding benchmarks and one auxiliary dialogue benchmark.

4.1 Training and Inference Setup

  • Training uses about 11,000 hours of ASR data, 3,767 hours of curated SFT data, and roughly 3,000 preference pairs, updating adapters and LoRA on LLaDA-8B-Instruct (~1.1% trainable parameters).Dataset breakdowns and prompts are in Appendix A; benchmark-specific inference configurations are in Appendix B.4.Source: §4.1 · p. 6

4.2 Baselines

4.3 Benchmarks

5 Experiments

Across MMSU, MMAU, and MMAR, DIFFA-2 consistently improves over DIFFA and often matches or exceeds similar-size open AR LALMs. Ablations show stronger holistic audio QA performance with a diffusion backbone, while AR retains an ASR word-error-rate advantage.

5.1 Results on Benchmarks

  • On MMSU, DIFFA-2 scores 60.45 overall accuracy, ahead of Kimi-Audio (59.28) and Qwen2.5-Omni (59.09), and leads open models in perception (45.58) and reasoning (76.40).It improves both perception and reasoning over DIFFA by 4.41 points; the paper attributes stronger speech understanding to the upgraded acoustic front-end and four-stage training pipeline.Source: §5.1 · pp. 6–7
  • On MMAR, DIFFA-2 reaches 50.80% average accuracy, above DIFFA (37.20%, +13.6 points), MiniCPM-O (48.60%), and Qwen2-Audio (30.00%).It underperforms Qwen2.5-Omni on the most challenging Sound–Music–Speech mixtures; the passage suggests the lack of mixed-modality supervision may explain this limitation.Source: Table 3 · p. 7, §5.1 · p. 7

5.2 Ablation Study

  • Ablations show DIFFA-2’s diffusion backbone is stronger for holistic audio QA under matched training, while AR retains a slight WER advantage on token-level ASR.Factor-based parallel decoding lowers DIFFA-2’s ASR real-time factor below the AR baseline while moderately increasing WER, and keeps audio-understanding accuracy close to standard decoding. Stage 2 already beats LLaMA-Audio on both benchmarks; Stage 3 improves both models, with larger DIFFA-2 reasoning gains, and Stage 4 VRPO yields DIFFA-2’s best overall performance. The paper notes that backbone pre-training effects are not completely disentangled.Source: Table 4 · p. 7, Table 5 · p. 8, §5.2 · p. 8

5.1 Results on Benchmarks

  • On MMAU, DIFFA-2 has the best average accuracy among open models on Test-mini (69.60) and Test (67.00), with especially strong sound and speech results.Its music performance is slightly behind Kimi-Audio and MiniCPM-O; versus DIFFA, Test-mini average accuracy rises from 49.71 to 69.60.Source: §5.1 · p. 7
  • On VoiceBench, DIFFA-2 improves over DIFFA and several open-source baselines but trails heavily instruction-tuned omni models, consistent with its audio-understanding focus.The paper suggests dialogue-centric supervision could further close the gap on interactive voice-assistant benchmarks.Source: §5.1 · pp. 7–8
  • Across MMSU, MMAU, and MMAR, DIFFA-2 consistently outperforms DIFFA and often matches or surpasses similar-size open AR LALMs, approaching larger proprietary systems on several metrics.Gains are particularly clear in semantic and phonological reasoning and in sound and speech understanding.Source: §5.1 · pp. 7–8

6 Related Work

Related work describes audio-language models as largely autoregressive, while diffusion language models offer iterative denoising and parallel updates but have mainly focused on text rather than audio-specific training.

6.1 Large Audio Language Models

6.2 Diffusion Large Language Models

  • Diffusion LLMs iteratively denoise corrupted tokens, enabling bidirectional context modeling and parallel updates; LLaDA scaled the approach to large models for understanding and reasoning (Zhu et al., 2025).Earlier discrete-text diffusion work established the feasibility of the paradigm (Shi et al., 2024; Sahoo et al., 2024).Source: §6.2 · p. 8
  • LLaDA and fast-dLLMs focus on text generation and do not use audio encoders or audio-specific training curricula.Related diffusion-LM work also explores training-free inference acceleration through confidence-aware parallel decoding with KV-cache-like reuse and adaptive length prediction (Li et al., 2025a; Wei et al., 2025).Source: §6.2 · p. 9

7 Conclusions

DIFFA-2 is an enhanced diffusion-based large audio language model that uses 1.1% (99M) trainable parameters and 14.8k hours of open-source data. It is competitive with leading autoregressive models on MMSU, MMAU, and MMAR.

7 Conclusions

  • DIFFA-2 is competitive with leading autoregressive models on the MMSU, MMAU, and MMAR benchmarks.The paper presents these results as evidence that diffusion-based modeling is a competitive alternative for universal audio understanding tasks.Source: §7 · p. 9
  • Using only 1.1% (99M) trainable parameters and 14.8k hours of open-source data, DIFFA-2 substantially improves over its predecessor.Source: §7 · p. 9
  • The authors present DIFFA-2 as an enhanced diffusion-based large audio language model for audio understanding.Source: §7 · p. 9

Limitations

DIFFA-2 is currently limited in dialogue supervision, task scope, and decoding speed relative to strong autoregressive audio LLMs in some settings.

Dialogue supervision

  • Limited conversational and alignment-style supervision leaves DIFFA-2 with mid-range VoiceBench performance compared with heavily instruction-tuned AR Omnimodels.Its training objectives and data curation prioritize fine-grained audio understanding; a more balanced recipe spanning audio understanding and spoken dialogue is a future direction.Source: Limitations · p. 9

Task scope

  • DIFFA-2 covers offline speech-in/text-out audio understanding only, leaving speech generation, streaming, and full-duplex interaction unaddressed.Future work includes end-to-end speech-in/speech-out systems and user-centric evaluation of latency and interaction quality.Source: Limitations · p. 9

Decoding speed

  • Factor-based parallel decoding reduces diffusion steps with clear latency gains and negligible accuracy loss, but does not make DIFFA-2 uniformly faster than strong AR audio LLMs.More advanced training-free acceleration methods from text dLLMs are a promising, orthogonal direction for audio.Source: Limitations · p. 9

A Data Details and Prompt Templates … B.2 Benchmark Details

The appendix details the corpora and synthetic data used for ASR, supervised fine-tuning, and preference optimization, and defines the benchmarks used to evaluate audio understanding. It also lists supplementary evaluation context and notes that the study’s models and datasets are open-source licensed.

A.1 ASR Data

  • Stage 1 ASR samples pair LibriSpeech and GigaSpeech (Chen et al., 2021) transcripts with 25 Qwen-32B-generated instruction templates.Examples include requests to transcribe audio or provide its exact content.Source: §A.1 · p. 14

A.2 SFT Data

  • Stage 2 audio-caption AQA combines speech, environmental-sound, and music datasets with selected Desta 2.5 question types and paralinguistic annotations answered by Qwen3-32B.The collected datasets include ParaSpeechCaps, AudioCaps, WavCaps, VocalSound, NSynth, FMA-medium, ESC-50, Clotho, AccentDB, EmoV-DB, IEMOCAP, MELD, VCTK, Meta FAIR ASR, and VoxCeleb1.Source: §A.2 · p. 14
  • Stage 2 also combines synthesized simple and complex QA, empathetic QA, English OpenS2S, 5% of Stage 1 ASR data, and AudioMCQ without chain-of-thought.Text-only QA sources include Alpaca, Natural Questions, TriviaQA, and WebQuestions; CosyVoice2 synthesizes speech using randomly sampled LibriSpeech training-set speaker prompts, while the ASR sample preserves acoustic grounding.Source: §A.2 · p. 14

A.3 Preference Data and Prompts

  • VRPO preference examples begin with high-quality SFT audio QA and use language-model-generated fluent but subtly audio-erroneous answers as incorrect responses.The supplied text notes that the full prompt templates and examples appear in Figure A.4.Source: §A.3 · pp. 14–15

B Additional Experimental Details

B.1 Baseline Models

  • The full evaluation includes baseline models beyond the representative systems highlighted in the main text, with the complete set summarized in Table B.1.The highlighted examples are Qwen2.5-Omni, Kimi-Audio, Qwen2-Audio, MiniCPM-O, and DIFFA.Source: §B.1 · p. 16

B.2 Benchmark Details

  • MMSU (Wang et al., 2025b) contains 5,000 audio-question-answer triplets across 47 tasks spanning linguistic and paralinguistic perception and reasoning.Its topics include phonetics, prosody, semantics, emotion, and speaker traits.Source: §B.2 · p. 16
  • MMAU (Sakshi et al., 2025) tests expert-level reasoning across speech, music, and environmental sounds on Test-mini; MMAR (Ma et al., 2025) assesses reasoning consistency and generalization across audio modalities.VoiceBench (Chen et al., 2024b) evaluates semantic dialogue, general knowledge, instruction following, and safety using TTS-converted audio queries, and is supplementary because it primarily measures semantic dialogue rather than audio understanding.Source: §B.2 · pp. 16–17

B.3 Training Details · B.4 Inference Details

DIFFA-2 uses a four-stage training schedule and diffusion-based iterative inference. Its decoding combines block-wise parallel generation with confidence-adaptive parallelism to improve efficiency while preserving generation quality.

B.3 Training Details

  • Training uses four stages with progressively reduced learning rates, rank-8 LoRA in Stage 3, preference optimization in Stage 4, and approximately five days on A100 GPUs.The first three stages use 64 A100 GPUs, while the final stage uses four.Source: §B.3 · p. 17

B.4 Inference Details

  • Inference starts with a fully masked response and iteratively denoises it over T steps, refining the output from coarse to fine.The prompt and audio input are padded to the target length before denoising.Source: §B.4 · p. 18
  • At each denoising transition, DIFFA-2 predicts masked tokens from the audio, prompt, and current corrupted response, then re-masks low-confidence tokens for further refinement.The re-masked fraction is proportional to ⌈s2/s1⌉, preserving full bidirectional context during iterative refinement.Source: §B.4 · p. 18
  • Following LLaDA, semi-autoregressive decoding generates responses in left-to-right blocks, with parallel token decoding and selective re-masking across diffusion steps.This strategy balances generation quality and efficiency.Source: §B.4 · p. 18
  • Factor-based parallel decoding from fast-dLLMs (Wu et al., 2025) adapts the number of parallel tokens to model confidence rather than using a fixed confidence threshold.It selects tokens by descending marginal confidence; the decoding factor f makes parallelism more aggressive when confidence is high and more conservative in uncertain regions.Source: §B.4 · p. 19
  • Applying factor-based parallel decoding on top of diffusion denoising improves inference efficiency while preserving generation quality.Source: §B.4 · p. 19

C Additional Experiments

DIFFA-2 is optimized for fine-grained audio understanding, not large-scale conversational tuning. On VoiceBench, it trails heavily instruction-tuned AR omnimodels but improves over DIFFA and remains competitive with several open-source baselines.

C Additional Experiments

  • On VoiceBench, DIFFA-2 scores 59.63, below GPT-4o-Audio, Kimi-Audio, and Qwen2.5-Omni, but competitive with or better than several open-source baselines.VoiceBench emphasizes spoken dialogue, instruction following, and safety rather than pure audio understanding.Source: C Additional Experiments · p. 20
  • DIFFA-2 improves markedly over DIFFA’s 48.22 overall VoiceBench score across most VoiceBench metrics.Source: C Additional Experiments · p. 20
  • DIFFA-2’s mid-range VoiceBench performance contrasts with its stronger MMSU, MMAU, and MMAR results, reflecting its fine-grained audio-understanding focus.The paper attributes the VoiceBench gap to DIFFA-2’s very limited dialogue-style audio training data.Source: C Additional Experiments · p. 20

Related papers

Loading 2601.23161v1…