PaperlayerView exact revision on arXiv

Source-linked AI summary

DIFFA: Large Language Diffusion Models Can Listen and Understand

Jiaming Zhou, Hongjie Chen, Shiwan Zhao, Jian Kang, Jie Li, Enzhi Wang, Yujie Guo, Haoqin Sun, Hui Wang, Aobo Kong, Yong Qin, Xuelong Li

arXiv:2507.18452v3cs.SDeess.AS

TL;DR

Diffusion models have not been explored for audio-language understanding, while existing audio-language model paradigms rely on autoregressive generation. DIFFA combines a frozen diffusion language model with dual adapters trained in two stages. It achieves competitive results on multiple benchmarks using 960 hours of ASR data and 127 hours of synthetic instruction data, suggesting diffusion models are a promising alternative to autoregressive audio-language models.

  • Problem

    Diffusion models had not been explored for audio-language understanding, while existing audio-language model paradigms relied on autoregressive generation.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    DIFFA combines a frozen diffusion language model with dual adapters, aligning speech representations through ASR training before instruction tuning on synthetic audio-caption data.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    DIFFA achieves competitive performance across MMSU, MMAU, and VoiceBench using 960 hours of ASR data and 127 hours of synthetic instruction data.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results suggest diffusion-based models are a promising alternative to autoregressive large audio-language models.

    Source: §6 · p. 9

    The modest training-data volume may limit generalization to low-resource accents, noisy environments, and specialized-domain speech.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have shown remarkable capabilities across textual and multimodal domains. In parallel, diffusion-based language models have emerged as a promising alternative to the autoregressive paradigm, offering improved controllability, bidirectional context modeling, and robust generation. However, their application to the audio modality remains underexplored. In this work, we introduce \textbf{DIFFA}, the first diffusion-based large audio-language model designed to perform spoken language understanding. DIFFA integrates a frozen diffusion language model with a lightweight dual-adapter architecture that bridges speech understanding and natural language reasoning. We employ a two-stage training pipeline: first, aligning semantic representations via an ASR objective; then, learning instruction-following abilities through synthetic audio-caption pairs automatically generated by prompting LLMs. Despite being trained on only 960 hours of ASR and 127 hours of synthetic instruction data, DIFFA demonstrates competitive performance on major benchmarks, including MMSU, MMAU, and VoiceBench, outperforming several autoregressive open-source baselines. Our results reveal the potential of diffusion-based language models for efficient and scalable audio understanding, opening a new direction for speech-driven AI. Our code will be available at https://github.com/NKU-HLT/DIFFA.git.

1 Introduction

Autoregressive audio-language models dominate despite limits in decoding flexibility, while diffusion-based language models remain largely unexplored for audio understanding. DIFFA tests this direction with a dual-adapter design and reports competitive benchmark performance using relatively little training data.

1 Introduction

  • Autoregressive audio-language models face exposure bias, slow generation, and limited flexibility for bidirectional or partially conditioned inference.Diffusion-based audio-language understanding remains underexplored despite audio’s acoustic variability, temporal structure, and paralinguistic information.Source: §1 · p. 2
  • DIFFA is introduced as the first diffusion-based large audio-language model, designed for large-scale audio-text understanding without autoregressive modeling.The paper describes audio-language modeling as useful for spoken interaction and more natural, robust, and versatile human-computer communication.Source: §1 · pp. 1–2
  • DIFFA uses lightweight semantic and acoustic adapters with a frozen diffusion language model, training first on ASR and then on synthetic instruction data.The speech encoder and language model remain frozen; the second stage uses a “What can you hear from the audio?” prompting scheme.Source: §1 · p. 2
  • DIFFA achieves competitive results on MMSU, MMAU, and VoiceBench using 960 hours of ASR data, 127 hours of synthetic instruction data, and 72 A800 GPU hours.Qwen2-Audio-Instruct uses 510,000 hours of data.Source: §1 · p. 2

2 Related Work

Prior audio-language systems rely on autoregressive modeling, while diffusion has been explored in language and vision but not audio-language understanding. Related work also motivates synthetic instruction data and end-to-end modality alignment as alternatives to human-supervised tuning and cascaded transcription.

2.1 Large Audio-Language Models (LALMs)

2.2 Diffusion-Based Language and Multimodal Models

  • Diffusion language models have been extended to vision-language modeling, motivating this paper’s first application of the framework to audio-language models.The paper notes that audio can benefit from bidirectional and nonsequential reasoning.Source: §2.2 · p. 3

2.3 Modality Alignment Without Explicit Supervised Fine-tuning

  • Synthetic instruction-tuning approaches use LLM-prompted audio captions to reduce reliance on paired human supervision, while DIFFA integrates semantic and acoustic information end to end.DESTA-2 uses a cascaded design with Whisper transcription before instruction response.Source: §2.3 · p. 3

3 Methods

DIFFA combines a frozen diffusion language model with speech adapters, trained to align speech and language before learning audio-caption instruction following. Its method uses masked-token prediction and iterative, partially parallel inference.

3.1 Preliminaries

  • LLaDA independently masks tokens and learns to reconstruct them, enabling bidirectional modeling and iterative denoising over a response sequence.During inference, low-confidence positions are remasked across T steps.Source: §3.1 · p. 3

3.2 Data Construction

  • Synthetic instruction responses are generated by prompting language models with audio transcriptions and acoustic attributes using “What can you hear from the audio?”.The generated response is paired with its audio as modality-alignment supervision, without explicit supervised fine-tuning data.Source: §3.2 · p. 4
  • LLaDA rewrites Qwen3-generated captions to align their textual style with its own internal data distribution.The authors call this variant LLaDA-rewrite-Qwen3.Source: §3.2 · p. 4

3.3 Model Architecture and Training Strategy

  • A semantic adapter compresses Whisper-small’s 50 Hz features to 12.5 Hz, while an acoustic Q-former extracts speech-specific features from intermediate states.The two adapter outputs are concatenated as prefix tokens; Whisper and LLaDA remain frozen during two-stage training.Source: §3.3 · p. 4
  • Stage 1 trains the semantic adapter with an ASR objective on 960 hours of Librispeech; stage 2 fine-tunes both adapters on 127 hours of synthetic audio-caption data.Audio and prompt tokens remain unmasked during training, while response tokens are masked for the diffusion-style prediction objective.Source: §3.3 · p. 4

3.4 Inference Procedure

  • Inference generates blocks from left to right, predicting tokens in parallel within each block and remasking low-confidence positions during iterative refinement.This semi-autoregressive strategy is described as balancing generation quality and efficiency while retaining parallel decoding and bidirectional context modeling.Source: §3.4 · p. 5

4 Experimental Setup

The setup uses open-source speech datasets for ASR and synthetic Q&A, pairs a frozen LLaDA backbone with lightweight adapters, and evaluates on benchmarks spanning spoken-language and broader audio understanding.

4.1 Datasets

  • Stage 1 uses Librispeech for ASR, while stage 2 draws on VCTK-Corpus, Accentdb, IEMOCAP, dailytalk, and VoxCeleb for Q&A data.The synthetic dataset includes 10 attributes, from speaker traits and emotion to intent and spoken text; it excludes PromptTTS and Mixed Noise & Reverb subsets used by DESTA-2 due to lack of access.Source: §4.1 · p. 5

4.2 Model Configuration and Training Setup

  • The model freezes LLaDA-8B-Instruct and trains a 14.4M-parameter semantic adapter and a 22.3M-parameter acoustic adapter alongside the Whisper-Small encoder.The Whisper-Small encoder has 88.2 million parameters; LLaDA-8B-Instruct has approximately 8.1 billion parameters.Source: §4.2 · p. 5

4.3 Benchmarks

  • Evaluation spans MMSU spoken-language perception and reasoning, MMAU audio questions across speech, music, and environmental sounds, and VoiceBench spoken queries for knowledge, instruction following, and safety.The experiments use MMAU's Test-mini split.Source: §4.3 · p. 6

5 Experiments

DIFFA is competitive across MMSU, MMAU, and VoiceBench, and the experiments compare backbone, adapter, and instruction-data choices. Results also show weaker performance on some acoustic and instruction-following dimensions.

5.1 Evaluation on MMSU

  • 56.04% average accuracy on MMSU trails Gemini-1.5-Pro (60.68%) but exceeds Qwen2-Audio-Instruct (53.27%) and Gemini-2.0-Flash (51.03%).DIFFA reaches 81.53% on semantic reasoning, while phonological and paralinguistic tasks remain weaker.Source: §5.1 · pp. 6–7

5.2 Evaluation on MMAU

  • 49.71% average accuracy on MMAU exceeds SALMONN (34.90%), GAMA-IT (22.83%), and LTU (17.44%), and approaches GPT-4o mini Audio (53.01%).DIFFA's strongest domain is speech-related tasks at 59.46%.Source: Table 3 · p. 7, §5.2 · pp. 7–8

5.3 Evaluation on VoiceBench

  • With only 960 hours of ASR data and 127 hours of synthetic instructions, DIFFA scores 34.45% on SD-QA and 35.60% on OBQA, versus Qwen2-Audio’s 35.72% and 49.45%, despite its hundreds of thousands of hours of proprietary data.Source: §5.3 · p. 8
  • 26.56% on IFEval slightly exceeds Qwen2-Audio (26.33%) and GLM-4-Voice (25.92%); 76.54% on AdvBench approaches GLM-4-Voice (88.08).The authors identify the gap to top IFEval models as a challenge for audio-conditioned instruction execution without large-scale supervised tuning.Source: Table 4 · p. 7, §5.3 · p. 8

5 Experiments

  • DIFFA establishes a competitive diffusion-based baseline across complex linguistic and audio reasoning benchmarks, including MMSU, MMAU, and VoiceBench.The VoiceBench conclusion notes that DIFFA uses orders of magnitude less training data than competing systems.Source: §5.1 · p. 7, §5.2 · p. 8, §5.3 · p. 8

5.4 Ablation Study

  • With dual adapters, LLaDA scores 51.71 on MMAU and 54.72 on MMSU, outperforming the LLaMA counterpart; adding an acoustic adapter gains +2.01 on MMAU and +3.16 on MMSU.The dual setup combines semantic representations with low-level acoustic cues from intermediate encoder states.Source: Table 5 · p. 8, §5.4 · pp. 8–9
  • Instruction-data variants perform comparably, with Qwen3-generated data yielding the best overall accuracy; rewriting that data with LLaDA brings only marginal improvement.Models trained on LLaDA-generated instructions outperform those trained on LLaMA-3 instructions.Source: Table 6 · p. 8, §5.4 · p. 9

6 Conclusion

DIFFA combines a frozen diffusion language backbone with lightweight dual adapters for audio understanding and instruction following. It performs competitively across several benchmarks with limited training data, while the experiments remain limited to relatively small-scale open-source corpora.

6 Conclusion

  • DIFFA combines a frozen diffusion language backbone with lightweight dual adapters for audio understanding and instruction following.The paper presents it as an initial exploration of diffusion-based modeling in the audio domain.Source: §6 · p. 9
  • DIFFA achieves competitive results on MMSU, MMAU, and VoiceBench using 960 hours of ASR data and 127 hours of synthetic instruction data.The paper contrasts this with Qwen2-Audio-Instruct, trained on over 500K hours.Source: §6 · p. 9
  • The experiments use relatively small-scale open-source corpora; the authors plan to scale to broader and more diverse data.Source: §6 · p. 9

Limitations

DIFFA is trained on a modest amount of data, which may constrain generalization to broader real-world audio phenomena. The authors plan to scale both ASR and synthetic instruction data to improve robustness and coverage.

  • DIFFA’s training data comprises 960 hours of ASR data and 127 hours of synthetic instruction data.Source: Limitations · p. 9
  • This constrained data volume may limit generalization to low-resource accents, noisy acoustic environments, and specialized-domain speech.Source: Limitations · p. 9
  • Future iterations will expand ASR corpora and diverse synthetic instruction datasets to strengthen robustness and coverage of complex audio scenarios.Source: Limitations · p. 9

A.1 Training Hyperparameter

Training uses two stages: first the semantic adapter is trained on LibriSpeech, then both adapters are jointly trained on the generated dataset. Both stages use Adam.

A.1 Training Hyperparameter

  • Stage 1 trains the semantic adapter on LibriSpeech for 10 epochs, using a 1e-4 learning rate, 1000 warm-up steps, and global batch size 128.Source: §A.1 · p. 12
  • Stage 2 jointly trains the semantic and acoustic adapters on the generated dataset for 10 epochs, using a 5e-5 learning rate, 2000 warm-up steps, and global batch size 32.Source: §A.1 · p. 12
  • Both stages use Adam, and experiments run on 4 NVIDIA A800 GPUs with 80GB memory each.Source: §A.1 · p. 12

A.2 Inference Hyperparameter

Inference configures answer length, block length, and denoising steps for each benchmark. AlpacaEval performance is sensitive to answer and block length, with moderate block sizes performing better.

A.2 Inference Hyperparameter

  • Inference settings include answer length, block length, and total denoising steps, with answer length defined as the maximum generated tokens and block length as tokens decoded per block.Source: §A.2 · p. 12, Table A.1 · p. 12
  • The number of decoding steps is set equal to answer length for best performance.Source: §A.2 · p. 12
  • Moderate block sizes, such as 32, consistently yield better AlpacaEval results, with performance sensitive to both answer length and block size.The authors suggest this may balance context stability and denoising granularity.Source: §A.2 · pp. 12–13, Table A.2 · p. 12
Loading 2507.18452v3…