Source-linked AI summary
Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Mengqi Wang, Zhan Liu, Zengrui Jin, Guangzhi Sun, Chao Zhang, Philip C. Woodland
TL;DR
Autoregressive ASR decoding is sequential and computationally costly, while non-autoregressive methods can trade accuracy for efficiency. This study evaluates LLaDA for transcript deliberation and as an ASR decoder; deliberation improves recognition, while direct decoding is efficient but remains less accurate than extensively pretrained autoregressive systems.
Problem
Autoregressive ASR decoding is inherently sequential and costly, while non-autoregressive approaches can reduce latency with some recognition-accuracy degradation.
Source: §1 · p. 1Method
The study evaluates LLaDA as an external module refining ASR hypotheses and as an internal diffusion-based decoder for ASR.
Source: §1 · pp. 1–2Results
Whisper-LLaDA consistently enhances recognition quality in deliberation, while acoustic-conditioned decoding is efficient but its accuracy still lags extensively pretrained autoregressive systems.
Source: §4.2 · p. 4, §5 · p. 4Takeaways & Limitations
The authors identify larger, more diverse training data and advanced masking/remasking policies as directions to narrow the accuracy gap while preserving efficiency.
Source: §5 · p. 4LLaDA's accuracy still lags behind extensively pretrained autoregressive systems; the authors propose scaling Whisper-LLaDA with data beyond LibriSpeech.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Diffusion-based large language models (DLLMs) have recently attracted growing interest as an alternative to autoregressive decoders. In this work, we present an empirical study on using the diffusion-based large language model LLaDA for automatic speech recognition (ASR). We first investigate its use as an external deliberation-based processing module for Whisper-LLaMA transcripts. By leveraging the bidirectional attention and denoising capabilities of LLaDA, we explore random masking, low-confidence masking, and semi-autoregressive strategies, showing that Whisper-LLaDA substantially reduces WER compared with the baseline. On LibriSpeech, the best cascade system achieves 2.25%/4.94% WER on test-clean/test-other, representing a 12.3% relative improvement over the Whisper-LLaMA baseline on the test-other split. In contrast, a plain-text LLaDA without acoustic features fails to improve accuracy, highlighting the importance of audio-conditioned embeddings. We further evaluate Whisper-LLaDA as a standalone decoder for ASR with diffusion-based and semi-autoregressive decoding. Most experimental configurations achieve faster inference than the Whisper-LLaMA baseline, although recognition accuracy is slightly lower. These findings offer an empirical view of diffusion-based LLMs for ASR and point to promising directions for improvements. Code and model are open-sourced at https://github.com/liuzhan22/Diffusion-ASR.
1. INTRODUCTION
Autoregressive ASR decoding is sequential and computationally costly, while non-autoregressive methods can trade recognition accuracy for efficiency. This work studies LLaDA as both an external transcript-refinement module and an internal ASR decoder.
1. INTRODUCTION
- Sequential autoregressive decoding reduces efficiency and raises computational cost, motivating alternatives that can mitigate the accuracy-efficiency trade-off.The introduction notes that existing non-autoregressive approaches generally reduce latency but slightly degrade recognition accuracy relative to autoregressive methods.Source: §1 · p. 1
- The study investigates LLaDA in two ASR roles: refining hypotheses from a base recognizer and replacing the conventional decoding backbone.The authors also study decoding strategies on LibriSpeech, including semi-autoregressive decoding.Source: §1 · pp. 1–2
- Prior work on external deliberation refines original hypotheses or N-best lists, and this study investigates LLaDA as an external language model for ASR.The authors also explore LLaDA internally as a diffusion-based decoder, with parallel decoding and efficiency as stated aims.Source: §1 · pp. 1–2
- The study evaluates a semi-autoregressive decoding strategy on LibriSpeech through systematic decoding-strategy and hyperparameter studies.The contribution description presents this alongside investigations of LLaDA as an external module and internal decoder.Source: §1 · pp. 1–2
2. DIFFUSION-BASED LARGE LANGUAGE MODELS
LLaDA models discrete token sequences through forward masking and learned reverse denoising. Its bidirectional Transformer predicts masked positions in parallel, with training objectives defined over masked tokens.
2. DIFFUSION-BASED LARGE LANGUAGE MODELS
- The forward process independently masks tokens with probability t, from a fully observed sequence at t = 0 to a fully masked one at t = 1; reverse denoising predicts masked tokens from unmasked context.The reverse process starts from a fully masked sequence and iteratively recovers a coherent sample as t approaches 0.Source: §2 · p. 2
- LLaDA uses a bidirectional attention-based Transformer to predict all masked positions in parallel, and pretraining uses cross-entropy for its mask predictor.The clean sequence is x_0, x_t is formed by independently masking positions with probability t, and S_t records the masked positions.Source: §2 · p. 2
- Supervised fine-tuning applies its loss to masked response tokens while leaving the prompt unmasked; the formulation allows prediction without taking t as input.The response-token positions used by the loss are recorded in S′_t.Source: §2 · p. 2
3. METHODS
Whisper-LLaDA combines a Whisper speech encoder with a diffusion language-model decoder and supports both direct ASR and transcript deliberation. Training and inference use masked response reconstruction, with diffusion-based and semi-autoregressive strategies for decoding and refinement.
3.1. Overall Architecture
- Whisper-LLaDA combines a Whisper-Large-v3 encoder with LLaDA-8B-Instruct, using a Q-Former and projection layer to align speech representations with the decoder.Only LoRA, the Q-Former, and projection layer are trainable, totaling about 87M parameters.Source: §3.1 · p. 2
3.2. Training
- Training uses LibriSpeech speech embeddings and independently masks response tokens, with LLaDA predicting masked tokens from the remaining context.The Whisper encoder is frozen while LLaMA/LLaDA are fine-tuned with LoRA; α denotes the input audio's acoustic features.Source: §3.2 · p. 2
3.3. Inference
- Inference combines a task instruction, acoustic features, and a response block; direct decoding starts from 128 masks, while deliberation starts from a transcript-length block.LLaDA denoises the block, with diffusion-based and semi-autoregressive approaches considered for decoding and deliberation.Source: §3.3 · p. 2
3.3.1. Deliberation-based Processing with LLaDA
- For deliberation, Whisper-LLaDA masks selected transcript tokens at random, by lowest confidence, or sequentially within sub-blocks, then reconstructs them using audio and unmasked context.The study also attempts a plain-text LLaDA version without Whisper embeddings.Source: §3.3.1 · p. 2
3.3.2. Diffusion-based Decoding
- Diffusion decoding predicts masked tokens in parallel and retains the top-K highest-confidence tokens per iteration, remasking the rest.An [EOS] token triggers early stopping by forcing later positions to [EOS].Source: §3.3.2 · pp. 2–3, Figure 2 · p. 3
3.3.3. Semi-autoregressive Decoding
- Semi-autoregressive decoding applies diffusion within equal-sized sub-blocks and proceeds autoregressively across them, with up to N/M denoising steps per block.The method varies the number of sub-blocks M and maximum denoising steps N, and applies the same early-stopping criterion.Source: §3.3.3 · p. 3, Figure 2 · p. 3
4. EXPERIMENTS
Experiments on LibriSpeech test-clean and test-other compare Whisper-LLaDA cascade refinement and standalone decoding with Whisper-LLaMA baselines. Cascade deliberation improves recognition, while diffusion decoding offers faster inference with some accuracy loss.
4.1. Baseline
- Whisper-LLaMA reaches 2.24%/5.63% WER on test-clean/test-other; Whisper-Large v3's 3.90% test-other score is reference-only because its training data make it not directly comparable.Experiments use LibriSpeech test-clean and test-other; the Whisper-Large systems are reported as using proprietary training datasets of up to ∼5M hours.Source: §4.1 · p. 3, §4 · p. 3
4.2. Deliberation-based Processing
- Plain-text LLaDA worsens WER to 3.89% on test-clean and 6.91% on test-other, indicating that audio embeddings are essential for effective deliberation.It was fine-tuned on Whisper-LLaMA transcripts paired with ground-truth text.Source: §4.2 · p. 3
- Whisper-LLaDA consistently improves over Whisper-LLaMA in cascade deliberation; random masking is best at 90%, while lowest-confidence masking yields marginal gains.Semi-autoregressive deliberation with sub-block partition further improves recognition accuracy.Source: §4.2 · p. 4
4.3. Diffusion-based Decoding
- 2.82%/5.79% WER on test-clean/test-other at 64 denoising steps runs 1.3× faster than Sys. 1 and 2.4× faster than Sys. 2.The 64-step setting is described as a favorable trade-off with moderate accuracy loss relative to most AR frameworks.Source: §4.3 · p. 4
4.4. Semi-Autoregressive Decoding
- 2.40%/4.96% WER on test-clean/test-other is achieved by the 4-block, 32-step semi-autoregressive setting.The test-other result is reported as significantly better than the LLaMA/Vicuna-based ASR baselines.Source: Figure 3 · p. 4, §4.4 · p. 4
5. CONCLUSION
The conclusion contrasts plain-text LLaDA, which does not improve transcripts, with audio-conditioned Whisper-LLaDA, which improves recognition; standalone decoding is faster but usually less accurate. The study also identifies data scale and masking policies as directions for further work.
5. CONCLUSION
- Whisper-LLaDA improves recognition, while plain-text LLaDA fails to improve outputs; high-ratio random masking and semi-autoregressive processing deliver the largest gains.The conclusion attributes effective deliberation-based processing to conditioning on acoustic features.Source: §5 · p. 4
- Standalone LLaDA decoding is faster than autoregressive baselines but has slightly higher WER in most settings.The conclusion notes that accuracy still trails extensively pretrained autoregressive systems.Source: §5 · p. 4
- Future work should scale Whisper-LLaDA beyond LibriSpeech with larger, more diverse training data and explore masking policies to narrow the accuracy gap while preserving efficiency.Source: §5 · p. 4