Diffsound: Discrete Diffusion Model for Text-to-sound Generation

I. INTRODUCTION

• The proposed framework maps text features to mel-spectrogram tokens, decodes them into a mel-spectrogram, and uses a vocoder to produce waveform audio.It comprises a text encoder, VQ-VAE, token-decoder, and vocoder.Source: §I · p. 1, Figure 1 · p. 2
• The AR decoder predicts tokens left-to-right, limiting use of context on both sides of sound events and allowing prediction errors to accumulate.The paper also identifies limited text-audio data: AudioCaps has about 49K samples, compared with over 200K audio-video pairs in VGGSound.Source: §I · pp. 1–2
• Diffsound predicts all mel-spectrogram tokens simultaneously, then iteratively revises them using context from the previous step.Its discrete diffusion process corrupts tokens toward a stationary distribution, then learns to recover the original tokens conditioned on text features.Source: §I · p. 2
• MBTG creates text descriptions from AudioSet event labels for pretraining, while curriculum learning adds single-event clips before multi-event clips.The pretrained model is fine-tuned on a smaller text-audio dataset such as AudioCaps.Source: §I · pp. 2–3
• The authors explore FID, KL-divergence, and audio caption loss as objective metrics for fidelity and relevance, alongside MOS for human evaluation.They describe human evaluation as expensive and tedious.Source: §I · pp. 2–3
• MOS: 3.56 v.s 2.786, and Diffsound generates sound five times faster than the AR token-decoder.The framework also reports MOS: 3.56 (ours) v.s 4.11 (ground truth).Source: §I · p. 2

II. RELATED WORK

• Related work covers GAN sound generation and discrete diffusion; RegNet is limited to single-domain sounds, while autoregressive methods, have been applied to sound generation.Discrete diffusion research includes D3PMs and VQ-Diffusion [13] for image generation.Source: §II · p. 3

III. PROPOSED TEXT-TO-SOUND FRAMEWORK

• The proposed framework consists of a text encoder, VQ-VAE, token-decoder, and vocoder.Figure 1(a) shows the overall architecture; this section introduces the design of each component.Source: §III · p. 3

A. Text encoder

• CLIP text features yielded better generation performance than BERT features, and the text encoder remains fixed during training.The authors suggest CLIP’s image-text contrastive training may give its representations more sound-event semantics. The encoders are pretrained BERT [39] and CLIP.Source: A. Text encoder · p. 3

B. Learning Discrete Latent Space of Mel-spectrograms Via VQ-VAE

• Unlike TTS, sound generation must extract event information because text and sound lack direct correspondence; VQ-VAE encodes event characteristics in a codebook.The proposed approach follows Liu et al. and Iashin and Rahtu in generating mel-spectrograms based on the codebook.Source: B. Learning Discrete Latent Space of… · p. 3
• A mel-spectrogram is approximated by a sequence of tokens, shifting spectrogram generation to token-sequence prediction.VQ-VAE [7] uses a spectrogram encoder, decoder, and finite codebook; its encoder compresses the input mel-spectrogram into a smaller-scale representation.Source: B. Learning Discrete Latent Space of… · p. 3
• The vector quantizer maps each encoded time-frequency vector to its nearest codebook entry, producing discrete spectrogram tokens that the decoder uses to reconstruct the mel-spectrogram.The reconstruction is expressed as ˆs = Ds(Q(ˆz)).Source: B. Learning Discrete Latent Space of… · p. 4
• VQ-VAE is trained end-to-end, with a patch-based adversarial loss added to preserve reconstruction quality when upsampling from a smaller-scale representation.A convolutional discriminator distinguishes original from reconstructed mel-spectrograms, and λd controls the adversarial-loss weight.Source: B. Learning Discrete Latent Space of… · p. 4

C. Token-decoder

• Given text features, the AR token-decoder predicts next-token probabilities over flattened pretrained VQ-VAE mel tokens, using cross-entropy and teacher forcing.Source: C. Token-decoder · p. 4
• The AR decoder’s teacher-forcing mismatch can accumulate inference errors, and its reliance on past tokens alone creates unidirectional bias [43].Source: C. Token-decoder · p. 4

D. Vocoder

• For waveform reconstruction, the paper selects non-autoregressive MelGAN for efficiency and quality, training it on AudioSet’s 527 sound events because speech- or music-trained models do not suit environmental sounds.Source: D. Vocoder · p. 4

IV. DIFFUSION-BASED DECODER

• Diffsound is proposed as a non-autoregressive token-decoder based on discrete diffusion to address AR decoders’ unnatural bias and accumulated errors.Source: §IV · p. 4

A. Diffusion Probabilistic Models

• Diffusion models are presented as powerful generation models in image and speech fields, [13],.Source: A. Diffusion Probabilistic Models · p. 4

1) Vanilla Diffusion Model:

• In vanilla diffusion, a fixed Markov chain adds scheduled Gaussian noise to x0 until xT approaches N(0, I), while a learned reverse process maps xT back toward x0.Source: 1) Vanilla Diffusion Model · p. 5
• The generative model pθ(x0) is trained to fit q(x0) by optimizing a variational upper bound on negative log-likelihood.Source: 1) Vanilla Diffusion Model · p. 5

2) Discrete Diffusion model:

• Because discrete tokens cannot be corrupted by adding Gaussian noise, discrete diffusion defines transition-probability matrices for each forward step.The matrices specify how tokens transition, enabling corruption into a meaningless sequence and subsequent recovery conditioned on text.Source: 2) Discrete Diffusion model · p. 5, B. Non-autoregressive Mel-spectrograms… · p. 5

B. Non-autoregressive Mel-spectrograms Generation Via Diffsound

• Diffsound predicts all mel-spectrogram tokens simultaneously and refines them over T iterations, using context to revise tokens rather than generating them in N autoregressive steps.The authors expect this to reduce unidirectional bias and accumulated prediction errors; in practice, N ≫ T.Source: B. Non-autoregressive Mel-spectrograms… · p. 5
• Diffsound combines uniform replacement and masking to avoid unrelated-token corruption that hinders denoising and mask-only corruption that may make the model ignore context.Its combined transition assigns probabilities to switching to [MASK], uniform resampling, or retaining the current token.Source: B. Non-autoregressive Mel-spectrograms… · pp. 5–6
• The uniform transition resamples tokens across the codebook or retains the previous value, while the mask strategy adds an absorbing [MASK] state.Instead of storing all uniform transition matrices, the method stores αt and βt; storing all matrices would require O(K^2T) memory.Source: B. Non-autoregressive Mel-spectrograms… · p. 6
• At inference, the model samples an initial token sequence from the stationary distribution, iteratively predicts earlier tokens conditioned on text, and decodes the result with the VQ-VAE decoder.The iteration can use the specified time stride Δt.Source: B. Non-autoregressive Mel-spectrograms… · p. 7
• Training minimizes a variational lower bound, while reparameterization predicts the noiseless token distribution and an auxiliary denoising loss is combined with the VLB for better performance.The experiments use a linear noise schedule; λ controls the weight of the auxiliary loss.Source: B. Non-autoregressive Mel-spectrograms… · p. 7
• Fast inference uses a larger time stride to skip reverse-diffusion steps, making inference more efficient with only a small decrease in quality.The final step is kept at x0.Source: B. Non-autoregressive Mel-spectrograms… · p. 7

C. Pre-training Diffsound on AudioSet dataset

• AudioSet is the largest open-source audio dataset but provides event labels rather than text, while collecting large-scale text-audio pairs is time-consuming.The passage contrasts this with CogView, which collects more than 30 million high-quality Chinese text-image pairs.Source: C. Pre-training Diffsound on AudioSet… · p. 8

Mask-based text generation method (MBTG)

• MBTG inserts 1 or 2 [MASK] tokens on either side of AudioSet event labels, prompting Diffsound to learn event relationships without specifying detailed descriptions.The method addresses the difficulty of automatically generating specific text descriptions from event labels, which may not capture the temporal relationships between events.Source: C. Pre-training Diffsound on AudioSet… · p. 8

Curriculum Learning Strategy

• Curriculum learning trains Diffsound first on the Single Event Set (SES), then on the Multiple Event Set (MES), progressing from simpler to more complex audio.AudioSet is split by the number of events in each audio sample.Source: C. Pre-training Diffsound on AudioSet… · p. 8

• The experiments use AudioSet and AudioCaps.The section then introduces both datasets and discusses data preprocessing.Source: §V · p. 8

A. AudioSet

• AudioSet uses an ontology of 527 sound classes.It is a large-scale audio dataset.Source: A. AudioSet · p. 8
• The collection contains more than 2 million 10-second audio snippets from YouTube videos.Source: A. AudioSet · p. 8
• The training set has roughly 1.9 million clips because some could no longer be downloaded, and clips may have one or more event labels.Source: A. AudioSet · p. 8

B. AudioCaps

• AudioCaps supplies 49,256 training clips, 494 validation clips, and 957 test clips; models are trained on the training set and evaluated on validation.Training clips have one human-annotated caption each, while validation and test clips have five captions each.Source: B. AudioCaps · p. 8

C. Data pre-processing

• Audio from both datasets is converted to 22.05 kHz, padded to 10 seconds, and represented as an 860 × 80 log-mel-spectrogram.Features use an 1,024-point Hanning window, a 256-point hop size, and 80 mel bins.Source: C. Data pre-processing · p. 8

VI. EVALUATION METRIC

• The study evaluates sound generation with human Mean Opinion Score (MOS) and objective assessment metrics.The objective metrics are Fréchet Inception Distance (FID), Kullback-Leibler (KL) divergence, and Audio Caption Loss.Source: §VI · p. 8, B. Objective Assessment Metrics · p. 8

A. MOS

• MOS averages ratings for relevance, fidelity, and intelligibility from 10 people who assess randomly selected AR and Diffsound sound sets.They grade each aspect from 0–5 without knowing whether a sound is real or generated; each set includes a text description, a real sample, and 1–2 generated sounds from each model.Source: A. MOS · p. 8

B. Objective Assessment Metrics

• FID measures the distance between real and generated feature distributions using InceptionV3, adapted for spectrograms and trained on AudioSet.The adaptation changes the input convolution from three channels to one and removes max pooling to preserve spectrogram resolution.Source: B. Objective Assessment Metrics · pp. 8–9
• KL divergence compares probability distributions for generated and real samples, using an AudioSet-pretrained InceptionV3 classifier to assess text–sound relevance.This metric is motivated by the presence of multiple events in a sound.Source: B. Objective Assessment Metrics · p. 9
• Audio Caption Loss uses an Audio Caption Transformer [53] trained on AudioCaps, with SPICE and CIDEr scores to assess generated samples.Higher SPICE and CIDEr indicate better generation quality; SPICE reflects semantic closeness and CIDEr caption fluency.Source: B. Objective Assessment Metrics · p. 9

C. The Effectiveness of FID and KL

• FID and KL scores increase as Gaussian noise, masked audio content, or interfering sound is increased, indicating sensitivity to these disturbances.The tests compare AudioCaps validation audio with versions modified by each disturbance; Figure 3 visualizes the results.Source: C. The Effectiveness of FID and KL · p. 9, Figure 3 · p. 9

A. Implementation Details

• The framework is trained in stages: VQ-VAE and vocoder first, then the token decoder, with the VQ-VAE and text encoder fixed during decoder training.The text encoder uses pre-trained BERT or CLIP models.Source: A. Implementation Details · p. 9

1) VQ-VAE:

• The VQ-VAE maps 860 × 80 spectrograms to 53 × 5 tokens, and the AudioSet-pretrained model is the default because it performs better than the AudioCaps-trained model.Its codebook dimension and size are both 256; training on AudioSet takes about 9 days on 8 Nvidia V100 GPUs.Source: 1) VQ-VAE · pp. 9–10

2) Autoregressive token-decoder:

• The autoregressive token decoder is a 19-layer, 16-head transformer with dimension 1024, trained for about 2 days on 8 Nvidia P40 GPUs.Text features are projected to the transformer’s 1024-dimensional hidden space, and a K-way softmax predicts output tokens.Source: 2) Autoregressive token-decoder · p. 10

3) Diffsound model:

• DiffSound uses a parameter-matched 19-layer, 16-head, 1024-dimensional transformer, with the mask-and-uniform transition matrix selected for best generation performance.The default uses T = 100 timesteps and loss weight λ = 1e−4; AudioCaps training takes about 2 days on 16 Nvidia V100 GPUs.Source: 3) Diffsound model · p. 10

4) Vocoder.:

• The MelGAN vocoder is trained for 200 epochs on 40% of AudioSet audio clips, using batches of 256 mel-spectrograms on one P40 GPU for about 20 days.Training inputs are random sequences of 8192 audio samples at a 22050 sample rate.Source: 4) Vocoder · p. 10

5) The duration of the generated sound:

• 10-second audio is represented by 265 generated mel-spectrogram tokens for both the AR token-decoder and Diffsound, ensuring duration-matched comparison with AudioCaps clips.The estimate follows from an 80 × 860 mel-spectrogram and 16 downsampling operations in both time and frequency, yielding approximately 5 × 53 tokens.Source: 5) The duration of the generated sound · p. 10

VIII. RESULTS AND ANALYSIS

• The results section reports MOS above 2.5 for both AR and Diffsound frameworks, then compares the decoders and conducts Diffsound ablations.Source: §VIII · p. 10

1) Subjective and objective metrics:

• MOS 3.56 v.s 2.786, FID 9.76 v.s 16.87, KL 4.21 v.s 5.31, and CIDEr 0.36 v.s 0.22 show Diffsound outperforming the AR token-decoder subjectively and on all reported objective metrics.Both the AR and Diffsound frameworks achieved MOS above 2.5; the authors note that background noise in many clips reduced Ground Truth intelligibility.Source: 1) Subjective and objective metrics · pp. 10–11, §VIII · p. 10
• CLIP as the text encoder performs better than BERT, and a VQ-VAE trained on AudioSet yields better generation quality than one trained on AudioCaps.The authors speculate that training on a larger dataset improves the VQ-VAE’s ability.Source: 1) Subjective and objective metrics · pp. 10–11

2) The generation speed:

• 0.53 seconds is Diffsound’s fastest reported generation time at T = 25 and ∆t = 7, 43 times faster than the AR token-decoder with a similar FID score.For 10-second audio, standard Diffsound generation takes about 5 seconds versus about 23 seconds for AR; using fewer timesteps and a larger stride speeds generation but reduces quality.Source: Table III · p. 11, 2) The generation speed · p. 11

3) Visualization:

• Figure 4 shows Diffsound generating more complete scene semantics and better detailed modelling than the AR token-decoder; in one example, AR omits the crickets-singing event.In that example, AR generates only the man-speaking event.Source: Figure 4 · p. 11, 3) Visualization · p. 11

1) Impact of different transition matrices for Diffsound model:

• The combined mask-and-uniform transition matrix performed best with γT = 0.9, outperforming the uniform matrix.The authors suggest uniform transitions make the reverse process hard to learn, while mask-only transitions can cause the model to focus on the mask token and ignore context.Source: 1) Impact of different transition… · p. 12

2) Impact of the final mask rate γT for the mask and uniform transition matrix:

• γT = 0.9 outperformed the other tested final mask rates for the mask-and-uniform transition matrix.The comparison covers lines 2–6 of Table IV.Source: 2) Impact of the final mask rate γT… · p. 12

3) Number of timesteps:

• More training and inference timesteps improved performance, but increased inference time; performance remained good with inference steps reduced from 100 to 25.The study set the maximum timestep count to 100 for training and inference, and identifies dropping 75 inference steps as a way to boost generation speed.Source: 3) Number of timesteps · p. 12

C. The effectiveness of pre-training the Diffsound model on AudioSet

• FID 8.27, KL 4.11, and SPICE 0.105 were the best reported scores when AudioSet pre-training and curriculum learning were combined.Pre-training used about 45% of the AudioSet training set; pre-training alone also lowered FID from 9.76 to 8.78 and KL from 4.21 to 4.15.Source: C. The effectiveness of pre-training… · p. 12, Table VI · p. 12

D. Choose high-relevance sound with text description based on audio caption loss

• Keeping the top 2 of 10 generated samples by summed SPICE and CIDEr improved KL from 5.31 to 5.03 for AR and from 4.21 to 3.86 for Diffsound.Diffsound’s FID slightly increased when keeping the top 2 rather than all 10; the authors suggest some high-fidelity samples may be text-irrelevant.Source: D. Choose high-relevance sound with… · p. 12, Table VII · p. 12

IX. CONCLUSION

• The authors conclude that Diffsound improves generation performance and speed over the AR token-decoder, with effectiveness verified by objective and subjective metrics.They also describe a simple pre-training strategy to further improve Diffsound and designed three objective evaluation metrics for text-to-sound generation.Source: §IX · p. 12
• The current framework is not end-to-end: its VQ-VAE, token decoder, and vocoder are trained separately, and the authors propose exploring end-to-end generation.The paper notes that separate training may not be optimal.Source: §IX · p. 12

APPENDIX A THE PROOF OF FORMULA (14)

• The text aims to prove Qtc(x0) = αtc(x0) + βt; for t = 1, (29) gives α1 + β1 when x = x0 and β1 when x ≠ x0.Source: Appendix A · p. 13
• For the induction step, considering both prior-state cases shows that the formula holds at t + 1 whether x = x0 or x ≠ x0.The two cases yield the expressions in (30) and (31), respectively.Source: Appendix A · p. 13
• When t is sufficiently large and α_t is close to 0, formula (14) gives Q_t^c(x0) = 1/K and the stationary distribution p(x_T) is uniform.The distribution is [1/K, 1/K, · · ·, 1/K].Source: Appendix A · p. 13

APPENDIX B THE PROOF OF FORMULA (17)

• The appendix uses induction to prove formula (17), Q_t^c(x_0) = β_t c(x_0) + γ_t c(K + 1), under the stated conditional information.It checks the base case t = 1 and assumes the formula holds at step t to establish it at t + 1.Source: Appendix B · p. 13
• The inductive step considers x = x_0 and x = K + 1 at step t + 1, with one and two respective possible states at step t.For x = x_0, the prior state is x = x_0; for x = K + 1, the prior states are x = x_0 and x = K + 1.Source: Appendix B · p. 13
• For sufficiently large t, β_t approaches 0 and γ_t = 1, yielding p(x_T) = [0, 0, · · · , 1] as the stationary distribution.The appendix first writes formula (17) as Q_t^c(x_0) = γ_t c(K + 1).Source: Appendix B · p. 13