PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffsound: Discrete Diffusion Model for Text-to-sound Generation

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, Dong Yu

arXiv:2207.09983v2cs.SDcs.AIeess.AS

TL;DR

Text-to-sound generation has limited prior research despite its importance for producing desired sound effects. The paper introduces a framework with Diffsound, a discrete-diffusion non-autoregressive decoder, and reports better generation performance and speed than an AR decoder. The framework is not end-to-end because its VQ-VAE, token decoder, and vocoder are trained separately.

  • Problem

    Text-to-sound generation has few prior studies, while generating desired sound effects is important for applications such as virtual reality, games, and production.

  • Method

    The paper combines a text encoder, VQ-VAE, token-decoder, and vocoder, and uses Diffsound with AudioSet pre-training and MBTG-based text-audio construction.

  • Results

    Diffsound achieves better performance than the AR token-decoder, including FID 9.76 v.s 16.87, KL 4.21 v.s 5.31, and CIDEr 0.36 v.s 0.22.

  • Takeaways & Limitations

    Objective and subjective metrics indicate that Diffsound improves text-to-sound generation performance and speed compared with the AR token-decoder.

  • Takeaways & Limitations

    The framework is not end-to-end because the VQ-VAE, token decoder, and vocoder are trained separately, which may not be optimal.

Abstract

from arXiv · show

Generating sound effects that humans want is an important topic. However, there are few studies in this area for sound generation. In this study, we investigate generating sound conditioned on a text prompt and propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a decoder, and a vocoder. The framework first uses the decoder to transfer the text features extracted from the text encoder to a mel-spectrogram with the help of VQ-VAE, and then the vocoder is used to transform the generated mel-spectrogram into a waveform. We found that the decoder significantly influences the generation performance. Thus, we focus on designing a good decoder in this study. We begin with the traditional autoregressive decoder, which has been proved as a state-of-the-art method in previous sound generation works. However, the AR decoder always predicts the mel-spectrogram tokens one by one in order, which introduces the unidirectional bias and accumulation of errors problems. Moreover, with the AR decoder, the sound generation time increases linearly with the sound duration. To overcome the shortcomings introduced by AR decoders, we propose a non-autoregressive decoder based on the discrete diffusion model, named Diffsound. Specifically, the Diffsound predicts all of the mel-spectrogram tokens in one step and then refines the predicted tokens in the next step, so the best-predicted results can be obtained after several steps. Our experiments show that our proposed Diffsound not only produces better text-to-sound generation results when compared with the AR decoder but also has a faster generation speed, e.g., MOS: 3.56 \textit{v.s} 2.786, and the generation speed is five times faster than the AR decoder.

I. INTRODUCTION

The paper addresses the limited research on text-conditioned sound generation by introducing a framework and Diffsound, a non-autoregressive token-decoder designed to improve over AR decoding.

  • Motivation: Text-to-sound generation has broad applications, but published work on generating sound from text descriptions was very limited.The paper motivates the task through applications including virtual reality, games, music, and background sound for speech synthesis.
  • Limitations of AR decoding: AR decoding predicts mel-spectrogram tokens sequentially, creating unidirectional bias and accumulated prediction errors while increasing generation time with sound duration.Information about a sound event may come from both left and right context, while earlier errors can propagate to later tokens.
  • Framework: The framework combines a text encoder, VQ-VAE, token-decoder, and vocoder to convert text input into waveform audio.The text encoder extracts features, the token-decoder generates mel-spectrogram tokens, and the vocoder transforms the generated mel-spectrogram into waveform.
  • Diffsound: Diffsound predicts all mel-spectrogram tokens simultaneously and iteratively revises them using contextual information from previous steps.It is a non-autoregressive token-decoder based on a discrete diffusion model and is intended to alleviate AR decoding weaknesses.
  • Data and pre-training: The paper proposes MBTG to build a larger text-audio dataset from AudioSet and pre-trains Diffsound before fine-tuning it on a small text-audio dataset.The pre-training strategy also uses curriculum learning, beginning with single-event clips and gradually adding multi-event clips.
  • Evaluation: Three objective metrics are introduced to assess generated-sound relevance and fidelity alongside MOS-based human evaluation.The metrics are FID, KL-divergence, and audio caption loss.

III. PROPOSED TEXT-TO-SOUND FRAMEWORK

The proposed text-to-sound framework uses text features, discrete mel-spectrogram representations, token decoding, and waveform reconstruction. VQ-VAE converts mel-spectrograms into discrete tokens, while the text encoder extracts sound-event information for generation.

  • The framework consists of a text encoder, VQ-VAE, token-decoder, and vocoder.
  • The vocoder transforms generated mel-spectrograms into waveforms.The framework uses a vocoder after token decoding and spectrogram reconstruction.
  • The text encoder extracts sound-event information from descriptions while excluding unrelated information.The study evaluates pretrained BERT and CLIP text encoders and reports better generation performance with CLIP.
  • VQ-VAE transfers mel-spectrogram generation into prediction of a sequence of discrete tokens.A mel-spectrogram is approximated by tokens retrieved from a finite codebook.
  • The VQ-VAE approximately reconstructs mel-spectrograms by decoding quantized codebook representations.The encoder, decoder, and codebook are trained end-to-end, with an added patch-based adversarial loss to preserve reconstruction quality after upsampling.

C. Token-decoder

The token-decoder maps text features to discrete mel-spectrogram tokens, with an autoregressive baseline and a diffusion-based alternative. Diffsound is introduced to address autoregressive error accumulation and unidirectional context limitations.

  • The token-decoder transfers text features into a discrete mel-spectrogram token sequence.
  • The autoregressive decoder predicts each token from preceding tokens and trains with cross-entropy and teacher forcing.During inference, previously predicted tokens replace the ground-truth tokens used during training.
  • Autoregressive decoding can suffer from training–inference mismatch, accumulated errors, and unidirectional bias.The current token depends only on previous tokens, so future context is ignored.
  • Diffsound is a non-autoregressive token-decoder based on a discrete diffusion model.The paper presents it as an alternative designed to address the weaknesses of autoregressive decoding.
  • Continuous diffusion corrupts data through a forward process and learns a reverse process that recovers the original data.The forward process uses Gaussian noise and the reverse process uses learnable parameters.

2) Discrete Diffusion model:

Discrete diffusion adapts diffusion processes to mel-spectrogram tokens, which take categorical rather than continuous values. Transition matrices define token corruption and allow distributions at arbitrary timesteps to be computed.

  • Gaussian corruption cannot directly operate on mel-spectrogram tokens because their values are discrete categories.
  • Discrete diffusion uses transition probability matrices to describe how categorical tokens change during each forward step.The token sequence is represented as a vector of discrete elements, and each matrix specifies transition probabilities between categories.
  • One-hot encoding and matrix multiplication give the categorical distribution of a token at the next timestep.The function c maps a scalar token to a one-hot column vector, and Qt c(xt−1) gives the distribution over xt.
  • The Markov property permits direct computation of q(xt|x0) using the product of transition matrices through timestep t.The resulting expression is q(xt|x0) = c^T(xt)Qt c(x0).
  • The reverse posterior q(xt−1|xt,x0) is computed from the forward-process distributions.

B. Non-autoregressive Mel-spectrograms Generation Via Diffsound

Diffsound generates all mel-spectrogram tokens simultaneously and refines them over diffusion iterations, using transition matrices, denoising objectives, and accelerated inference. The design combines masking and uniform resampling to support contextual token recovery.

  • Non-autoregressive generation: Diffsound predicts all mel-spectrogram tokens simultaneously and refines them across multiple iterations.Unlike an autoregressive decoder requiring N token steps, Diffsound uses T diffusion steps and can revise any token using all-token context.
  • Diffusion process: The model corrupts the original token sequence into a meaningless stationary distribution, then learns to recover it from text conditioning.
  • Transition matrices: Uniform transition matrices resample tokens across categories, while mask transition matrices add [MASK] as an absorbing state.
  • Transition matrices: The combined mask-and-uniform transition matrix masks tokens, uniformly resamples categories, or leaves tokens unchanged.The design assigns probabilities to [MASK], uniform resampling, and retaining the current token.
  • Training objective: The model is trained to estimate posterior transitions and minimize a variational lower bound, with an auxiliary noiseless-token objective.The final loss combines Lx0 and Lvlb, and experiments indicate that this combination improves performance.
  • Pre-training: Pre-training on AudioSet uses mask-based text generation to create text descriptions from event labels before fine-tuning on smaller text-audio data.
  • Inference: Fast inference uses a larger timestep stride to reduce sampling steps while ensuring the final step is x0.The authors report improved efficiency with only a small quality decrease.

C. Pre-training Diffsound on AudioSet dataset

The paper proposes mask-based text generation to turn AudioSet event labels into text descriptions, enabling construction of a large-scale text-audio dataset for pre-training.

  • Mask-based text generation creates text descriptions from AudioSet event labels.Examples express event identity and temporal relationships in natural-language descriptions.
  • The generated descriptions can represent different relationships between the same labeled events.For example, descriptions can indicate simultaneous events or sequential events.
  • The strategy supports building large-scale text-audio data from AudioSet despite its event-label-only annotations.

V. DATASET AND DATA PRE-PROCESSING

The experiments use the AudioSet and AudioCaps datasets and describe their preprocessing before model evaluation.

  • AudioSet and AudioCaps are used in the experiments.
  • The dataset section introduces the two datasets before discussing data preprocessing.
  • The datasets provide the experimental data for the paper’s text-to-sound generation models.

A. AudioSet

AudioSet supplies large-scale labeled audio, while AudioCaps provides human captions for training and evaluation; the study also defines its objective and human assessments.

  • AudioSet: AudioSet contains more than 2 million 10-second YouTube audio snippets organized across 527 sound classes.
  • AudioSet: Each AudioSet clip may carry one or more labels for the audio events it presents.
  • AudioCaps: AudioCaps contains around 50k AudioSet-sourced clips with human captions and separate training, validation, and test sets.
  • Preprocessing: All clips are resampled to 22.05k Hz, padded to 10 seconds, and represented as 860 × 80 log mel-spectrograms.
  • Evaluation: The study evaluates generated sound with human Mean Opinion Scores and objective assessment metrics.
  • Evaluation: Objective evaluation uses FID, KL-divergence, and audio caption loss to assess generated-sound quality.FID measures feature-distribution distance, while KL-divergence compares predicted event-probability distributions.

C. The Effectiveness of FID and KL

The paper tests whether FID and KL respond to controlled degradations while describing the separately trained model components and comparison settings.

  • Effectiveness of FID and KL: FID and KL scores increase as Gaussian noise, masked audio content, or interfering sound increases.These controlled tests examine sensitivity to noise, missing acoustic information, and irrelevant acoustic content.
  • Model training: The framework trains the text encoder, VQ-VAE, vocoder, and token-decoder separately rather than end-to-end.The text encoder and VQ-VAE are fixed when training the token-decoder.
  • VQ-VAE: VQ-VAE converts an 860 × 80 spectrogram into 53 × 5 tokens using a 256-vector-dimensional codebook of size 256.
  • Token decoders: The autoregressive token-decoder uses a 19-layer, 16-head transformer with dimension 1024.
  • Token decoders: Diffsound uses a similarly sized transformer with full attention, text cross-attention, feedforward blocks, and timestep-conditioned AdaLN.
  • Vocoder: The vocoder uses MelGAN and is trained on 40% of AudioSet because of time-complexity considerations.

5) The duration of the generated sound:

The experiments fix generated sound duration at 10 seconds and compare AR and Diffsound using subjective, objective, speed, and visualization analyses. Diffsound achieves stronger quality and relevance results while generating mel-spectrograms faster than the AR decoder.

  • Experimental setup: 10 seconds: generated sound duration is fixed for fair comparison, producing 265 mel-spectrogram tokens for both decoders.The setup uses 80 × 860 mel-spectrograms with VQ-VAE downsampling.
  • Subjective and objective metrics: FID 9.76 v.s 16.87, KL 4.21 v.s 5.31, and CIDEr 0.36 v.s 0.22: Diffsound outperforms AR across objective metrics.Both objective and subjective metrics indicate Diffsound’s effectiveness.
  • The generation speed: 23 seconds v.s 5 seconds: Diffsound generates a mel-spectrogram substantially faster than the AR token-decoder.With T = 25 and Δt = 7, Diffsound reaches 0.53 seconds, 43 times faster than AR with a similar FID score.
  • Visualization: Complete semantics: Diffsound includes events such as a man speaking and crickets singing when AR omits the cricket event.The figure also indicates better detailed modelling ability for Diffsound in other examples.

B. Ablation study for Diffsound model

Ablation studies examine transition matrices, mask rates, timestep counts, pre-training, and sample selection. Combined transitions, suitable mask rates, more timesteps, AudioSet pre-training, and relevance-based selection each affect performance or efficiency.

  • Transition matrices: Mask and uniform transition matrices achieve the best results when γT = 0.9.The combined matrix outperforms the uniform matrix, while γT = 0.9 outperforms other tested settings.
  • Training and inference steps: More training and inference timesteps improve performance but increase inference time; good performance remains after dropping 75% of inference steps.The study sets a maximum of 100 timesteps for training and inference when considering generation speed.
  • Pre-training: AudioSet pre-training lowers FID from 9.76 to 8.78 and KL from 4.21 to 4.15.The experiments use about 45% of the AudioSet training set because of time complexity.
  • Sample selection: Ranking 10 generated samples by SPICE and CIDEr allows retaining a smaller subset of high-relevance sounds.Table VII evaluates retaining the top K samples according to these scores.
  • Conclusion and limitation: The framework’s objective and subjective metrics verify Diffsound’s effectiveness, while the overall system remains non-end-to-end.The VQ-VAE, token decoder, and vocoder are trained separately, which may not be optimal.

APPENDIX A THE PROOF OF FORMULA (14)

Appendix A proves formula (14) by mathematical induction for the discrete diffusion transition process. As the timestep grows, the resulting distribution approaches a uniform stationary distribution.

  • Transition parameters: The transition parameters include βt = (1 − αt)/K.This parameterization appears in the discrete diffusion process used by the proof.
  • Inductive proof: Formula (14) is established by verifying the base case at t = 1 and assuming the result at step t.The induction then considers t + 1.
  • Inductive step: At step t + 1, the proof separately handles x = x0 and x ≠ x0 cases.The two cases derive the corresponding transition expressions.
  • Stationary distribution: When t is sufficiently large, αt approaches 0 and formula (14) becomes Qtc(x0) = 1/K.This yields the stationary distribution p(xT) = [1/K, 1/K, · · ·, 1/K].

APPENDIX B THE PROOF OF FORMULA (17)

Appendix B proves formula (17) by induction for a transition process involving the original token and an additional state K + 1. At large timesteps, the process converges to that additional state.

  • Inductive proof: Formula (17) is proved by mathematical induction using γt ∈ [0, 1] and βt = 1 − γt.The proof begins from the stated conditional information and verifies the induction structure.
  • Inductive step: The induction step considers x = x0 and x = K + 1 at timestep t + 1.These cases account for the transitions involving the original token and the additional state.
  • Stationary distribution: When t is sufficiently large, βt approaches 0 and formula (17) becomes Qtc(x0) = γtc(K + 1), with γt = 1.The resulting stationary distribution is p(xT) = [0, 0, · · ·, 1].
Loading 2207.09983v2…