PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Zhou Zhao

arXiv:2105.02446v6eess.AScs.LGcs.SD

TL;DR

Existing SVS acoustic models struggle with over-smoothing or unstable adversarial training. DiffSinger uses diffusion modeling with an adaptive shallow-diffusion start, and evaluations report improvements over prior SVS systems plus generalization to TTS.

  • Problem

    Simple-loss SVS models produce over-smoothed outputs, while GAN-based models can train unstably, hindering singing naturalness.

  • Method

    DiffSinger conditions a diffusion-based acoustic model on music scores and begins reverse denoising at an adaptively predicted shallow intersection step.

  • Results

    DiffSinger outperforms prior SVS work, while DiffSpeech generalizes the methods to TTS; shallow diffusion yields 45.1% speedup in SVS.

  • Takeaways & Limitations

    Diffusion modeling provides a stable SVS acoustic-modeling approach, and shallow diffusion improves quality while reducing inference cost.

  • Takeaways & Limitations

    The decoder prior does not maintain variable aperiodic parameters, although it usually preserves a ground-truth-matching harmonic skeleton.

Abstract

from arXiv · show

Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing. In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs. To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively. The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger. The old title of this work: "Diffsinger: Diffusion acoustic model for singing voice synthesis".

1 Introduction

DiffSinger applies diffusion modeling to SVS to address over-smoothing and unstable GAN training, then accelerates and improves generation with shallow diffusion. Experiments report stronger SVS performance and generalization to TTS.

  • Previous SVS acoustic models face over-smoothing from simple losses and unstable training from GAN-based reconstruction, limiting synthesized singing naturalness.
  • Shallow diffusion starts denoising from the intersection of ground-truth and decoder-predicted diffusion trajectories, using boundary prediction to select the step adaptively.
  • DiffSinger evaluations on a Chinese singing dataset outperform prior SVS work, while DiffSpeech experiments demonstrate generalization to TTS.The reported SVS result includes 0.11 MOS gains over a state-of-the-art acoustic model; shallow diffusion adds 0.14 MOS, 0.5 CMOS, and 45.1% speedup.
  • DiffSinger is an SVS acoustic model based on diffusion probabilistic modeling, which converts noise into mel-spectrograms conditioned on a music score.
  • The shallow diffusion mechanism improves voice quality and accelerates inference by starting from a better point than Gaussian white noise.

2 Diffusion Model

Diffusion models gradually transform data into Gaussian noise, then learn a reverse Markov process to reconstruct data. Their closed-form noising process and variational training support efficient, stable learning.

  • Diffusion Process: A diffusion model gradually converts data into Gaussian noise through a fixed-parameter Markov chain and learns a reverse process to restore the data.The forward process adds Gaussian noise over T steps; the reverse process uses learnable parameters.
  • Diffusion Process: If the variance schedule is well designed and T is sufficiently large, the final noisy distribution approaches isotropic Gaussian noise.
  • Diffusion Process: The conditional distribution q(y_t|y_0) can be calculated in closed form in O(1) time, enabling direct sampling at arbitrary diffusion steps.The formulation uses α_t := 1 − β_t and the cumulative product ᾱ_t.
  • Reverse Process: Because the exact reverse transition is intractable, a neural network with shared parameters θ approximates each reverse transition.
  • Training: Training minimizes a variational bound on negative log likelihood and efficiently optimizes a randomly selected loss term with stochastic gradient descent.
  • Conditional SVS: DiffSinger conditions the diffusion denoiser on the music score while operating on mel-spectrograms at each diffusion step.

3 DiffSinger

DiffSinger applies conditional diffusion to singing voice synthesis and uses a shallow start point to exploit the auxiliary decoder’s prior while reducing reverse-process work. A boundary predictor adaptively selects that start point.

  • 3 DiffSinger: DiffSinger models the conditional distribution of mel-spectrograms given a music score and includes naive diffusion, shallow diffusion, and adaptive boundary prediction.
  • Naive DiffSinger: The naive model trains on noisy mel-spectrograms to predict diffusion noise and begins inference from Gaussian white noise, iterating T denoising steps.
  • Motivation: The L1-trained auxiliary decoder produces blurry mel-spectrograms that retain a skeleton matching the ground truth but omit variable aperiodic details.
  • Trajectory Intersection: The diffusion trajectories from auxiliary-decoder and ground-truth mel-spectrograms intersect at sufficiently large but nonterminal steps.
  • Shallow Diffusion Mechanism: Shallow diffusion starts reverse denoising from f M_k at the trajectory intersection rather than Gaussian noise, requiring k iterations where k < T.The auxiliary decoder generates f M, which is noised to step k before reverse denoising begins.
  • Boundary Prediction: A boundary predictor classifies noisy samples as originating from M or f M and uses cross-entropy training to locate their intersection adaptively.
  • Boundary Prediction: The boundary is estimated from the earliest step whose later-step predictions satisfy a 95% threshold criterion, then averaged across training examples.

1 repeat

The training and inference procedures sample score–spectrogram pairs, generate auxiliary predictions, and use a selected shallow boundary to initialize denoising. Boundary selection can also be treated as preprocessing or tuned manually.

  • Training Procedure: Training samples a music-score and mel-spectrogram pair from the training set before applying gradient descent.
  • Boundary Selection: Boundary prediction can be treated as dataset preprocessing for choosing k across the dataset, while k can alternatively be selected by validation-set brute-force search.
  • Encoder: The encoder combines lyric and pitch embeddings with duration-based length regulation to produce a condition sequence aligned to mel-spectrogram length.
  • Inference Procedure: Inference samples a music score, generates f M with the auxiliary decoder, and uses the selected intersection boundary k for shallow diffusion.
  • Inference Procedure: The intermediate sample is formed as f M_k(f M, ϵ) = √ᾱ_k f M + √(1 − ᾱ_k)ϵ.

10 end

DiffSinger uses explicit embeddings and neural modules to condition denoising on diffusion steps and music-score features. Its auxiliary decoder produces the initial mel-spectrogram estimate, while a ResNet classifier predicts the boundary.

  • Step Embedding: A sinusoidal embedding followed by two linear layers converts the discrete diffusion step t into a continuous C-channel representation E_t.
  • Auxiliary Decoder: The auxiliary decoder uses stacked feed-forward Transformer blocks to generate f M as its final mel-spectrogram output.
  • Denoiser: The denoiser predicts diffusion noise from M_t, conditioned on the step embedding and music condition sequence, using a non-causal WaveNet architecture.
  • Boundary Predictor: The boundary-predictor classifier combines step embeddings, a ResNet with stacked convolutional layers, and a linear layer to classify M_t versus f M_t.

4 Experiments

Experiments evaluate DiffSinger on Chinese singing and extend it to TTS, measuring perceptual quality, spectrogram detail, inference speed, and shallow-diffusion effectiveness. DiffSinger improves over prior SVS systems and its shallow mechanism improves quality while accelerating inference.

  • Experimental Setup: The evaluation uses PopCS, a 5.89-hour Chinese Mandarin pop-song dataset recorded by a qualified female vocalist, with 24kHz studio audio.
  • Audio Performance: DiffSinger outperforms the simple-loss FFT-Singer baseline and the state-of-the-art GAN-Singer in song-sample MOS evaluation.GT (Mel + PWG) achieves 4.04 ± 0.11 as the acoustic-model upper limit.
  • Audio Performance: DiffSinger and GAN-Singer preserve more delicate between-harmonic details than FFT-Singer, while DiffSinger is more competitive at mid and low frequencies.DiffSinger maintains similar high-frequency quality to GAN-Singer.
  • Audio Performance: 45.1% inference speedup is achieved with shallow diffusion, reducing RTF from 0.348 to 0.191.RTF denotes the seconds required to generate one second of audio.
  • Ablation Studies: Removing shallow diffusion causes a -0.500 CMOS quality drop, supporting the mechanism’s effectiveness in ablation studies.
  • Extensional Experiments on TTS: On LJSpeech, DiffSpeech outperforms FastSpeech 2 and Glow-TTS, while shallow diffusion provides a 29.2% speedup from RTF 0.171 to 0.121.The speech evaluation uses ten Amazon Mechanical Turk testers.

5 Related Work

Related work spans early concatenative and HMM-based SVS, neural and adversarial SVS systems, and diffusion models for image and waveform generation. DiffSinger applies diffusion modeling to conditional acoustic modeling for SVS.

  • SVS Systems: Early SVS systems used concatenated or HMM-based parametric methods, which were cumbersome and lacked flexibility and harmony.
  • SVS Systems: Deep neural SVS methods introduced feed-forward Transformer and adversarial approaches for faster inference, end-to-end spectrogram generation, and improved voice quality.
  • Diffusion Models: Diffusion models are parameterized Markov chains trained by variational lower-bound optimization and have been applied to image generation and neural vocoders.
  • Diffusion Models: DiffSinger extends diffusion modeling to acoustic-feature generation conditioned on a music score for singing voice synthesis.

6 Conclusion

The conclusion presents DiffSinger and shallow diffusion as effective for SVS, reports successful extension to TTS, and identifies direct vocoder-free synthesis as future work.

  • DiffSinger uses a diffusion probabilistic model for singing voice synthesis and shallow diffusion to improve voice quality and speed up inference.
  • Starting reverse diffusion at the intersection of ground-truth and auxiliary-decoder trajectories alleviates the reverse-process burden.
  • Experiments on PopCS support DiffSinger’s superiority over previous works and the effectiveness of shallow diffusion, while LJSpeech experiments support DiffSpeech on TTS.
  • Direct synthesis without a vocoder remains future work.

A Theoretical Proof of Intersection

The theoretical proof shows that the divergence between diffused ground-truth and auxiliary-decoder mel-spectrograms decreases rapidly with diffusion step. This supports locating an intersection and selecting a shallow reverse-process start point.

  • The forward diffusion distributions for the ground-truth mel-spectrogram and auxiliary prediction are modeled as Gaussian conditionals at step t.
  • The KL-divergence between these distributions decreases rapidly as t increases, guaranteeing an intersection of their diffusion trajectories.
  • Simple reconstruction training minimizes the auxiliary decoder’s distance to the ground truth, facilitating trajectory intersection.
  • The auxiliary diffused sample need only lie near the mode of the ground-truth distribution rather than exactly match it.
  • The smallest step k satisfying the proposed condition can be used as the shallow start point, with a TTS-specific choice based on ground-truth F0 and duration.

D Model Size

DiffSinger has a model footprint similar to other state-of-the-art comparison systems. Its architecture includes a music-score-conditioned Encoder and Auxiliary Decoder, plus a Denoiser.

  • DiffSinger has a similar number of learnable parameters to other state-of-the-art comparison systems.
  • The Encoder and Auxiliary Decoder process the music score and auxiliary mel-spectrogram information within the model structure.The music score is x, the music condition sequence is E_m, and the auxiliary decoder generates a blurry mel-spectrogram using L1 loss.
  • The Denoiser uses step embeddings and music-condition sequences in a simplified non-causal WaveNet-derived structure.Its residual-layer count is denoted by N, and naive convolution replaces the dilation layer.

E Details of Training and Inference

DiffSinger is trained with Adam on one NVIDIA V100 GPU using a 48-sample batch and a learning rate of 10^-3. Training takes about 28 hours across warmup and main stages, while inference achieves the reported SVS and TTS real-time factors.

  • Training: DiffSinger is trained on 1 NVIDIA V100 GPU with batch size 48 using Adam at learning rate lr = 10^-3.
  • Training: The warmup stage takes about 16 hours, and the main training stage takes about 12 hours.
  • Inference: The acoustic model reaches RTFs of 0.191 for SVS and 0.121 for TTS during inference.
  • Model footprint: In the model-footprint table, Param denotes the number of learnable parameters.
Loading 2105.02446v6…