PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural source-filter-based waveform model for statistical parametric speech synthesis

Xin Wang, Shinji Takaki, Junichi Yamagishi

arXiv:1810.11946v4eess.AScs.SDstat.ML

TL;DR

WaveNet's autoregressive generation is slow, while faster non-autoregressive alternatives can require complex distillation and mixed training criteria. The paper proposes a sine-excitation source-filter model trained with spectral criteria. Experiments report much faster generation with speech quality similar to AR WaveNet.

  • Problem

    Autoregressive WaveNet generates waveform samples one by one and is slow; faster non-autoregressive alternatives use distillation and mixed criteria that raise implementation cost and reduce interpretability.

    Source: §1 · p. 1

  • Method

    A source module generates sine-wave excitation at specified F0, a dilated-convolution filter transforms it into speech, and training minimizes spectral amplitude and phase distances with SGD.

    Source: §1 · p. 1, §2.1.3 · p. 2, §2.2 · p. 2

  • Results

    The proposed model generated speech with quality similar to AR WaveNet at much faster speed.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Sine-wave excitation was essential for harmonic structure, while multiple spectral criteria and the filter transformation contributed to model performance.

    Source: §4 · p. 4

    When converting linguistic features to waveforms, the model showed high-frequency oversmoothing, which the authors leave for future investigation.

    Source: §4 · p. 4

Abstract

from arXiv · show

Neural waveform models such as the WaveNet are used in many recent text-to-speech systems, but the original WaveNet is quite slow in waveform generation because of its autoregressive (AR) structure. Although faster non-AR models were recently reported, they may be prohibitively complicated due to the use of a distilling training method and the blend of other disparate training criteria. This study proposes a non-AR neural source-filter waveform model that can be directly trained using spectrum-based training criteria and the stochastic gradient descent method. Given the input acoustic features, the proposed model first uses a source module to generate a sine-based excitation signal and then uses a filter module to transform the excitation signal into the output speech waveform. Our experiments demonstrated that the proposed model generated waveforms at least 100 times faster than the AR WaveNet and the quality of its synthetic speech is close to that of speech generated by the AR WaveNet. Ablation test results showed that both the sine-wave excitation signal and the spectrum-based training criteria were essential to the performance of the proposed model.

1. INTRODUCTION

WaveNet generation is slow because it is autoregressive, while earlier faster non-autoregressive approaches rely on complex distillation setups. The paper proposes a sine-excitation source-filter model trained with spectral criteria, reporting faster generation and quality close to WaveNet.

1. INTRODUCTION

  • WaveNet directly models waveform-sample distributions and showed outstanding performance, but its autoregressive generation is slow.It generates waveform samples one by one.Source: §1 · p. 1
  • Parallel WaveNet and ClariNet use distillation from a teacher WaveNet, and combining large models with mixed criteria raises implementation cost and reduces interpretability.Source: §1 · p. 1
  • The proposed source module generates sine-based excitation at specified F0; a dilated-convolution filter transforms it into speech, trained with spectral amplitude and phase distances.The criteria are efficiently implemented using discrete Fourier transforms.Source: §1 · p. 1
  • The model generated waveforms much faster than AR WaveNet, achieved close MOS quality, and ablations found sine excitation and spectral amplitude distance crucial.Source: §1 · p. 1

2. PROPOSED MODEL AND TRAINING CRITERIA

The model maps acoustic features to a waveform through separate source and filter modules, with a condition module supplying inputs to both. It generates an F0-conditioned sine excitation, transforms it with dilated convolutions, and trains using spectral amplitude and phase distances computed via DFTs.

2.1. Model structure

  • The model converts acoustic features into a waveform through source and filter modules, while a condition module processes features for both.The modules do not take previously generated waveform samples as input.Source: §2.1 · p. 1

2.1.1. Condition module

  • The condition module sends upsampled F0 to the source and processed, upsampled acoustic features to the filter.Its bidirectional LSTM and convolutional layer can be replaced with a feedforward layer in practice.Source: §2.1.1 · p. 1

2.1.2. Source module

  • The source module generates a sine-based excitation from F0, using Gaussian noise for unvoiced frames and a random initial phase.The paper also tests a training-selected phase and additional harmonics; generation uses a randomly generated phase.Source: §2.1.2 · pp. 1–2

2.1.3. Neural filter module

  • The filter modulates the excitation through multiple dilated-convolution and affine-transformation stages to produce the waveform, without distillation.The first stage produces signals a and b and transforms the excitation as e ⊙ b + a.Source: §2.1.3 · p. 2

2.2. Training criteria in frequency domain

  • Training minimizes spectral amplitude and phase distances, which are implemented using DFTs and optimized with stochastic gradient descent.Multiple spectral losses can be combined into the ultimate training criterion.Source: §2.2 · p. 2, §2.2.2 · p. 3

2.2.1. Spectral amplitude distance

  • Gradients for spectral-amplitude loss can be computed across framing and DFT configurations and summed; three amplitude losses were found essential.The configurations specify framing and DFT settings for spectral-amplitude and phase losses.Source: §2.2.1 · p. 3, Table 1 · p. 3

2.2.2. Phase distance

  • Phase distance is computed from the spectra, and multiple phase and amplitude losses with different framing and DFT configurations can be added together.The corresponding gradients can be computed by the same procedure as for spectral-amplitude distance.Source: §2.2.2 · p. 3

3. EXPERIMENTS

Across experiments, NSF was slightly below WAD in MOS but much faster, while ablations showed that its spectral training criteria and sine-wave excitation mattered to speech quality.

3.1. Corpus and features

  • Experiments used 15 hours of neutral female Japanese speech for training, 480 utterances each for validation and test, 48 kHz inputs, and 16 kHz model waveforms.The ablation training set was reduced to five hours; acoustic features included 60 MGC dimensions and one F0 dimension at a 5 ms frame shift.Source: §3.1 · p. 3

3.2. Comparison of proposed model, WaveNet, and WORLD

  • NSF outperformed WOR and WAC but was slightly below WAD, with an approximately 0.12 mean-MOS gap under both natural and generated acoustic features.Differences were significant at p < 0.01 except between NSF and WAC with generated features.Source: Figure 2 · p. 3, §3.2 · p. 3
  • NSF generated waveforms much faster than WAD in tests of 80 utterances around five seconds long, although memory-save mode incurred time-consuming memory operations.Normal mode could not generate waveforms longer than six seconds on a single GPU card.Source: Table 3 · p. 3, §3.2 · p. 4

3.3. Ablation test on proposed model

  • Using multiple spectral losses improved speech quality; the L3-only variant showed pulse-train noise, which adding Ls2 with a 5 ms frame length alleviated.Phase distance did not improve speech quality despite decreasing on training and validation data.Source: Figure 4 · p. 4, §3.3 · p. 4
  • A single sine-wave excitation with random initial phase also worked, while removing sine excitation left speech intelligible but without stable harmonic structure.N1 slightly outperformed NSFs, suggesting its skip-connection-equivalent transformation may help training.Source: §3.3 · p. 4

4. CONCLUSION

The conclusion describes a neural waveform model with separate sine-excitation source and dilated-convolution filter modules, reporting similar speech quality to AR WaveNet at much faster speed. It also notes high-frequency oversmoothing when converting linguistic features to waveforms.

4. CONCLUSION

  • The proposed model generated speech with similar quality to AR WaveNet at a much faster speed.Its source module produces sine-wave excitation with specified F0, and its filter module uses dilated convolution to transform it into a waveform.Source: §4 · p. 4
  • Sine-wave excitation was essential for generating waveforms with harmonic structures.Multiple spectral-based training criteria and the filter-module transformation also contributed to model performance.Source: §4 · p. 4
  • When converting linguistic features into waveforms, the authors observed oversmoothing in the high-frequency band.They identify this issue as future work and suggest exploring simpler dilated-convolution blocks and classical excitation or multi-band approaches.Source: §4 · p. 4

A. FORWARD COMPUTATION

The forward computation derives spectra from generated waveforms through framing/windowing and a DFT. The passages identify waveform-frame and spectrum representations, while the figure caption specifies the relevant lengths and DFT-bin count.

A. FORWARD COMPUTATION

  • The method denotes the n-th waveform frame of length M as a vector in R^M.The passage introduces a corresponding frame representation for the generated waveform.Source: A. FORWARD COMPUTATION · p. 6
  • The spectrum of a generated frame is represented as a vector in C^K and calculated using a K-point DFT.For FFT computation, K is set to a power of 2 and the frame is zero-padded to length K.Source: A. FORWARD COMPUTATION · p. 6
  • Figure 5 presents framing/windowing followed by DFT as the two steps for deriving a spectrum from the generated waveform.The caption defines T, M, and K as waveform length, frame length, and number of DFT bins.Source: Figure 5 · p. 6, A. FORWARD COMPUTATION · p. 6

A.1. Framing and windowing

The framing and windowing operation is parallelized and can be described as a matrix operation. Figure 6 identifies the coefficients as those of a Hann window.

A.1. Framing and windowing

  • The framing and windowing operation is parallelized over the generated waveform frames using a CUDA/Thrust for-each command.Source: §A.1 · p. 6
  • For explanation, the framing and windowing computation is represented as a matrix operation in Figure 6.Source: §A.1 · p. 6, Figure 6 · p. 6
  • The transformation matrix is described by its row and column indices, and Figure 6 identifies w1:M as Hann-window coefficients.Source: §A.1 · p. 6, Figure 6 · p. 6

A.2. DFT

The implementation uses cuFFT to compute the DFT outputs in parallel.

A.2. DFT

  • cuFFT computes the set of DFT outputs in parallel.Source: §A.2 · p. 6
  • The implementation names cufftExecR2C and cufftPlan1d as the cuFFT functions used for the parallel computation.Source: §A.2 · p. 6
  • The parallel cuFFT computation produces {y(1), ···, y(N)}.Source: §A.2 · p. 6

B. BACKWARD COMPUTATION

The backward computation obtains the waveform-sample gradient through the steps in Figure 7, including a step that maps frame-level gradients back through framing and windowing.

B. BACKWARD COMPUTATION

B.1. The 2nd step: from ∂L

  • The second step relates each output-sample gradient to frame-level gradients through the framing and windowing coefficients.The coefficients determine how each output sample relates to the framed signals.Source: §B.1 · p. 7
  • Each output sample appears in only a few frames, so its gradient can be computed over that frame range using its position in each frame.The implementation launches T CUDA/Thrust threads to compute the output-sample gradients in parallel.Source: §B.1 · p. 7

B.2. The 1st step: compute ∂L

This section computes the loss gradient with respect to framed signals from a log spectral amplitude distance, then uses it in the backward computation.

B.2. The 1st step: compute ∂L

  • The gradient computation sums over frequency-bin index k, with the stated range k ∈ [1, K].The text notes that a sum term can be safely ignored.Source: §B.2 · p. 7
  • Zero-padding can be avoided by setting K = M, but K is usually a power of 2 while frame length M is not.The passage gives this as the practical reason zero-padding is used.Source: §B.2 · p. 7
  • The log spectral amplitude distance is computed over N frames, and its gradient with respect to each framed signal is obtained using the chain rule.The section describes this as the first step in computing the gradient.Source: §B.2 · p. 8

B.3. Implementation of the 1st step using inverse DFT

The section implements the first gradient-computation step with an inverse DFT, using conjugate symmetry to obtain real-valued results and support common distance metrics.

B.3. Implementation of the 1st step using inverse DFT

  • A K-point inverse DFT maps the complex-valued signal g to b, and the implementation uses inverse DFT rather than direct matrix multiplication.The inverse DFT can be executed for all N frames in parallel.Source: §B.3 · pp. 8–9
  • The inverse DFT output is real-valued when the real components of paired frequency bins match and their imaginary components are opposites, with the specified endpoint imaginary parts zero.These conditions make the imaginary part of the inverse DFT expression zero.Source: §B.3 · p. 8
  • Conjugate symmetry makes the inverse DFT real-valued, allowing the implementation to omit imaginary-part computation and use cuFFT’s C2R command.The implementation also avoids computing and allocating memory for the redundant conjugate-symmetric frequency bins.Source: §B.3 · p. 9

B.4. Conjugate symmetry complex-valued gradient vector

  • Common distance metrics can be used when the loss is chosen to satisfy the conjugate-symmetry conditions.The text identifies this symmetry property as the requirement for the inverse-DFT implementation.Source: §B.4 · p. 9, §B.4.1 · p. 9

B.4.1. Log spectral amplitude distance

  • The log spectral amplitude distance produces gradients satisfying the conjugate-symmetry conditions used for the real-valued inverse DFT.The passage connects the conjugate symmetry of the real-signal DFT spectrum and its amplitude to those conditions.Source: §B.4.1 · p. 9

B.4.2. Phase distance

  • The phase distance is defined using the phases of the framed spectra, and its gradients satisfy the stated conjugate-symmetry conditions.The text attributes this property to the conjugate symmetry of both the signal and its DFT spectrum.Source: §B.4.2 · p. 10

C. MULTIPLE DISTANCE METRICS

Different distance metrics can be combined, even when they use different transform and framing settings, because their waveform gradients share a common real-valued dimension.

C. MULTIPLE DISTANCE METRICS

  • The combined loss can sum multiple log spectral amplitude and phase distances, including distances computed with different DFT-bin counts, frame lengths, or frame shifts.Figure 8 illustrates the use of multiple distances.Source: C. MULTIPLE DISTANCE METRICS · p. 11, Figure 8 · p. 11
  • Each metric’s gradient with respect to the output waveform is a real-valued vector of dimension T after de-framing and windowing.The framed-signal gradient dimensions may differ across metrics.Source: C. MULTIPLE DISTANCE METRICS · p. 11
  • The gradients from the different distances are then merged together.The section presents this as the final step after de-framing and windowing.Source: C. MULTIPLE DISTANCE METRICS · p. 11
Loading 1810.11946v4…