PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural source-filter waveform models for statistical parametric speech synthesis

Xin Wang, Shinji Takaki, Junichi Yamagishi

arXiv:1904.12088v2eess.AScs.SDstat.ML

TL;DR

AR waveform models generate samples sequentially, and IAF approaches shift sequential computation to training or require additional training criteria. This paper proposes NSF models built from a sine-based source, a dilated-convolution filter, and a conditioning module; hn-NSF matches WaveNet’s quality while generating much faster.

  • Problem

    AR models generate waveforms slowly through sequential sampling, while IAF approaches can require sequential training transformations or additional criteria and complex knowledge distillation.

    Source: §I · p. 1

  • Method

    NSF models combine a sine-based excitation source, a dilated-convolution filter, and a conditional module, and train using spectral-amplitude distances.

    Source: §I · p. 1, §III · p. 3, §V · p. 12

  • Results

    hn-NSF performed comparably well to WaveNet on a large single-speaker Japanese corpus and generated waveforms much faster.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    The hn-NSF’s F0 input to the source module allows easy control of the generated waveform’s pitch.

    Source: §V · p. 12

Abstract

from arXiv · show

Neural waveform models such as WaveNet have demonstrated better performance than conventional vocoders for statistical parametric speech synthesis. As an autoregressive (AR) model, WaveNet is limited by a slow sequential waveform generation process. Some new models that use the inverse-autoregressive flow (IAF) can generate a whole waveform in a one-shot manner. However, these IAF-based models require sequential transformation during training, which severely slows down the training speed. Other models such as Parallel WaveNet and ClariNet bring together the benefits of AR and IAF-based models and train an IAF model by transferring the knowledge from a pre-trained AR teacher to an IAF student without any sequential transformation. However, both models require additional training criteria, and their implementation is prohibitively complicated. We propose a framework for neural source-filter (NSF) waveform modeling without AR nor IAF-based approaches. This framework requires only three components for waveform generation: a source module that generates a sine-based signal as excitation, a non-AR dilated-convolution-based filter module that transforms the excitation into a waveform, and a conditional module that pre-processes the acoustic features for the source and filer modules. This framework minimizes spectral-amplitude distances for model training, which can be efficiently implemented by using short-time Fourier transform routines. Under this framework, we designed three NSF models and compared them with WaveNet. It was demonstrated that the NSF models generated waveforms at least 100 times faster than WaveNet, and the quality of the synthetic speech from the best NSF model was better than or equally good as that from WaveNet.

I. INTRODUCTION

Neural waveform models face a trade-off between slow autoregressive generation and costly or complicated IAF training. The paper introduces NSF models and reports that hn-NSF matches WaveNet’s quality while generating much faster.

I. INTRODUCTION

  • AR models generate waveform samples sequentially, while IAF models avoid sequential generation but require sequential waveform transformation during training; teacher-student approaches add training criteria and implementation complexity.The proposed NSF framework instead uses a sine-based source, a dilated-convolution filter, and a conditional module, trained with spectral-amplitude distances.Source: §I · p. 1
  • hn-NSF outperformed the other NSF models in a large-scale listening test and generated speech with comparably good quality but much faster than WaveNet.The hn-NSF uses separate source-filter pairs for harmonic and noise components.Source: §I · pp. 1–2

A. Naive neural waveform model

  • The naive upsample-and-predict model, trained with MSE under an independent-sample assumption, produced unintelligible, over-smoothed waveforms in pilot tests.The authors suggest this assumption may mismatch the strong temporal correlation of natural waveforms.Source: A. Naive neural waveform model · p. 2

B. Neural AR waveform models

AR models capture temporal dependence by conditioning on previous waveform samples, but generate samples sequentially. IAF-flow models generate waveforms without that sequential process, while their training requires sequential transformation.

B. Neural AR waveform models

  • AR models condition each waveform sample on previous samples and use teacher forcing in training, but generation remains sequential and takes O(T) time.WaveNet and SampleRNN are examples; other architectures reduce or parallelize computation but remain linearly proportional to waveform length.Source: B. Neural AR waveform models · p. 2

C. IAF-flow-based models

  • CNN-based AR training is O(1), whereas flow-model training is O(T), making their training and generation complexities dual with respect to waveform length.The cited comparison is theoretical and concerns time complexity with respect to T.Source: B. Neural AR waveform models · p. 2, C. IAF-flow-based models · p. 3
  • IAF-flow models transform Gaussian noise into a waveform without sequential generation, but training sequentially inverts the natural waveform into noise and takes O(T) time.The passage cites a Japanese-speech WaveGlow study that trained on four V100 GPUs for one month.Source: C. IAF-flow-based models · pp. 2–3

D. AR plus IAF

The paper contrasts the complications of AR-plus-IAF distillation with an NSF framework that uses a sine-based source, a dilated-convolution filter, and acoustic conditioning. NSF training uses spectral-amplitude distances computed from STFTs, with gradients that can be efficiently propagated.

D. AR plus IAF

  • Parallel WaveNet and ClariNet use a pre-trained AR teacher to evaluate an IAF student without sequential transformation, but require additional spectral criteria and are complicated to implement.The paper reports that knowledge distillation alone is insufficient for these models.Source: D. AR plus IAF · p. 3

III. NEURAL SOURCE-FILTER MODELS

  • The NSF framework combines acoustic-feature conditioning, sine-based excitation, and a dilated-convolution filter that transforms excitation into a waveform.The filter also uses feedforward transformation.Source: §III · p. 3
  • NSF models avoid AR and IAF approaches, use O(1) time complexity in both training and generation, and require no knowledge distillation.The framework trains with a spectral-domain criterion using stochastic gradient descent.Source: §III · p. 3

A. Training criterion based on spectral amplitude distances

B. Baseline NSF model

The b-NSF maps acoustic features to a waveform through conditioning, sine/noise excitation, and a dilated-convolution filter. Its source treats voiced and unvoiced regions differently, while the filter transforms excitation into the output waveform.

B. Baseline NSF model

  • The b-NSF converts input acoustic features into a waveform using condition, source, and filter modules.The condition module processes features for the source and filter; the source generates excitation that the filter transforms.Source: B. Baseline NSF model · p. 5, 1) Condition module · p. 5
  • Voiced regions use F0-based sine harmonics combined by a trainable feedforward layer, while unvoiced regions use Gaussian noise.The source constructs harmonic overtones from multiples of F0 before combining them into the excitation signal.Source: 2) Source module · p. 5
  • Five baseline dilated-convolution filter blocks transform excitation into the waveform, with ten dilated-convolution stages in each block.The block outputs feed the following block, and the final block produces the waveform.Source: 3) Neural filter module · p. 6
  • Although the filter layers can be non-causal without distillation or invertibility requirements, the experiments used causal layers to match WaveNet configurations.The passage says NSF does not need to compute signal distributions during forward propagation or make filter blocks invertible.Source: 3) Neural filter module · p. 6

C. Simplified NSF model

The s-NSF replaces the baseline filter blocks with simplified blocks that retain dilated convolutions, skip connections, and feedforward layers for dimension changes. This yields fewer parameters and faster generation than b-NSF.

C. Simplified NSF model

  • Simplified filter blocks retain dilated-convolution layers, skip connections, and feedforward layers for dimension changes.Their output adds a residual signal to the input rather than applying the baseline affine transformation.Source: C. Simplified NSF model · pp. 6–7
  • Compared with b-NSF, s-NSF has fewer parameters and a faster generation speed.The models share the same network structure except for the simplified filter blocks.Source: C. Simplified NSF model · p. 7
  • The cascade of simplified filter blocks forms a deep residual network.The residual-sum design was motivated by an ablation test on b-NSF.Source: C. Simplified NSF model · p. 7

D. Harmonic-plus-noise NSF model

The hn-NSF separates harmonic and noise components, addressing cases where b-NSF and s-NSF produce poor unvoiced sounds or silence for fricatives. It merges the components with voicing-specific filters, and its harmonic component resembles natural speech over the reported 0–7 kHz range.

D. Harmonic-plus-noise NSF model

  • b-NSF and s-NSF may produce low-quality unvoiced sounds or silence for fricatives, motivating different filter transformations for voiced and unvoiced sounds.The authors report waveform-like hidden features for voiced sounds and noisy hidden features for unvoiced sounds.Source: D. Harmonic-plus-noise NSF model · p. 7
  • The hn-NSF generates periodic harmonic and noise components separately, then merges them in different ratios for voiced and unvoiced sounds.The paper refers to the periodic component as the harmonic component.Source: D. Harmonic-plus-noise NSF model · p. 7
  • The hn-NSF filters the harmonic component with a low-pass filter and the noise component with a high-pass filter, using separate filter pairs for voiced and unvoiced sounds.The FIR filter coefficients are fixed after calculation, and the voicing flag is extracted from the input F0 sequence.Source: D. Harmonic-plus-noise NSF model · p. 7
  • From 0 to 7 kHz, the harmonic component dominates the generated voiced frame and has spectral amplitude similar to natural speech.Although the low-pass passband ends at 5 kHz, the neural filter generates harmonic energy between 5 and 7 kHz to compensate for attenuation.Source: D. Harmonic-plus-noise NSF model · p. 7

IV. EXPERIMENTS

The experiments describe the corpus and data configuration, test b-NSF through ablation, and compare the three NSF models with WaveNet. They also examine F0 controllability and model-internal behavior.

IV. EXPERIMENTS

  • The experimental sequence begins with corpus and data configuration, followed by an ablation test on b-NSF.The data configuration is described in Section IV-A and the ablation in Section IV-C.Source: §IV · pp. 7–8
  • Section IV-D compares the three NSF models with WaveNet.Source: §IV · p. 8
  • The experiments also investigate input-F0 controllability and examine the NSF models' internal behaviors.Source: §IV · p. 8

A. Corpus and data configuration

The experiments used Japanese female-speaker reading speech, with multiple training-set sizes and acoustic-feature configurations. Acoustic features included F0 and either MGCs or Mel-spectrograms.

A. Corpus and data configuration

  • The corpus contained about 50 hours of Japanese female-speaker reading speech, recorded at 48 kHz across 30,016 utterances.Waveform-model training used data downsampled to 16 kHz.Source: A. Corpus and data configuration · p. 8
  • The waveform-model training subsets contained 15 hours, 5 hours, or 1.6 hours, with separate validation and test sets of 500 and 480 utterances.The 15-hour and 1.6-hour sets were used for NSF–WaveNet comparison; the 5-hour set was used for ablations.Source: A. Corpus and data configuration · p. 8
  • Models received F0 with either order-60 MGCs extracted using WORLD or order-80 Mel-spectrograms.Features were extracted at a 5 ms frame shift; F0 came from an ensemble of pitch trackers.Source: A. Corpus and data configuration · p. 8
  • The TTS acoustic models predicted acoustic features from text-derived linguistic features aligned to the acoustics using hidden Markov models.Linguistic inputs included quin-phone identity and phrase accent type.Source: A. Corpus and data configuration · p. 8

B. Model configurations

The study compared three NSF variants with WaveNet and used shared training and implementation conditions. The NSF variants differed in filter structure and in whether harmonic and noise components had separate source-filter paths.

B. Model configurations

  • The benchmark comparison included b-NSF, s-NSF, hn-NSF, and WaveNet; IAF-flow models and Parallel WaveNet or ClariNet were not included.The authors cited high training-time and GPU-resource demands for IAF models and lack of authentic implementations for Parallel WaveNet and ClariNet.Source: B. Model configurations · p. 8
  • b-NSF used five baseline dilated-convolution filter blocks, while s-NSF replaced them with simplified blocks and hn-NSF used separate harmonic and noise source-filter paths.hn-NSF used the s-NSF network for its harmonic component and one simplified block for its noise component.Source: B. Model configurations · p. 8
  • All NSF models were trained with the multi-resolution spectral-amplitude distance L = L1+L2+L3, using configurations that trade off temporal and frequency resolution.The authors note that the selected configurations may not suit another corpus and may require trial and error.Source: B. Model configurations · p. 8
  • WaveNet used 40 dilated-convolution layers and generated 10-bit quantized µ-law waveform values from a categorical distribution.The models were trained with Adam on a single Nvidia Tesla P100; training stopped when validation error rose continually for five epochs.Source: B. Model configurations · pp. 8–9
  • The acoustic models generated MGCs, F0, or Mel-spectrograms from linguistic features using recipes based on prior work.Their training data comprised about 28,000 utterances, or 47 hours.Source: B. Model configurations · p. 9

C. Ablation test on b-NSF

The b-NSF ablations identify sine excitation, multi-resolution spectral distances, and filter-block skip connections as important components. The tests also report artifact and waveform-structure differences across variants.

C. Ablation test on b-NSF

  • Using only L1, L2, or L3 produced pulse-train-like artifacts and perceptually worse voiced and unvoiced sounds; the authors hypothesize that multiple temporal-spatial resolutions mitigate them.The pulse-train-like artifact was observable in unvoiced segments.Source: C. Ablation test on b-NSF · pp. 9–10, Figure 9 · p. 9
  • The subjective evaluation involved 245 paid Japanese native speakers and 1,444 valid rounds; reported differences between b-NSF and other models were significant at p < 0.01.The significance test was a two-sided Mann–Whitney test.Source: C. Ablation test on b-NSF · p. 9
  • Removing sine-based excitation destabilized perceived pitch, and the resulting S2 waveforms lacked the periodic structure expected in voiced sounds.The authors suggest sine excitation gives the model a better starting point for generating periodic waveforms.Source: Figure 9 · p. 9, C. Ablation test on b-NSF · p. 10
  • N1 outperformed b-NSF despite simpler filter-block transformations, while N2 produced unnatural voiced segments lacking stable periodic structure.The authors suggest skip-connections may let sine excitation reach later filter blocks without attenuation by nonlinear transformations.Source: C. Ablation test on b-NSF · p. 10
  • The ablation results identify sine excitation, multi-resolution spectral distances, and skip-connections inside the filter modules as crucial to NSF performance.The variants were designed to identify essential components, using b-NSF as the reference and subjective MOS evaluation of generated test-set waveforms.Source: C. Ablation test on b-NSF · pp. 9–10

D. Comparison between WaveNet and NSF models

Across the comparison, hn-NSF matched or exceeded WaveNet's speech quality while NSF models generated waveforms much faster. The experiments also found that NSF output F0 closely followed the source-module input, allowing direct F0 control.

D. Comparison between WaveNet and NSF models

E. Consistency between input F0 and F0 in generated waveforms

  • Generated NSF waveform F0 was highly consistent with the source-module F0 input, whereas WaveNet waveform F0 correlated with F0 information in its Mel-spectrogram input.The evaluation used hn-NSF and WaveNet trained on 15 hours of natural Mel-spectrogram and F0, with three generated F0 contours per test utterance.Source: E. Consistency between input F0 and F0… · p. 11, Table VI · p. 11

F. Investigation of hidden features of hn-NSF

Hidden-feature analysis shows the filter progressively shapes sine excitation into waveform structure while retaining F0. The conclusion reports hn-NSF’s WaveNet-comparable quality, faster generation, and study scope.

F. Investigation of hidden features of hn-NSF

  • The dilated-convolution filter progressively adds formant structure to sine excitation while retaining its F0 in the output waveform.This explains the observed consistency between output F0 and the input F0.Source: F. Investigation of hidden features of… · p. 12
  • Skip connections make the filter a deep residual network that gradually transforms excitation; their removal was associated with poor b-NSF performance.Source: F. Investigation of hidden features of… · p. 12
  • Sine excitation eases waveform modeling because the filter need not reproduce the voiced-sound periodic structure that evokes F0 perception.The authors connect this to the poor performance of b-NSF without sine excitation.Source: F. Investigation of hidden features of… · p. 12

V. CONCLUSION

  • NSF uses source, filter, and condition modules, and trains efficiently with merged spectral-amplitude distance implemented using STFT routines.The framework avoids AR and flow-based approaches and knowledge distillation.Source: §V · p. 12
  • hn-NSF performed comparably well to WaveNet on a large single-speaker Japanese speech corpus and generated waveforms much faster.Its separate modules model harmonic and noise waveform components.Source: §V · p. 12
  • The study compares NSF models with a verified WaveNet implementation; broader comparisons on multi-speaker corpora remain future work.The authors cite the time required to implement and train other models.Source: §V · p. 12
Loading 1904.12088v2…