PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequence-to-Sequence Acoustic Modeling for Voice Conversion

Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, Li-Rong Dai

arXiv:1810.06865v5cs.SDeess.AS

TL;DR

Conventional frame-by-frame acoustic mapping keeps converted durations equal to source durations and constrains modeling of dynamic speech. SCENT uses attention to map source speech features to target acoustic sequences, with ASR bottleneck inputs and WaveNet waveform synthesis. The authors report better objective and subjective results than GMM- and DNN-based baselines and their previous top-ranked Voice Conversion Challenge 2018 system.

  • Problem

    Conventional frame-by-frame acoustic mapping keeps converted utterance durations equal to source durations and constrains modeling of dynamic speech.

    Source: §I · p. 1

  • Method

    SCENT uses attention to map source to target Mel-spectrogram sequences, appends ASR bottleneck features as auxiliary inputs, and uses WaveNet for waveform synthesis.

    Source: §I · p. 1, §V · p. 12

  • Results

    SCENT achieved better objective and subjective performance than GMM- and DNN-based baselines and outperformed the authors’ previous work, which ranked first in Voice Conversion Challenge 2018.

    Source: §I · p. 2

  • Takeaways & Limitations

    SCENT can generate target sequences with durations different from the source, a capability described as difficult for conventional methods.

    Source: §I · pp. 1–2

    Long frame-level input and output sequences make it difficult for attention to find the correct hidden entries, and abnormal skips or repetitions can cause conversion errors.

    Source: E. Discussions · p. 12

Abstract

from arXiv · show

In this paper, a neural network named Sequence-to-sequence ConvErsion NeTwork (SCENT) is presented for acoustic modeling in voice conversion. At training stage, a SCENT model is estimated by aligning the feature sequences of source and target speakers implicitly using attention mechanism. At conversion stage, acoustic features and durations of source utterances are converted simultaneously using the unified acoustic model. Mel-scale spectrograms are adopted as acoustic features which contain both excitation and vocal tract descriptions of speech signals. The bottleneck features extracted from source speech using an automatic speech recognition (ASR) model are appended as auxiliary input. A WaveNet vocoder conditioned on Mel-spectrograms is built to reconstruct waveforms from the outputs of the SCENT model. It is worth noting that our proposed method can achieve appropriate duration conversion which is difficult in conventional methods. Experimental results show that our proposed method obtained better objective and subjective performance than the baseline methods using Gaussian mixture models (GMM) and deep neural networks (DNN) as acoustic models. This proposed method also outperformed our previous work which achieved the top rank in Voice Conversion Challenge 2018. Ablation tests further confirmed the effectiveness of several components in our proposed method.

I. INTRODUCTION

Conventional frame-by-frame voice-conversion mapping constrains dynamic speech modeling and keeps converted durations equal to source durations. SCENT uses attention to map feature sequences without explicit frame alignment, and the paper reports better objective and subjective performance than GMM- and DNN-based baselines.

I. INTRODUCTION

  • Conventional frame-by-frame mapping aligns parallel utterances and trains on paired frames, but limits dynamic modeling and leaves converted durations unchanged.Earlier methods often converted spectral features while adjusting F0 trajectories linearly in the logarithm domain.Source: §I · p. 1
  • SCENT uses an attention-based encoder-decoder to predict target acoustic sequences without explicit frame-to-frame alignment, followed by post-filtering and speaker-dependent WaveNet waveform reconstruction.The method uses Mel-spectrograms and appends ASR-derived bottleneck features; attention enables target durations to differ from source durations.Source: §I · p. 1
  • SCENT achieved better objective and subjective performance than GMM- and DNN-based baselines and outperformed the authors’ previous work, which ranked first in Voice Conversion Challenge 2018.The authors also report appropriate duration conversion, which is difficult for conventional methods, and say ablations confirmed several components’ effectiveness.Source: §I · p. 2

A. Relationship with sequence-to-sequence learning for text-to-speech

  • Voice conversion differs from sequence-to-sequence text-to-speech because its acoustic inputs carry speaker identity and its alignments can be one-to-many or many-to-one.The passage also notes that voice-conversion training data is typically smaller than text-to-speech training data.Source: A. Relationship with… · p. 2

B. Relationship with voice cloning

  • Voice cloning learns unseen speakers from a few samples for text-to-speech, whereas this work converts speech between a speaker pair using source speech rather than text.The authors focus on one-to-one conversion and describe extension to multiple pairs through speaker-identity codes as possible future work.Source: B. Relationship with voice cloning · p. 2, §I · p. 2

C. Sequence-to-sequence learning for voice conversion

Earlier voice-conversion sequence-to-sequence methods either retained DTW alignment or required target phone durations at conversion. This work uses Mel-spectrogram features, an ASR-feature-augmented SCENT model, and a speaker-dependent WaveNet vocoder to generate target speech.

C. Sequence-to-sequence learning for voice conversion

  • Unlike prior sequence-to-sequence approaches that retained DTW alignment or required target phone durations, this method predicts target Mel-spectrograms without DTW and determines output duration automatically.The authors position their method as most similar to Ramos, while noting differences in features and duration handling.Source: C. Sequence-to-sequence learning for… · p. 2

D. Voice conversion using WaveNet

  • The proposed WaveNet vocoder reconstructs target-speaker waveforms from Mel-spectrogram conditions and is trained speaker-dependently without the adaptation technique in.Prior work also used WaveNet for voice conversion through GMM-based mapping, model adaptation, or direct acoustic-feature-to-waveform mapping.Source: D. Voice conversion using WaveNet · pp. 2–3

A. Overall architecture

  • Conversion has two stages: SCENT maps concatenated source Mel-spectrogram and ASR bottleneck sequences to target Mel-spectrograms, then WaveNet synthesizes speech.The source bottleneck features provide additional linguistic-related descriptions and are extracted without requiring text transcriptions.Source: A. Overall architecture · p. 3, B. Feature extraction · p. 3

B. Feature extraction

  • Mel-spectrograms are computed from waveform STFT magnitudes using Mel-filterbanks and logarithmic compression, while framewise ASR bottleneck activations are concatenated as SCENT inputs.The bottleneck features come from the ASR model’s final hidden layer before its softmax output.Source: B. Feature extraction · p. 3

C. Structure of SCENT

SCENT uses an attention-based encoder-decoder to predict target acoustic sequences from source features, followed by a bidirectional post-filter that refines the prediction.

C. Structure of SCENT

  • SCENT combines an attention-based encoder-decoder with a bidirectional post-filtering network that produces the final prediction.The encoder-decoder predicts features in a left-to-right manner; the PostNet refines its output.Source: C. Structure of SCENT · pp. 3–4
  • The decoder models each target Mel-spectrogram frame conditioned on prior output frames and the source input.The figure illustrates one predicted frame per decoder step and generated autoregressive inputs during conversion.Source: C. Structure of SCENT · p. 3, Figure 2 · p. 3
  • The encoder maps concatenated source Mel-spectrograms and bottleneck features into hidden representations used by the decoder.The hidden sequence is shorter than the input because the encoder uses a pyramid structure.Source: C. Structure of SCENT · pp. 3–4

1) Encoder:

The encoder uses a pyramid bidirectional LSTM to shorten source sequences and add explicit position information. The decoder uses autoregressive prediction and attention informed by prior alignments.

1) Encoder:

  • The pyramid bidirectional LSTM concatenates consecutive lower-layer outputs to downsample the sequence, then adds a location code to its top-layer representation.The location code gives the model explicit information about which sequence portion is being processed.Source: 1) Encoder · p. 4
  • The shorter hidden sequence is intended to capture linguistic-related information while easing attention convergence and reducing computation.The authors note that a phone typically spans tens of acoustic frames.Source: 1) Encoder · p. 4

2) Decoder with attention mechanism:

  • The autoregressive decoder predicts non-overlapping groups of acoustic frames, while hybrid attention combines encoder-query matching with the previous alignment.Forward attention is also used to stabilize alignment and speed its convergence, following the monotonic nature of speech alignments.Source: 2) Decoder with attention mechanism · pp. 4–5
  • The decoder projects its context, query, and recurrent outputs into Mel-spectrogram frames and predicts sequence completion at each step.Source: 2) Decoder with attention mechanism · p. 5

3) Post-filtering network:

The PostNet refines the decoder’s Mel-spectrogram predictions using bidirectional context and a residual connection.

3) Post-filtering network:

  • The PostNet uses convolutional context and a residual connection to refine the decoder’s Mel-spectrogram output.Its convolutional filter-bank outputs pass through a two-layer 1-D CNN before being added to the input Mel-spectrograms.Source: 3) Post-filtering network · p. 5

D. Loss function of SCENT

SCENT is trained with a weighted combination of decoder, post-filter, and completion losses. Decoder training compares MSE and GMM-based likelihood criteria, while the post-filter uses MSE.

D. Loss function of SCENT

  • SCENT uses a weighted sum of decoder acoustic-feature loss, PostNet loss, and completion-probability loss.The completion loss is binary cross-entropy.Source: D. Loss function of SCENT · p. 5
  • The decoder loss uses either MSE or GMM maximum likelihood, with MSE described as a single-component, fixed-unit-variance special case of GMM-ML.The paper notes that MSE can yield oversmoothed predictions through averaging, while GMM-ML models more general distributions.Source: D. Loss function of SCENT · p. 5
  • For GMM-ML, the decoder parameterizes a Gaussian mixture and minimizes negative log-likelihood.The mixture covariance matrices are diagonal.Source: D. Loss function of SCENT · p. 5
  • Training supplies natural target acoustic histories to the decoder; the PostNet uses MSE, and only the final decoder step is labeled complete.Source: D. Loss function of SCENT · p. 5

E. WaveNet-based vocoder

A WaveNet-based vocoder reconstructs waveforms from SCENT-predicted Mel-spectrograms, using them as conditioning features.

E. WaveNet-based vocoder

  • WaveNet reconstructs time-domain waveforms from the Mel-spectrogram features predicted by SCENT.Source: E. WaveNet-based vocoder · p. 6
  • A ConditionNet upsamples conditioning vectors by repetition to waveform sampling rate before feeding them into WaveNet’s dilated convolution blocks.The vectors control waveform generation.Source: E. WaveNet-based vocoder · p. 6
  • The WaveNet vocoder is trained on target-speaker speech, without the adaptation technique used in prior work.Source: E. WaveNet-based vocoder · p. 6

IV. EXPERIMENTS

Experiments use Mandarin and English parallel-speech datasets to assess SCENT and decoder loss choices. On validation data, two-mixture GMM-ML generally performs best among the tested decoder settings, while larger mixtures can destabilize attention alignment.

A. Experimental conditions

  • The experiments use 1,060 parallel Mandarin utterances and 1,132 parallel English CMU ARCTIC utterances, with 1,000 training utterances in each dataset.Analytical experiments use Mandarin; the main objective and subjective evaluations use both datasets.Source: A. Experimental conditions · p. 6
  • Baselines include JD-GMM, DNN, bottleneck-input DNN, and VCC2018, with conventional MCC, F0, and BAP features and STRAIGHT waveform reconstruction for the described GMM and DNN systems.VCC2018 uses a speaker-dependent WaveNet vocoder.Source: A. Experimental conditions · pp. 6–7

B. Comparison between different decoder loss functions

  • Decoder-loss evaluation compares MSE and GMM-ML using MCD and F0 RMSE, extracting features from reconstructed waveforms and aligning them to validation references.F0 RMSE uses only frames voiced in both converted and reference utterances.Source: B. Comparison between different… · p. 7
  • Two-mixture GMM-ML achieves the best validation performance except for male-to-female MCD; larger mixtures may destabilize attention, so two mixtures are used thereafter.Some larger-mixture cases show attention stuck on one frame, and reweighting only slightly improves alignment.Source: B. Comparison between different… · p. 7, Table II · p. 7
  • SCENT learns source-target utterance alignments implicitly with soft attention, which matches the DTW path while changing smoothly across decoder steps.The heat map depicts attention probabilities and the dashed line depicts the downsampled DTW path.Source: B. Comparison between different… · pp. 7–8, Figure 3 · p. 7

C. Comparison between baseline and proposed methods

Objective tests compare SCENT with baseline systems on Mandarin and English, including duration conversion. SCENT leads on both Mandarin objective metrics and English F0 RMSE, and has lower duration errors than baselines with fixed or globally adjusted speaking rates.

1) Objective evaluation:

  • On English CMU ARCTIC, SCENT achieves the best F0 RMSE, but some baselines have better MCD.The paper notes that MCD may not align with human perception and follows with subjective evaluation.Source: Table IV · p. 8, 1) Objective evaluation · p. 8
  • SCENT yields lower converted-to-target duration errors than baselines that leave duration unchanged or apply global speaking-rate compensation, across Mandarin and English.Source: Table V · p. 8, 1) Objective evaluation · p. 9
  • SCENT has the lowest MCD and F0 RMSE among the tested methods on the Mandarin test set.Source: 1) Objective evaluation · p. 8
  • Linear interpolation brings baseline output lengths closer to target lengths, but interpolation uses one shared global ratio and does not change baseline durations utterance by utterance.Without interpolation, baseline output duration equals source duration.Source: 1) Objective evaluation · pp. 8–9
  • For an example Mandarin utterance, SCENT produces F0 contours more like the natural reference and adjusts duration toward it.Source: 1) Objective evaluation · p. 9

2) Subjective evaluation:

Subjective tests compare naturalness and similarity across Mandarin and English. SCENT outperforms the VCC2018 baseline on both measures in both languages, while VCC2018 is the strongest of the three baselines, especially for naturalness.

2) Subjective evaluation:

  • SCENT outperforms VCC2018 on naturalness and similarity in both Mandarin and English subjective evaluations.Source: Table VI · p. 9, 2) Subjective evaluation · p. 9
  • The evaluations use 20 randomly selected test utterances; Mandarin ratings come from ten native listeners, while English ratings come from at least twenty native listeners on AMT.Listeners score naturalness and similarity on a five-point scale.Source: 2) Subjective evaluation · p. 9
  • Among the three baselines, VCC2018 performs best, especially on naturalness; the authors point to its WaveNet rather than STRAIGHT vocoder as one reason.The bottleneck-input DNN and JD-GMM have similar naturalness and similarity scores.Source: 2) Subjective evaluation · p. 9
  • The authors connect SCENT’s better prosody similarity and naturalness to duration conversion, contrasting it with globally adjusted speaking rate.Source: 2) Subjective evaluation · p. 10

D. Ablation tests

The ablations, evaluated on the Mandarin dataset, examine Mel-spectrogram and bottleneck inputs and the attention module. They report benefits from both input types and better duration, F0, and similarity outcomes with attention.

1) Mel-spectrograms and bottleneck features:

  • Mel-spectrograms benefited acoustic-feature prediction, while removing bottleneck features worsened F0 RMSE and MCD, with a more serious degradation in F0 RMSE and serious mispronunciation.In Fig. 7, the no-bottleneck output shows spectral distortion at 1–2 s, while the proposed method's F0 contour is closer to the natural reference.Source: 1) Mel-spectrograms and bottleneck… · p. 10, Figure 7 · p. 11

2) Attention module:

  • Attention learns input-output alignment during training and predicts target durations at conversion; its ablation replaced attention with DTW-aligned frame-by-frame modeling.The decoder's attention LSTM became a plain unidirectional LSTM in the comparison model.Source: 2) Attention module · pp. 10–11
  • Without attention, prediction errors increased; attention produced durations and F0 contours closer to natural target speech, and preference tests favored attention over source interpolation.The preference tests also found that global speaking-rate adjustment by interpolation improved similarity in both conversion pairs.Source: Table VIII · p. 10, 2) Attention module · p. 11, Table IX · p. 11

3) Location code:

SCENT outperformed the baselines in objective and subjective results, especially for duration, while removing the location code worsened measured outcomes. The discussion notes attention-related conversion errors and risks tied to training data and ASR features.

3) Location code:

  • Removing the location code slightly raised MCD and F0 RMSE, while DDUR increased 5.4% for female-to-male and 15.3% for male-to-female conversion.The experiments identify these outcomes as evidence of the location code's positive effects.Source: 3) Location code · p. 11

E. Discussions

  • Long frame-level sequences challenge attention's search; abnormal skips or repetitions can cause conversion mistakes, motivating forward attention, location features, and linguistic bottleneck cues.The paper reports that these errors still cannot be completely avoided.Source: E. Discussions · p. 12
  • The roughly 7.5 M-parameter model may generalize poorly to unseen utterances when training data are insufficient, and ASR accuracy limits may make bottleneck features misleading.The authors identify reducing conversion errors as future work.Source: E. Discussions · p. 12

V. CONCLUSION

  • Objective and subjective results showed SCENT outperforming baseline methods, especially in duration; ablations supported Mel-spectrograms, bottleneck features, attention, and the location code.Future work targets investigating training-set size and improving attention calculation to reduce conversion errors.Source: §V · p. 12
Loading 1810.06865v5…