PaperlayerView exact revision on arXiv

Source-linked AI summary

Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling

Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, Helen Meng

arXiv:2009.02725v3eess.AScs.CLcs.SD

TL;DR

Voice conversion seeks to change speaker identity while preserving linguistic content, but PPG-based any-to-many systems have limitations in alignment and framewise acoustic prediction. This paper combines bottleneck features with MoL-attention seq2seq synthesis and reports strong naturalness and speaker-similarity results, with an extension to any-to-any conversion.

  • Problem

    PPG-based any-to-many conversion can suffer from mispronunciations when phonetic alignments are inaccurate and from framewise prediction that is unfavorable for sparse training data.

    Source: §I · p. 2

  • Method

    A CTC-attention phoneme recognizer supplies bottleneck content features to a multi-speaker seq2seq synthesizer with MoL location-relative attention.

    Source: §I · p. 2, 2) MoL attention-based decoder · p. 6

  • Results

    The proposed approach has the best average subjective results for naturalness and speaker similarity, and the best average performance on all four objective metrics.

    Source: E. Subjective evaluations · p. 9, D. Objective evaluations · p. 9

  • Takeaways & Limitations

    The approach is reported to support any-to-any conversion and shorten the seq2seq VC pipeline to an ASR encoder and synthesis decoder.

    Source: §I · p. 2, §VI · p. 11

Abstract

from arXiv · show

This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach, which utilizes text supervision during training. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. During the training stage, an encoder-decoder-based hybrid connectionist-temporal-classification-attention (CTC-attention) phoneme recognizer is trained, whose encoder has a bottle-neck layer. A BNE is obtained from the phoneme recognizer and is utilized to extract speaker-independent, dense and rich spoken content representations from spectral features. Then a multi-speaker location-relative attention based seq2seq synthesis model is trained to reconstruct spectral features from the bottle-neck features, conditioning on speaker representations for speaker identity control in the generated speech. To mitigate the difficulties of using seq2seq models to align long sequences, we down-sample the input spectral feature along the temporal dimension and equip the synthesis model with a discretized mixture of logistic (MoL) attention mechanism. Since the phoneme recognizer is trained with large speech recognition data corpus, the proposed approach can conduct any-to-many voice conversion. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity. Ablation studies are conducted to confirm the effectiveness of feature selection and model design strategies in the proposed approach. The proposed VC approach can readily be extended to support any-to-any VC (also known as one/few-shot VC), and achieve high performance according to objective and subjective evaluations.

I. INTRODUCTION

Voice conversion changes non-linguistic speech attributes while preserving linguistic content; this paper targets conversion from arbitrary source speakers to target speakers seen in training. It proposes a BNE-based, autoregressive seq2seq synthesis approach to address limitations of PPG-based conversion.

I. INTRODUCTION

  • Conventional one-to-one conversion requires parallel source-target data and frame alignment, while seq2seq modeling can bypass that alignment stage.The paper describes PPG-based and non-parallel seq2seq methods as text-supervised approaches, contrasting them with methods that do not require text supervision.Source: §I · p. 1
  • Any-to-many conversion maps speech from an arbitrary source speaker to a target speaker appearing in the training set.The paper notes few previously reported approaches for this setting.Source: §I · p. 1
  • PPG-based systems combine a speaker-independent PPG extractor with multi-speaker framewise conversion, whose independent frame predictions are unfavorable when training data are sparse.The paper also identifies inaccurate phonetic alignments as a possible source of downstream mispronunciations.Source: §I · p. 2
  • The proposed approach combines a phoneme-recognizer bottleneck feature extractor with multi-speaker autoregressive seq2seq synthesis and MoL location-relative attention.Input spectral features are down-sampled by a factor of four, and speaker identity is represented with one-hot vectors.Source: §I · p. 2
  • The approach is presented as a shorter, simpler-training alternative that can be extended from any-to-many to any-to-any conversion.For the any-to-any extension, speaker embeddings from a speaker encoder provide speaker representations.Source: §I · p. 2

II. RELATED WORK

The paper situates its approach among location-relative attention methods, seq2seq conversion systems, and recognition-synthesis pipelines. It also introduces a Seq2seqPR-DurIAN baseline and contrasts it with other recognition-synthesis approaches.

A. Attention mechanisms in seq2seq models

  • Location-relative attention extends location-based attention, and the proposed model represents attention weights with a discretized mixture-of-logistics distribution.The distribution parameters are computed from the attention RNN state by an MLP.Source: A. Attention mechanisms in seq2seq… · p. 2

B. A Seq2seq baseline model

  • Non-parallel seq2seq conversion cascades ASR and multi-speaker synthesis, but may incur long pipelines, slow inference, and unstable attention alignment.The prior rhythm-model approach adds explicit temporal expansion and local attention; this paper redesigns it as Seq2seqPR-DurIAN.Source: B. A Seq2seq baseline model · pp. 2–3

C. Recognition-synthesis VC approaches

  • Recognition-synthesis VC extracts spoken content with ASR and predicts acoustic features with synthesis; Seq2seqPR-DurIAN uses explicit phoneme-level duration alignment and a simpler training procedure than.The compared approach uses multiple losses and alternates generation and adversarial steps.Source: C. Recognition-synthesis VC approaches · p. 3

III. SEQ2SEQPR-DURIAN-BASED VOICE CONVERSION

  • Seq2seqPR predicts a phoneme sequence from spectral features, and DurIAN generates spectral features from that sequence conditioned on speaker representations.The pipeline is the paper’s Seq2seqPR-DurIAN conversion model.Source: §III · p. 3

A. Seq2seq phoneme recognizer

The phoneme recognizer combines CTC and attention-based modeling to predict phoneme sequences from speech features. Its training objective jointly weights both objectives.

1) CTC and attention-based modeling:

  • CTC monotonically maps an input sequence to a shorter output sequence using latent framewise phoneme paths.The recognizer combines this CTC objective with an attention-based objective.Source: 1) CTC and attention-based modeling · p. 3

2) Model structure and training objective:

  • The hybrid recognizer uses CTC as an auxiliary task for training the attention encoder, with a VGG-Prenet and BiLSTM encoder.The VGG-Prenet subsamples spectral features by four in time, and the attention decoder uses location-sensitive attention.Source: 2) Model structure and training… · p. 3
  • The training objective is a logarithmic linear combination of the CTC and attention objectives, weighted by λ and 1 − λ.The paper sets λ to 0.5.Source: 2) Model structure and training… · pp. 3–4

B. DurIAN synthesis model

DurIAN uses phoneme-level durations to expand encoder representations before autoregressive mel-spectrogram generation. For any-to-any conversion, a speaker encoder supplies an embedding to condition synthesis.

B. DurIAN synthesis model

  • DurIAN adds a duration module to address alignment errors that can cause missing or repeated words, incomplete synthesis, or difficulty with longer utterances.The model predicts mel-spectrograms from phoneme sequences.Source: B. DurIAN synthesis model · p. 4
  • The duration module expands encoder representations according to phoneme-level durations, and an autoregressive decoder generates mel-spectrograms conditioned on speaker representations.The any-to-many setting uses one-hot speaker vectors, while the any-to-any setting uses vectors from a speaker encoder.Source: B. DurIAN synthesis model · p. 4

C. Extension to support any-to-any conversion

  • For any-to-any conversion, a speaker encoder produces a fixed-dimensional embedding from speech to condition DurIAN on the desired target identity.The encoder is trained with a generalized end-to-end speaker verification loss.Source: C. Extension to support any-to-any… · p. 4

IV. BNE-SEQ2SEQMOL-BASED VOICE CONVERSION

The approach pairs a bottleneck feature extractor with a multi-speaker MoL-attention seq2seq synthesizer. Its encoder combines content and pitch features with speaker representations to condition mel-spectrogram generation.

IV. BNE-SEQ2SEQMOL-BASED VOICE CONVERSION

  • The BNE-Seq2seqMoL approach extracts dense content features from mel-spectrograms and generates mel-spectrograms autoregressively with a multi-speaker MoL-attention synthesis model.The section illustrates BNE training in Fig. 4.Source: §IV · p. 4, Figure 4 · p. 5

A. Bottle-neck feature extractor

  • A CTC-attention phoneme recognizer supplies the BNE; after training, its CTC module and attention decoder are removed, leaving a 256-unit bottleneck layer.The resulting bottleneck features are treated as spoken-content representations and presumed speaker-invariant.Source: A. Bottle-neck feature extractor · p. 5

B. Seq2seqMoL synthesis

1) Bottle-neck feature prenet and pitch encoder:

  • Pitch features are down-sampled by a factor of four to match BNF resolution, then combined element-wise with prenet outputs before speaker conditioning.The pitch encoder uses two stride-2 convolution layers, with instance normalization after each to remove possible speaker information.Source: 1) Bottle-neck feature prenet and… · p. 5

2) MoL attention-based decoder:

The decoder uses location-relative discretized MoL attention to align its outputs with down-sampled encoder states. It generates mel-spectrogram frames autoregressively and is trained with spectrogram and stop-token losses.

2) MoL attention-based decoder:

  • The attention RNN state produces an alignment over encoder states, whose weighted average forms the context used by the decoder RNN.The encoder sequence has one-quarter as many frames as the mel-spectrogram input.Source: 2) MoL attention-based decoder · p. 5
  • A location-relative discretized MoL distribution supplies attention weights at each decoder step, using mixture coefficients, means, and scales.The model uses five mixtures, and positive shifts move component centers monotonically toward the end of the input sequence.Source: 2) MoL attention-based decoder · p. 6
  • Training minimizes mel-spectrogram MSE together with binary cross-entropy on stop-token predictions.Stop-token probability reaching 0.5 triggers decoding to stop.Source: 2) MoL attention-based decoder · p. 6

C. Conversion procedure

Conversion extracts content and pitch features from an arbitrary source speaker, conditions the synthesis model on the target speaker, and generates a converted mel-spectrogram for waveform synthesis.

C. Conversion procedure

  • The procedure begins by computing a mel-spectrogram, continuous Log-F0, and UV flags from an arbitrary source-speaker utterance.Source: C. Conversion procedure · p. 6
  • Source Log-F0 is converted linearly in log scale using source and target speakers’ log-F0 statistics.The statistics are the mean and standard deviation of log-scaled F0.Source: C. Conversion procedure · p. 6
  • BNF content features, converted Log-F0, and UV flags feed the encoder with the target speaker embedding; the MoL-attention decoder generates a mel-spectrogram autoregressively.A neural vocoder converts the generated mel-spectrogram into a waveform.Source: C. Conversion procedure · p. 6

D. Extension to any-to-any conversion

The any-to-any extension replaces one-hot speaker identity control with speaker-encoder embeddings for arbitrary target speakers. The evaluation setting trains the speaker encoder on data from more than 8K speakers and uses four unseen CMU ARCTIC speakers for conversion.

D. Extension to any-to-any conversion

  • A speaker encoder replaces one-hot identity vectors with embeddings, allowing the synthesis module to condition conversion on an arbitrary target speaker.The same speaker encoder model is used to generate the target-speaker vector.Source: D. Extension to any-to-any conversion · p. 6

A. Datasets

  • Any-to-many experiments use VCTK and CMU ARCTIC, with non-overlapping training utterances selected for the four ARCTIC speakers.LibriSpeech (960 hours) trains the phoneme recognizer; the VCTK data used here includes 105 speakers.Source: A. Datasets · pp. 6–7
  • More than 8K speakers from LibriSpeech train-other-500, VoxCeleb1, and VoxCeleb2 train the speaker encoder; four CMU ARCTIC speakers are unseen during synthesis training.LibriTTS train-clean-100 and train-clean-360 [43], together with VCTK training data, train the synthesis models; CMU ARCTIC is used only for conversion.Source: A. Datasets · p. 7

B. Features and neural vocoder model

The experiments define the acoustic features and comparison systems, then report that BNE-Seq2seqMoL leads the any-to-many objective metrics on average, while any-to-any performance varies by metric.

B. Features and neural vocoder model

  • Speech features are 80-dimensional log mel-spectrograms, while the speaker encoder uses 40-dimensional log mel-spectrograms; WaveRNN is trained on ground-truth mel-spectrograms from VCTK.The 80-dimensional features use a 50 ms Hanning window; the 40-dimensional features use a 25 ms window, both with a 10 ms frame shift. WaveRNN models 512-way µ-law quantized waveforms.Source: B. Features and neural vocoder model · p. 7

C. Comparisons

  • The comparisons include PPG-VC, NonParaSeq2seq-VC, and the proposed Seq2seqPR-DurIAN and BNE-Seq2seqMoL approaches.The NonParaSeq2seq-VC baseline uses disentangled acoustic content and speaker representations, preserving source content while replacing the speaker representation.Source: C. Comparisons · p. 7

D. Objective evaluations

  • BNE-Seq2seqMoL achieves the best average performance on all four any-to-many objective metrics, and its autoregressive property boosts performance across those metrics.PPG-VC has the worst F0-RMSE; NonParaSeq2seq-VC has significantly worse CER and WER, with repeating, skipping, and truncation observed in preliminary listening.Source: Table I · p. 8, D. Objective evaluations · p. 9
  • In any-to-any conversion, PPG-VC leads on MFC, CER, and WER, while BNE-Seq2seqMoL has the lowest F0-RMSE and good results on the other reported metrics.The models use VCTK training data combined with LibriTTS train-clean-100 and train-clean-360; the authors state that more training data can overcome PPG-VC's independent-frame prediction deficiency.Source: Table II · p. 8, D. Objective evaluations · p. 9
  • Objective evaluation uses MCD for spectral conversion, F0-RMSE for F0 conversion, and ASR-derived CER and WER for intelligibility.The ASR engine is trained on LibriSpeech; its CER and WER on the CMU ARCTIC test set are 2.71% and 4.30%, respectively.Source: D. Objective evaluations · p. 8

E. Subjective evaluations

In any-to-many tests, BNE-Seq2seqMoL has the best average naturalness and speaker-similarity scores, though Seq2seqPR-DurIAN performs better for male-to-female conversion. The proposed approach also has good absolute MOS results in the reported any-to-any tests.

E. Subjective evaluations

  • The MOS tests use a five-point Likert scale, 10 CMU ARCTIC test utterances per conversion pair, and 35 English-proficient raters.Naturalness stimuli include target-speaker recordings shuffled with converted samples; similarity tests directly compare converted speech with target-speaker recordings.Source: E. Subjective evaluations · p. 9
  • BNE-Seq2seqMoL achieves the best average MOS for both naturalness and speaker similarity in any-to-many conversion.It performs better for F-F, M-M, and F-M pairs, while Seq2seqPR-DurIAN performs better for M-F; only the proposed approach is evaluated by MOS in the any-to-any setting, where it achieves good absolute MOS values.Source: E. Subjective evaluations · p. 9
  • The ablation table covers the proposed BNE-Seq2seqMoL approach and identifies instance normalization and location-sensitive attention as factors under study.Source: Table V · p. 10

F. Cross-speaker property of bottle-neck features

The paper uses t-SNE to examine bottleneck-feature structure across speakers. The features show cross-speaker clustering and similar manifold patterns, which the authors interpret as evidence of speaker-agnostic content representations.

F. Cross-speaker property of bottle-neck features

  • t-SNE maps high-dimensional bottleneck features into a two- or three-dimensional space to visualize their structure.The paper uses this nonlinear dimensionality-reduction technique to explore the features extracted by the BNE.Source: F. Cross-speaker property of… · p. 10
  • The 256-dimensional bottleneck features form a similar manifold pattern across four speakers, indicating cross-speaker structure.The visualization uses frames from utterance “arctic-a0001” by bdl, rms, slt, and clb.Source: F. Cross-speaker property of… · p. 10
  • The authors interpret the cross-speaker pattern as evidence that bottleneck features provide speaker-agnostic content representations for synthesis.They suggest this lets the synthesis module focus more on generating the target voice.Source: F. Cross-speaker property of… · p. 10

G. Ablation Studies

The proposed approach shows superior voice-conversion performance in both any-to-many and any-to-any settings, while ablations support its feature-selection and model-design choices. The conclusion notes that its spectral-feature and independent-vocoder pipeline may constrain synthesis quality.

G. Ablation Studies

  • The proposed feature selection and model design achieve the best F0-RMSE and WER results and near-the-best MCD and CER results among the ablations.The three tested choices concern Log-F0 and UV input, pitch-encoder instance normalization, and attention type.Source: G. Ablation Studies · p. 10

H. Training and Inference Speed

  • BNE-Seq2seqMoL synthesis training took 28 hours for 56 epochs, versus 44 hours for 200 Seq2seq-DurIAN epochs.Folding input into two segments gave the greatest inference-speed gain; preliminary listening found no conversion-performance harm.Source: H. Training and Inference Speed · p. 10

VI. CONCLUSION

  • The redesigned approach connects a seq2seq phoneme recognizer to a multi-speaker duration-informed attention network for synthesis.The conclusion also reports an extension to any-to-any voice conversion.Source: §VI · p. 10
  • Objective and subjective evaluations show superior VC performance in both any-to-many and any-to-any settings.The paper reports that ablations confirm the effectiveness of its feature-selection and model-design strategies.Source: §VI · p. 11
  • The system still uses mel spectrograms and an independently trained neural vocoder, which may reduce synthesis quality.The authors identify end-to-end waveform-to-waveform training as a future direction.Source: §VI · p. 11
Loading 2009.02725v3…