Source-linked AI summary
AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanisms
Kou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko, Nobukatsu Hojo
TL;DR
Conventional VC may require time-aligned parallel speech, and transcript-based Seq2Seq approaches require many transcripts. This paper proposes attention-based Seq2Seq VC with context preservation; its speech quality exceeded GMM-based VC and was comparable to recurrent-NN TTS.
Problem
GMM-based VC requires time-aligned parallel source and target speech, while transcript-based Seq2Seq VC approaches require many transcripts.
Source: §1 · p. 1, §4.2 · p. 5Method
The method applies Seq2Seq with attention and context preservation losses to convert acoustic features and durations without phoneme labels or time-aligned speech data in advance.
Source: §1 · pp. 1–2Results
The proposed VC produced higher-quality speech than GMM-based VC and speech comparable to recurrent-NN TTS in naturalness and speaker similarity.
Source: §1 · p. 2Takeaways & Limitations
The reported comparison places the proposed VC above GMM-based VC and comparable to recurrent-NN TTS while also converting speech durations.
Source: §1 · p. 2Extending the method to semi-supervised learning remains ongoing work.
Source: §5 · p. 5
Abstract
from arXiv · showhide
This paper describes a method based on a sequence-to-sequence learning (Seq2Seq) with attention and context preservation mechanism for voice conversion (VC) tasks. Seq2Seq has been outstanding at numerous tasks involving sequence modeling such as speech synthesis and recognition, machine translation, and image captioning. In contrast to current VC techniques, our method 1) stabilizes and accelerates the training procedure by considering guided attention and proposed context preservation losses, 2) allows not only spectral envelopes but also fundamental frequency contours and durations of speech to be converted, 3) requires no context information such as phoneme labels, and 4) requires no time-aligned source and target speech data in advance. In our experiment, the proposed VC framework can be trained in only one day, using only one GPU of an NVIDIA Tesla K80, while the quality of the synthesized speech is higher than that of speech converted by Gaussian mixture model-based VC and is comparable to that of speech generated by recurrent neural network-based text-to-speech synthesis, which can be regarded as an upper limit on VC performance.
1. INTRODUCTION
The paper proposes Seq2Seq voice conversion with attention and context preservation, aiming to convert acoustic features and durations without context labels or time-aligned speech. Its synthesized speech outperformed GMM-based VC and was comparable to recurrent-NN TTS in naturalness and speaker similarity.
1. INTRODUCTION
- GMM-based VC requires time-aligned parallel source and target speech, while parallel-data-free VC can require many transcripts and may have difficulty converting durations.The paper presents Seq2Seq VC as an alternative to these constraints.Source: §1 · p. 1
- Seq2Seq models map input sequences to output sequences of different lengths and can be trained end-to-end on source and target sequences.The paper motivates this approach as a way to convert speech durations as well as acoustic features.Source: §1 · p. 1
- The method uses guided attention and context preservation losses to stabilize and accelerate training.It converts spectral envelopes, fundamental frequency contours, and speech durations without phoneme labels or time-aligned source and target data in advance.Source: §1 · p. 2
- The proposed VC produced speech with higher quality than GMM-based VC and quality comparable to recurrent-NN TTS in naturalness and speaker similarity.Training took one day on one NVIDIA Tesla K80 GPU.Source: §1 · p. 2
2. CONVENTIONAL VC
Frame/sequence-based VC frameworks do not convert source durations and acoustic features together. Seq2Seq-based VC can convert durations, but transcript-based approaches require many transcripts; the proposed model uses none.
2.1. Frame/Sequence- based VC
- Frame/sequence-based VC frameworks do not convert source durations and acoustic features at the same time.This applies to both parallel-data and parallel-data-free frameworks described here.Source: §2.1 · p. 2
2.2. Seq2Seq-based VC
- Seq2Seq-based VC can convert source durations along with acoustic features, unlike frame/sequence-based VC.The described Seq2Seq approaches include ASR and TTS modules trained on speech-transcript pairs.Source: §2.2 · p. 2
- Transcript-based Seq2Seq VC converts context-posterior sequences that include duration information, but requires many transcripts; the proposed model uses no transcript.Other approaches either regenerate durations from text after conversion or directly convert context-posterior sequences.Source: §2.2 · p. 2
3. ATTS2S-VC
AttS2S-VC combines a Seq2Seq model with attention and added decoders for context preservation. Guided attention encourages near-diagonal alignments, while the added decoders help retain source linguistic information.
3. ATTS2S-VC
- The model adds a source decoder and a second target decoder to the four basic Seq2Seq components; Fig. 2 distinguishes original from added components.The figure uses solid lines for training and conversion, dashed lines for training-difference calculations, green boxes for original components, and red boxes for additions.Source: §3 · p. 2, Figure 2 · p. 3
3.1. Seq2Seq Model with Attention Mechanism
- Source and target acoustic sequences are encoded as embeddings; attention relates each target embedding to the source embeddings, and an autoregressive decoder generates the output sequence.The decoder uses a seed that incorporates long-range temporal dependencies, together with the seed, target embeddings, and its previous outputs.Source: §3.1 · p. 3
3.2.1. Guided Attention Loss
- Guided attention encourages the attention matrix to be nearly diagonal, reflecting the assumed near-linear progression between source and target speech frames.The parameter σ_g controls how close the attention matrix is to diagonal.Source: §3.2.1 · p. 3
3.2.2. Context Preservation Loss
- The context preservation loss adds source and target decoders to address cases where Seq2Seq training fails to preserve source context.The source decoder reconstructs source speech from its embeddings, while the target decoder predicts target speech from the seed; the authors report the target decoder was more important in preliminary experiments.Source: §3.2.2 · pp. 3–4
4. EXPERIMENTS
In subjective tests, the proposed system outperformed GMM-VC-wGV in naturalness and speaker similarity and was comparable to LSTM-TTS. Experiments used CMU Arctic speakers and evaluated naturalness and speaker similarity preferences.
4.1. Experimental Conditions
- CMU Arctic training used about 1,000 sentences per speaker, with 132 evaluation sentences per speaker across four independently trained source-target pairs.The pairs were rms-bdl, clb-slt, rms-slt, and clb-bdl.Source: §4.1 · p. 4
4.2. Experimental Results
- Proposed outperformed GMM-VC-wGV in naturalness and speaker similarity and was comparable to LSTM-TTS.The authors attribute the GMM comparison partly to duration conversion: GMM-VC-wGV preserves source durations, which can increase errors when those durations are unsuitable for the target.Source: Figure 3 · p. 4, §4.2 · pp. 4–5
- Fifteen listeners evaluated 80 shots, comparing Proposed with GMM-VC-wGV and LSTM-TTS across naturalness and speaker similarity.The evaluations used randomly selected speech samples across four intra- and cross-gender speaker pairs.Source: §4.2 · p. 4, Figure 3 · p. 4
5. CONCLUSIONS
The paper proposes Seq2Seq voice conversion with attention and context preservation mechanisms. Experiments found it outperformed conventional GMM-based VC and was comparable to LSTM-based TTS; semi-supervised extension and further convolutional-version evaluations remain ongoing.
5. CONCLUSIONS
- The proposed Seq2Seq method combines attention and context preservation mechanisms for voice conversion.Source: §5 · p. 5
- The proposed method outperformed conventional GMM-based VC and was comparable to LSTM-based TTS.Source: §5 · p. 5
- Extending the method to semi-supervised learning and evaluating the convolutional version remain ongoing work.Source: §5 · p. 5