Source-linked AI summary
The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods
Jaime Lorenzo-Trueba, Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Tomi Kinnunen, Zhenhua Ling
TL;DR
Voice Conversion Challenge 2018 addresses the need for a common framework to compare systems that convert speaker identity while preserving linguistic information. It combines parallel Hub and non-parallel Spoke tasks with crowdsourced naturalness and similarity evaluation. One submitted system achieved 4.1/5 quality and about 80% target-speaker judgments, while listening-test results did not directly reflect spoofing capability.
Problem
Comparing voice conversion systems requires a common framework for evaluating speaker-identity conversion while preserving linguistic information across parallel and non-parallel data.
Method
The challenge summarized VC approaches, defined parallel Hub and non-parallel Spoke tasks, and evaluated converted speech through crowdsourced naturalness and target-similarity tests.
Results
One submitted system achieved an average quality score of 4.1/5, and about 80% of its converted samples were judged the same as target speakers.
Takeaways & Limitations
VCC 2018 continued a common benchmark and showed converted speech quality similar to natural speech for a leading submitted system.
Takeaways & Limitations
Listening-test results do not directly reflect spoofing capability.
Abstract
from arXiv · showhide
We present the Voice Conversion Challenge 2018, designed as a follow up to the 2016 edition with the aim of providing a common framework for evaluating and comparing different state-of-the-art voice conversion (VC) systems. The objective of the challenge was to perform speaker conversion (i.e. transform the vocal identity) of a source speaker to a target speaker while maintaining linguistic information. As an update to the previous challenge, we considered both parallel and non-parallel data to form the Hub and Spoke tasks, respectively. A total of 23 teams from around the world submitted their systems, 11 of them additionally participated in the optional Spoke task. A large-scale crowdsourced perceptual evaluation was then carried out to rate the submitted converted speech in terms of naturalness and similarity to the target speaker identity. In this paper, we present a brief summary of the state-of-the-art techniques for VC, followed by a detailed explanation of the challenge tasks and the results that were obtained.
1. Introduction
VCC 2018 extended a common evaluation framework for voice conversion by reducing training data, adding non-parallel conversion, and linking VC evaluation with speaker-verification spoofing. It used shared tasks and evaluation procedures to compare systems and address practical challenges.
- Motivation: Voice conversion transforms source speaker identity while preserving the source speech’s linguistic information.The technique has potential applications including speaking aids, expressive voice generation, singing effects, silent speech interfaces, and accent conversion.
- Prior challenge: VCC 2016 established common evaluation methodologies and protocols for benchmarking voice conversion systems using clean parallel speech data.The challenge involved source and target speakers reading the same utterances in professional recordings, with 17 participating research groups.
- Challenge updates: VCC 2018 reduced participant training data by half and added the non-parallel Spoke task alongside the parallel Hub task.The Spoke task used source and target speakers reading different utterance sets, while both tasks shared a large-scale crowdsourced listening evaluation.
- Challenge updates: VCC 2018 also assessed whether newly developed conversion systems could support automatic speaker-verification spoofing research.The challenge sought to bridge the VC and ASV communities because the systems could enhance the ASVspoof 2015 database.
- Paper scope: The paper summarizes VC techniques, describes the challenge tasks and evaluation design, and reports the resulting system comparisons.Its evaluation framework covered participant guidelines, the common dataset, and assessment of diverse VC systems.
2. Voice Conversion
Voice conversion commonly uses data-driven mappings to alter segmental and prosodic speech features while preserving linguistic content. Parallel systems learn from aligned utterance pairs, whereas non-parallel systems use different utterances and require specialized approaches.
- Frameworks: Data-driven VC develops a conversion function that modifies segmental and prosodic speech features to achieve speaker conversion.Parallel and non-parallel VC are the two main frameworks described for this purpose.
- Parallel VC: Parallel VC trains on source-target utterance pairs aligned by dynamic time warping so corresponding frames share linguistic information.The aligned frames form a supervised training dataset for learning the conversion function.
- Non-parallel VC: Non-parallel VC uses different source and target utterances, making it more valuable but more challenging than parallel VC.The paper identifies pseudo-parallel construction, adaptation of canonical functions, and latent speaker-independent phonemic variables as typical approaches.
- Feature extraction and conversion: VC can map source acoustic features directly to target acoustics or map speaker-independent phonetic posteriors to target acoustic features.The posterior-to-acoustic framework can be trained using only target-speaker speech and therefore supports both parallel and non-parallel VC.
- Waveform generation: Converted speech waveforms can be generated with deterministic vocoders or neural data-driven vocoders such as WaveNet and SampleRNN.Neural waveform modeling was developed to address quality degradation associated with source-filter approximations.
3. The Voice Conversion Challenge 2018
VCC 2018 evaluated speaker conversion in parallel Hub and non-parallel Spoke tasks using shared speech data, participant rules, and crowdsourced tests. Naturalness and target-speaker similarity were measured across submitted systems.
- Hub task: The Hub task used parallel source-target corpora, included phonetic transcriptions, and required systems for all source-target pair combinations.Participants converted source speakers toward target speakers using the common challenge dataset.
- Participation guidelines: Manual editing during conversion and manual transcriptions of evaluation data were prohibited, although automatic speech recognition could generate linguistic information.Manual system optimization was permitted only during training.
- Evaluation: The mandatory Hub entries were evaluated for target-speaker similarity and naturalness through organizer-run listening tests.Participants submitted waveforms and system descriptions after generating converted evaluation materials.
- Spoke task: The Spoke task evaluated non-parallel conversion with the same target-speaker data but different source speakers and different utterances.The Spoke task followed the Hub task’s participation guidelines.
- Common dataset: The challenge dataset used clean DAPS recordings from professional US English speakers, with about 13 minutes of speech per speaker downsampled to 22.05 kHz.Most non-speech sounds were manually removed, and the recordings were freely available online.
- Speaker selection: Twelve speakers were selected, with four target speakers shared across Hub and Spoke and separate source-speaker groups for each task.Target selection emphasized distinctive timbre, while source speakers were divided between the two tasks.
- Dataset design: Each speaker had 81 sentences; Hub source and target speakers shared sentences, whereas Spoke source and target speakers used different sentences.The evaluation used 35 test sentences released roughly one week before submission, with 16 Hub source-target combinations.
- Perceptual evaluation: Crowdsourced listening tests measured naturalness on a 1–5 scale and similarity using a Same/Different judgment paradigm.Evaluation sets contained 44 utterances: 32 naturalness items and 12 similarity items from four systems.
4. Evaluation results
The challenge evaluated diverse VC systems across Hub and Spoke tasks using perceptual measures of naturalness and target-speaker similarity. N10 generally led performance, while gender and non-parallel conditions exposed substantial difficulty.
- Participation and systems: 23 teams submitted Hub systems, and 11 additionally participated in the Spoke task.Participants included university research teams and industry-related companies, with several using Merlin or sprocket baselines.
- Participation and systems: Only 3 teams used additional data, while teams using WaveNet reported external training data.Other reported approaches included LSTM-RNNs, GMMs, feed-forward DNNs, CycleGAN, and DRM plus ARBM.
- Baseline and systems: N10 used speaker-independent content-posterior extraction, speaker-dependent LSTM-RNN prediction, and a speaker-dependent WaveNet vocoder.The extractor was trained with hundreds of hours of external speech and aligned phonetic transcriptions.
- Hub task results: N10 outperformed the other Hub systems in same-gender naturalness, while the sprocket baseline remained competitive and exceeded most systems overall.Only N10 and natural source and target speech significantly outperformed the baseline in overall naturalness.
- Hub task results: 0.3 MOS points separated same-gender and cross-gender Hub naturalness, falling from 3.08 to 2.78 for the other systems.N10 remained the top-performing cross-gender system without a significant naturalness reduction.
- Hub task results: N10 achieved the best Hub similarity, near natural target speech, while N17 provided comparable similarity and naturalness; both used WaveNet.Most teams exceeded 50% similarity, and the baseline remained competitive despite several systems surpassing it.
- Spoke task results: Spoke naturalness dropped by 0.09 MOS points on average versus Hub, or 0.17 points when N10 was excluded.The results were described as reflecting the greater complexity of non-parallel conversion.
- Spoke task results: Excluding N10, cross-gender Spoke naturalness fell from 2.95 to 2.59, a 0.36-point MOS decrease; N10 remained stable.The baseline performed very well for same-gender pairs but declined significantly for cross-gender pairs.
5. Conclusion
VCC 2018 provided a common framework for developing and evaluating voice conversion systems. One submitted system achieved high perceived quality and target-speaker similarity, while spoofing capability requires separate analysis.
- VCC 2018 continued the use of a common framework for developing and evaluating voice conversion systems.
- One submitted system achieved an average quality score of 4.1 on a five-point scale.
- About 80% of its converted speech samples were judged to match the target speaker.
- Listening-test results do not directly reflect the spoofing capability of the voice conversion systems.
A. Breakdown of listeners
The listener sample was characterized by age categories and self-reported accents. The supplied passages identify the relevant table and its purpose but provide no category counts.
- Table 3 breaks down the listening-test subjects by age category and self-reported accent.
B. Similarity to source speakers
The listening test measured whether converted samples sounded different from their source speakers. Almost all samples were judged to represent different speakers, except for D02.
- Almost all voice-converted samples were judged different from their source speakers, except D02.
- Figures 11 and 12 report similarity-to-source-speaker results for the Hub task.
C. Scatter plots of ASR WER and MOS
Scatter plots compare naturalness with ASR word error rates for the Hub and Spoke tasks. The reported reasonable correlation indicates that voice conversion frameworks generally need to process linguistic information properly.
- Figures 13 and 14 plot naturalness scores against ASR word error rates for the Hub and Spoke tasks, respectively.
- The results show a reasonable correlation between systems’ word error rates and mean opinion scores.
- The correlation indicates that voice conversion frameworks generally need to process linguistic information properly.
D. Statistical significance
The study groups systems by whether their naturalness and target-speaker similarity scores differ significantly, separately for the Hub and Spoke tasks. These comparisons show that voice conversion remains statistically distinguishable from natural target speech, although N10 generally outperforms other systems.
- Wilcoxon signed-rank tests with Bonferroni correction (α = 0.01) grouped systems without significant differences in naturalness or target-speaker similarity.The groupings cover Hub-task naturalness and similarity, and Spoke-task naturalness and similarity.
- The differences between N10 and T00, the target speaker’s natural speech, remain statistically significant.This indicates that converted speech has not yet reached the statistical level of natural target speech.
- N10 is statistically better than every other system across all evaluations except Spoke-task similarity.
- For Spoke-task similarity, N10 and N17 do not differ significantly.