Source-linked AI summary
Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion
Yi Zhao, Wen-Chin Huang, Xiaohai Tian, Junichi Yamagishi, Rohan Kumar Das, Tomi Kinnunen, Zhenhua Ling, Tomoki Toda
TL;DR
VC research needs comparable evaluation because system performance depends on the dataset, especially for challenging cross-lingual conversion. VCC 2020 created a common database and evaluated 33 submissions across intra-lingual semi-parallel and cross-lingual tasks. Several systems matched target-speaker similarity in the intra-lingual task, none reached human-level naturalness there, and the best cross-lingual MOS exceeded 4.0.
Problem
VC systems are difficult to compare because performance strongly depends on the speech datasets, while cross-lingual VC requires conversion across different languages and nonparallel data.
Method
VCC 2020 constructed and distributed a common multilingual database for intra-lingual semi-parallel and cross-lingual VC, then evaluated submissions with large-scale crowd-sourced listening tests.
Results
Several systems achieved target-level speaker similarity in intra-lingual semi-parallel VC, none achieved human-level naturalness there, and the best cross-lingual systems exceeded 4.0 MOS.
Takeaways & Limitations
Cross-lingual VC remained more difficult overall, while additional analyses identified reference-audio language as important for interpreting speaker-similarity scores.
Takeaways & Limitations
In cross-lingual VC, all systems had significantly lower similarity than natural speech, and fewer than 80% of converted samples were judged the same as target speakers.
Abstract
from arXiv · showhide
The voice conversion challenge is a bi-annual scientific event held to compare and understand different voice conversion (VC) systems built on a common dataset. In 2020, we organized the third edition of the challenge and constructed and distributed a new database for two tasks, intra-lingual semi-parallel and cross-lingual VC. After a two-month challenge period, we received 33 submissions, including 3 baselines built on the database. From the results of crowd-sourced listening tests, we observed that VC methods have progressed rapidly thanks to advanced deep learning methods. In particular, speaker similarity scores of several systems turned out to be as high as target speakers in the intra-lingual semi-parallel VC task. However, we confirmed that none of them have achieved human-level naturalness yet for the same task. The cross-lingual conversion task is, as expected, a more difficult task, and the overall naturalness and similarity scores were lower than those for the intra-lingual conversion task. However, we observed encouraging results, and the MOS scores of the best systems were higher than 4.0. We also show a few additional analysis results to aid in understanding cross-lingual VC better.
1. Introduction
VC transforms speaker or other non-linguistic information while preserving linguistic content, but comparing systems is difficult because performance depends strongly on datasets. VCC 2020 introduced more challenging evaluation, including cross-lingual conversion, and assessed systems through large-scale listening tests.
- Motivation: VC transforms non- or para-linguistic information in speech while preserving its linguistic information.Applications include speaking aids, accent-conversion language learning, voice changers, singing effects, and silent-speech telecommunication.
- Motivation: Comparing VC techniques is difficult because their performance strongly depends on the speech datasets used by individual researchers.
- Voice Conversion Challenge: VCC compares different VC techniques on a freely available common dataset while bringing teams together to study shared challenges.Earlier challenges progressively moved from parallel to nonparallel training.
- VCC 2020: VCC 2020 focused on cross-lingual VC, requiring speaker conversion between different languages with completely nonparallel training.The challenge included neural vocoders, encoder-decoder networks, GANs, and seq2seq mapping networks, followed by large-scale perceptual evaluation.
2. Past voice conversion challenges and what we learned
The earlier VCC editions established progressively more challenging datasets and training conditions for comparing VC systems. Their listening tests documented substantial progress in naturalness and speaker similarity by 2018.
- VCC 2016: VCC 2016 used a parallel database of four American English speakers, each recording 162 common sentences.Participants produced converted speech for all source-target pairs, with 54 converted samples per speaker.
- Evaluation: VCC 2016 evaluated converted speech using five-point naturalness MOS and a four-point same/different speaker-similarity test.
- VCC 2018: VCC 2018 introduced both a smaller parallel database and a nonparallel database with 23 participating teams.The speakers were four American English speakers different from those used in 2016.
- VCC 2018: 4.1 naturalness MOS and about 80% same-speaker judgments were reported for the best 2018 system in both parallel and nonparallel tasks.The best system used a phone encoder and neural vocoder.
3. Tasks, databases, and timeline for Voice Conversion Challenge 2020
VCC 2020 defined intra-lingual semi-parallel and cross-lingual speaker-conversion tasks using a multilingual database. The challenge supplied controlled training and evaluation data for source-target conversion across 16 and 24 speaker pairs.
- Objectives: The VCC 2020 objective was to make converted speech sound like a target speaker while retaining the source sentence’s linguistic content.
- Task 1: Intra-lingual semi-parallel VC: Task 1 used same-language data combining a smaller parallel corpus with a larger nonparallel corpus.The setting was intended to assess whether limited parallel recordings aid model learning alongside nonparallel VC methods.
- Task 2: Cross-lingual VC: Task 2 used source English speech and target-speaker speech in Finnish, German, or Mandarin, making the task nonparallel across languages.Systems had to disentangle source content and speaker characteristics, then replace the source identity with the target identity.
- Database construction: 56 speakers from the EMIME bilingual Finnish/English, German/English, and Mandarin/English database were recorded in a semianechoic chamber and down-sampled to 24 kHz.
- Database construction: Each source and target speaker had 70 training sentences, while Task 1 included 20 parallel sentences and evaluation used 25 shared test sentences.Participants built systems for 4 × 4 = 16 Task 1 and 4 × 6 = 24 Task 2 source-target pairs.
- Timeline: Training data were released on March 9, 2020, and converted evaluation audio was due May 29, 2020, after two months and two weeks of system development.
4. Participants and submitted systems
VCC 2020 received 33 submissions spanning diverse feature-conversion and waveform-generation methods. Encoder-decoder systems were most common, while teams also used GANs, parallel mappings, and several neural or traditional vocoders.
- Participants: 33 submissions included three organizer baselines, with 31 teams entering Task 1 and 28 entering Task 2.Twenty-six teams participated in both tasks.
- Participants: The organizers anonymized participants as Team IDs T01 to T33 to support scientific analysis of submitted VC methods.
- System components: VC systems generally contain feature-conversion and waveform-generation modules, both of which mainly use neural networks.
- Feature conversion: Feature-conversion models comprised encoder-decoder, GAN-based, and parallel spectral feature-mapping categories.Encoder-decoder and GAN models can use nonparallel data, whereas parallel spectral mapping requires paired source-target data.
- Feature conversion: Encoder-decoder models were the most popular feature-conversion structure, reported by 23 Task 1 teams and 22 Task 2 teams.The reported subtypes included PPG-VC, ASR-TTS, TTS-assisted VC, and auto-encoder approaches.
5. Subjective evaluation
The study used subjective listening tests to evaluate converted speech, focusing on naturalness and speaker similarity. Objective evaluation was also conducted but was reported separately.
- Subjective listening tests evaluated the naturalness and speaker similarity of converted speech.
- The evaluation methodology was designed to compare converted samples using listener judgments.
- An objective evaluation was also performed, with its results reserved for a separate paper.
5.1. Motivations and evaluation methodology
The evaluation measured naturalness and speaker similarity, with refinements to handle cross-lingual conversion. These refinements included multilingual references and natural-speech ratings.
- The evaluation aimed to measure naturalness and speaker similarity of converted samples.
- Task 2 additionally rated natural speech in German, Finnish, and Mandarin alongside English speech.
- Task 2 additionally presented German, Finnish, and Mandarin reference speech for judging speaker similarity across languages.
5.2. Experimental setup
The experiment used crowd-sourced English and Japanese listeners, standardized five-point naturalness and four-point similarity ratings, and selected samples and references to cover the challenge systems.
- English and Japanese listeners were recruited to reflect possible cross-lingual speech-to-speech translation audiences.
- Naturalness was rated on a five-point scale from Bad to Excellent, with pronunciation quality separated from sound quality.
- Speaker similarity used a four-level same/different paradigm based on speaker characteristics rather than content or language.
- Task 2 used English and target-speaker L2-language references, selecting three English and two L2 recordings per speaker pair.
- Each evaluation set contained three audio samples, covering quality, speaker similarity, and a similarity reference across 62 webpages.
- Task 1 samples were rated 6 times, whereas Task 2 samples were rated 4 times.
5.3. Evaluation results – English listeners –
Listening tests show substantial progress in intra-lingual voice conversion: leading systems reached human-level speaker similarity, but not human-level naturalness. Cross-lingual conversion remained more difficult, with lower overall scores despite encouraging naturalness results.
- Task 1 — Naturalness: T10 and T13 achieved the highest Task 1 naturalness MOS values, significantly outperforming T11, the best system from VCC 2018.T29, T25, T27, and T30 were also not significantly different from T11.
- Task 1 — Naturalness: None of the Task 1 systems achieved human-level naturalness, because the highest scores remained significantly below natural source and target speech.The authors conclude that basic intra-lingual VC naturalness was not completely solved.
- Task 1 — Speaker similarity: Eight Task 1 systems significantly improved speaker similarity over T11, with no significant differences among those eight systems.The systems were T10, T22, T27, T13, T33, T23, T29, and T07.
- Task 1 — Speaker similarity: Over 90% of converted Task 1 samples from the eight best teams were judged the same as target speakers, with similarity not significantly different from natural target speech.This was described as human-level speaker similarity for the basic intra-lingual task.
- Task 1 — Joint analysis: Task 1 systems showed naturalness–similarity trade-offs, while T10 was closest to natural target speech on both measures.Most systems still needed to improve either similarity or naturalness.
- Task 2 — Cross-lingual conversion: Cross-lingual conversion produced lower overall naturalness and similarity than intra-lingual conversion, although the best systems exceeded 4.0 MOS.T10’s naturalness dropped by 0.75 MOS points relative to the intra-lingual task, and all cross-lingual systems remained significantly below natural-speech similarity.
6. Further analysis of VCC 2020 results
The analysis examines listener-group agreement, reference-language effects, and target-speaker-language effects in cross-lingual voice conversion. Japanese and English judgements were generally correlated, while L2 references and target-language differences affected similarity and naturalness.
- Listener-group agreement: Correlation values of 0.965, 0.984, 0.973, and 0.968 showed strong agreement between Japanese and English listeners across naturalness and similarity in Tasks 1 and 2.The figures compare listener-group scores for each task and evaluation dimension.
- Listener-group agreement: English listeners rated T28, T19, and T03 lower than Japanese listeners in both tasks, coinciding with very poor speech intelligibility.These systems were identified as exceptions to the general listener-group agreement.
- Listener-group agreement: Non-native listeners were considered acceptable for assessing cross-lingual VC performance to some extent despite some inconsistencies with native listeners.This conclusion follows from the observed correlations and identified discrepancies.
- Reference-language effects: Subjects generally gave lower speaker-similarity scores when the same target speaker’s reference audio was in an L2 language.The analysis attributes this pattern to the difficulty of human speaker verification across languages, with exceptions for T24, T19, and T18.
- Reference-language effects: The language of reference audio is important because a different language from the input may lower subjects’ speaker-similarity scores through a language barrier.This conclusion concerns subjective similarity evaluation in cross-lingual VC.
- Target-speaker-language effects: The target speaker’s language affected both speaker similarity and naturalness, with most systems highest for German targets and lowest in similarity for Mandarin targets.The authors note that linguistic distance from English may partially explain this pattern, but further investigation is required.
7. Conclusion
VCC 2020 introduced a new database for intra-lingual semi-parallel and cross-lingual voice conversion and evaluated 33 submissions. The results showed strong progress in similarity, persistent naturalness gaps in Task 1, and lower but encouraging cross-lingual performance.
- Conclusion: 33 submissions, including 3 baselines, were received for the two VCC 2020 tasks.Participants had two months and two weeks to build their systems.
- Conclusion: Several systems matched target-speaker similarity in intra-lingual semi-parallel VC, but none achieved human-level naturalness.These findings came from crowd-sourced listening tests.
- Conclusion: Cross-lingual VC had lower overall naturalness and similarity than intra-lingual VC, although the best systems exceeded 4.0 MOS.The authors characterize cross-lingual conversion as the more difficult task.
- Conclusion: Additional analyses addressed listener-group agreement, L2 reference audio, and target-speaker language to improve understanding of cross-lingual VC.The authors state that these insights could help future improvement and evaluation.
A. Vocoder type comparisons
The vocoder comparison presents Task 1 and Task 2 naturalness results with systems colored by vocoder category. MOS values are ordered by their means.
- Vocoder type comparisons: Figure 18 presents naturalness results for Task 1 and Task 2, with MOS scores arranged by mean.Bars are colored according to the vocoder category used by each system.
- Vocoder type comparisons: Vocoder categories are encoded by bar color in the naturalness comparison.The figure compares the corresponding vocoder types used for each system.
B. Evaluation results Japanese listeners –
Japanese listeners showed the same general evaluation tendencies as English listeners across the two tasks. Task 1 included systems reaching target-speaker similarity, while Task 2 systems remained below target-speaker similarity.
- Evaluation results Japanese listeners: 206 unique valid Japanese listeners evaluated naturalness and speaker similarity in Tasks 1 and 2.Figures 19–22 report the Japanese-listener results using the same formats as the English results.
- Evaluation results Japanese listeners: Figures 19 and 20 show Task 1 naturalness and similarity results, while Figures 21 and 22 show the corresponding Task 2 results.The figures also report groupings or ordering of systems according to the captions.
- Evaluation results Japanese listeners: Several systems achieved target-speaker-level similarity in Task 1, whereas all systems scored below target-speaker similarity in Task 2.These were the general tendencies among Japanese listeners.
- Evaluation results Japanese listeners: T10 was rated as natural as the target speakers in Task 1, with no significant difference in naturalness or speaker similarity.This result was specific to the Japanese-listener evaluation.