Source-linked AI summary
AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, Ming Li
TL;DR
Multi-speaker Mandarin TTS needs suitable annotated data because Mandarin’s tonal and prosodic complexity is not well served by existing English-only resources. The paper introduces AISHELL-3 and a Tacotron-2-based speaker-aware baseline, whose evaluations report high speaker similarity and generalization to unseen speakers. The corpus and baseline are intended to support robust multi-speaker Mandarin synthesis and zero-shot voice-cloning research.
Problem
The lack of a publicly available multi-speaker Mandarin dataset suitable for TTS training makes research more difficult and limits comparable objective evaluation.
Method
The paper constructs AISHELL-3 and trains a Tacotron-2-based multi-speaker TTS system with speaker-embedding feedback constraint.
Results
The system achieves high voice similarity by speaker embedding similarity and equal error rate, and generalizes to speakers unseen during training.
Takeaways & Limitations
AISHELL-3 provides an open Mandarin resource and baseline for multi-speaker TTS, including research on zero-shot voice cloning.
Takeaways & Limitations
Speaker genders are biased toward females, and the age distribution centers on young adults around 20 years old.
Abstract
from arXiv · showhide
In this paper, we present AISHELL-3, a large-scale and high-fidelity multi-speaker Mandarin speech corpus which could be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. We present a baseline system that uses AISHELL-3 for multi-speaker Madarin speech synthesis. The multi-speaker speech synthesis system is an extension on Tacotron-2 where a speaker verification model and a corresponding loss regarding voice similarity are incorporated as the feedback constraint. We aim to use the presented corpus to build a robust synthesis model that is able to achieve zero-shot voice cloning. The system trained on this dataset also generalizes well on speakers that are never seen in the training process. Objective evaluation results from our experiments show that the proposed multi-speaker synthesis system achieves high voice similarity concerning both speaker embedding similarity and equal error rate measurement. The dataset, baseline system code and generated samples are available online.
1. INTRODUCTION
The paper addresses the need for controllable multi-speaker TTS, especially for Mandarin, by introducing AISHELL-3 and a baseline system for speaker-conditioned synthesis. The corpus fills a lack of publicly available multi-speaker Mandarin data suitable for training and comparable evaluation.
- Motivation: TTS maps input text specifications to target speech, but its one-to-many nature permits multiple valid voices and speaking styles.The paper motivates additional specifications to obtain more flexible and consistent behavior.
- Motivation: Speaker identity is an intuitive control signal that lets multi-speaker acoustic models separate perceptual voice identity from textual content.These systems explicitly condition synthesis on the desired speaker.
- Research gap: Mandarin TTS faces challenges from complex tonal and prosodic structures, while existing VCTK resources contain only English recordings.The lack of a public multi-speaker Mandarin dataset increases research difficulty and limits comparable objective indicators.
- Contribution: AISHELL-3 provides roughly 85 hours of high-fidelity Mandarin recordings from 218 native speakers with Chinese-character and pinyin transcriptions.The corpus is introduced to fill the vacancy in publicly available Mandarin multi-speaker TTS resources.
- Contribution: The paper also presents a multi-speaker TTS baseline trained on AISHELL-3, with objective evaluations showing behavior consistent with prior VCTK experiments using the same architecture.The baseline is positioned as a reference system for the new corpus.
2. THE AISHELL-3 DATASET
AISHELL-3 is a high-fidelity Mandarin multi-speaker corpus containing recordings, speaker information, and pronunciation-aligned transcripts. Its data cover varied textual domains, while speaker attributes are unevenly distributed.
- Corpus overview: AISHELL-3 contains 88035 neutral-emotion recordings from 218 native Mandarin speakers, recorded indoors with high-fidelity microphones.The recordings use 44.1 kHz, 16-bit audio, with microphones positioned 20 cm from speakers.
- Corpus overview: The textual content spans domains including smart-home voice commands, news reports, and geographic information.
- Annotations: Each recording includes Chinese-character transcripts and human-verified pinyin corresponding to the speaker’s actual pronunciation.The pinyin annotations address pronunciation ambiguities that dictionary lookup from characters can miss, including homographs and tone sandhi.
- Speaker attributes: Speaker genders are biased toward females, and the age distribution is centered on young adults around 20 years old.These distributions constrain how broadly speaker attributes are represented in the corpus.
3. BASELINE SYSTEM
The baseline system combines speaker-agnostic text processing and waveform generation with a speaker-aware Tacotron-2 acoustic model. Speaker embeddings condition synthesis and an additional feedback loss encourages similarity between synthesized and reference voices.
- Baseline architecture: The baseline comprises a textual frontend, acoustic model, and neural vocoder that map text and speaker specifications to audio waveforms.The frontend analyzes textual input, the acoustic model predicts feature vectors, and the vocoder converts them into waveforms.
- Speaker-agnostic subsystem: The speaker-agnostic frontend combines pinyin parsing, phonetization, and LSTM-based prosody prediction, while MelGAN vocoding ignores speaker labels.Prosodic annotations are added to the phoneme sequence before acoustic modeling.
- Speaker-agnostic subsystem: Prosodic prediction uses character embeddings, BMES word segmentation, and part-of-speech tags encoded by two bidirectional LSTM layers.Two fully connected output layers produce hierarchical BMES-tagged prosodic predictions.
- Speaker-aware subsystem: The speaker-aware acoustic model extends Tacotron-2 with a speaker encoder that extracts discriminative information from variable-length reference audio.The encoder uses a ResNet-based speaker-verification network with global mean-std pooling and linear projection.
- Feedback constraint: The feedback objective combines mel-spectrogram error, stop-token binary cross-entropy, and cosine similarity between synthesized and ground-truth speaker embeddings.The cosine-similarity term is weighted by α = 1.0, encouraging voice similarity during training.
4. EXPERIMENTS
Experiments evaluate speaker similarity on seen validation speakers and unseen test speakers. Data preparation addresses alignment speed and long-sentence robustness, while objective results show high similarity and a roughly 5% EER gap between unseen and seen speakers.
- Experimental setup: The acoustic model and vocoder are trained on AISHELL-3, while the separately trained speaker encoder uses 775289 utterances from two Mandarin corpora.The evaluations assess synthetic samples produced by this baseline system.
- Experimental setup: 44 of 218 speakers form the unseen test set, while the remaining 174 speakers train the model and provide seen-speaker validation data.The split is made broadly at the speaker-identity level.
- Data preparation: Energy-based VAD trims initial silence from training Mel-spectrograms, accelerating attention-alignment formation by 10 times under 2 GTX-1080Ti GPUs.The alignment controls timing and rhythm in synthesized utterances.
- Data preparation: Long-sentence instability produces unreliable attention alignment or unnatural speaking rates, so training concatenates randomly selected same-speaker sentences as augmentation.The augmentation generates 30000 longer samples using N probabilities P2 = 0.6, P3 = 0.2, and P4 = 0.2.
- Objective evaluation: Evaluations use seen-speaker validation and unseen-speaker test groups to measure voice imitation and generalization to reference speakers absent from training.Each trial includes text-dependent and text-independent synthesis with textual content unseen during training.
- Objective evaluation: Same-speaker embeddings form dense, separated clusters in t-SNE visualizations, and synthesized samples are compared with recordings using cosine similarity and SV-EER.The evaluation compares synthesized embeddings with ground-truth audio embeddings.
- Objective evaluation: Around 5% separates test-set and validation-set EER, while validation-speaker EERs are only 0.65% and 0.60% higher than the SV-baseline.The authors report this pattern as consistent with a comparable VCTK experiment despite different speaker-embedding training data.
5. CONCLUSION
The paper presents AISHELL-3 as a public Mandarin corpus for multi-speaker TTS and evaluates a speaker-embedding-feedback baseline. The authors conclude that the corpus supports Mandarin multi-speaker TTS research and voice imitation with generalization evaluation.
- Conclusion: AISHELL-3 contains 88035 high-fidelity utterances from 218 native speakers and includes hand-labeled full-pinyin annotations.The paper presents it as a publicly available Mandarin speech corpus for training multi-speaker TTS systems.
- Conclusion: The baseline combines speaker-embedding feedback constraints with prosodic label prediction, silence trimming, and long-form sentence concatenation during data preparation.The trained model is evaluated for speaker similarity and generalization capacity.
- Conclusion: Objective and perceptual experiments support the conclusion that the corpus is valuable for multi-speaker TTS research in Mandarin.The authors report that the system produces natural speech imitating reference speakers.