Source-linked AI summary
Unified Mandarin TTS Front-end Based on Distilled BERT Model
Yang Zhang, Liqun Deng, Yasheng Wang
TL;DR
Mandarin TTS front-ends require complex pipelines for prosodic structure prediction and grapheme-to-phoneme conversion, creating substantial construction and deployment burdens. The paper uses a Chinese BERT encoder with multi-task learning and then applies TinyBERT distillation. The resulting unified model has 25% of benchmark pipeline size while achieving competitive performance on both tasks.
Problem
Mandarin TTS front-ends separately implement prosodic structure prediction and grapheme-to-phoneme conversion through long, laborious pipelines with large storage and computation requirements.
Source: §1 · p. 1Method
A Chinese BERT encoder jointly models PSP and G2P with multi-task learning, followed by two-stage TinyBERT knowledge distillation.
Source: §1 · p. 1Results
Takeaways & Limitations
Abstract
from arXiv · showhide
The front-end module in a typical Mandarin text-to-speech system (TTS) is composed of a long pipeline of text processing components, which requires extensive efforts to build and is prone to large accumulative model size and cascade errors. In this paper, a pre-trained language model (PLM) based model is proposed to simultaneously tackle the two most important tasks in TTS front-end, i.e., prosodic structure prediction (PSP) and grapheme-to-phoneme (G2P) conversion. We use a pre-trained Chinese BERT[1] as the text encoder and employ multi-task learning technique to adapt it to the two TTS front-end tasks. Then, the BERT encoder is distilled into a smaller model by employing a knowledge distillation technique called TinyBERT[2], making the whole model size 25% of that of benchmark pipeline models while maintaining competitive performance on both tasks. With the proposed the methods, we are able to run the whole TTS front-end module in a light and unified manner, which is more friendly to deployment on mobile devices.
1. Introduction
Mandarin TTS front-ends traditionally separate prosodic structure prediction and grapheme-to-phoneme conversion into complex pipelines. This work unifies both tasks with BERT-based multi-task learning and distills the model for smaller deployment.
Motivation
- The front-end predicts prosodic boundaries and character pronunciations, but existing rule-based and statistical components form a laborious pipeline that hinders mobile deployment.PSP covers prosodic word, phrase, and intonational phrase boundaries, while G2P converts Mandarin characters into pronunciation.Source: §1 · p. 1
Proposed approach
- A unified Chinese BERT encoder jointly models PSP and G2P through multi-task learning.The approach replaces separate front-end components with shared text representations and task-specific outputs.Source: §1 · p. 1
Model compression
- Two-stage TinyBERT distillation compresses the unified front-end to facilitate deployment on edge devices.The distillation strategy targets a more compact model while retaining the unified architecture.Source: §1 · p. 1
Results
- 25% model size and improvements of 0.31% in polyphone disambiguation accuracy, 1.45% in PPH F1, and 0.52% in IPH F1 are reported for the distilled model.These results are presented as state-of-the-art performance on both PSP and G2P tasks.Source: §1 · p. 1
2. Proposed Method
The proposed front-end unifies PSP and G2P with a Chinese BERT encoder, parallel task-specific prediction layers, multi-task training, and TinyBERT compression. It uses contextual character representations to handle pronunciation and prosodic prediction while reducing deployment cost.
2. Proposed Method
- The unified model uses a pre-trained Chinese BERT encoder with parallel prediction layers for polyphone disambiguation and prosodic structure.The whole model is fine-tuned with multi-task learning and then compressed by distillation.Source: §2 · pp. 1–2
2.1. Pre-trained BERT for text representations
- Character-based BERT generates one linguistic representation per input character without requiring word segmentation.The encoder is pre-trained on large-scale Mandarin text and is used to capture contextual and semantic information for PSP and G2P.Source: §2.1 · p. 2
2.2. Polyphone disambiguation
- Polyphone disambiguation treats pronunciation prediction as classification, using a BERT-based MLP with softmax to produce probabilities over pronunciation classes.Monophonic characters are handled by a dictionary during inference, while polyphonic characters receive the maximal-probability pronunciation class.Source: §2.2 · p. 2
2.3. Prosodic structure prediction
- Prosodic structure prediction assigns one of four break labels—NO BREAK, prosodic word, prosodic phrase, or intonational phrase—after each character.An MLP or BLSTM prediction layer replaces the traditional sequence of three prediction steps.Source: §2.3 · p. 2
2.4. Multi-task learning
- Multi-task learning attaches the polyphone and prosody outputs to BERT in parallel and combines their losses with balancing weights.Mixed batches from separate pronunciation- and prosody-labelled datasets update the shared model continuously, while only the labelled task's loss is counted for each sentence.Source: §2.4 · p. 2
2.5. TinyBERT distillation
- TinyBERT compression uses two-stage general and task-specific distillation, transferring BERT's attention and hidden-state knowledge to a smaller student model.The student is first distilled on open-domain data, then adapted using TTS-specific data and fine-tuned in the front-end; attention and hidden-state losses are summed across student layers.Source: §2.5 · p. 3
3. Experiments and discussion
Experiments evaluate datasets, baseline and unified systems, task-specific prediction layers, model size, and inference time. TinyBERT-MTL achieves strong polyphone and prosody results while reducing model size for deployment.
3.1. Datasets and experiment settings
- The evaluation uses separate 9:1 train-test splits for 340581 polyphone-labelled sentences and 231964 prosody sentences, reporting ACC, SENT ACC, and three prosody F1 scores.The polyphone corpus contains 661 polyphone characters, and each sentence contains at least one polyphone character.Source: §3.1 · p. 3
3.2. Systems built
- The systems include WFST and BLSTM G2P baselines, CRF and BLSTM-CRF PSP baselines, single-task BERT variants, and BERT-MTL or TinyBERT-MTL.BERT-MTL uses a BERT encoder, while TinyBERT-MTL uses a distilled TinyBERT encoder.Source: §3.2 · p. 3
3.3. Results and analysis
- 0.31% higher character accuracy and 0.62% higher sentence accuracy let TinyBERT-MTL outperform WFST-based G2P and BLSTM, with a slight loss versus BERT-MTL.BERT-polyphone performs best, while BERT-MTL is only 0.3 accuracy lower; Table 1 reports the polyphone comparison.Source: §3.3 · p. 4, Table 1 · p. 4
- Nearly 2% higher F1 across PW, PPH, and IPH lets TinyBERT-MTL outperform baseline systems in prosodic structure prediction.Sentence-final prosodic breaks are excluded from F1, and CRF-based baselines use Jieba-predicted word segmentation.Source: §3.3 · p. 4, Table 2 · p. 4
- Within BERT-MTL, MLP and BLSTM perform similarly for PSP, while adding a CRF layer slightly degrades results; MLP performs better for polyphone prediction than BLSTM.The authors relate this to finetuned BERT capturing contextual label and polyphone information.Source: §3.3 · p. 4, Table 3 · p. 4, Table 4 · p. 4
- 25% model size makes TinyBERT-MTL more feasible for mobile deployment than benchmark front-end modules.The benchmark comprises WFST-based G2P, BLSTM-CRF PSP, Jieba segmentation, and POS processing.Source: §3.3 · p. 4, Table 5 · p. 4
- TinyBERT-MTL takes 21.8 ms per 64-character sentence on the evaluated CPU server, although it is slower than benchmark models.The timing averages 1000 runs on an Intel Xeon E5-2690 v3 at 2.60GHz.Source: §3.3 · p. 4
4. Conclusions
The paper concludes with a unified end-to-end front-end that jointly predicts polyphone pronunciation and prosodic structure from raw characters. Distillation produces a smaller model that outperforms the benchmark system with 25% of its model size.
4. Conclusions
- The unified end-to-end model accepts raw characters and simultaneously predicts polyphone disambiguation and prosodic structure using BERT and task-specific layers.The authors identify acceleration, closer teacher-model performance, and joint distillation with quantization as future directions.Source: §4 · p. 4
- The final distilled model outperforms the benchmark system on both tasks while using only 25% of the benchmark model size.The model is intended to support smaller-scale deployment, including mobile devices.Source: §4 · p. 4