Source-linked AI summary
Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese
Shiyu Zhou, Linhao Dong, Shuang Xu, Bo Xu
TL;DR
The paper addresses Mandarin Chinese ASR by applying the self-attention Transformer and comparing syllable and CI-phoneme output units. It proposes a cascading decoder for mapping sub-word sequences to words; on HKUST, the syllable-based model reports 28.77% CER, compared with 28.0% for the cited state-of-the-art system.
Problem
The paper notes that sequence-to-sequence attention models had been less explored for Mandarin ASR, where syllables are logical pronunciation units and offer an alternative to CI-phoneme outputs.
Source: §1 · p. 1Method
The paper applies a self-attention Transformer to Mandarin ASR, compares syllable and CI-phoneme modeling, and uses a greedy cascade to map sub-word sequences into words.
Source: §1 · p. 1Results
28.77% CER is reported for the syllable-based Transformer on HKUST, better than its CI-phoneme counterpart and competitive with the 28.0% state-of-the-art result.
Source: §1 · p. 1Takeaways & Limitations
On HKUST, the syllable-based Transformer performs better than the CI-phoneme-based counterpart, supporting syllables as the better of these two modeling units in this setting.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Sequence-to-sequence attention-based models have recently shown very promising results on automatic speech recognition (ASR) tasks, which integrate an acoustic, pronunciation and language model into a single neural network. In these models, the Transformer, a new sequence-to-sequence attention-based model relying entirely on self-attention without using RNNs or convolutions, achieves a new single-model state-of-the-art BLEU on neural machine translation (NMT) tasks. Since the outstanding performance of the Transformer, we extend it to speech and concentrate on it as the basic architecture of sequence-to-sequence attention-based model on Mandarin Chinese ASR tasks. Furthermore, we investigate a comparison between syllable based model and context-independent phoneme (CI-phoneme) based model with the Transformer in Mandarin Chinese. Additionally, a greedy cascading decoder with the Transformer is proposed for mapping CI-phoneme sequences and syllable sequences into word sequences. Experiments on HKUST datasets demonstrate that syllable based model with the Transformer performs better than CI-phoneme based counterpart, and achieves a character error rate (CER) of \emph{$28.77\%$}, which is competitive to the state-of-the-art CER of $28.0\%$ by the joint CTC-attention based encoder-decoder network.
1. Introduction
The paper applies the self-attention Transformer to Mandarin Chinese ASR and compares syllable and CI-phoneme modeling units. It proposes a cascading decoder for word-sequence output and reports that the syllable-based model achieves 28.77% CER on HKUST.
1. Introduction
- The Transformer replaces recurrent sequence processing with attention over global input-output dependencies, allowing more parallelization.The paper applies it to Mandarin ASR after its reported NMT performance.Source: §1 · p. 1
- Mandarin syllables are logical pronunciation units; the paper cites their longer duration, fixed inventory, character correspondence, and avoidance of OOVs as modeling advantages.The work uses around 1,400 tone-marked pinyins; the introduction says each written character corresponds to a syllable.Source: §1 · p. 1
- 28.77% CER is reported for the Transformer syllable-based model, which outperforms its CI-phoneme counterpart and is competitive with the 28.0% state-of-the-art result.The comparison is on HKUST; the 28.0% result is from a joint CTC-attention encoder-decoder network.Source: §1 · p. 1
- The paper compares Transformer models using syllables or CI-phonemes and proposes a greedy cascading decoder to map either sub-word sequence to words.The decoder is intended to approximate Pr(W|X) by cascading sub-word and word-sequence models.Source: §1 · p. 1
2. Related work
Prior Mandarin ASR work explored character and syllable approaches, while syllable-based acoustic and CTC models had shown promise. This paper extends syllable-versus-CI-phoneme comparison to sequence-to-sequence attention models and situates syllable-to-word conversion as sequence labeling.
2. Related work
- Earlier Mandarin attention-based work used Pinyin during training for character models or emitted Chinese characters directly, rather than modeling syllables as the output unit.The passage describes Character-Pinyin work and direct character output.Source: §2 · pp. 1–2
- The paper extends prior CTC-based comparisons of syllable and CI-phoneme modeling units to sequence-to-sequence attention-based models.Earlier Mandarin CTC work reported that syllable-based models can outperform CI-phoneme-based models.Source: §2 · p. 2
- Syllable-based acoustic models had been studied with GMM and context-dependent approaches, with prior work reporting promising performance.The passage also notes earlier evidence of syllable-based CTC outperforming CI-phoneme-based CTC on Mandarin ASR.Source: §2 · p. 2
- Converting Chinese syllable sequences into words can be treated as labeling unsegmented sequence data, and prior work used RNNs with CTC for direct conversion.Source: §2 · p. 2
3. System overview
The system uses a self-attention encoder-decoder Transformer, with multi-head attention and position-wise feed-forward layers. A second Transformer maps predicted CI-phoneme or syllable sequences to words in a greedy cascade.
3.1. Transformer model
- The encoder maps input symbols to continuous representations, and the decoder generates output symbols one at a time using those representations.The Transformer relies on self-attention and position-wise fully connected layers rather than recurrence.Source: §3.1 · p. 2
3.1.1. Multi-head attention
- Multi-head attention uses learned projections to run attention in parallel across heads, then concatenates and projects their outputs.The paper describes it as allowing attention to information from different representation subspaces and positions.Source: §3.1.1 · p. 2
3.1.2. Transformer model architecture
- The encoder stacks attention and feed-forward sublayers with residual connections and normalization; the decoder adds encoder-output attention and masks future positions.Positional encodings are added to both stacks; an input-dimension transformation is also used for 80-dimensional log-Mel features.Source: §3.1.2 · pp. 2–3, Figure 1 · p. 2
3.2. Greedy cascading decoder with the Transformer
- The greedy cascade treats acoustic-to-subword and subword-to-word mapping as sequence-to-sequence transformations, then combines their estimates to approximate Pr(W|X).The first Transformer selects a sub-word sequence from observation X; a second selects the word sequence from that sub-word sequence.Source: §3.2 · p. 3
4. Experiment
Experiments on HKUST compare Transformer-based CI-phoneme and syllable systems, including training configurations, word conversion, speed perturbation, and frame rates. The syllable system outperforms its CI-phoneme counterpart, and its 28.77% CER is competitive with prior systems.
4.1. Data
- The HKUST conversational telephone-speech corpus contains 150 hours, with 873 training calls and 24 test calls.Features are 80-dimensional log-Mel filterbanks, stacked with three preceding frames and downsampled to a 30 ms frame rate.Source: §4.1 · p. 3
4.2. Training
- Training uses base (D512-H8) and big (D1024-H16) Transformer models, but only the base model for subword-to-word conversion.CI-phoneme and syllable alignments are generated using GMM-HMM alignments and a lexicon before Transformer training.Source: §4.2 · p. 3
- 4.70% and 4.15% CER are lower bounds for the CI-phoneme and syllable models, respectively, from converting test alignments into word sequences.The experiment uses trained Pr(W|s) models for this conversion.Source: §4.2 · p. 3
4.3. Results of CI-phoneme and syllable based model
- 28.77% CER is the best syllable-model result with 0.9 and 1.1 speed perturbation; the CI-phoneme model becomes very slightly worse with perturbation.The authors attribute the contrast to syllables' longer duration and greater invariance than CI-phonemes.Source: §4.3 · p. 4
- The syllable-based Transformer outperforms its CI-phoneme counterpart on HKUST, and the big model performs better than the base model for both units.The results are summarized in Table 2.Source: §4.3 · pp. 3–4, Table 2 · p. 4
4.4. Comparison with previous works
- 28.77% CER is comparable to the joint CTC-attention model's 28.9% without an external language model, but below its 28.0% with a separate RNN-LM.The CI-phoneme Transformer is comparable to deep multidimensional residual learning with 9 LSTM layers in a hybrid system; the syllable model provides over 6% relative CER improvement over the referenced result.Source: Table 3 · p. 4, §4.4 · p. 4
4.5. Comparison of different frame rates
- CER worsens as frame rate increases for both units, with relatively slow degradation from 30 to 50 ms and rapid deterioration from 50 to 70 ms.The authors report that frame rates between 30 and 50 ms perform relatively well.Source: Table 4 · p. 4, §4.5 · p. 4
5. Conclusions
The paper applies the Transformer to Mandarin Chinese ASR on HKUST and compares syllable and CI-phoneme modeling units. Syllable-based modeling performs better, and experiments verify the effectiveness of a greedy cascading decoder.
5. Conclusions
- Syllable-based Transformer modeling performs better than its CI-phoneme-based counterpart on HKUST datasets.The comparison uses syllable and CI-phoneme sequences as modeling units.Source: §5 · p. 4
- The paper applies the Transformer, based entirely on self-attention without RNNs or convolutions, to Mandarin Chinese ASR on HKUST datasets.The authors report verifying its effectiveness on these datasets.Source: §5 · p. 4
- The proposed greedy cascading decoder maximizes Pr(W|s)Pr(s|X), and experiments with both output units verify its effectiveness.The conclusion states that this permits maximizing the posterior Pr(W|X).Source: §5 · p. 4