Source-linked AI summary
Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes
Bo Li, Yu Zhang, Tara Sainath, Yonghui Wu, William Chan
TL;DR
Large grapheme vocabularies and language-specific pronunciation resources complicate multilingual speech processing. This paper models text as UTF-8 bytes in end-to-end A2B recognition and B2A synthesis. The multilingual A2B model outperforms monolingual baselines by 4.4% relatively on average, and B2A matches single-language grapheme systems in subjective evaluations.
Problem
Large grapheme vocabularies create label sparsity in multilingual models, while phonetic systems require costly, language-specific pronunciation dictionaries.
Source: §1 · p. 1Method
A2B converts speech to UTF-8 Unicode bytes and B2A generates speech from byte sequences, using a 256-value output vocabulary.
Source: §1 · p. 1Results
4.4% relatively on average: multilingual A2B outperforms monolingual baselines, and byte models outperform grapheme models in monolingual and multilingual settings.
Source: §5 · p. 4Takeaways & Limitations
Byte representations support a single model across Unicode languages without changing input processing, and the compact models are described as suitable for on-device applications.
Source: §1 · p. 1The reported 1MOS is worse than because the test set contains OOV items.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We present two end-to-end models: Audio-to-Byte (A2B) and Byte-to-Audio (B2A), for multilingual speech recognition and synthesis. Prior work has predominantly used characters, sub-words or words as the unit of choice to model text. These units are difficult to scale to languages with large vocabularies, particularly in the case of multilingual processing. In this work, we model text via a sequence of Unicode bytes, specifically, the UTF-8 variable length byte sequence for each character. Bytes allow us to avoid large softmaxes in languages with large vocabularies, and share representations in multilingual models. We show that bytes are superior to grapheme characters over a wide variety of languages in monolingual end-to-end speech recognition. Additionally, our multilingual byte model outperform each respective single language baseline on average by 4.4% relatively. In Japanese-English code-switching speech, our multilingual byte model outperform our monolingual baseline by 38.6% relatively. Finally, we present an end-to-end multilingual speech synthesis model using byte representations which matches the performance of our monolingual baselines.
1. INTRODUCTION
Large grapheme vocabularies create label sparsity in multilingual speech processing, motivating Unicode-byte representations. The paper introduces byte-based A2B recognition and B2A synthesis; its multilingual A2B outperforms language-dependent baselines, while B2A performs similarly to grapheme-based single-language systems.
1. INTRODUCTION
- Grapheme vocabularies can be very large across languages, leaving long-tail graphemes poorly covered and worsening label sparsity in multilingual models.The passage notes that Chinese, Japanese, and Korean have tens of thousands of graphemes.Source: §1 · p. 1
- A2B maps speech to UTF-8 Unicode bytes, while B2A generates speech from byte sequences; both use a 256-value vocabulary and require no input-processing changes for additional Unicode languages.The authors describe the compact models as suitable for on-device applications.Source: §1 · p. 1
- For Japanese and Korean, A2B performs better than A2C, while English and Spanish have the same performance.The authors attribute the difference to label sparsity for large grapheme vocabularies and byte sharing across graphemes.Source: §1 · p. 1
- The progressively expanded multilingual A2B model outperforms all language-dependent systems.The authors add languages after convergence, retaining small mixing portions for existing languages; they report that this approach also beats training from scratch.Source: §1 · p. 2
- B2A has similar quantitative subjective evaluation performance to grapheme-input systems trained on single languages across English, Mandarin, and Spanish.The authors present this as a more compact multilingual TTS model.Source: §1 · p. 2
2. MULTILINGUAL AUDIO-TO-BYTE (A2B)
The A2B section presents Unicode bytes as a language-independent output representation and describes the model architecture and multilingual input strategy. It explains how UTF-8 byte sequences represent graphemes and why this avoids changing the model structure for new languages.
2.1. Model Structure
- The LAS-based A2B encoder uses five unidirectional LSTM layers, and its two-layer unidirectional LSTM decoder uses four-head additive content-based attention.The output softmax has 256 dimensions, one for each possible byte value.Source: §2.1 · p. 2
- The A2B front end stacks three preceding log-mel frames and downsamples to a 30 ms frame rate; language sampling ratios are adjusted to address data imbalance.The system also concatenates a one-hot language ID vector to every encoder and decoder layer; sampling ratios are selected from intuition and empirical observations.Source: §2.1 · p. 2
2.2. Output Unit
- Word-piece models require rebuilding when a language or data is added, and multilingual word-piece training is challenging with unbalanced grapheme distributions.The passage contrasts these issues with fixed-size data-driven word-piece vocabularies.Source: §2.2 · p. 2
- Graphemes can still require large output vocabularies: the paper's Japanese vocabulary contains over 4.8k characters.The authors explore decomposing graphemes into Unicode-byte sequences.Source: §2.2 · p. 2
- A2B emits variable-length UTF-8 bytes one at a time; multi-byte graphemes require learning within-grapheme and longer-range byte dependencies.For single-byte languages such as English, byte output is equivalent to grapheme output.Source: §2.2 · p. 2
- Unicode bytes represent scripts across languages without changing the model structure, unlike grapheme systems that require output-softmax changes for new symbols.The paper identifies this language independence as useful for multilingual and code-switching speech modeling.Source: §2.2 · p. 2
3. MULTILINGUAL BYTE-TO-AUDIO (B2A)
The B2A model adapts Tacotron 2 to byte-sequence inputs for speech synthesis. Its text encoder and attention supply the decoder, which is conditioned on a speaker embedding; a separately trained WaveRNN converts mel spectrograms to waveform.
3.1. Model Structure
- B2A encodes byte embeddings with three convolutional layers and a bidirectional LSTM, then uses location-sensitive attention to aggregate text encodings for the decoder.The attention mechanism accounts for attention history when computing normalized aggregation weights.Source: §3.1 · p. 2
- The autoregressive decoder receives aggregated byte encodings and a fixed speaker embedding, which functions as language ID because training uses one speaker per language.Source: §3.1 · p. 2
- A separately trained WaveRNN converts the predicted mel spectrograms into time-domain waveforms.Source: §3.1 · p. 2
4.1. Byte for ASR
The ASR experiments compare byte and grapheme outputs across four languages, then evaluate multilingual training and code-switching. The final multilingual byte model outperforms the language-dependent baselines on average, while byte models also improve Japanese-English code-switch recognition.
4.1.1. Data
- The supervised ASR training set covers English, Japanese, Spanish, and Korean, with around 76,000 hours; each language’s test set has roughly 15,000 utterances, or 20 hours.Evaluation uses WER except for Japanese, where TER avoids word-segmentation ambiguity.Source: §4.1.1 · pp. 2–3, Table 2 · p. 3
4.1.2. Language Dependent Systems
- In monolingual comparisons, A2B byte-output models with the same model structure achieved 4.0% relative WER reduction on Japanese and 2.6% on Korean versus grapheme systems; English and Spanish performance was identical.Byte outputs use a 256-dimensional softmax and have no OOV problem for any language.Source: §4.1.2 · p. 3
4.1.3. Multilingual ASR Systems
- Continuing training from the English-Japanese byte model with Spanish data trains faster and performs better than training from scratch; the resulting C2 matches Japanese and slightly exceeds Spanish language-dependent models.Adding Korean by continuing C2 brings D1 closer to language-dependent models except for English, while giving the best multilingual English performance reported at that point.Source: §4.1.3 · p. 3
- Adding a 1-hot language ID vector to all A2B layers yields dramatic error reduction over the larger decoder without the vector.Increasing decoder layers from 2 to 6 improved English but degraded Japanese, potentially from over-fitting; the vector combines language-independent weight matrices with language-dependent biases.Source: §4.1.3 · p. 4
- 4.4% relatively: final multilingual model D4 outperforms the language-dependent models on average.The comparison uses Table 1.Source: Table 1 · p. 3, §4.1.3 · p. 4
4.1.4. Error analysis
- 38.6% relatively: monolingual A2B outperforms A2C on the 476 English-Japanese code-switching sentences, measured by TER.The multilingual A2B model outperforms A2C by 4.2% relatively on the same subset; Table 3 reports these results. In examples, byte models correctly recognize English proper nouns such as “wi-fi” and “Google” in Japanese utterances where grapheme models miss characters.Source: §4.1.4 · p. 4, Table 3 · p. 4
4.2. Byte for TTS
The section evaluates multilingual byte-to-audio synthesis with crowdsourced naturalness ratings and reports MOS comparable to monolingual grapheme models. It also notes that code-switching TTS quality lacks a good evaluation metric.
4.2.1. Data
- The TTS models used 44 hours of North American English, 37 hours of Mandarin, and 44 hours of North American Spanish speech, each recorded by a female speaker.Audio was synthesized at 24 kHz in 16-bit format; crowdsourced MOS ratings used the Absolute Category Rating scale from 1 to 5 in 0.5-point increments.Source: §4.2.1 · p. 4
4.2.2. Multilingual TTS System
- The multilingual B2A model achieved subjective naturalness MOS comparable to the state-of-the-art monolingual model for English, Mandarin, and Spanish.The model also read code-switching text, but the authors lack a good metric for evaluating its quality.Source: §4.2.2 · p. 4
5. CONCLUSIONS
The authors conclude that Unicode-byte models outperform grapheme models in monolingual and multilingual settings, with multilingual A2B gains over monolingual baselines and B2A matching monolingual TTS. They note that MOS is lower than because the test set contains OOV items.
5. CONCLUSIONS
- 4.4% relative average improvement over monolingual baselines was achieved by multilingual A2B, and it improved 38.6% relatively over monolingual baselines on Japanese-English code-switching speech.The paper also reports byte models outperforming grapheme models in both multilingual and monolingual models.Source: §5 · p. 4
- Multilingual B2A models matched the performance of monolingual baselines in TTS.The authors investigated Unicode bytes as a language representation for end-to-end ASR and TTS.Source: §5 · p. 4
- A single byte-based model can serve many languages without model-structure changes, sharing representations that are crucial for languages with large grapheme vocabularies.The authors connect this representation sharing to multilingual processing.Source: §5 · p. 4
- MOS was worse than because the test set contained OOV items.Source: §5 · p. 4