The Zero Resource Speech Challenge 2019: TTS without T

1. Introduction

• The task addresses low-resource languages that lack the textual resources or expertise needed for traditional synthesis systems.The introduction frames the challenge around children's ability to produce novel sentences without speech annotated with text.Source: §1 · p. 1
• ZR19 tests speech synthesis without text or labels: participants discover subword units from raw target-voice audio, align incoming speech to them, and synthesize novel utterances from novel speakers.It continues the subword-unit discovery track of ZeroSpeech 2017 and 2015.Source: §1 · p. 1
• Submissions may replace the unit-discovery component, the TTS component, or both, including with an end-to-end system.The baseline combines automatic discrete acoustic-unit discovery with a standard TTS system.Source: §1 · p. 1

2. Related work

• Earlier proof-of-concept systems provide the challenge baseline, which combines out-of-the-box acoustic unit discovery with Merlin synthesis.The cited unit-discovery and synthesis components are and, respectively; the prior proof of concept is.Source: §2 · p. 1
• Related methods include binarized autoencoders and siamese networks for unit discovery, plus newer waveform-generation systems such as Wavenet and Tacotron 2.Speaker normalization has also been used to improve discovered categories; the methods cited are –.Source: §2 · p. 1
• ZR19 shares the unannotated source-to-target conversion setting of voice conversion, but requires a small unannotated target corpus, novel-speaker test inputs, phoneme discriminability, and low bitrate.Related work includes joint ASR/TTS training with reconstruction losses [15] and voice-conversion approaches including VAEs, disentangling autoencoders, and GANs.Source: §2 · pp. 1–2

3. Datasets

• English is the Development language and Indonesian is the Surprise language; only Development data may be used for model development.Participants must treat English as low-resource and avoid existing resources; Surprise-language training must use the same procedure, with hyperparameter optimization on Development data or integrated into training.Source: §3 · p. 2
• Each language has Train Voice, Train Unit Discovery, optional Train Parallel, and Test datasets, with new speakers and utterances in the test set.Train Voice has two talkers for Development and one for Surprise; Train Unit Discovery provides read speech from multiple speakers, around 10 minutes per speaker. The test requires resynthesis to rule out returning the original audio unchanged.Source: §3 · p. 2

4. Metrics

• Systems submit resynthesized audio and a pseudo-text embedding, which can use fixed-rate or collapsed-symbol sequences and continuous or discrete vector values.Quantization to a discrete subset is advised to reduce bitrate; the optional Train Parallel set was not used by submitted systems.Source: §4 · p. 2

4.1. Synthesis intelligibility, quality and speaker similarity

• Intelligibility is measured by Character Error Rate from transcription, while naturalness and target-voice similarity use 1-to-5 ratings.Sentence allocation differs by task, and each system, topline, baseline, and original recording is evaluated; participants with catch-trial CER of 0.80 or higher are excluded.Source: §4.1 · pp. 2–3

4.2. Embedding bitrate and quality

• Embedding bitrate is computed as symbol-sequence entropy cost divided by total duration, with the dictionary built from values in the submitted test-set embeddings.A fixed frame-rate phonetic transcription can have higher bitrate than a textual representation because symbols repeat across frames.Source: §4.2 · p. 3
• Unit quality uses theory-neutral machine ABX error rates on cross-speaker minimal pairs, with 0% for gold transcription and 50% at chance.The comparison uses extracted triphone items; distance may be based on cosine divergence, KL divergence, or normalized Levenshtein distance.Source: §4.2 · p. 3
• Because the baseline TTS requires textual input, unit embeddings may be converted to one-hot form before synthesis, potentially losing information.Auxiliary embeddings can expose representations from before quantization for ABX and bitrate evaluation.Source: §4.2 · p. 3
• For 2019, triphone audio is pre-extracted to support systems without fixed frame rates, unlike earlier challenges that supplied longer recordings for later extraction.The topline phone-level language model performs sub-optimally on these files because they begin with unlikely phone sequences.Source: §4.2 · p. 3

5. Toplines and Baselines

• The topline uses Kaldi ASR trained on original transcriptions, with tri-state triphones, 15,000 Gaussian mixtures, and a trigram phone-level language model, then TTS trained on gold labels.Source: §5 · p. 3
• The baseline pairs DPGMM-based acoustic unit discovery with Merlin parametric speech synthesis.Source: §5 · p. 3
• Baseline linguistic inputs encode neighboring phones and their positions in the sentence, excluding prosody, phonetic categories, and part-of-speech.Source: §5 · p. 3

6. Results

• Only five final embeddings beat MFCCs in Surprise; Figure 2 plots embedding quality against log bitrate, with lower-left better.The reported MFCC reference scores are 25.01% for Development and 18.21% for Surprise.Source: §6 · p. 3, Figure 2 · p. 4
• ASR phone decoding scored worse than posteriorgrams and MFCCs (29.85% Dev; 16.09% Surprise), which the authors attribute to a fairly simple supervised model.Source: §6 · p. 4
• FE-11-A outperformed all others in Surprise, CH-14 led final embeddings, and non-discrete auxiliaries generally improved ABX over discrete counterparts at higher bitrate.Source: §6 · p. 4
• MOS correlated with lower CER (Dev: r = −0.83; Surprise: −0.93), while CER–similarity correlations weakened without Gold (Dev: −0.28; Surprise: −0.08).Similarity measures closeness to the target voice only.Source: §6 · p. 4
• Worst-case 95% CI half-widths were 0.07 CER, 0.17 MOS, and 0.28 Similarity for Dev, so system differences require CI context.Source: §6 · p. 4
• CER improved monotonically with bitrate; Figure 3 plots CER against log bitrate for Development and Surprise.Source: Figure 3 · p. 4, §6 · p. 4

• Text-free TTS is possible, with quality on the level of a simple supervised comparison system.Source: §7 · p. 4
• Systems with bitrates similar to phonemic annotations tend to perform poorly, revealing a tradeoff between discretization and synthesis quality.Source: §7 · p. 4
• ABX phone discrimination on discovered representations is correlated with speech synthesis quality.Source: §7 · p. 4
• High-bitrate representations may suffice for low-resource dialogue, while gold phonemic transcriptions show room to improve low-bitrate embeddings.The authors suggest the baseline Ossian system's need for textual input may explain some, but not all, of the weaker low-bitrate performance.Source: §7 · p. 4