Source-linked AI summary
The Zero Resource Speech Challenge 2017
Ewan Dunbar, Xuan Nga Cao, Juan Benjumea, Julien Karadayi, Mathieu Bernard, Laurent Besacier, Xavier Anguera, Emmanuel Dupoux
TL;DR
The paper addresses how speech systems can discover subword and word units without textual or linguistic resources, especially across languages and speakers. It presents a two-track challenge with cross-language and talker-adaptation tests, and reports that simple bottom-up clustering remained strongest in the unsupervised subword setting while spoken-term systems traded matching quality against coverage and segmentation quality.
Problem
Many languages lack sufficient textual or linguistic resources, motivating speech systems that discover useful units directly from raw speech.
Method
The challenge evaluates subword modelling and spoken term discovery, testing systems tuned on three development languages against two surprise languages and new speakers.
Results
Simple bottom-up clustering still seems to work best in the unsupervised case, while spoken-term systems show trade-offs between matching, coverage, grouping, and segmentation quality.
Takeaways & Limitations
The challenge extends prior successful strategies with multilingual training and additional neural and time-series tools while testing generalization beyond development conditions.
Takeaways & Limitations
The spoken-term-discovery strategy for improving subword features fares relatively poorly, possibly because its autoencoder architecture had previously been reported as suboptimal.
Abstract
from arXiv · showhide
We describe a new challenge aimed at discovering subword and word units from raw speech. This challenge is the followup to the Zero Resource Speech Challenge 2015. It aims at constructing systems that generalize across languages and adapt to new speakers. The design features and evaluation metrics of the challenge are presented and the results of seventeen models are discussed.
1. INTRODUCTION
The challenge targets speech technology without textual or linguistic resources, motivated by the scarcity of such resources for many languages. It evaluates systems for discovering subword and word-like units while testing generalization across languages and speakers.
- Zero-resource speech systems aim to operate without textual or linguistic resources, addressing languages where transcripts are sparse, unavailable, or limited by absent orthographies.
- Such systems could support speech services for speakers of under-resourced languages and assist field linguists in documenting endangered languages.
- They may also provide quantitative tools for studying how sociolinguistic input affects normal or abnormal language and cognitive development.
- The 2017 challenge has subword-modelling and spoken-term-discovery tracks, requiring systems to discover useful units without linguistic resources.Subword models represent speech sounds robustly to talker variation, while spoken term discovery finds recurring fragments with timestamps and class labels.
- Its innovations test architecture and hyperparameter generalization on unseen surprise languages after development on three languages.
2. THE CHALLENGE
The challenge defines two discovery tasks: learning language-relevant speech representations and finding word-like fragments. Its design separately probes cross-language learning and adaptation to unseen speakers.
- Track one: Subword modelling: Subword modelling transforms acoustic features into representations evaluated for phoneme discrimination on novel speakers.Training uses unlabelled speech in a particular language, and the learned representation is applied to new samples of running speech.
- Track two: Spoken term discovery: Spoken term discovery outputs boundaries and category labels for proposed word tokens, without requiring an exhaustive parse of the speech.
- Participants tune architectures and hyperparameters on three development languages, then submit fixed systems for evaluation on two undisclosed surprise languages.
- The subword task uses skewed speaker distributions in training and only new speakers in testing to measure adaptation from limited evidence.
3. DATA SETS
The datasets are designed to test generalization across languages, data scales, and speaker distributions. Development resources cover English, French, and Mandarin, while surprise data uses two undisclosed languages.
- Development data contains English, French, and Mandarin corpora with evaluation software and train/test splits for the two tracks.
- Surprise data contains two new languages whose identities are withheld, providing speech-only training resources and a test set for track-one evaluation.
- Training-set sizes range from 2.5 to 45 hours, supporting evaluation with both limited and large data quantities.
- Speaker distributions include a high-frequency family and lower-frequency outsiders, while test files span 1-second, 10-second, and 120-second durations.
- The surprise languages are German, sourced from LibriVox, and Wolof, from the corpus described in.
4. EVALUATION
Evaluation covers subword discriminability and three spoken-term-discovery perspectives: matching, lexicon discovery, and word segmentation. The metrics measure phoneme-level similarity, coverage, clustering quality, token recovery, and boundary accuracy.
- Subword modelling: Subword modelling is evaluated with an unsupervised ABX task that measures discriminability between sound categories using a dissimilarity metric.The challenge emphasizes phoneme structure while de-emphasizing speaker identity, emotion, and channel effects.
- Subword modelling: Track-one results are organized by within- and between-talker scores, development and surprise languages, and test-file durations across submitted systems.
- Subword modelling: ABX evaluation uses phone-triplet minimal pairs and reports within-talker and across-talker error rates after averaging across speakers, contexts, and phone contrasts.
- Spoken term discovery: Spoken term discovery is evaluated as matching, lexicon discovery, and word segmentation, allowing distinct ways of finding word-like units.
- Matching: Matching uses NED for phonemic similarity and coverage for the fraction of the discoverable corpus covered by discovered fragments.
- Lexicon discovery: Lexicon discovery uses grouping precision, recall, and F-score plus type precision, recall, and F-score, restricting types to three-to-twenty-segment words.
- Word segmentation: Word segmentation uses token precision, recall, and F-score together with boundary precision, recall, and F-score for three-to-twenty-segment tokens.
5. DESCRIPTION OF SUBMITTED SYSTEMS
Seventeen submitted systems used diverse strategies for zero-resource speech discovery, spanning clustering, multilingual neural embeddings, spoken-term-discovery refinement, and out-of-domain supervision.
- Submission overview: Sixteen systems were received for track one and three for track two, although two track-one submissions were excluded from the overview.Their results remained available on the challenge leaderboard.
- Track-one strategies: Bottom-up frame-level clustering used PLP or transformed acoustic representations with a parallel Dirichlet process Gaussian mixture model sampler.Heck et al.'s system combined PLP representations with three learned feature transformations before clustering.
- Track-one strategies: Unsupervised multilingual embedding systems clustered frames by language, trained multitask neural networks on the resulting labels, and used a bottleneck layer as the representation.The systems varied their acoustic inputs, including MFCC-based configurations.
- Evaluation: Track-two evaluation was organized around matching and grouping discovered fragments, with results summarized using scores computed over corpus subparts.The caption warns that the reported F-scores are not directly related to the precision and recall scores.
- Track-one strategies: Spoken-term-discovery refinement trained stacked autoencoders on DTW-aligned frame pairs from matched acoustic motifs to improve bottleneck features.The matched pairs were supplied by an STD system.
- Cross-track strategies: Other approaches used supervised acoustic models trained on Japanese or ten languages, while track two included exhaustive-segmentation k-means and supervised-ASR-based matching systems.Track two's baseline used locally sensitive hashing, graph clustering, and a Mandarin-specific DTW threshold.
6. RESULTS
Track-one results favored simple unsupervised clustering, while track two exposed a trade-off between matching quality, coverage, grouping, and segmentation quality.
- Track one: 9.7 average ABX score on development languages made Heck et al.'s unsupervised baseline better than second-ranked [C2] at 10.7.The 9.7 result was not evaluated on the surprise languages.
- Track one: Monolingual training performed better within language, but multilingual systems generally improved substantially over the baseline.All remaining systems except [P] used multilingual training and showed substantial baseline improvements.
- Track one: The spoken-term-discovery refinement strategy performed relatively poorly, partly because Yuan et al. used autoencoders previously reported as suboptimal for improving acoustic features.The authors suggest Siamese networks as a possible alternative.
- Track two: System [K] achieved development-language NED comparable to partly supervised [G1] and [G2], while those systems achieved higher grouping recall than [K].The comparison reflects a trade-off between exhaustive-parse coverage and grouping quality.
- Track two: System [K] produced better token and boundary segmentation quality than the other systems, presumably because its coverage was higher.Its exhaustive parse guarantees full coverage but risks poorer matches.
7. CONCLUSION
The challenge extends earlier successful strategies while introducing multilingual training, but simple bottom-up clustering remains strongest among unsupervised track-one systems.
- Contributions: The 2017 challenge builds on bottom-up clustering and spoken-term-discovery training from the previous challenge while adding multilingual training.Additional tools include neural-network training and time-series models.
- Conclusions: Simple bottom-up clustering still appears to work best in the unsupervised case.This conclusion is stated at the challenge level for track one.
- Conclusions: Strong out-of-domain supervised labels remain a benchmark for unsupervised systems in both tracks.The conclusion points to Japanese and Hungarian supervision as examples.
- Future work: Future work could combine spoken-term discovery with subword modelling and exploit the feature improvements demonstrated in track one.The challenge remains open to new strategies for discovering phoneme- and word-like units without linguistic resources.
http://arxiv.org/ps/1712.04313v1
The passage identifies a figure named "zr2speakers.png" and states that it is available in PNG format.
- The figure is named "zr2speakers.png".
- The figure is available for access.
- The figure is provided in PNG format.