PaperlayerView exact revision on arXiv

Source-linked AI summary

The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling

Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, Emmanuel Dupoux

arXiv:2011.11588v2cs.CLcs.SDeess.AS

TL;DR

The paper addresses the lack of benchmarks for evaluating linguistic representations learned from raw speech beyond phonetics. It introduces four zero-shot tests and a CPC–clustering–language-model baseline, which performs above chance across the tests but below text-based systems trained on the same data.

  • Problem

    No established benchmark assessed linguistic representations learned from raw audio beyond the acoustic or phonetic level.

    Source: §1 · p. 2

  • Method

    The paper introduces four zero-shot tests and evaluates a baseline combining CPC representations, k-means-derived pseudo-text, and an LSTM or BERT language model.

    Source: §1 · p. 2

  • Results

    The CPC–clustering–language-model baseline performs above chance on all four tests but worse than text-based models trained on the same data.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results show that spoken language modeling from raw speech is feasible, while leaving room for improvement.

    Source: §5 · p. 8

    Constructing the test sets and metrics still requires significant resources, including a phonetic dictionary.

    Source: §5 · p. 8

Abstract

from arXiv · show

We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics. We present the results and analyses of a composite baseline made of the concatenation of three unsupervised systems: self-supervised contrastive representation learning (CPC), clustering (k-means) and language modeling (LSTM or BERT). The language models learn on the basis of the pseudo-text derived from clustering the learned representations. This simple pipeline shows better than chance performance on all four metrics, demonstrating the feasibility of spoken language modeling from raw speech. It also yields worse performance compared to text-based 'topline' systems trained on the same data, delineating the space to be explored by more sophisticated end-to-end models.

1 Introduction

Self-supervised audio methods raise the possibility that raw-speech systems learn linguistic structure beyond acoustics, but no established benchmark had assessed such representations beyond phonetics. The paper introduces four zero-shot metrics and a CPC–clustering–language-model baseline that performs above chance across them, while trailing text-based toplines.

1 Introduction

  • Raw-audio self-supervision’s strong low-label ASR results suggest these systems may learn language modeling as well as acoustic representations, but evaluation beyond phonetics lacked an established benchmark.The paper identifies this gap in the zero-resource setting of learning linguistic representations with little or no textual resources.Source: §1 · pp. 1–2
  • Four black-box, zero-shot metrics assess learned representations at phonetic, lexical, syntactic, and semantic levels using interpretable scores and no trained evaluation classifier.The metrics use either distances between embeddings or pseudo-probabilities over test inputs.Source: §1 · p. 2, Table 1 · p. 2
  • A CPC–k-means–language-model baseline performs above chance on all four metrics, while a text-based BERT topline trained on the same data performs better.The language model is trained on pseudo-text produced by clustering learned speech representations; the paper presents the benchmark and baselines as open source.Source: §1 · p. 2

2 Related work

Earlier zero-resource benchmarks concentrated on acoustic and lexical levels, while black-box language-model evaluations offered methods for probing syntax and lexical knowledge. This work draws on those evaluation traditions while addressing higher-level linguistic abilities.

Zero Resource Speech Challenge Series

  • Earlier zero-resource dialogue-system benchmarks mainly evaluated acoustic and lexical abilities, using ABX, TTS opinion scores, and word-segmentation metrics.Word-segmentation measures assume that models explicitly discover words, whereas character-based language models can learn higher-level concepts without explicit word segmentation.Source: §2 · pp. 2–3

Black box NLP

  • Black-box language-model studies use minimal grammatical contrasts to probe syntax, including agreement, and have also applied word probes to lexical knowledge.Prior examples include probability comparisons for grammatical forms in English and Basque agreement tests.Source: §2 · p. 3

3 Methods

The methods pair LibriSpeech training data and four speech-based evaluation metrics with a composite baseline that learns acoustic representations, discretizes them, and trains language models on the resulting units. Text-trained models provide toplines for comparison.

3.1 Training set

  • LibriSpeech 960 hours supplies training speech, with clean-6k Libri-light audio and phonetic transcriptions used to enable comparative text-based toplines.The transcriptions combine the original lexicon with G2P-generated pronunciations, and forced alignment supports the topline systems.Source: §3.1 · p. 3

3.2 Metrics

  • The benchmark evaluates phonetics with ABX, lexical knowledge with sWUGGY, syntax with sBLIMP, and semantic similarity with sSIMI.ABX compares within-category and between-category acoustic distances; sWUGGY contrasts words with nonwords, sBLIMP contrasts grammatical with ungrammatical sentences, and sSIMI compares word-representation similarity with human judgments.Source: §3.2 · pp. 3–5
  • sWUGGY reports pairwise accuracy when a word receives a higher pseudo-probability than its matched nonword.The nonwords are designed to match phonotactics and syllabic structure, with additional constraints for speech synthesis quality and phoneme-frequency matching.Source: §3.2 · p. 4
  • sBLIMP measures whether pseudo-probabilities favor grammatical over ungrammatical sentences, while sSIMI correlates model-derived semantic similarity with human judgments.sSIMI uses Spearman’s rank correlation coefficient ρ.Source: §3.2 · pp. 4–5

3.3 Models

  • The baseline combines CPC acoustic modeling, k-means clustering, and an LSTM or BERT language model trained on discretized speech units.CPC learns representations by predicting future embeddings from an autoregressive context; its contrastive loss uses negative embedding samples.Source: §3.3 · p. 5
  • K-means clusters CPC output features, assigns each feature a discrete unit, and converts audio into unit sequences for language-model training.Clustering is trained on 100 hours of clean LibriSpeech speech.Source: §3.3 · pp. 5–6
  • The language-model baselines include low-budget LSTM and small BERT systems and higher-budget BERT systems; toplines use BERT trained on phonetic transcriptions or forced-aligned phonemes.The paper also compares against pretrained RoBERTa large, trained on a much larger dataset.Source: §3.3 · p. 6

4 Results

All four baseline models perform above chance, but performance varies across tasks; lexical results are substantially above chance, while syntax and semantics leave room for improvement against text-based toplines. The component results show trade-offs in clustering and variation in semantic-layer choices.

4.1 Libri-light ABX

  • 50 clusters gave the ABX sweet spot; CPC-big outperformed CPC-small, while clustering increased error by 60–100%.The reported within- and across-speaker ABX errors cover Libri-light dev-clean and dev-other, before and after clustering.Source: §4.1 · pp. 6–7, Table 4 · p. 7

4.2 sWUGGY spot-the-word

  • All four baselines performed better than chance and n-gram controls on sWUGGY; the selected BERT settings were Δt = 5 and Md = 15.These settings were retained for subsequent pseudo-probability experiments.Source: §4.2 · p. 7

4.3 sBLIMP acceptability

  • sBLIMP results were above chance but considerably below text-based toplines.Source: §4.3 · p. 7

4.4 sSIMI semantic similarity

  • Semantic-similarity scores were modest against BERT trained on larger BPE units; the best transformer layers were in its first third, with max pooling preferred.This contrasts with ABX, where the best layers occur at the transformer’s extremities.Source: §4.4 · p. 7

4.5 Model comparison

  • All four baseline models are above chance across four tasks, but lexical performance is substantially above chance while syntax and semantics leave room against text-based toplines.The text-based toplines were trained on similar amounts of data.Source: §4.5 · p. 8

5 Discussion

The discussion reports that the baseline performs above chance and n-gram controls but below text-based toplines, and identifies both research directions and benchmark-construction costs.

5 Discussion

  • The CPC-plus-clustering-plus-language-model baseline performs above chance and n-gram controls, but below text-based models trained on the same data.Table 5 covers dev and test results across four zero-shot metrics and includes speech- and text-based toplines.Source: Table 5 · p. 8, §5 · p. 8
  • Representation learning, clustering, and transformer components remain untuned for this benchmark, while larger unlabelled audio datasets are identified as a potential way to improve performance.The discussion also points to trying end-to-end models and reducing the resources needed to scale the benchmark beyond English.Source: §5 · p. 8
  • Constructing the benchmark still requires substantial resources, including a phonetic dictionary, aligned speech, grammar, and TTS or trained speakers.The authors call for work to reduce this footprint and scale the benchmark to languages other than English.Source: §5 · p. 8

Broader Impact

The authors frame zero-resource research as relevant to low-resource and textless languages, while noting that open-source benchmarking is intended to limit the risk of concentrating its benefits.

Broader Impact

  • Zero-resource research may help develop speech technology for low-resource languages and languages without textual resources.The authors contrast this with the supervised setting, which they say cannot address such languages.Source: Broader Impact · p. 8
  • Learning language models from raw speech could help address dialect variation, including minority dialects, even for high-resource languages.The authors connect this possibility with more inclusive speech technology.Source: Broader Impact · p. 8
  • The authors note that proprietary development could increase the economic dominance of already-large actors and use an open-source benchmark to engage the community.Source: Broader Impact · p. 8

A Sampling method to balance ngram scores

The sampling procedure builds word–nonword pairs to balance n-gram scores near chance, then stratifies sampling to reduce score imbalances across word lengths and frequencies.

A Sampling method to balance ngram scores

  • The objective selects matching word–nonword pairs so that, for each of M scores, the proportion favoring the word is as close to 50% as possible.The scores may include unigram or bigram character and phone scores.Source: A Sampling method to balance ngram… · p. 12
  • The procedure iteratively pairs an unchosen word with a matching nonword candidate, retaining additions that improve the objective and resampling otherwise.If all candidates fail to produce a new pair, it randomly selects a candidate and continues.Source: A Sampling method to balance ngram… · p. 12
  • Sampling all words together produced near-50% overall accuracy but favored high-frequency or short words, so the authors sampled within frequency and length subcategories.The subcategory approach yielded more balanced scores across those levels.Source: A Sampling method to balance ngram… · p. 12
  • Balancing by sub-paradigm still did not perfectly balance the scores for some linguistic paradigms.Source: A Sampling method to balance ngram… · p. 12

B Supplementary ABX methods and results

The supplementary ABX analysis uses representation sequences to compare sounds and reports that the best phonetic layer and cluster count need not follow the most obvious configurations.

B Supplementary ABX methods and results

C Supplementary spot-the-word results

The supplementary spot-the-word analysis examines how masking settings affect evaluation accuracy and reports frequency and word-length effects for the best system.

C Supplementary spot-the-word results

  • Table S3 reports sWUGGY dev accuracy across masking parameters and estimates runtime relative to the Md = ∆t = 10 base setting.The selected setting is identified as the compromise between accuracy and speed.Source: Table S3 · p. 13
  • Overlapping masking spans improve spot-the-word performance, and a decoding mask size of 15 was best; Md = 15 and ∆t = 5 were selected for an accuracy–speed compromise.Runtime is theoretically inversely proportional to ∆t.Source: C Supplementary spot-the-word results · p. 13, Table S3 · p. 13
  • The CPC-big+BERT system performs below text-based toplines but above chance, with accuracy increasing for more frequent and longer words.The authors suggest longer words may have sparser phonetic neighborhoods, making nonwords harder to salvage into real words.Source: C Supplementary spot-the-word results · p. 13

D Supplementary grammaticality results

Supplementary results show that the best model beats chance on most sBLIMP subsets and that semantic and phonetic performance favor different BERT layers.

D Supplementary grammaticality results

  • 11/12 sBLIMP subsets score above chance and beat phoneme n-gram controls; 9/12 and 10/12 beat unigram and bigram word n-gram controls, respectively.Table S4 compares sentence-acceptability accuracy for the best CPC+k-means-50+BERT model with phone n-gram, text-transformer, and human scores.Source: D Supplementary grammaticality results · pp. 13–14, Table S4 · p. 14
  • Figure S1 compares sWUGGY dev accuracy with phone n-gram baselines and text-based transformer toplines across word-frequency and word-length effects.The caption identifies the comparisons and plot dimensions but does not state specific outcomes.Source: Figure S1 · p. 14

E Supplementary semantic similarity results

  • The best layers for phonetics and semantics are complementary: layers strongest for ABX are weakest for semantic similarity, and vice versa.Table S5 reports sSIMI correlations with human judgments and ABX scores across embedding levels and pooling functions.Source: E Supplementary semantic similarity… · p. 14, Table S5 · p. 14
Loading 2011.11588v2…