Source-linked AI summary
Do speech foundation models really learn words?
Robin Huo, Ewan Dunbar
TL;DR
Word-identity discrimination in speech foundation models may be explained by phonological form rather than word representations. Using residualization to remove phoneme information, the paper finds that later HuBERT and wav2vec 2.0 layers retain word information and that phoneme residualization benefits lexicon discovery.
Problem
Word-identity probe success may reflect phonological form, so it does not by itself establish representations of words beyond local phonetic content.
Method
The study linearly residualizes phoneme, diphone, and triphone information from HuBERT and wav2vec 2.0 representations before probing word identity and evaluating word discovery.
Results
Later layers retain substantial word-identity information after phoneme residualization, with peak accuracies above 90%, and lexicon discovery benefits from phoneme residualization.
Takeaways & Limitations
The findings indicate that these models encode word information partly independently of local phonological content, while not specifying its semantic or syntactic nature.
Takeaways & Limitations
Practical use requires aligned phonemic labels, which are not easily available for most tasks.
Abstract
from arXiv · showhide
Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.
1. Introduction
Word-identity probing cannot establish that speech models encode words beyond phonological form, because phonemes often identify words and predict their properties. This study uses residualization to test whether HuBERT and wav2vec 2.0 retain word information after phonological information is removed.
- 1. Introduction: Word-identity probe success may reflect phoneme encoding rather than representations of syntactic, semantic, or identity-related word properties.A phonological representation can identify most words and support prediction of their properties from word identity.
- 1. Introduction: The study residualizes phonological information before probing frame-level word identity in English-pretrained HuBERT and wav2vec 2.0.The authors remove a component encoding phonological information and test whether later representations remain predictive of words.
- 1. Introduction: After residualization, later layers generally remain predictive of word identity, indicating word information beyond diphones or triphones.The authors interpret this as context-integrated information that creates a locally identifiable token.
- 1. Introduction: The study does not determine whether the retained word information is lexical-semantic or syntactic.The result separates word information from local phonological content but does not specify what information about words is encoded.
2. Related methods
The paper uses linear residualization to remove phoneme-related information from existing speech-model representations rather than training separately specialized disentangled representations. The authors note that residualization can either leave phoneme information behind or remove additional information.
- 2. Related methods: Residualization estimates embeddings from categorical phoneme labels and subtracts those predictions to remove phoneme-level information linearly.The procedure can be viewed as subtracting phoneme-level means from representation frames.
- 2. Related methods: Residualization may fail in two directions: incomplete phoneme removal or removal of information beyond phonemes.The study prioritizes the first error because excessive removal would strengthen its conclusion.
- 2. Related methods: Unlike modular disentanglement methods, this approach interprets existing representations that were not generally trained to specialize in separate factors.The paper presents this as appropriate for understanding and using pretrained representations.
- 2. Related methods: The authors distinguish their approach from methods that collapse selected representation axes associated with a factor.They note that such axes are not true subspaces and require choosing an arbitrary number of dimensions.
3. Method
The experiments probe frame-level representations from HuBERT and wav2vec 2.0 across layers and remove phoneme, diphone, or triphone information before predicting word identity. Standardized representations and cross-validation support both residualization and probe validation.
- 3. Method: The study analyzes the convolutional output and all 12 transformer layers of base HuBERT and wav2vec 2.0 on LibriSpeech dev-clean.Frames are aligned to 42 phoneme categories and 8,217 word categories plus silence.
- 3. Method: Representations are scaled and centered before residualization and probing, and the main experiment always includes this standardization.The method excludes factor examples that exactly contain the corresponding complete word when fitting the regression.
- 3. Method: Ridge regression predicts representation frames from one-hot phoneme, diphone, or triphone factors, whose predictions are then subtracted.The four tested variants are phonemes, left diphones, right diphones, and triphones.
- 3. Method: A five-fold cross-validated linear probe predicts word identity from individual frames after residualization.A separate phoneme probe checks whether phoneme information remains linearly recoverable.
4. Results
Residualizing phonemes preserves strong word-identity classification in later layers, while triphone removal lowers accuracy but leaves performance above simple baselines in several settings. Validation shows phoneme information is substantially reduced, though not completely eliminated.
- 4. Results: Residualization substantially reduces phoneme-probe accuracy, although it does not reach chance level.The modal phoneme baseline is 11.6%, and the authors argue that differing word- and phoneme-probe patterns indicate use of different information.
- 4. Results: More than 90% word-identity accuracy remains in peak layers after phoneme residualization: HuBERT layers 9–10 and wav2vec 2.0 layers 7–8.Residualization preserves the raw representations’ overall layer pattern, including weak convolutional-layer performance and sharp transformer-layer increases.
- 4. Results: Diphone removal decreases accuracy slightly more in earlier layers across both models.The effect is smaller than the substantial reductions observed after triphone removal.
- 4. Results: Triphone removal substantially reduces accuracy, but word classification remains above modal-word and triphone baselines for word lengths 3–6.The reported baselines are 14.9%, 0.4%, 0.1%, and ≪0.1% by lengths 3–6, and 50.3% by triphone.
- 4. Results: The residualized representations therefore retain word information partly independently of local phonological information.The result is not merely explained by encoding short phoneme sequences.
5. Word discovery
Residualizing phoneme information from HuBERT representations improves unsupervised word discovery, whereas removing diphone or triphone information harms segmentation. The phoneme-based improvement is robust across segmentation settings but depends on aligned phonemic labels.
- 5. Word discovery: The evaluation uses HuBERT layer 9 representations, aligned phonemic labels, boundary detection, and k-means clustering into hypothesized word categories.Performance is assessed against aligned word boundaries and types using token F1, R-value, and normalized edit distance.
- 5. Word discovery: Residualizing phoneme information improves NED, token F1, and R-value across several segmentation hyperparameter settings.These metrics indicate improved lexicon quality and word-boundary segmentation after phoneme removal.
- 5. Word discovery: The observed improvement is a validation of the insight that removing phonemic information can expose higher-level information, not a direct estimate of practical performance.The method requires aligned phonemic labels, which are typically unavailable in practice.
- 5. Word discovery: Phoneme residualization improves word discovery, but diphone and triphone residualization degrades segmentation performance.Local phonemic transition information may support boundary detection, so removing it can harm segmentation.
6. Discussion
Later HuBERT and wav2vec 2.0 layers retain word-predictive information after local phonemic content is removed, suggesting that these models treat words as units. Residualization also benefits lexicon discovery, although the experiments do not establish what specific word information is encoded.
- 6. Discussion: Later HuBERT and wav2vec 2.0 layers predict words fairly robustly after phoneme or short-phoneme-sequence information is erased.The result separates word prediction from the models’ encoding of local phonological form and context.
- 6. Discussion: The analysis cannot determine what word properties are encoded, because representations may integrate longer-range context sufficient for word prediction without forming context-independent word representations.Residualization was limited to phoneme sequences of length ≤3 because higher-order regression fits were unreliable.
- 6. Discussion: Residualization largely removes phoneme-prediction ability while preserving word-identity prediction, and it improves lexicon discovery rather than the already-strong word probe.This motivates evaluating disentanglement with more complex downstream tasks than linear probing alone.
7. Summary of contributions
The study dissociates word-identity information from local phonological form in English-pretrained HuBERT and wav2vec 2.0, and applies this insight to improve word discovery.
- 7. Summary of contributions: HuBERT and wav2vec 2.0 encode substantial local word-identity information that cannot be fully explained by local phonological features.A straightforward application of the finding improves performance in word discovery.
8. Limitations
The main practical limitation is that the residualization procedure requires aligned phonemic labels, while the residualization design can also leave some phoneme information in the representations.
- 8. Limitations: Applying the approach in practice is constrained by the need for aligned phonemic labels, which are not easily available for most tasks.Future work could investigate approximate or inferred labels instead of precise phonemic alignments.
- 8. Limitations: The exclusion of frames whose phoneme, diphone, or triphone fully contains the word can leave phoneme identity information after residualization.Without that exclusion, phoneme accuracy falls below 16% in layers 1–10, indicating that the exclusion policy affects removal effectiveness.