Source-linked AI summary
Comparative layer-wise analysis of self-supervised speech models
Ankita Pasad, Bowen Shi, Karen Livescu
TL;DR
Self-supervised speech models achieve strong downstream results, but the properties encoded across their layers and differences among models remain insufficiently understood. The paper uses CCA to analyze acoustic, phonetic, and word-level content across 11 models, then compares these trends with frozen-layer downstream performance. Information patterns vary with model and pre-training objective, while CCA trends help identify useful layers and single layers often match or outperform all-layer representations.
Problem
The properties encoded by successful self-supervised speech models and the differences across models remain limitedly understood.
Method
The study applies CCA to measure acoustic, phonetic, and word-level content across individual layers of 11 pre-trained models and relates the measures to frozen-layer downstream performance.
Results
Layer-wise property patterns differ across models and relate to pre-training objectives; CCA trends correlate with downstream performance, and single intermediate layers often match or outperform all-layer representations.
Takeaways & Limitations
CCA trends can guide layer selection for speech recognition and spoken language understanding, supporting more efficient use of pre-trained models.
Takeaways & Limitations
Task performance depends on representation quality as well as modeling and optimization, so CCA scores do not fully determine downstream outcomes.
Abstract
from arXiv · showhide
Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.
1. INTRODUCTION
The paper addresses limited understanding of what self-supervised speech models encode beyond downstream performance. It analyzes 11 models layer by layer and connects acoustic, phonetic, and word-level content to downstream utility.
- The study examines acoustic, phonetic, and word content in individual layers of 11 pre-trained speech models using lightweight CCA analysis.
- Layer-wise phonetic and word information concentrates in different layers across models, with trends related to pre-training objectives despite differing training data.
- Phone and word content both correlate well with layer-wise speech recognition performance, while word content strongly correlates with spoken language understanding performance.
- A single intermediate frozen layer can match or outperform using all layers, suggesting more efficient application of pre-trained models.
2. BACKGROUND
The study compares eleven speech models that differ in training objectives, modalities, languages, and size. Their architectures generally combine convolutional or linear local-feature extraction with transformer self-attention layers and masking-based pretext tasks.
- The eleven models vary in training objective, speech or image-speech modality, English or multilingual data, and model size.
- Most architectures pass raw audio or filter banks through convolutional layers or a linear projection, followed by 12 or 24 self-attention layers.
- All models use masking-based pretext tasks, but W2V2, XLSR-53, and FaST-VGS+ recover local features with contrastive loss, whereas HuBERT, WavLM, and AV-HuBERT classify discrete cluster IDs.
- WavLM additionally simulates noisy or overlapped speech, XLSR-53 uses 53 languages, and the audio-visual models are analyzed using their audio branches alone.
- W2V2 and HuBERT small models use 960 hours of LibriSpeech, while corresponding large models use 60k hours of LibriLight.
3. ANALYSIS METHODS
The analysis uses CCA-based similarity to quantify how closely model-layer representations align with acoustic, phonetic, and word-level variables. It also tests whether these layer-wise measures predict performance of frozen representations on related downstream tasks.
- CCA measures similarity between two continuous-valued views through correlations between their linear projections.
- The study applies projection-weighted CCA, which weights canonical correlations by the input variance represented by each direction and regularizes covariance-matrix inversions.
- PWCCA compares each layer with mel filter banks, phone labels, and word labels to assess acoustic, phone-identity, and word-identity information.
- Downstream utility is evaluated by training predictors on frozen layer representations and correlating CCA-phone or CCA-word scores with layer-wise task performance using Spearman’s ρ.
4. EXPERIMENT DETAILS
The experiments analyze representations from every layer, including randomly initialized models, across frame-, phone-, and word-level samples. They evaluate both CCA similarity and downstream performance on phonetic, recognition, intent, action, and scenario tasks.
- The study analyzes every model layer and includes randomly initialized architectures to test whether observed trends arise from architecture alone.
- Frame-level analyses use approximately 180k frames from 500 LibriSpeech dev-clean utterances, while phone- and word-level analyses use approximately 7k segments each.
- CCA estimates are averaged across nine correlation scores produced by three repeated train-development-test partitions.
- Downstream evaluation covers phonetic recognition, automatic speech recognition, intent classification, and SLURP action and scenario classification.
5. FINDINGS
Across models, layer-wise acoustic, phonetic, and word representations vary with pre-training objectives, and CCA trends help identify layers useful for downstream tasks. Intermediate single-layer representations often match or exceed all-layer performance while reducing adaptation costs.
- Acoustic content: Initial transformer layers or final CNN layers remain highly correlated with spectrogram features across models.Randomly initialized models show a similar CNN trend, but their correlations can be as low as the lowest pre-trained-model scores.
- Phonetic and word-level content: Models recovering local features tend to peak in phonetic and word content at intermediate layers, whereas discrete-unit prediction concentrates both properties toward higher layers.These patterns suggest that the latent feature layer used in pre-training may influence where information is concentrated.
- Downstream-task relationships: CCA-phone and CCA-word scores correlate well with speech-recognition performance, while semantic-task performance correlates more strongly with CCA-word than CCA-phone.The reported correlations use Spearman’s ρ to account for potentially non-linear relationships; SLURP-action and SLURP-scenario have similar trends.
- Downstream-task relationships: Most top-performing layers overlap between CCA measures and task performance, although outliers can weaken Spearman correlations.Scatter plots show that rank correlation may not fully capture agreement among the best layers.
- Single-layer efficiency: Best single-layer ASR performance exceeds all-layers performance by 4-9% relative for W2V2-small, HuBERT-small, and HuBERT-large.For semantic tasks, the best single layer generally matches or exceeds all-layers performance, except SLURP-action with HuBERT models.
- Single-layer efficiency: CCA-word and CCA-phone can narrow layer selection because CCA requires fewer samples and has a closed-form solution, reducing task-specific adaptation costs.Intermediate-layer use also discards upper layers, reducing space and compute requirements; task performance nevertheless also depends on modeling and optimization.
6. CONCLUSION
The study provides broad cross-model evidence that useful phone and word information concentrates in model-dependent layers, especially in relation to pre-training objectives. Its task-agnostic analyses also guide downstream application and extend to additional properties and large-scale studies.
- Useful phone and word information tends to concentrate in different layers depending on the model and its pre-training objective.
- The layer-wise findings can help guide application of pre-trained speech models for downstream tasks.
- The task-agnostic analysis derives implications for a variety of tasks from a few fundamental acoustic-linguistic properties.
- The method is more scalable than popular task-specific probing approaches and can be extended to properties such as speaker, prosody, and syntax.
- Publicly available code is intended to enable additional large-scale analyses of representation models.