PaperlayerView exact revision on arXiv

Source-linked AI summary

Layer-wise Analysis of a Self-supervised Speech Representation Model

Ankita Pasad, Ju-Chieh Chou, Karen Livescu

arXiv:2107.04734v3cs.CLcs.LGeess.AS

TL;DR

The paper addresses limited understanding of what information self-supervised speech representations encode and how that information changes across layers and after ASR fine-tuning. It analyzes wav2vec 2.0 with canonical correlation, mutual information, and simple downstream tasks, finding layered acoustic-to-linguistic structure and motivating a fine-tuning modification that improves low-resource ASR performance.

  • Problem

    The information content of pretrained self-supervised speech representations is not well understood, despite their empirical downstream success.

  • Method

    The authors analyze wav2vec 2.0 intermediate representations with CCA, mutual information, and simple downstream tasks using non-parametric probes.

  • Results

    The analyses find layer-wise acoustic and linguistic information, autoencoder-style behavior, some word-meaning information, and improved low-resource ASR after modifying fine-tuning.

  • Takeaways & Limitations

    Layer-wise representation analyses can help characterize the abilities and limitations of speech models trained without external supervision and guide modifications.

  • Takeaways & Limitations

    The analyses focus on locally extracted frame, phone, and word representations and do not measure information delocalization from self-attention.

Abstract

from arXiv · show

Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models. The utility of these learned representations has been observed empirically, but not much has been studied about the type or extent of information encoded in the pre-trained representations themselves. Developing such insights can help understand the capabilities and limits of these models and enable the research community to more efficiently develop their usage for downstream applications. In this work, we begin to fill this gap by examining one recent and successful pre-trained model (wav2vec 2.0), via its intermediate representation vectors, using a suite of analysis tools. We use the metrics of canonical correlation, mutual information, and performance on simple downstream tasks with non-parametric probes, in order to (i) query for acoustic and linguistic information content, (ii) characterize the evolution of information across model layers, and (iii) understand how fine-tuning the model for automatic speech recognition (ASR) affects these observations. Our findings motivate modifying the fine-tuning protocol for ASR, which produces improved word error rates in a low-resource setting.

1. INTRODUCTION

This work analyzes wav2vec 2.0 representations to determine what information they encode across layers and how fine-tuning changes that information. The findings include an acoustic-linguistic hierarchy, autoencoder-style layer behavior, and improved low-resource ASR after modifying fine-tuning.

  • Motivation: The study addresses limited understanding of information encoded in successful self-supervised speech representations.Such representations are learned from unlabeled data and used for supervised downstream tasks, often with fine-tuning.
  • Approach: The authors analyze wav2vec 2.0 intermediate representations across layers, linguistic properties, and downstream fine-tuning effects.They emphasize tools that study representations directly rather than relying on additional classifier probes.
  • Findings: Wav2vec 2.0 layers exhibit an acoustic-linguistic hierarchy, while deeper layers show autoencoder-style behavior and some word-meaning information.The reported progression moves from acoustic features through phonetic and word-level information, followed by a reverse trend in deeper layers.
  • Findings: The authors use analyses of layer-wise properties to motivate a modified ASR fine-tuning protocol that improves performance in low-resource settings.The supplied introduction identifies improved low-resource ASR performance as a main finding.

2. RELATED WORK

Prior work analyzed supervised speech models and some SSL representations, but this study extends layer-wise analysis to SSL speech using multiple representation relationships and non-parametric probes.

  • Prior SSL analysis: Research on analyzing self-supervised speech models has been limited relative to extensive work on supervised speech models.Recent studies examined phonetic, paralinguistic, and semantic content using classifier probes.
  • Methodological scope: This work applies CCA and discrete mutual-information estimates to relationships involving discrete labels, continuous embeddings, and pretrained versus fine-tuned models.The methods are used to analyze several kinds of representation relationships rather than only classifier-probe performance.
  • Contribution: The authors describe the study as the first analysis of an SSL speech model across a range of linguistic properties using non-parametric probes.This distinguishes the work from prior analyses of visually grounded speech and SSL text models.

3. ANALYSIS METHODS

The paper combines representation-similarity measures, mutual information, and simple downstream tasks to assess acoustic and linguistic content in wav2vec 2.0 layers.

  • Representation similarity: CCA measures relationships between continuous-valued vectors through correlations between their linear projections.The study applies it to layer representations, other model layers, fine-tuned models, and external representations.
  • Representation similarity: The authors use projection-weighted CCA and report a symmetric average of directional similarities, with a maximum CCA similarity of 1.PWCCA weights directions by the proportion of input they explain and is described as more robust to spurious correlations.
  • Information dependence: Mutual information estimates dependence between clustered continuous representations and corresponding phone or word labels.The resulting measures are identified as MI-phone and MI-word.
  • Downstream validation: Word discrimination evaluates whether two speech segments represent the same word using cosine-similarity thresholds and average precision.The task primarily corroborates findings from MI-word.
  • Downstream validation: Word similarity evaluates context-independent wav2vec 2.0 word embeddings by comparing cosine similarities with human judgments using Spearman’s ρ.The paper uses a suite of 11 standard word-sim tasks.

4. EXPERIMENTAL SETUP

The experiments analyze wav2vec 2.0 representations across model variants, layers, and fine-tuning conditions using curated LibriSpeech data and several representation-level measures. The setup combines acoustic and linguistic analyses with word discrimination and discretized mutual-information experiments.

  • Models: Wav2vec 2.0 maps raw waveforms through convolutional and transformer layers into contextual features, trained with a contrastive masked-segment objective.Self-attention incorporates context surrounding each masked segment.
  • Models: The study analyzes Base, Large-960, and Large-60k models, plus ASR-fine-tuned versions using 10 minutes, 100 hours, or 960 hours of labeled LibriSpeech.Standard fine-tuning adds a randomly initialized linear layer, uses character-level CTC loss, and freezes the CNN layers.
  • Data and extraction: Experiments use LibriSpeech utterances, extract outputs from every layer, and repeat analyses across four sample sets.The reported findings primarily use dev-clean and generalize to dev-other and the Large-960 model unless stated otherwise.
  • Data and extraction: Word representations average all frames in word segments, while phone representations average the central third of each phone to reduce coarticulation effects.Montreal forced-aligner alignments define the phone and word boundaries.

5. FINDINGS

Layer-wise analyses show that pre-trained wav2vec 2.0 representations first diverge from local CNN features and later become more similar to them, suggesting an autoencoder-style pattern. These findings are evaluated on multiple model and fine-tuning variants, with the last two layers identified as an exception to the broader trend.

  • 5. FINDINGS: The findings are reported for Base, Base-ft-960, Large-60k, and Large-60k-ft-960 on dev-clean, and generally extend to dev-other and Large-960.Plots report means across four sample sets, with typical variation below 0.02 for CCA measures, 0.07 for MI, and 2% for word discrimination.
  • 5.1. How do the representations evolve across layers?: Pre-trained representations initially become less similar to local CNN features with depth, then deeper layers reverse course and become more similar to the input features.This pattern resembles reconstructing the input, although it appears to break in the last two layers.
  • 5.1. How do the representations evolve across layers?: The layer-wise local-feature comparison is measured with CCA similarity between transformer representations and the CNN output designated as layer 0.The figure caption identifies this comparison as CCA similarity with local features.
  • 5.1. How do the representations evolve across layers?: Masked-objective context encoding and reconstruction-like trends have also been observed in BERT, whose objective is based on masked reconstruction rather than contrastive prediction.The comparison concerns a related pattern in text-model representations, not an identical training objective.

5.2. Where is acoustic/linguistic information encoded?

The analysis tracks acoustic and linguistic information across wav2vec 2.0 layers using local representations and complementary similarity, information, and discrimination measures. Acoustic-phonetic content is concentrated in particular layers, while deeper layers can emphasize word-level or contextual information.

  • Layer-wise interpretation: Local frame-, phone-, and word-level outputs can gain information through self-attention contextualization, while decreases may reflect transformations of local content.The analysis cautions that layer-wise information changes need not reflect simple addition or deletion of information.
  • Frame-level acoustic content: CCA similarity with fbank increases through the first few layers, exceeding 0.75 across Large-model layers C4-T2.This suggests early layers compute features resembling engineered filter-bank features.
  • Phonetic information: Phone and AGWE curves broadly follow similar trends, supporting their use as complementary measures of phonetic information.The study measures phonetic content with MI-phone and CCA-agwe.
  • Phonetic information: Phonetic information is most salient around layers 6-7 for Base, while Large-60k shows similarly strong peaks at layer 11 and layers 18/19.Large-60k therefore has a dip between two regions of high phonetic information.
  • Word identity: Word discrimination follows the MI-word trend, and the two measures are consistently highly correlated across W2V2 variants.Word discrimination is reported through average precision.
  • Contextual and word-level information: For Large-60k, acoustic-phonetic measures dip around layers 13-17, which are also the layers with the most word content.The authors interpret this pattern as increased contextual and high-level processing, followed by deeper reconstruction of local representations.
  • Contextual and word-level information: The intermediate phonetic-content drop is absent in Base and Large-960, suggesting it is associated with Large-60k’s larger training set rather than model size.The comparison uses MI-phone and CCA-agwe scores for the additional Large-960 experiments.

5.3. Does the pre-trained model learn word meaning?

The paper tests whether wav2vec 2.0 representations encode word meaning using embedding-based similarity and word-similarity benchmarks. The results indicate some semantic information, although W2V2 is weaker than models trained specifically for semantics.

  • Word similarity: The best word-similarity performance for both Large-60k and Large-60kft-960 occurs at layer 15.This agrees with the authors’ hypothesis that layers 14-16 contain the most semantic information.
  • Evaluation setup: The evaluation compares W2V2 with character edit distance, AGWE, Speech2Vec, and GloVe baselines.The baselines distinguish orthographic, acoustic-phonetic, and explicitly semantic sources of similarity.
  • Results: W2V2 is outperformed by Speech2Vec and GloVe but performs better than the nonsemantic baselines, suggesting that it encodes some word meaning.W2V2 was not trained with an explicit semantic criterion.

5.4. How does fine-tuning affect the above observations?

Fine-tuning for ASR changes the layer-wise representation structure, especially in higher layers, and selectively increases word-identity information. These observations motivate reconsidering how the top layers are initialized for ASR.

  • Representation changes: Fine-tuning breaks the pre-trained autoencoder-style behavior, with deeper layers diverging from the input and presumably learning task-specific information.The changed behavior is observed through CCA-intra analysis.
  • Representation changes: Higher layers change the most during fine-tuning, suggesting that they may be a poor pre-trained initialization for ASR.The authors state that this finding motivates re-initializing these layers.
  • Information changes: MI-word consistently improves across top layers 19-24 after fine-tuning, whereas phone identity does not always improve similarly.Semantic measures also lack the same large improvement, consistent with ASR’s more direct relation to word identity than word meaning.

5.5. What about those peculiar last two layers?

The final two pre-trained layers often depart from the trends of earlier layers, but masking changes this pattern and increases their measured phonetic and word content. The effect disappears after fine-tuning, linking it to the pre-training objective while leaving its exact relationship unclear.

  • Observed peculiarity: At least one of the final two layers often fails to follow preceding CCA-similarity trends in pre-trained W2V2 models.This peculiarity is observed across most CCA similarity curves.
  • Effect of masking: When analyzing masked segments, the peculiarity disappears and MI-measured phonetic and word content improves in the last two layers.The rest of the layers show reduced measured content under this masked-segment analysis.
  • Relation to training: The discrepancy is absent in fine-tuned models, suggesting a connection to the training objective, although the exact relationship remains unclear.The authors note a similar peculiarity in local representations extracted from BERT.

6. PRACTICAL IMPLICATIONS FOR ASR

The authors modify standard ASR fine-tuning by re-initializing the top layer(s), based on evidence that the final layers change most during fine-tuning and represent less ASR-relevant linguistic content.

  • The last few W2V2 layers change the most during fine-tuning.This pattern is reported in Fig. 10.
  • The final few layers represent linguistic content that should be helpful for ASR less well.The authors cite Figs. 5a and 6a for this observation.
  • The authors test re-initializing the top layer(s) before fine-tuning as a modification to the standard approach.

7. CONCLUSION

The paper analyzes layer-specific information in wav2vec 2.0 and finds that acoustic and linguistic properties are distributed across layers, with some word-meaning information also encoded. The analyses have practical implications for low-resource ASR, while local representations leave self-attention-driven information delocalization unmeasured.

  • 7. CONCLUSION: Acoustic and linguistic properties tend to be encoded in different layers of wav2vec 2.0.The pre-trained model also follows an autoencoder-style behavior.
  • 7. CONCLUSION: The model encodes some non-trivial word meaning information, although the nature of this semantic content remains unclear.
  • 7. CONCLUSION: The analyses are corroborated by multiple analytical measures and certain downstream tasks.
  • 7. CONCLUSION: A motivated modification to fine-tuning improves downstream ASR performance in the very low-resource setting.
  • 7. CONCLUSION: The local frame-, phone-, and word-level analyses do not measure information delocalization caused by self-attention layers.In-depth self-attention analysis and evaluation on additional models and downstream tasks are left for future work.
Loading 2107.04734v3…