Source-linked AI summary
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-Wen Li, Hung-yi Lee
TL;DR
Large pretrained speech models provide useful representations but are costly in size, motivating more efficient alternatives. The paper introduces Audio ALBERT with shared parameters, evaluates it on phoneme and speaker tasks, and finds comparable performance with 91% fewer parameters while probing reveals richer information in intermediate layers than the last layer.
Problem
BERT-like pretrained speech models are often prohibitively large, creating a need for more parameter-efficient speech representation models.
Method
Audio ALBERT applies parameter sharing across transformer layers to learn pretrained audio representations, which are evaluated on phoneme and speaker classification and with frozen-layer probing classifiers.
Results
91% fewer parameters yields comparable speaker-classification results to Mockingjay, while Audio ALBERT and Mockingjay show similar phoneme-classification performance.
Takeaways & Limitations
Intermediate-layer representations encode richer phonetic and speaker information than the last layer, suggesting they may be more suitable for downstream adaptation.
Abstract
from arXiv · showhide
For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utilized in acoustic model training in order to achieve better performance. In this paper, we propose Audio ALBERT, a lite version of the self-supervised speech representation model. We use the representations with two downstream tasks, speaker identification, and phoneme classification. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks while utilizing 91\% fewer parameters. Moreover, we use some simple probing models to measure how much the information of the speaker and phoneme is encoded in latent representations. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.
1. INTRODUCTION
Audio ALBERT addresses the prohibitive size of BERT-like speech models by sharing parameters across layers for parameter-efficient representation learning. It achieves comparable downstream performance with much smaller networks and shows that intermediate representations can contain richer task information than the last layer.
- BERT-like speech models are often prohibitively large, motivating more efficient pretrained speech representations.
- ALBERT reduces parameters through layer sharing and embedding-matrix factorization while retaining robust, task-agnostic representations.
- Audio ALBERT transfers ALBERT’s parameter-sharing idea to speech after finding redundancy among Mockingjay’s learned layer parameters.
- Audio ALBERT yields comparable downstream performance to other pretrained models using much smaller networks.
- Intermediate Audio ALBERT layers contain more phonetic and speaker information than the last layer, which may make them better suited for downstream adaptation.
2. RELATED WORK
Prior work develops self-supervised speech representations and increasingly compact transformer architectures. Probing methods assess what information representations encode, while weight sharing reduces parameters while preserving task performance.
- 2.1. Self-supervised learning representation: Self-supervised learning has produced pretrained representations across vision, language, and speech, including masked-reconstruction models for speech.
- 2.1. Self-supervised learning representation: Existing pretrained speech networks are generally large and improve performance through more parameters or pretraining data, motivating compact alternatives.
- 2.2. Weight sharing: Weight sharing across transformer layers reduces parameters and can preserve comparable task performance, while also supporting faster inference and training.
- 2.3. Probing task: Probing extracts representations, trains simple classifiers for downstream probing tasks, and uses classifier performance to measure encoded information.
3. METHOD
The method builds Audio ALBERT by sharing Transformer parameters across layers after identifying redundancy in Mockingjay, then evaluates compact representations through feature extraction, fine-tuning, and probing.
- 3.1. Mockingjay: Mockingjay uses 3-, 6-, or 12-layer Transformer encoders to reconstruct masked spectrograms during pre-training.The model takes masked spectrograms as input and reconstructs the original spectrogram.
- 3.1. Mockingjay: JS-divergence analysis finds similar attention distributions across Mockingjay layers, indicating redundant parameter usage.The analysis supports compressing the model through weight sharing without sacrificing model expressiveness.
- 3.2. AALBERT: AALBERT shares self-attention, feed-forward, and layer-normalization parameters across Transformer layers to reduce network size.It uses mel-spectrogram features and reconstructs the corresponding log-linear spectrogram after masking and normalization.
- 3.3. Downstream tasks: Downstream evaluation compares frozen feature extraction with fine-tuning, including weighted sums of representations from multiple layers.Feature extraction feeds fixed representations into a trainable layer, whereas fine-tuning updates the entire pre-trained model.
- 3.4. Probing: Probing classifiers with linear, one-layer, and two-layer prediction heads measure information encoded in representations from each pre-trained layer.The study varies prediction-layer architectures to reduce bias from a particular probe design.
4. EXPERIMENT RESULTS AND DISCUSSIONS
Across phoneme and speaker classification, AALBERT matches or approaches Mockingjay while using substantially fewer parameters. Probing shows that intermediate representations often encode more phonetic and speaker information than the last layer.
- Phoneme classification: AALBERT achieves phoneme classification accuracy comparable to Mockingjay with far fewer parameters in both fine-tuning and weighted-sum settings.The comparison uses phoneme classification accuracy against the number of network parameters.
- Phoneme classification: Phoneme classification performance saturates at six layers, with neither 12-layer AALBERT nor Mockingjay improving over their three- and six-layer counterparts.The authors conjecture that limited pre-training data contributes to this saturation.
- Phoneme classification: AALBERT and Mockingjay outperform the input acoustic feature and show similar phoneme classification performance across training-data proportions.This pattern appears in both feature-extraction and fine-tuning conditions.
- Speaker classification: AALBERT and Mockingjay encode substantially more speaker information than raw acoustic features, yielding nearly perfect speaker classification while AALBERT uses fewer parameters.Speaker classification evaluates utterance-level representations for subsets containing 251 and 921 speakers.
- Speaker classification: AALBERT uses 91% fewer parameters than Mockingjay while achieving comparable speaker-classification results.The comparison uses weighted-sum representations for speaker classification.
- Probing task: Intermediate representations outperform last-layer representations in all four probing tasks for both AALBERT-6L and Mockingjay-6L.The fifth AALBERT-6L layer and fourth Mockingjay-6L layer contain the most speaker information, while phoneme information rises and then falls with depth.
6. CONCLUSION
The paper presents Audio ALBERT (AALBERT), a compact pretrained model for extracting audio representations. It reconstructs masked acoustic features and achieves comparable performance to Mockingjay with fewer parameters.
- AALBERT extracts latent representations that encode audio information.
- AALBERT learns by reconstructing masked input acoustic features to the linear spectrogram.
- AALBERT achieves comparable performance to Mockingjay while using substantially fewer parameters.
- Future work will investigate model architectures to further improve computational and parameter efficiency.