Source-linked AI summary
Masked Autoencoders that Listen
Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, Christoph Feichtenhofer
TL;DR
Audio representation learning needs an audio-specific alternative to image-pretrained initialization and efficient self-supervision. Audio-MAE adapts masked autoencoding to spectrograms with sparse encoding, spectrogram-aware local decoder attention, and lower-ratio masked fine-tuning. It achieves state-of-the-art performance on six audio and speech classification tasks, including AudioSet-2M.
Problem
ImageNet-pretrained initialization may be sub-optimal for audio, while efficient self-supervised learning requires large-scale data and reduced Transformer sequence lengths.
Method
Audio-MAE masks most spectrogram patches, encodes only visible patches, restores their order with mask tokens, and decodes the spectrogram using local attention.
Results
Audio-MAE achieves state-of-the-art performance on six audio and speech classification tasks and reaches state-of-the-art AudioSet-2M mAP without external supervision.
Takeaways & Limitations
Audio spectrograms support masked autoencoding, with local decoder attention and masking during pre-training and fine-tuning yielding stronger representations and reduced training computation.
Abstract
from arXiv · showhide
This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.
1 Introduction
Audio-MAE adapts masked autoencoding to audio spectrograms, addressing the limitations of transferring image-pretrained models to audio. It uses sparse encoding, spectrogram-aware decoding, and achieves state-of-the-art results across six audio and speech classification tasks.
- Motivation: Audio models initialized from ImageNet weights may be sub-optimal because audio spectrograms differ from images.The paper highlights discrepancies between spectrograms and images and questions out-of-domain supervised initialization.
- Approach: Audio-MAE masks most spectrogram patches, encodes only visible embeddings, then reconstructs the spectrogram with a Transformer decoder.The decoder restores frequency-time order and combines encoded patches with learnable mask embeddings.
- Audio-specific design: Local decoder attention improves reconstruction because spectrogram patches contain strong local time-frequency correlations.Formants and other spectrogram features are grouped locally, while their positions carry semantic information.
- Training: Audio-MAE minimizes patch-normalized mean square error and discards the decoder when fine-tuning the encoder for downstream tasks.The encoder is fine-tuned with patch masking on target datasets.
- Results: Audio-MAE achieves state-of-the-art performance on six audio and speech classification tasks, including AudioSet-2M mAP without external supervision.It outperforms recent models using external supervised pre-training.
2 Related Work
Prior audio representation models use out-of-domain image supervision or audio-specific objectives over raw waveforms, features, or spectrograms. Audio-MAE differs from earlier spectrogram methods by encoding only non-masked patches during self-supervised pre-training.
- Visual masked pre-training: Visual masked pre-training reconstructs masked image content, while MAE reduces computation by encoding only non-masked patches.These methods motivate extending masked autoencoding beyond images.
- Out-of-domain pre-training: Audio models commonly transfer ImageNet-pretrained ViT or ResNet weights to spectrogram inputs.This involves adapting patch embeddings from RGB images to single-channel spectrograms.
- In-domain pre-training: Audio self-supervised methods vary by input type and objective, including waveform, frame-level, and spectrogram inputs with contrastive or reconstruction losses.SSAST is the closest benchmark and combines contrastive and reconstructive objectives on masked spectrogram patches.
- Audio-MAE distinction: Unlike previous audio methods that encode full-view masked and unmasked segments, Audio-MAE encodes only non-masked spectrogram patches.This distinction follows the efficient masked-encoding strategy of image MAE.
3 Audio Masked Autoencoders (Audio-MAE)
Audio-MAE transforms recordings into Mel-spectrogram patches, applies substantial masking, encodes visible patches, and reconstructs masked content. Its decoder incorporates local attention to reflect spectrogram structure, while masking strategies differ between pre-training and fine-tuning.
- Spectrogram Patch Embeddings: Audio recordings become non-overlapping Mel-spectrogram patches that are flattened, linearly embedded, and given fixed sinusoidal positional embeddings.The patches form the token sequence processed by Audio-MAE.
- Masking Strategies: Audio-MAE explores random masking and structured masking over time, frequency, or both during pre-training and fine-tuning.The masking strategies are illustrated on Mel-spectrograms.
- Decoder with Local Attention: The decoder restores time-frequency order, adds positional embeddings, and predicts the input spectrogram from encoded patches and trainable masked tokens.A linear head produces the reconstructed spectrogram.
- Decoder with Local Attention: Local attention groups spectrogram patches into windows, with shifted windows moving by 50% between consecutive decoder layers.The design targets the positional and local structure of audio spectrogram features.
- Objective: The reconstruction objective is mean squared error between predicted and input spectrogram values, averaged over masked patches.The paper reports that reconstruction loss alone is sufficient without an additional contrastive objective.
- Fine-tuning for Downstream Tasks: During fine-tuning, Audio-MAE retains only the encoder and can mask patches to regularize learning and reduce computation.This differs from SpecAug because Audio-MAE processes a subset of real-valued input patches rather than a full input with zeros.
4 Experiments
Audio-MAE is evaluated across six audio and speech classification tasks, with ablations identifying effective masking, decoder, data, and initialization choices. It achieves strong results using audio-only self-supervised pre-training on AudioSet, including state-of-the-art comparisons without external non-audio data.
- Evaluation setup: Audio-MAE is evaluated on six tasks spanning AudioSet, ESC-50, Speech Commands, and VoxCeleb.The evaluation includes AS-2M, AS-20K, ESC-50, SPC-1, SPC-2, and SID.
- Masking strategies: 80% random masking is optimal for pre-training, while structured time+frequency masking at a lower ratio is preferred for fine-tuning.The default fine-tuning masking ratio is 0.3.
- Decoder design: Shifted-window local attention gives the best decoder performance, while hybrid local-global attention also improves over vanilla global attention.Qualitatively, local attention better reconstructs vowel harmonics and recovers more spectrogram context.
- Pre-training data and duration: More pre-training data generally improves performance, while balanced versus unbalanced class distributions have similar effects at comparable small data scales.Using 1% balanced AS-20K versus randomly sampled 20K unbalanced data yields 39.4 versus 39.6 mAP; performance saturates after epoch 24.
- Pre-training initialization: Audio-only pre-training from scratch is best; ImageNet initialization does not help Audio-MAE and can degrade accuracy, possibly because of domain shift.The comparison considers ImageNet self-supervised and supervised initialization alongside audio-only training.
- State-of-the-art comparison: 37.1 mAP on AudioSet-20K surpasses competing approaches, while Audio-MAE also outperforms prior systems on AudioSet-2M, ESC-50, and speech tasks.The model uses only AudioSet for pre-training and outperforms models relying on external supervised ImageNet pre-training.
5 Conclusion
Audio-MAE extends MAE to audio spectrograms and achieves strong self-supervised representations through masking, local decoder attention, and same-modality pre-training and fine-tuning.
- Audio-MAE reconstructs masked spectrogram patches and achieves state-of-the-art performance on six audio and speech classification tasks.
- Local self-attention in the decoder yields stronger audio representations.
- Masking during both pre-training and fine-tuning improves accuracy while reducing training computation.
- The optimal masking strategy depends on modality and learning type.
- Best performance comes from pre-training and fine-tuning within the same modality, without cross-modality transfer learning.
Checklist
The checklist records the paper’s disclosures on reproducibility, assets, ethics, and limitations, including references to supplemental details and experimental sections.
- The paper states that limitations are discussed in the supplemental material and that ethics requirements are satisfied.
- The paper states that code, data, and pre-trained models are available or will be released for reproducibility.
- Training details, error bars, and compute resources are reported in the paper or its referenced sections.
- The authors report citing existing assets, mentioning licenses, and including new assets.
- The checklist marks consent, personally identifiable information, offensive content, human-subject procedures, and theoretical-results requirements as not applicable where indicated.
- The appendix provides additional visualizations, complete experimental details, hyperparameters, and extra ESC-50 and packet-loss-concealment experiments.
A Additional Reconstruction Details and Results by Audio-MAE Decoder
Additional experiments visualize Audio-MAE reconstructions under heavy masking, showing reasonable recovery and easier reconstruction for predictable music and event patterns than speech.
- The visualizations use ground truth, masked input, and reconstruction output, with columns grouped by speech, music, event, and other sounds.
- Waveform reconstruction uses inverse Mel-spectrograms and Griffin-Lim, which can introduce artifacts from imperfect phase estimation.
- Audio-MAE produces reasonable reconstructions even with 70%–80% masking.
- Music and event sounds are easier to reconstruct because their spectrogram patterns are relatively predictable.
- Repeating tempos and frequency harmonics are reconstructed well in representative music, siren, and elephant examples.
- Speech recordings are more challenging to reconstruct than the illustrated music and event examples.
B Experimental Details and Hyperparameter Settings
The appendix specifies preprocessing, optimization, augmentation, sampling, and supplementary evaluation procedures used for Audio-MAE experiments.
- All recordings are converted to mono at a 16K sampling rate for consistency across pre-training and fine-tuning.
- The effective learning rate is determined by the base learning rate and batch size.
- Strong pre-training augmentations were discarded because they produced similar or worse performance, including an approximately 0.5 mAP degradation from CutMix on AudioSet-2M.
- Fine-tuning on unbalanced AudioSet-2M uses weighted sampling with probabilities proportional to inverse label frequency.
- The appendix adds ESC-50 comparisons with supervised AudioSet-2M pre-training and a preliminary packet-loss-concealment case study.
C.1 ESC-50 with AudioSet-2M Supervised Pre-training
Audio-MAE achieves state-of-the-art accuracy on ESC-50 with additional supervised pre-training on AudioSet-2M, while remaining an audio-only model without ImageNet pre-training.
- Audio-MAE achieves state-of-the-art accuracy on ESC-50 under additional AudioSet-2M supervised pre-training.
- The comparison includes models with additional AudioSet supervised pre-training and identifies ImageNet-pre-trained models separately.
C.2 Qualitative Results for a practical generation task
The paper explores Audio-MAE for packet loss concealment by reconstructing time-corrupted speech. In simulations, its decoder produces reasonable speech reconstruction, while deeper analysis remains future work.
- Packet loss can corrupt VoIP speech during transmission, motivating techniques that recover missing or corrupted waveform segments.
- The PLC experiment masks speech in time and reconstructs the corrupted recordings with Audio-MAE.
- Audio-MAE produces reasonable speech reconstruction in both simulated cases, but in-depth PLC and bandwidth-expansion analysis is left for future work.
- Figure 8 presents original, packet-loss-corrupted, and Audio-MAE-restored spectrogram and time sequences for two speech recordings at 25% loss.
C.3 Negative Results: Directions that did not work well
The study finds that adding a contrastive objective does not improve Audio-MAE beyond reconstruction alone. Reconstruction remains the sufficient and best-performing pre-training objective.
- The contrastive objective draws masked patches toward their contextualized embeddings while separating them from mismatched patches.
- The experiments compare reconstruction alone, contrastive learning alone, and a weighted combination of both objectives.
- Reconstruction alone yields the best performance, whereas contrastive-only and multi-task objectives provide no empirical improvement.
- Table 5 reports the measured impact of adding the contrastive objective.
D Limitations
Audio-MAE is limited by the scale and duration of its AudioSet pre-training data, coverage of rare or unseen events, and the computational demands of Transformer-based audio modeling.
- AudioSet is around two orders of magnitude smaller than the text corpora used by language-modeling counterparts.
- AudioSet’s 10-second recordings may not adequately support learning distant temporal dependencies in audio.
- Audio-MAE may be sub-optimal for tasks involving rare or unseen audio events because AudioSet is unbalanced and covers only 527 annotated classes.
- Modeling lengthy, high-dimensional audio with Transformers remains computationally demanding despite improved self-supervised learning efficiency.