MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

Motivation

• 75% masking makes SSAST perform most self-attention computation on mask tokens, motivating a more efficient encoder-decoder design.Transformer self-attention has O(N^2) memory and time complexity, while MAE discards masked tokens during encoding.Source: §1 · p. 1

Contribution

• MAE-AST runs only unmasked tokens through a large encoder, then combines encoder outputs with mask tokens in a shallow decoder.Fine-tuning uses only the encoder, excluding the decoder’s reconstruction layers.Source: §1 · p. 1

Results

• 3× faster pretraining and 2× lower memory usage are reported for MAE-AST than SSAST with similar model architectures.The comparison concerns pretraining efficiency under the paper’s audio pretraining setup.Source: §1 · p. 1
• MAE-AST outperforms SSAST on several downstream tasks under a shared encoder depth and fixed other factors.The paper also evaluates masking, loss, and audio-versus-vision pretraining strategies.Source: §1 · p. 1

2.1. Model Architecture

• Fixed one-dimensional sinusoidal positional embeddings are added to tokens ordered by channel and then time, supporting variable-length temporal inputs.The same positional-embedding strategy is used for patch- and frame-based tokenization.Source: §2.1 · p. 1
• The encoder uses linearly projected 768-dimensional unmasked latents, while the pretraining decoder combines encoder outputs with learned mask tokens and predicts masked-token targets.Decoder outputs use separate linear projections for the two loss functions.Source: §2.1 · pp. 1–2
• The MAE-AST pipeline splits log-Mel spectrograms into patch- or frame-based tokens, masks a subset, encodes unmasked tokens, and decodes masked-token outputs.For fine-tuning, encoder hidden states are mean pooled into a classification representation and the decoder is discarded.Source: §2.1 · pp. 1–2, Figure 1 · p. 2

2.2. Mask Sampling

• Masking experiments vary patch- versus frame-based inputs, random versus chunked masking, and masking ratios across eight strategies.Higher masking ratios shorten MAE decoder inputs and can improve efficiency, while insufficient masking or chunking can make reconstruction too easy.Source: §2.2 · p. 2
• Patch-based random masking follows MAE, patch-based chunking follows SSAST, and frame-based chunking follows Wav2Vec2.0 span masking with overlap-adjusted proportions.The batch uses the same chunked mask to reduce compute for the otherwise slow unbiased sampling procedure.Source: §2.2 · p. 2

2.3. Joint Discriminative and Generative Pretraining

• MAE-AST retains SSAST’s joint reconstruction and classification objectives, using mean-squared error, InfoNCE, and λ = 10 to weight reconstruction.Its decoder permits single-layer output projections for each objective instead of SSAST’s separate two-layer heads.Source: §2.3 · pp. 2–3

3.1. Datasets

• Pretraining uses shuffled, unlabeled AudioSet-2M and Librispeech, while fine-tuning evaluates six audio and speech datasets including AudioSet, ESC-50, Speech Commands, VoxCeleb, and IEMOCAP.The same pretraining and downstream datasets are used as SSAST for a fair comparison.Source: §3.1 · p. 3
• The downloaded AudioSet contained 86.4% of the original data, so missing balanced and evaluation examples were resampled from the unbalanced partition.This assumes deleted videos are uncorrelated with label type; the resulting sets contain 1,762,073 unbalanced, 22,680 balanced, and 20,920 evaluation examples.Source: §3.1 · p. 3

3.2. Results

• MAE-AST converges after 8 epochs (500,000 iterations) and maintains performance when pretrained for twice that duration.Performance is evaluated on AudioSet using checkpoints from epochs 1 through 16.Source: Figure 2 · p. 3
• MAE-AST matches or outperforms SSAST on all downstream tasks except SID, where the 12-layer patch-based model failed to converge.The comparison uses equivalent input processing and masking techniques; the authors report the SID exception explicitly.Source: §3.2 · p. 3, Table 1 · p. 4
• Decoder depth yields only small improvements overall, supporting a shallow decoder, although AS and SID improve with fewer layers while ER improves with more.The ablation changes only decoder depth from the default model.Source: §3.2 · p. 3, Table 2 · p. 4
• Patch tokenization performs better for audio tasks and frame tokenization for speech tasks, while chunking and high masking ratios consistently benefit patch-based models.Frame-based masking results are more varied, and chunking improves patch-based audio models across the tested strategies.Source: §3.2 · p. 3, Table 3 · p. 4
• 3× speedup and 2× memory reduction are achieved versus MAE-AST training with mask tokens included at every layer, while convergence occurs in the same iterations as SSAST.The speed comparison controls for library differences using a decoder-only MAE-AST implementation and ordinary model sizes on a Quadro RTX 8000.Source: Figure 2 · p. 3, §3.2 · p. 4, Table 4 · p. 4
• MAE-AST is significantly better than SSAST with only a generative loss, while discriminative loss is more useful for audio and generative loss for speech.The authors associate the generative-only advantage with MAE-AST’s information bottleneck, while noting weak performance on the low-level SID task.Source: §3.2 · p. 4, Table 5 · p. 4

• MAE-AST significantly speeds pretraining, reduces memory costs, and matches or beats SSAST on most downstream tasks.The model uses MAE-like pretraining and is intended to support scalable multi-domain self-supervised learning.Source: §4 · p. 4
• MAE-like pretraining is more robust to different loss functions, outperforming SSAST with only a generative task but performing worse on low-level speaker identification.The authors hypothesize that the MAE architecture encourages additional abstraction.Source: §4 · p. 4