EAT: Self-Supervised Pre-Training with Efficient Audio Transformer

• Prior audio SSL pre-training remains costly; Audio-MAE's high masking can require a complex decoder, while BEATs tokenization may lose objective information and require more iterations.Source: §1 · p. 1
• EAT's Utterance-Frame Objective combines global utterance-level and local frame-level representations to enhance audio-clip understanding.Source: §1 · p. 1
• EAT applies an 80% masking ratio with inverse-block multi-masking, preserving unmasked patches in blocks and using masked-input clones to improve data utilization.Source: §1 · pp. 1–2
• A complex Transformer encoder paired with a lightweight CNN decoder is designed to decode features efficiently for frame-level prediction.Source: §1 · p. 2
• EAT achieves state-of-the-art performance on several audio and speech classification datasets while requiring significantly fewer total training hours.Source: §1 · p. 2

2.1 Bootstrap Method

• BYOL's bootstrap architecture pairs a target encoder with a predictor and momentum updates; EAT applies the bootstrap methodology to audio feature learning.Source: §2.1 · p. 2

2.2 Self-supervised Audio Pre-training

• Audio SSL uses joint speech-audio or audio-only pre-training, with waveform or Mel inputs and masked-patch or augmented-view tasks.Source: §2.2 · p. 2
• EAT adapts representation prediction into UFO, combining global and local spectrogram information rather than reconstructing patches or predicting discretized semantic features.Source: §2.2 · p. 2

3.1 Model Architecture

• EAT uses a Transformer encoder for visible patches and a lightweight CNN decoder to decode masked and visible features.A CNN encoder first converts spectrograms into patch embeddings; the teacher processes complete patches while the student receives masked inputs.Source: §3 · pp. 2–3, Figure 1 · p. 3

Utterance-Frame Objective

• The Utterance-Frame Objective combines global utterance-level and local frame-level losses to predict audio representations.The teacher target averages outputs across Transformer layers, and the student CLS token predicts the target's patch-dimension mean.Source: Utterance-Frame Objective · pp. 3–4

Masking Strategies in Pre-training

• EAT masks up to 80% of patch embeddings before encoding, reducing the data processed by the Transformer and increasing the masked-prediction challenge.Inverse block masking preserves patches in blocks while maintaining time-frequency correlation; the Figure 2 example uses 2×2 blocks at an 80% masking ratio.Source: Masking Strategies in Pre-training · p. 4, Figure 2 · p. 4
• Multi-mask training creates differently masked clones of each spectrogram for concurrent student processing, increasing data utilization through parallel computing.The teacher processes complete patch embeddings, which requires more computation than the student.Source: Masking Strategies in Pre-training · p. 4

3.2 Pre-training Details

• During pre-training, the teacher is updated by an exponential moving average of student parameters; EAT has 93M parameters before and 88M after fine-tuning.The encoder is a 12-layer ViT-B, and pre-training uses a 6-layer 2D CNN decoder.Source: §3.2 · p. 4

3.3 Fine-tuning Details

• For fine-tuning, EAT replaces the CNN decoder with a linear category-prediction layer and applies audio augmentations.The listed augmentations include SpecAug, mixup, droppath, audio rolling, and random noise.Source: §3.3 · pp. 4–5

4 Experiments

• EAT is pre-trained on AudioSet-2M (AS-2M).Source: §4 · p. 5
• Audio-classification fine-tuning is evaluated on AS-2M, AS-20K, and ESC-50.Source: §4 · p. 5
• Speech-classification fine-tuning is evaluated on Speech Commands V2 (SPC-2).Source: §4 · p. 5

4.1 Experimental Setups

• AudioSet evaluation uses mAP, ESC-50 uses five-fold average validation accuracy, and SPC-2 uses accuracy with the SUPERB data split.The processed AudioSet sets contain 1,912,134 unbalanced clips and 20,550 balanced clips; ESC-50 has 2,000 clips, and SPC-2 has 35 commands.Source: §4.1 · p. 5

Training Details

• Waveforms are resampled to 16 kHz and converted to 128-dimensional Mel-frequency bands using a 25 ms window and 10 ms shift.Spectrogram patches are normalized to mean 0 and standard deviation 0.5.Source: Training Details · p. 5
• EAT is pre-trained on AS-2M for 10 epochs with batch size 12, using 16 differently masked clones per clip and four RTX 3090 GPUs.The reported total training time is around 58 hours.Source: Training Details · p. 5

Pre-training Efficiency

• After 5 epochs, EAT outperforms Audio-MAE (32 epochs) and BEATsiter3 (342 epochs) on AS-20K, reaching 40.2% mAP after 10 epochs.EAT matches Audio-MAE’s performance after two epochs and surpasses BEATsiter3 by the fifth epoch.Source: Figure 3 · p. 5, Pre-training Efficiency · pp. 5–6

Model Performance

• EAT leads audio classification on AS-2M (48.6% mAP), AS-20K (40.2% mAP), and ESC-50 (95.9% accuracy), while matching prior SOTA on SPC-2 (98.3%).The paper reports a 0.6% mAP gain over the previous SOTA on AS-2M, a 1.9% mAP gain on AS-20K, and a reduction in ESC-50 average error rate from 4.4% to 4.1%.Source: Model Performance · p. 5, Table 1 · p. 6

Pre-training Efficiency

• EAT reduces total pre-training time 15.65 times versus BEATsiter3 and 10.02 times versus Audio-MAE after 10 epochs.The comparison reports wall-clock costs across different GPU setups.Source: Pre-training Efficiency · p. 5, Table 2 · p. 6
• EAT’s 80% masking ratio reduces the data processed by the student encoder, while its Utterance-Frame Objective enables lightweight CNN decoding instead of Transformer decoding.The paper attributes the efficiency gains to both high masking and the lightweight decoder.Source: Pre-training Efficiency · p. 6

4.3 Ablation Study

• The ablations pre-train configurations for 10 epochs on AS-2M and then fine-tune them on AS-20K.Source: §4.3 · p. 6

Utterance-level Learning

• λ=1 improves mAP by 1.3% over λ=0 and 1.0% over λ=0.01, while λ=10 diminishes performance.The objective combines utterance loss with frame loss; the caption identifies Figure 4 as comparing loss weight and fine-tuning prediction methods.Source: Figure 4 · p. 7, Utterance-level Learning · p. 7
• CLS-token prediction outperforms mean pooling in fine-tuning, with the paper attributing the advantage to the CLS token’s ability to extract global features.The caption identifies the two prediction methods compared in Figure 4.Source: Figure 4 · p. 7, Utterance-level Learning · p. 7

Inverse Block Masking on Audio Patches

• Inverse-block masking with S>1×1 outperforms random masking, and S=5×5 achieves the highest evaluation mAP at 40.2%.Flexible block shapes perform similarly to 5×5; block size and the resulting number of preserved blocks matter more than shape.Source: Table 3 · p. 7, Inverse Block Masking on Audio Patches · p. 7

5 Conclusion

• EAT achieves state-of-the-art results on AudioSet, ESC-50, and SPC-2, surpassing existing base audio SSL models in overall performance.The conclusion also reports more than tenfold time-efficiency gains over Audio-MAE and BEATs.Source: §5 · p. 7
• Inverse block multi-masking with a high mask ratio contributes to pre-training that is more than tenfold faster than Audio-MAE and BEATs.The conclusion identifies the Utterance-Frame Objective loss and balanced utterance- and frame-level learning as central to EAT’s learned representations.Source: §5 · p. 7
• Future work will scale up EAT and investigate joint audio-speech training to explore performance potential and the interplay between the domains.Source: §5 · p. 7

A.1 Hyperparamter Settings

• The experimental setup uses four GPUs for pre-training and one GPU for fine-tuning.The authors describe this as an efficiency measure for maintaining a lightweight setup.Source: §A.1 · p. 9
• Pre-training and fine-tuning are assigned different GPU counts: four and one, respectively.Source: §A.1 · p. 9
• Table 4 presents hyperparameter settings for both EAT pre-training and fine-tuning.Source: §A.1 · p. 9, Table 4 · p. 10