Source-linked AI summary
Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
Alexei Baevski, Arun Babu, Wei-Ning Hsu, Michael Auli
TL;DR
Self-supervised learning is often modality-specific and computationally expensive, motivating a more efficient general objective. data2vec 2.0 combines contextualized target prediction with efficient encoding, convolutional decoding, and target reuse. It reaches comparable accuracy across vision, speech, and language while reducing training time by 2–16x.
Problem
Self-supervised algorithms are often designed for one modality and require substantial computational training effort.
Method
data2vec 2.0 uses contextualized teacher targets, encodes only unmasked inputs, applies a fast convolutional decoder, and reuses targets across masked versions.
Results
2–16x faster training achieves comparable downstream accuracy across image classification, speech recognition, and natural language understanding.
Takeaways & Limitations
Contextualized target representations support an efficient self-supervised learner using one objective across vision, speech, and language.
Takeaways & Limitations
The teacher-student setup requires processing each sample twice, including the full unmasked input for teacher representations.
Abstract
from arXiv · showhide
Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8\% with a ViT-L model trained for 150 epochs.
1. Introduction
data2vec 2.0 targets inefficient, modality-specific self-supervised learning with a shared objective and contextualized target prediction. Its efficiency improvements preserve accuracy while substantially reducing training time across vision, speech, and language.
- Self-supervised pre-training has become computationally expensive because strong models rely on increased model capacity, larger datasets, and extensive training.
- data2vec 2.0 combines efficient data encoding, a fast convolutional decoder, and reused teacher targets for multiple masked sample versions.
- A shared learning objective generalizes across modalities, while separate Transformer models use modality-specific feature encoders.
- Contextualized targets capture information from the entire sample, including context-dependent meanings that noncontextualized targets represent less effectively.
- 2–16x efficiency gains at similar accuracy span image classification, speech recognition, and natural language understanding.
2. Related Work
Prior self-supervised methods are commonly organized around modality-specific inputs and objectives, while data2vec contributes a unified objective based on contextualized target representations. Related efficiency work reduces encoding or model costs, but multi-modal training objectives remain a distinct direction.
- Vision, speech, and language methods traditionally define modality-specific reconstruction, token-prediction, discretization, or masked-prediction tasks.
- Multi-mask training relates to prior work using multiple crops or masks, but data2vec 2.0 reuses targets for multiple masked versions of one sample.
- data2vec unifies the self-supervised objective across vision, speech, and text by predicting contextualized representations containing information from the entire input.
- Joint multi-modal models combine modalities within one model, whereas data2vec and this work train separately for each modality.
- Efficiency approaches include lightweight objectives, weight sharing, skipping masked patches, sparse attention, and more efficient feature encoders.
3. Method
data2vec 2.0 predicts contextualized teacher representations from masked inputs while reducing computation through asymmetric encoding, convolutional decoding, and multi-mask training. Inverse block masking structures the visible regions, and teacher-target reuse amortizes the teacher’s cost.
- 3.1. Contextualized Target Prediction: The student regresses contextualized targets formed from normalized averages of the teacher’s top K feed-forward blocks.
- 3.1. Contextualized Target Prediction: The teacher is an exponentially moving average of the student, and training uses an L2 loss between teacher targets and student predictions.
- 3.2. Asymmetric Encoder/Decoder Architecture: Only unmasked patches or time-steps are encoded, while a lightweight convolutional decoder predicts representations for masked portions.
- 3.3. Multi-mask Training: Teacher-student training requires processing each sample twice, creating an overhead that multi-mask reuse is designed to reduce.
- 3.3. Multi-mask Training: Teacher-target computation is amortized by reusing one unmasked representation across M masked versions of each training sample.
- 3.4. Inverse Block Masking: Inverse block masking preserves contiguous regions rather than selecting masked patches directly, with block sizes controlled by modality-specific parameters.
4. Experiments
Experiments evaluate data2vec 2.0 across vision, speech, and NLP using accuracy, word error rate, GLUE performance, and wall-clock pre-training time. Across modalities, it improves the speed–accuracy trade-off, with gains over established baselines and favorable ablation results.
- 4.1. Efficiency: 16.4x lower pre-training time yields 83.7% ImageNet top-1 accuracy versus 83.6% for MAE after 50.7 hours.The data2vec 2.0 model reaches 83.7% after just over 3 hours, compared with 50.7 hours for MAE.
- 4.1. Efficiency: 10.6x less wall-clock time achieves comparable speech-recognition word error rate to wav2vec 2.0.For Base models, data2vec 2.0 trains for 43.3 hours versus 57.3 hours for wav2vec 2.0 in the cited comparison, while broader results report up to 26% relative word error-rate reduction.
- 4.2. Computer Vision: Compared with MAE, data2vec 2.0 increases ImageNet accuracy by 0.9% while reducing ViT-B pre-training time from 50.7 to 32 hours and ViT-L time from 93.3 to 63.3 hours.The comparison covers ViT-B and ViT-L models trained without external data.
- 4.2. Computer Vision: For ViT-H, data2vec 2.0 outperforms MAE by 0.5% while using 40% less training time and 1/16 as many epochs.This result represents a reported speed–accuracy trade-off for ViT-H models.
- 4.5. Ablations: Ablations show that CLS loss slightly improves vision accuracy, pixel regression alone substantially reduces it, and inverse block masking outperforms block or random masking.The masking comparison identifies inverse block masking with B = 3 as the default, while the speech ablation finds ALiBi embeddings outperform convolutional embeddings alone.
5. Conclusion and Future Work
data2vec 2.0 is an efficient, general pre-training technique using one learning objective across modalities, improving training speed without losing downstream accuracy.
- 2-16x training speed with the same accuracy as many popular existing algorithms across modalities.The approach applies to vision, speech, and text, with future work extending it to other modalities.
A. Pre-training Hyper-parameters
The paper reports pre-training hyper-parameters for vision, speech, and natural language processing experiments.
- Vision, speech, and NLP experiments use separate modality-specific pre-training hyper-parameter tables.The tables define settings such as normalization and pooling choices for each modality.
B. Effect of Pre-training Dataset Size
Increasing pre-training data benefits larger models more, indicating that base-size models underfit the data2vec-style pre-training task on ImageNet-1K.
- Increasing pre-training data helps larger models more, implying that base-size models underfit the data2vec-style task.Figure 4 measures top-1 accuracy after fine-tuning on the full ImageNet-1K while varying the pre-training subset.