Source-linked AI summary
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli
TL;DR
Existing self-supervised methods use modality-specific objectives, leaving uncertainty about whether their learning biases generalize across speech, vision, and language. data2vec addresses this with masked prediction of contextualized full-input representations using an EMA teacher and Transformer student, achieving state-of-the-art or competitive results across the three modalities. The method still relies on modality-specific input encoders and masking strategies.
Problem
Self-supervised algorithms use modality-specific designs and learning biases, and it is unclear whether these biases generalize across modalities.
Method
data2vec uses one masked-prediction self-distillation method in which a student predicts contextualized representations of the full input produced by an EMA teacher.
Results
data2vec achieves state-of-the-art or competitive performance across vision, speech recognition, and natural language understanding benchmarks.
Takeaways & Limitations
A single self-supervised learning method can be effective across images, speech, and text while predicting continuous contextualized targets instead of modality-specific discrete or local targets.
Takeaways & Limitations
The approach still uses modality-specific input encoders and masking strategies, and speech is more prone to collapse when adjacent targets are highly correlated.
Abstract
from arXiv · showhide
While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.
1. Introduction
data2vec addresses modality-specific self-supervised designs with one learning objective for images, speech, and text. It predicts contextualized latent representations from masked inputs and performs competitively across all three modalities.
- Motivation: Self-supervised algorithms often use modality-specific designs and learning biases for speech, vision, and language.Examples include learned speech-unit inventories, visual tokens, input regression, and augmentation-invariant representations.
- Motivation: It is unclear whether modality-specific learning biases generalize across modalities, motivating more general learning processes.The motivation is also connected to evidence that general neural architectures can perform well across modalities.
- Contribution: data2vec uses an identical self-supervised learning objective for images, speech, and text while retaining modality-specific representations.The framework is intended as a step toward more general machine learning across modalities.
- Method: The method uses a Transformer in teacher and student modes: the teacher encodes the full input, while the student predicts those representations from a masked input.Teacher parameters are an exponentially moving average of student weights, and targets average K teacher layers.
- Novelty: data2vec predicts continuous, contextualized representations rather than fixed visual tokens, normalized inputs, or discrete speech units.Self-attention allows targets to incorporate information from the entire input rather than only local content.
- Results: Experiments report data2vec state-of-the-art results for specified ImageNet-1K ViT settings, speech-recognition improvements, and GLUE gains over a like-for-like RoBERTa baseline.The reported evaluation spans computer vision, speech processing, and natural language understanding.
2. Related Work
Prior self-supervised methods commonly use modality-specific targets or objectives, whereas data2vec predicts contextualized latent representations across vision, language, and speech. Its targets are continuous, input-adaptive, and incorporate broader context than many prior targets.
- Computer Vision: Vision self-supervised learning includes contrastive methods, online clustering, and masked prediction of visual tokens or input pixels.These approaches may use separately learned, jointly learned, or directly observed visual targets.
- Related Methods: Unlike BYOL and DINO, data2vec combines momentum-encoder regression with masked prediction and multiple-layer targets across modalities.The paper reports multiple-layer regression as more effective than regressing only the top layer.
- Computer Vision: data2vec differs from visual-token and pixel prediction by predicting contextualized latent representations that incorporate information from the entire image.The targets are not isolated to the current patch.
- NLP: In NLP, data2vec predicts continuous contextualized representations instead of discrete words, subwords, or bytes.The targets are not predefined or limited in number and can adapt to each input example.
- Speech: In speech, wav2vec 2.0 and HuBERT predict discrete speech units, whereas data2vec directly predicts contextualized latent representations without quantization.data2vec averages representations across multiple layers rather than predicting discretized units.
3. Method
data2vec trains a student Transformer to predict contextualized representations of the full input produced by an EMA teacher from an unmasked view. Targets are normalized averages of multiple teacher layers, and masked positions are optimized with Smooth L1 regression.
- Teacher-Student Framework: The student encodes a masked sample while the EMA teacher encodes the corresponding unmasked sample to provide full-input targets.The teacher and student use the same model architecture in different parameterization modes.
- Input Processing: The framework uses a standard Transformer with modality-specific input encoders and masking strategies for vision, speech, and language.This preserves modality-specific input handling while unifying the learning objective.
- Training Targets: Only masked student positions are predicted, using contextualized teacher representations that encode both the target position and information from the sample.The contextual information comes from self-attention over the unmasked input.
- Training Targets: Targets are formed by normalizing and averaging outputs from the top K teacher blocks.Averaging performed as well as separate projections while being more efficient.
- Stabilization: Target normalization helps prevent constant-representation collapse and prevents high-norm layers from dominating target features.Speech uses instance normalization, while NLP and vision use parameter-less layer normalization.
- Objective: The objective uses Smooth L1 loss to regress student predictions toward contextualized targets.Its transition between squared and L1 behavior is controlled by β and is less sensitive to outliers, but β must be tuned.
4. Experimental Setup
The experiments use Base and Large Transformer models with modality-specific preprocessing, masking, optimization, and fine-tuning procedures. Computer-vision evaluation uses ImageNet-1K top-1 accuracy with ViT-B and ViT-L models.
- Model Configurations: The study evaluates data2vec Base and Large models with 12 or 24 Transformer blocks and hidden dimensions of 768 or 1024.EMA updates are performed in fp32 for numerical stability.
- Computer Vision: Images are represented as 16x16 patches from 224x224 inputs, with block-wise masking applied to 60% of patches.The vision models use ViT-style patch embeddings and are pretrained for 800 epochs for ViT-B and 1,600 epochs for ViT-L.
- Computer Vision: Image classification mean-pools the final Transformer output and fine-tunes ViT-B for 100 epochs or ViT-L for 50 epochs.Optimization uses Adam with a cosine learning-rate schedule.
- Speech: Speech inputs are 16 kHz waveforms processed by a seven-layer temporal convolutional encoder producing 50 Hz representations.The encoder uses strides (5,2,2,2,2,2,2) and a receptive field of 25ms.
- Speech: For the speech Base model, masking starts at p = 0.065 of time-steps and masks approximately 49% of a typical training sequence.The setup averages K = 8 teacher blocks and trains for 400K updates.
- NLP: Language experiments build on RoBERTa with 50K byte-pair-encoded types and BERT-style masking of 15% of tokens.Pretraining uses 1M updates with batches of up to 256 sequences of 512 tokens.
- Evaluation: Computer-vision evaluation reports top-1 validation accuracy on ImageNet-1K for ViT-B and ViT-L, distinguishing individual models from multi-model setups.The comparison includes methods that use separate visual tokenizers or additional distilled models.
5. Results
Across vision, speech, and NLP, data2vec performs strongly by predicting contextualized latent representations, with especially large gains in low-resource speech and competitive results in vision and language.
- Computer Vision: ImageNet-1K results show data2vec outperforming prior work with ViT-B and ViT-L in the single-model setting.
- Speech and Audio Processing: 20% relative WER improvement is the largest Base-model gain with 10 minutes of labeled Librispeech data.Large models also improve strongly in the smallest labeled-data settings, while performance is comparable in resource-rich 100-hour and 960-hour settings.
- Speech and Audio Processing: AudioSet evaluation uses pre-training on AudioSet and/or Librispeech followed by fine-tuning on the 20K subset.
- Natural Language Processing: data2vec outperforms the retrained RoBERTa baseline on GLUE, with further improvement from masking spans of four BPE tokens.The evaluation reports average development-set accuracy over five fine-tuning runs across GLUE tasks.
- Natural Language Processing: The NLP model predicts continuous contextualized representations from the entire unmasked text sequence rather than discrete linguistic units.The targets are not fixed or limited to a closed vocabulary, allowing them to adapt to each input example.
- Ablations: Averaging multiple teacher layers improves performance over using only the top layer across all modalities.Using all layers is generally effective and only slightly worse than a carefully tuned number of layers.
- Ablations: Richer teacher context improves downstream performance, with the best accuracy when the entire input sample is visible.The ablation restricts teacher self-attention to a fraction of the input during pre-training, then restores full context for fine-tuning.
6. Discussion
data2vec uses contextualized, continuous targets built from the full unmasked sequence rather than predefined discrete units, while retaining modality-specific input processing. Its self-distillation setup must address representation collapse, especially for speech with highly correlated adjacent targets.
- The framework retains modality-specific feature extractors and masking strategies despite using one learning mechanism across modalities.
- data2vec builds contextualized training targets with self-attention over the entire unmasked input, making each target specific to its sequence.
- Unlike HuBERT and other discrete-unit methods, data2vec has no fixed limit on target units and can vary representations by sequence context.
- Representation collapse is more likely with excessive learning rates, short warmup, low τ, or highly correlated targets requiring longer masked spans.
- Target normalization promotes variance for correlated modalities such as speech, whereas momentum tracking is sufficient for vision and NLP.
7. Conclusion
data2vec applies one self-supervised learning regime to vision, speech, and language by predicting contextualized representations from partial inputs. It reports improvements over prior work across image classification, low-resource speech recognition, and natural language understanding, while leaving input encoders and masking modality-specific.
- A single self-supervised learning regime works across vision, speech, and language by regressing contextualized representations from partial input views.
- data2vec outperforms prior self-supervised work on ImageNet-1K for ViT-B and ViT-L single models, improves low-resource Libri-light speech recognition, and outperforms RoBERTa on GLUE.
- The approach still uses modality-specific input encoders and masking strategies, which the authors identify as targets for future unification.
A. Extended speech processing results
The extended speech results evaluate word error rate on Librispeech dev/test sets across low-resource labeled-data regimes using LS-960 as unlabeled data.
- The evaluation measures word error rate on Librispeech dev/test sets with 10-minute, 1-hour, 10-hour, and clean-100h labeled-data setups.
- All models use Librispeech LS-960 audio as unlabeled pre-training data.
B. Comparison of loss functions
The paper compares different pre-training losses on Librispeech dev-other without a language model. It reports that loss-function choices have relatively little effect on final performance.
- Different pre-training losses have a relatively small effect on final performance.
- The comparison uses Librispeech dev-other and excludes a language model.
C. Speech masking parameter ablation
The speech masking ablation found no significant improvement from tuning masking hyperparameters, with any small gains likely disappearing after adding a language model.
- Speech masking hyperparameter tuning did not produce significant improvements.The evaluation used Librispeech dev-other without a language model.
- The small improvements observed are likely to disappear after adding a language model.
- The ablation results were measured on Librispeech dev-other without a language model.