PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Feature Prediction for Self-Supervised Visual Pre-Training

Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, Christoph Feichtenhofer

arXiv:2112.09133v2cs.CVcs.LG

TL;DR

Transformer-based video models can overfit and often rely on supervised pre-training on large image datasets. MaskFeat instead predicts features of masked visual regions, with HOG performing particularly well; on unlabeled videos it closes the supervised-versus-scratch gap and achieves strong video and image results.

  • Problem

    Transformer-based video models are prone to over-fitting and heavily rely on supervised pre-training on large-scale image datasets.

  • Method

    MaskFeat masks portions of visual input and directly regresses features of the masked regions, using features extracted from the intact sample as supervision.

  • Results

    MaskFeat closes the supervised-versus-scratch gap for video understanding and achieves 86.7% top-1 accuracy on Kinetics-400 without external data, alongside strong transfer and image-domain results.

  • Takeaways & Limitations

    HOG is a particularly effective and efficient prediction target, while MaskFeat generalizes across video and image domains and scales to large models.

  • Takeaways & Limitations

    Using supervised features would make pre-training technically not self-supervised and requires extra computation for masked feature generation.

Abstract

from arXiv · show

We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer-based models. Without using extra model weights or supervision, MaskFeat pre-trained on unlabeled videos achieves unprecedented results of 86.7% with MViT-L on Kinetics-400, 88.3% on Kinetics-600, 80.4% on Kinetics-700, 39.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageNet.

1. Introduction

MaskFeat adapts mask-and-predict pre-training to continuous visual signals by regressing features of masked content, especially HOG, without tokenization or external model weights. It enables large-scale video Transformers to pre-train directly on unlabeled videos and achieves strong video, transfer, and image results.

  • Motivation and contribution: MaskFeat is motivated by predicting masked features from visible spatiotemporal information, analogous to humans inferring missing visual structure.The paper illustrates predictions of gradient histograms from masked inputs while original images are not used for prediction.
  • Motivation and contribution: MaskFeat directly regresses features of masked visual content instead of requiring a discrete visual vocabulary.The approach uses a vision Transformer to predict feature representations for masked regions.
  • Findings: HOG is a particularly effective MaskFeat target in both performance and efficiency.The study also finds that local pattern characterization is important, while supervised deep features can degrade performance.
  • Practical properties: MaskFeat uses one network and one view per sample, with minimal data augmentation compared with contrastive methods.It therefore avoids the siamese structure and multiple views required by the compared contrastive approaches.
  • Practical properties: MaskFeat with HOG avoids external model weights and the additional pre-training stage and overhead associated with dVAE tokenizers.This removes a computational requirement present in previous masked visual prediction methods.
  • Results: 86.7% top-1 accuracy on Kinetics-400 was achieved by MaskFeat-pre-trained MViT-L without external data, surpassing the best prior number of this kind by +5.2%.The method also reports 39.8 mAP on AVA, 75.0% top-1 accuracy on SSv2, and competitive ImageNet results of 84.0% with ViT-B and 85.7% with ViT-L.

2. Method

MaskFeat randomly masks video space-time cubes and predicts features extracted from the intact sample, using feature regression to define the visual prediction task. It supports video and image inputs and evaluates pixel, HOG, token, deep-feature, and pseudo-label targets.

  • Masked Feature Prediction: MaskFeat masks space-time cubes and predicts features of the corresponding masked regions from the remaining visual input.The target features are extracted from the original intact sample, and the choice of target shapes the pre-trained model.
  • Video instantiation: Masked tokens are processed by a Transformer, whose outputs for masked cubes are projected to target features with a linear layer.The loss operates only on masked cubes and predicts the spatial patch temporally centered in each masked cube.
  • Image instantiation: MaskFeat extends to images by treating them as one-frame videos without a temporal dimension.Each image token represents a spatial patch rather than a space-time cube.
  • Target features: The five target-feature types span direct one-stage targets, including pixel colors and HOG, and two-stage targets produced by trained networks or teachers.Two-stage targets introduce additional computation for teacher pre-training and feature generation.
  • HOG target: HOG describes local gradient-orientation distributions and captures local shapes while providing invariance to several geometric and photometric changes.Its local contrast normalization helps absorb brightness and foreground-background contrast variation.
  • HOG target: For HOG prediction, the method computes a whole-image HOG map, partitions it into patches, and minimizes the ℓ2 distance between predicted and original masked-patch features.HOG is collected in each RGB channel to include color information.
  • Alternative targets: dVAE targets discretize each patch into one of 8192 tokens but require extra computation for dVAE pre-training and image tokenization.Deep-feature targets instead use a pre-trained CNN or ViT teacher and cosine distance on normalized features.

3. Study: Target Features for MaskFeat

The target-feature study compares MaskFeat variants across video and image recognition under matched pre-training recipes. HOG provides a strong and efficient target, while pixel, discrete, unsupervised, and supervised targets show distinct accuracy and computational trade-offs.

  • One-stage targets: RGB prediction changes from-scratch accuracy by -0.4% for video classification and +0.7% for image classification.On IN-1K, pixel prediction reaches 82.5%, compared with 81.0% reported for BEiT.
  • Two-stage targets: +0.6% on K400 and +1.0% on IN-1K are obtained by dVAE over from-scratch baselines, but HOG outperforms it without an external model.This comparison highlights HOG’s accuracy-efficiency trade-off among the studied targets.
  • Two-stage targets: +1.4% on K400 and +2.2% on IN-1K are achieved by the DINO target over its baselines.The comparison uses unsupervised features from models pre-trained on IN-1K, including for video pre-training.
  • Two-stage targets: Supervised features and token labels lag behind unsupervised features and HOG, with significant over-fitting observed during fine-tuning.The study suggests that features learned from class labels are unsuitable MaskFeat targets in this setting.
  • Discussion: The study concludes that discretization is unnecessary because continuous unsupervised features and image descriptors can provide strong masked-prediction targets.Image descriptors such as HOG provide this benefit without the extra model cost of teacher-based targets.

4. Experiments: Video Recognition

MaskFeat improves video recognition by pre-training MViT models on unlabeled videos, achieving strong results across Kinetics and downstream tasks without external supervised image pre-training. Its gains scale with model and dataset size, while cube masking and longer schedules particularly benefit larger models.

  • Kinetics-400: 81.1% top-1 accuracy scratch MViT-S improves by +1.1% with 300 epochs of MaskFeat pre-training on Kinetics-400.The comparison uses direct MViT baselines, with 16×4 denoting 16 input frames at temporal stride 4.
  • Kinetics-400: 86.7% top-1 accuracy on Kinetics-400 is achieved by MViT-L without external data, exceeding the previous best without external data by +5.2%.The model uses larger spatial sizes and longer temporal durations during fine-tuning.
  • Kinetics-400: 87.0% top-1 accuracy is obtained by fine-tuning an MViT-L pre-trained on 387K unlabeled Kinetics-600 videos, while Kinetics-600 pre-training adds +0.8% over Kinetics-400 pre-training.The reported comparison highlights gains from scaling the unlabeled video pre-training dataset.
  • Kinetics-600 and Kinetics-700: 88.3% top-1 on Kinetics-600 and 80.4% top-1 on Kinetics-700 set new state-of-the-art results without external supervised pre-training.These results use larger input resolution and longer duration, while the corresponding models use over 10× fewer FLOPs than previous Transformer-based methods in the standard setting.
  • Downstream transfer: 39.8 mAP on AVA and 75.0% top-1 on SSv2 are achieved after transferring MaskFeat-pre-trained models to action detection and human-object interaction classification.Kinetics-600 pre-training produces the reported best results on both downstream tasks.
  • Ablations: Cube masking performs best among the evaluated masking strategies, suggesting that both spatial and temporal cues help masked spatiotemporal prediction.The masking-strategy comparison uses a shared 40% masking ratio; tube masking reaches 81.9% accuracy.
  • Ablations: Longer pre-training degrades MViT-S accuracy by 0.2% but improves MViT-L accuracy by +1.2%, indicating stronger schedule benefits for larger models.The comparison is between 300 and 800 pre-training epochs on Kinetics-400.

5. Experiments: Image Recognition

Image experiments show that MaskFeat is competitive with supervised and self-supervised pre-training while remaining simpler, and that HOG design choices strongly affect performance.

  • Comparison with previous work: MaskFeat at 2242 is on par with ViT-B or outperforms ViT-L supervised pre-training on IN-21K at 3842, despite using IN-1K without labels.The comparison uses 10× fewer images and no labels for MaskFeat.
  • Comparison with previous work: MaskFeat is more accurate and simpler than contrastive methods, using a single view and minimal augmentation, and avoids BEiT’s dVAE tokenizer.BEiT’s tokenizer requires an extra pre-training stage on 250M DALL-E images.
  • Comparison with previous work: 84.0% top-1 accuracy is achieved by MaskFeat with HOG without an extra model, compared with 84.2% using MaskFeat with DINO and a separately trained ViT-B.The DINO variant uses approximately 1535 additional effective epochs, whereas HOG does not use an extra model.
  • HOG ablations: Local contrast normalization is essential for HOG: ℓ1 normalization drops accuracy by 0.8%, while removing normalization causes a 1.4% decrease.The normalization is applied to histogrammed vectors from local 8×8 pixel cells.
  • HOG ablations: Color information provides approximately a 0.4% gain over grayscale intensity, while HOG remains robust to orientation-bin counts and performs best with 8×8 cells.The default descriptor uses 9 evenly spaced unsigned orientation bins from 0° to 180°.
  • Target analysis: Pixel predictions can be reasonable but are more ambiguous than HOG, including penalties when a plausible color differs from the target.A balloon prediction receives a large loss penalty for predicting black instead of red.
  • Target analysis: Combining pixel and HOG targets reaches 82.3% versus an 81.8% scratch baseline but underperforms either target alone.The equal-weight multi-task setup uses separate linear prediction heads.

6. Related Work

Related work traces masked visual prediction from autoencoding and inpainting through ViT-based objectives, while contrasting MaskFeat with tokenization and contrastive learning.

  • Masked prediction: Masked prediction originated with stacked autoencoders and inpainting, then regained attention after ViT and BERT demonstrated scalable masked modeling.BERT masks input tokens and predicts them for downstream-transferable representations.
  • Masked visual prediction: Vision methods have predicted pixels, mean colors, or dVAE-encoded tokens, whereas MaskFeat directly regresses selected features of masked regions.BEiT and VIMPAC rely on discrete variational autoencoders.
  • Contrastive learning: Contrastive learning instead enforces similarity between augmented views while avoiding model collapse.This related-work family uses augmentation invariance as its central principle.

7. Conclusion

MaskFeat regresses features of masked regions, with HOG emerging as an effective target. The approach is efficient, generalizes across video and image domains, and scales to large models.

  • Conclusion: MaskFeat regresses features of masked regions as a simple visual pre-training objective.The approach is presented for self-supervised visual pre-training.
  • Conclusion: HOG works surprisingly well as the prediction target despite being a hand-designed feature from before deep learning.The conclusion identifies HOG as the characteristic target for MaskFeat.
  • Conclusion: MaskFeat closes the over-5% gap between supervised image pre-training and training from scratch by directly pre-training on unlabeled videos.Transfer learning improves by +5.7 mAP on AVA and +1.1% top-1 on SSv2 over the corresponding IN-21K counterpart.

A. Ablations on Image Classification

Image ablations examine training schedules, augmentation, and representation depth, showing larger gains from longer training in large models, near-minimal augmentation, and stronger linear probing in lower layers.

  • Pre-training schedule: +1% accuracy is obtained by the large model when pre-training increases from 300 to 800 epochs.Both base and large models improve with longer schedules, but the large model benefits more.
  • Data augmentation: A relatively large random-crop scale provides a small gain, while MaskFeat is robust to the crop-scale hyperparameter.The best reported range is [0.5, 1.0], covering a large fraction of each sample.
  • Data augmentation: MaskFeat works best with random resized cropping alone; color jitter has no effect and RandAugment lowers performance by 0.4%.The results support nearly augmentation-free pre-training.
  • Linear probing: Lower transformer layers, such as the 8th block, tend to achieve higher linear-probing accuracy than later layers.The comparison uses outputs from the 8th, 16th, and 24th blocks of a ViT-L.
  • Linear probing: The authors hypothesize that contrastive instance-discrimination losses can produce embeddings dominated by class-level information.This hypothesis is offered to explain the observed linear-probing pattern.

B.1. ImageNet and Kinetics Experiments

ImageNet uses standard ViT models, while Kinetics uses an improved multi-scale MViT with modified positional embeddings, residual pooling, and coordinated masking. Pre-training configurations are largely shared across tasks, supporting MaskFeat’s cross-task generality.

  • Architecture: ImageNet experiments use standard ViT architectures in base and large sizes with a single linear layer for target predictions.Relative positional bias and layer scaling are not used.
  • Architecture: Kinetics experiments use an improved MViT architecture with relative positional embeddings and residual pooling inside attention blocks.The relative embeddings are decomposed across height, width, and temporal axes, while pooled queries are added to self-attention outputs.
  • Architecture: MViT uses four multi-scale stages, but MaskFeat pre-training removes final-stage query pooling to produce a spatial size of 14^2 matching ViT models.Fine-tuning restores the unchanged MViT configuration with a 7^2 output, and the pre-training modification adds little computation.
  • Pre-training: Mask sampling begins at the final output resolution and is nearest-neighbor resized to earlier stages so tokens contributing to one output are masked together.This design avoids trivial predictions from unmasked corresponding input tokens.
  • Pre-training: Most pre-training configurations are shared between ImageNet and Kinetics without task-specific tuning, indicating cross-task generality.Gradient clipping differs by target type: 0.02 for HOG and 0.3 for pixel-color or deep-feature prediction.
  • Fine-tuning: Extra-large long-term video models are initialized from lower-resolution counterparts and fine-tuned for 30 epochs with specified optimization and regularization settings.The recipe disables mixup and uses a learning rate of 1.6e-5, batch size 128, weight decay 1e-8, and drop path rate 0.75.

B.2. AVA Experiments

AVA evaluates spatiotemporal human-action localization using mean Average Precision over 60 classes. MaskFeat uses an MViT-L backbone and a Faster R-CNN-style detection architecture with framewise RoI feature extraction.

  • Dataset and evaluation: AVA contains 211K training and 57K validation video segments for spatiotemporal human-action localization.Methods are evaluated on AVA v2.2 using mean Average Precision over 60 classes.
  • Model and training: The AVA detector uses MViT-L↑312, 40×3 features with framewise RoIAlign on final-layer spatiotemporal feature maps.Maxpooled RoI features are passed to a per-class sigmoid classifier for action prediction.

B.3. SSv2 Experiments

SSv2 fine-tuning evaluates pre-trained MViT-L models on human-object interactions using a specified 40-epoch synchronized-SGD recipe with cosine learning-rate decay.

  • Dataset and model: SSv2 contains 169K training and 25K validation videos spanning 174 human-object interaction classes.The experiments fine-tune pre-trained MViT-L↑312, 40×3 Kinetics models.
  • Training: Fine-tuning runs for 40 epochs with batch size 128, synchronized SGD, cosine decay, weight decay 1e-4, and drop path rate 0.75.The base learning rate is 0.02 per 128-sample batch, with Kinetics-matched training augmentation.

C. Qualitative Experiments

Qualitative experiments show image and video HOG predictions on ImageNet-1K and Kinetics-400 validation data, while updated AVA results reflect changes to the official evaluation code without changing the models.

  • HOG predictions: Image HOG predictions are qualitatively evaluated on ImageNet-1K validation images, and video HOG predictions on Kinetics-400 validation videos.The results are presented in Fig. 4 and Fig. 5, respectively.
  • AVA update: AVA results are updated according to changes in the official evaluation code, while the evaluated models remain unchanged.The update is recorded in Table 19 and Table 5.
Loading 2112.09133v2…