PaperlayerView exact revision on arXiv

Source-linked AI summary

Extreme Masking for Learning Instance and Distributed Visual Representations

Zhirong Wu, Zihang Lai, Xiao Sun, Stephen Lin

arXiv:2206.04667v2cs.CV

TL;DR

ExtreMA addresses the challenge of learning both spatially distributed token representations and holistic instance representations without relying on masked modeling’s token-prediction objective. It uses extreme random masking in a Siamese BYOL-style architecture, with a masked view predicting an intact view through ViT and cross-attention representations. The method reports stronger masked-modeling comparisons and transfer results, while extreme masking and multi-masking improve efficiency but introduce overfitting and tuning constraints.

  • Problem

    Existing approaches emphasize either distributed token representations through masked modeling or holistic instance representations through Siamese learning, leaving their connection and masking’s role as an augmentation insufficiently addressed.

    Source: §1 · p. 1

  • Method

    ExtreMA uses 75%-90% random masking, a full-view momentum encoder, masked-view base encoder, ViT token features, and cross-attention instance aggregation with asymmetric BYOL-style prediction.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ExtreMA outperforms prior masked modeling approaches on ImageNet1k finetuning and linear probing and achieves superior semi-supervised and semantic-segmentation transfer results.

    Source: §1 · p. 2, §5.3 · p. 10

  • Takeaways & Limitations

    Random masking can serve as a computationally efficient Siamese augmentation that learns both instance and distributed representations while preserving useful spatial and generative properties.

    Source: §1 · p. 2, §6 · p. 11

    Extreme masking can overfit, and the approach requires CaiT-style architecture plus dataset-specific tuning of training epochs and masking ratio.

    Source: §3 · pp. 4, 6

Abstract

from arXiv · show

The paper presents a scalable approach for learning spatially distributed visual representations over individual tokens and a holistic instance representation simultaneously. We use self-attention blocks to represent spatially distributed tokens, followed by cross-attention blocks to aggregate the holistic image instance. The core of the approach is the use of extremely large token masking (75\%-90\%) as the data augmentation for supervision. Our model, named ExtreMA, follows the plain BYOL approach where the instance representation from the unmasked subset is trained to predict that from the intact input. Instead of encouraging invariance across inputs, the model is required to capture informative variations in an image. The paper makes three contributions: 1) It presents random masking as a strong and computationally efficient data augmentation for siamese representation learning. 2) With multiple sampling per instance, extreme masking greatly speeds up learning and improves performance with more data. 3) ExtreMA obtains stronger linear probing performance than masked modeling methods, and better transfer performance than prior contrastive models.

1 Introduction

ExtreMA addresses the gap between masked modeling’s distributed token features and Siamese learning’s holistic instance features by using extreme random masking as self-supervision. It combines token-level spatial representations with instance-level prediction and reports strong downstream performance, efficient training, and useful generative and locality properties.

1 Introduction

  • Masked modeling provides strong finetuning but weaker off-the-shelf retrieval, whereas Siamese networks directly learn holistic instance representations for such use.This distinction motivates learning both distributed token features and an explicit instance representation.Source: §1 · p. 1
  • ExtreMA trains a masked-view instance representation to predict the intact-view representation, while ViT and cross-attention jointly support distributed and holistic features.The instance-level objective also supervises distributed representations implicitly, without masked modeling’s explicit token-prediction loss.Source: §1 · p. 2
  • 75%-90% masking makes random masking a strong augmentation: it preserves unmasked content and geometry while creating a large information gap for richer self-supervision.Multiple masks produce complementary visible content, and the full-view teacher needs only one forward pass.Source: §1 · pp. 1–2
  • Unlike conventional contrastive learning, ExtreMA does not seek augmentation-induced invariance; it preserves information needed to recover the full image and remains sensitive to scale and location.The model can also inpaint masked pixels through network inversion and learn representations capturing scale, location, and color intensities.Source: §1 · p. 2
  • ExtreMA outperforms prior masked modeling approaches on ImageNet1k finetuning and linear probing, while using one 8×V100 node for about two days with ViT-Base.The paper also reports superior transfer results for semi-supervised learning and semantic segmentation.Source: §1 · p. 2

2 Related Works

Prior self-supervised vision methods construct pretext tasks through masked prediction or instance discrimination, with different strengths in token-level structure, invariance, and off-the-shelf utility. Recent work combines masking and Siamese learning, but the role of masking as a standalone augmentation remains unresolved.

2 Related Works

  • Masked modeling predicts hidden tokens from visible tokens using targets such as dVAE tokens, pixels, or encoder features, and often improves finetuning performance.Its targets and objectives vary across BERT-like and online-encoder formulations.Source: §2 · p. 3
  • Contrastive learning learns view-invariant instance representations through augmented views, with methods differing in their use of negatives, clusters, or momentum encoders.Its flexible augmentation framework has supported strong performance at scale and multiple-crop variants.Source: §2 · p. 3
  • Masking in Siamese networks has been explored, but MSN uses only 30% masking and depends heavily on other augmentations, while ADIOS optimizes masking adversarially.Hybrid methods combine contrastive and masked-modeling objectives rather than treating masking itself as the augmentation.Source: §2 · p. 3
  • ExtreMA differs by using random masking as a Siamese augmentation with instance-level supervision and no explicit token-level prediction objective.This targets the unresolved connection between distributed representations from masking and holistic representations from Siamese learning.Source: §2 · p. 3

3 The ExtreMA Approach

ExtreMA uses a full-view momentum encoder and an extremely masked base encoder, combining ViT self-attention with cross-attention aggregation and asymmetric BYOL-style instance prediction. The design supports distributed token structure and generative reconstruction, but requires careful architectural and training choices because extreme masking can destabilize or overfit learning.

3 The ExtreMA Approach

  • ExtreMA’s full-view momentum encoder supplies the target while multiple masked base encoders process different partial views, enabling efficient multi-masking.Shared teacher targets make multi-masking more efficient than masked modeling, whose decoder must run separately for each mask.Source: §3 · pp. 3–4
  • The model performs best with 75%-90% masking, leaving the base encoder only 10%-25% of patches and creating a harder pretext task than masked image modeling.The authors hypothesize that full-view processing by the momentum encoder helps avoid masked-modeling train-test discrepancy.Source: §3 · p. 4
  • Extreme masking can overfit, especially with multi-masking, yielding strong training classification but weaker generalization; more unlabeled data can mitigate this.The authors characterize this as learning training-set semantics through self-supervision rather than cheating.Source: §3 · pp. 4–5
  • ViT self-attention forms distributed patch representations, while CaiT-style cross-attention uses a class token to aggregate them with O(N) rather than O(N^2) complexity.Only the class token queries the cross-attention blocks, while patch tokens remain frozen there.Source: §3 · p. 5
  • The asymmetric objective minimizes cosine distance from the masked-view prediction to the intact-view representation, encouraging extrapolation of masked regions instead of shared-view invariance.This differs from symmetric BYOL because only the masked view predicts the intact view.Source: §3 · p. 6
  • Attention maps show meaningful semantic clusters across patch tokens despite the absence of explicit token-level supervision.The visualization averages attention across 12 heads for four query patches.Source: Figure 1 · p. 4, §3 · p. 6
  • Stable convergence depends on CaiT-style architecture, while conventional ViT can fail; training epochs and the optimal masking ratio require dataset-specific tuning.These architectural and hyperparameter dependencies are stated limitations of the approach.Source: §3 · pp. 5–6

4 Representation Properties

ExtreMA learns distributed features that support visual reconstruction and an instance representation sensitive to spatial and scale configuration, rather than only semantic identity.

Generative Properties

  • ExtreMA’s distributed representations can reconstruct masked image content, with smoother spatial structure and more accurate colors than the compared encoder representations.The inversion method reconstructs masked patches from visible distributed features; reconstruction degrades at very large masking ratios because unseen semantic areas are difficult to inpaint.Source: §4 · p. 6

Locality Properties

  • The instance representation retrieves image crops with the closest spatial and scale configuration to a query crop, indicating sensitivity beyond semantic similarity.The retrieval gallery varies crop location and scale, and the top-ranked crop matches the query’s spatial and scale configuration.Source: §4 · p. 7, Figure 3 · p. 7

5 Experiments

Experiments show that extreme masking, multiple masks, and suitable augmentation choices make ExtreMA efficient and competitive across classification and transfer tasks. Performance depends on data scale and augmentation design: larger datasets reduce multi-mask overfitting, while shared crops with different colors yield the best ablation results.

5.1 Ablation Studies

  • With a single mask, finetuning plateaus from 70% to 90% masking, linear probing peaks at 80%, and performance degrades beyond 90%.The 90% masking ratio still achieves reasonably good performance despite the extreme information reduction.Source: §5.1 · p. 8
  • Multi-masking improves finetuning but can reduce linear probing through overfitting on ImageNet1k, whereas ImageNet22k removes this overfitting as masks increase and improves performance.At 90% masking, multi-masking converges steeply but saturates and degrades after 120 epochs on ImageNet1k; the larger dataset preserves the faster learning benefit.Source: §5.1 · p. 8, Figure 4 · p. 8
  • A shared spatial crop with different color augmentations achieves the best overall finetuning and linear probing performance.Adding spatial and scale invariance reduces finetuning by 0.4% and linear probing by 2.0%, whereas color augmentation raises finetuning from 82.5% to 83.3%.Source: §5.1 · pp. 8–9, Table 4 · p. 8
  • Two cross-attention blocks improve linear probing by 2% before saturation, while finetuning is unaffected by cross-attention depth; replacing BYOL with MoCo-v3 raises finetuning 0.2% but lowers linear probing 2.0%.The ablations use ViT-Base with the stated optimizer, augmentation, and 300-epoch training setup.Source: §5.1 · pp. 7–9, Table 5 · p. 8, Table 6 · p. 8

5.2 ImageNet Comparisons with Previous Methods

  • ExtreMA outperforms MoCo-v3, DINO, MSN, MAE, and BEiT on ImageNet finetuning, beats masked modeling methods by a large margin in linear probing, and trains 5× to 10× faster.Data2vec remains better for finetuning, while contrastive methods remain better for linear probing; the speed comparison uses a single node of 8×V100 GPUs.Source: §5.2 · pp. 9–10, Table 7 · p. 9, Table 8 · p. 9

5.3 Transfer Learning Results

  • On transfer tasks, ExtreMA outperforms MAE and other baselines in semi-supervised classification, semantic segmentation, and same-budget COCO transfer, while ImageNet22k improves segmentation by 0.5 mIoU and COCO by about 1% AP.For semi-supervised classification, the gains over MAE and BEiT are 12% and 30% with 1% labels; longer MAE pretraining can surpass ExtreMA on COCO.Source: §5.3 · pp. 10–11, Table 10 · p. 10, Table 11 · p. 11

6 Conclusions

ExtreMA uses extreme masking as a Siamese-learning augmentation to jointly learn strong instance and distributed representations. It offers fast convergence with multi-masking, low resource consumption, and potential for precise locality in detection transfer.

  • ExtreMA learns strong instance and distributed representations through masking-based Siamese learning without masked-modeling supervision.Source: §6 · p. 11
  • Extreme masking uses ratios of 75%-90%, while multi-masking accelerates convergence and supports scalability to large datasets.Source: §6 · p. 11
  • The model has low computational resource consumption and may support detection transfer through precise locality encoding.Source: §6 · p. 11

Broader Impacts

The method may inherit biases from its training data and downstream fine-tuning datasets. The study is limited to curated datasets, leaving uncurated-data research for future work.

  • Training on a specific dataset may transmit its biases into the learned representations and downstream applications.Source: Broader Impacts · p. 11
  • The study is limited to curated datasets, so research on uncurated datasets remains future work.Source: Broader Impacts · p. 11

A Discussions on CaiT-Style Architecture

The appendix examines architecture stability, multi-masking efficiency, and the generative behavior of ExtreMA. CaiT-style class-token placement stabilizes learning, multi-masking preserves similar performance while reducing training time, and inversion enables masked-region inpainting.

A Discussions on CaiT-Style Architecture

B Multi-masking Performance Efficiency

C Technical Details of the Generative Property

D Additional Comparisons of Locality Properties

The locality comparison evaluates ExtreMA against MAE, DINO, and MoCo-v3 using class-token representations. DINO performs favorably, while MAE and MoCo-v3 show degraded localization performance under this comparison.

E Details of Evaluation Protocols

The evaluation protocols specify separate end-to-end fine-tuning and linear probing setups, with configurations detailed in Tables 13 and 14. Training schedules and optimization settings vary by model pretraining scale.

End-to-end fine-tuning

Optimization settings

Linear probing

Optimization settings

F Additional Visualizations

Additional visualizations examine ExtreMA’s distributed representations through inpainting examples, reconstruction failures, and attention maps. They show token-level correspondence and semantic relationships, while exposing limits when masking is large.

Distributed representations

  • The distributed representations preserve accurate input-token correspondences while inferring meaningful semantic relationships among tokens.These properties are revealed by the generative and attention visualizations.Source: F Additional Visualizations · p. 18

Reconstruction failures

Inpainting visualizations

  • Figure 7 provides additional inpainting examples across various masking ratios.Source: Figure 7 · p. 19

Attention maps

  • The attention maps show diverse, distributed responses, unlike DINO, which mainly attends to the foreground object.The maps average responses from 12 attention heads, and border colors correspond to colored queries in the input image.Source: Figure 9 · p. 21

Related papers

Loading 2206.04667v2…