PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning

Anthony Fuller, Scott C. Lowe, Daniel G. Kyrollos, Graham W. Taylor, Evan Shelhamer, James R. Green

arXiv:2609.38278v1cs.CV

TL;DR

Autoencoding reconstructs inputs after an information obstacle, but no autoencoding method directly encourages view-agnostic representations. Masked Swingers independently augments and masks two views, exchanges their CLS tokens before reconstruction, and outperforms MAE across 67 datasets as a frozen feature extractor, including a +45% relative gain on instance retrieval.

  • Problem

    No autoencoding method directly encourages view-agnostic representations, although representing two views similarly underlies many self-supervised learning algorithms.

    Source: §1 · p. 1

  • Method

    Masked Swingers independently augments and masks two views, encodes them, exchanges their CLS tokens, and reconstructs each view’s masked portions.

    Source: §1 · p. 1

  • Results

    Across 67 datasets, Masked Swingers outperforms MAE as a frozen feature extractor, with +45% relative gain on instance retrieval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that Swingers shows potential as a frame-encoder in world modeling and reduces state-prediction error relative to MAE.

    Source: §1 · p. 2, §6 · p. 9

    The study pre-trains only on natural images, leaving other settings for future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Self-supervised learning (SSL) removes the need for annotations and makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches. In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms MAE by +3-5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error -64% relative to MAE on three new state-probing datasets, opening the door to world modeling. Welcome to our Swingers party.

1 INTRODUCTION: AUTOENCODING AND VIEWS

Autoencoding learns representations by reconstructing inputs after an information obstacle, while joint embedding compares representations from two views; no autoencoding method directly encourages view-agnostic representations. Masked Swingers addresses this gap by independently augmenting and masking two views, exchanging their CLS tokens, and reconstructing masked portions.

1 INTRODUCTION: AUTOENCODING AND VIEWS

  • Autoencoding learns useful representations by reconstructing its input after an obstacle such as masked encoder inputs or a low-dimensional bottleneck.Joint embedding separately encodes two views and encourages similar representations, but no autoencoding method directly encourages view-agnostic representations.Source: §1 · p. 1
  • Masked Swingers independently augments and masks two views, encodes each, exchanges their CLS tokens, and reconstructs each view’s masked portions.The exchanged summaries are intended to help reconstruct an arbitrary view and support efficient, broad transfer.Source: §1 · p. 1
  • Masked Autoencoders have been used across natural, satellite, time-series, and point-cloud data, motivating efforts to improve the framework.MAE also has low device-memory requirements and grounded targets, and does not require cross-sample negatives or statistics to avoid collapse.Source: §1 · pp. 1–2
  • Masked Swingers outperforms MAE across 67 datasets as a frozen feature extractor, including +45% relative gain on instance retrieval and −64% error relative to MAE on state prediction.The authors also report gains on animal re-identification and species classification, and study mask rate, model size, token exchange, effective rank, and state prediction.Source: §1 · p. 2

2 BACKGROUND: MASKED AUTOENCODERS

MAE divides an image into patch tokens, randomly drops a subset before encoding, and decodes all patch positions to predict the masked content. Its reconstruction loss is evaluated only on masked positions.

2 BACKGROUND: MASKED AUTOENCODERS

  • An image is split into non-overlapping patches, flattened, projected to the token dimension, and combined with positional embeddings.The patch sequence forms the input representation used by the MAE pipeline.Source: §2 · p. 2
  • MAE randomly drops a subset of patches at masking ratio ρ, leaving visible patches for the encoder and masked positions for reconstruction.The background also describes color masking, which can create more contiguous visible and masked regions than uniform masking.Source: §2 · p. 2
  • The encoder processes a prepended learnable CLS token and visible patch tokens, while the decoder receives representations for all patch positions and the CLS token.Visible patches are projected to the decoder width; masked positions receive a shared learnable mask token and decoder positional embeddings.Source: §2 · p. 3
  • MAE uses mean squared error on masked positions only, with targets formed from image patches normalized by each patch’s mean and standard deviation.The decoder discards its CLS output and maps patch latents to pixel predictions.Source: §2 · p. 3

3 MASKED SWINGERS: TRADING CROSS-VIEW CLS TOKENS

Masked Swingers changes MAE by supplying the decoder with the CLS token encoded from the other independently augmented and masked view. This makes each view’s summary useful for reconstructing content under a different view’s transformations.

3 MASKED SWINGERS: TRADING CROSS-VIEW CLS TOKENS

  • The two augmented views are patchified and masked independently, then encoded separately with the same encoder.Each view yields its own CLS and patch-token representations.Source: §3 · p. 3
  • Masked Swingers trades the two views’ CLS tokens before decoding, while retaining each view’s own patch tokens and mask tokens.The decoder’s remaining processing is unchanged from MAE.Source: §3 · p. 3
  • Full-Swingers requires each view’s masked patches to be inferred using its own visible patches and the other view’s CLS summary.This promotes a summary that remains useful under a different augmentation and mask; reconstruction loss is averaged over both views.Source: §3 · p. 3
  • Half-Swingers uses one view’s CLS token to decode both views, equivalent to exchanging CLS tokens for half the images and keeping same-view CLS tokens for the rest.The encoder may balance summarizing visible content for arbitrary views against inferring its own view’s masked patches.Source: §3 · p. 3

4 EXPERIMENTING WIDELY WITH CONTROLS AND CONTEXT

The section compares MAE and Swingers under controlled training and evaluation settings, then examines masking, model size, representation structure, and state prediction. It also tests fine-tuning and several design alternatives.

4.1 PRE-TRAINING: VARYING AUGMENTATION, COLOR MASKING, AND EPOCHS

  • Half-Swingers outperforms MAE on all 10 frozen-feature tasks; its IN-1K kNN gains are +5.1/+5.2/+2.8 pp at 100/200/800 epochs.Gains tend to be larger on the other tasks; natural-image gains are largest on shorter schedules, while gains on other tasks maintain or grow with longer pre-training.Source: Figure 2 · p. 4, §4.1 · p. 5
  • Both Swinger variants fine-tune as well as MAE on ImageNet-1K in the controlled experiments.Pre-training length affects downstream accuracy more than the differences among controlled methods.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 PRE-TRAINING: VARYING MASK RATE AND MODEL SIZE, ALONGSIDE AUGMENTATION

  • At a 65% mask rate, Full-Swingers reaches 50% IN-1K kNN accuracy, matching Half-Swingers’ best result; Omniglot instead favors higher masking.The lower mask rate also improves Full-Swingers on 10-shot IN-21K, all mini-VTAB subsets, retrieval, animal re-identification, and fine-grained species classification.Source: §4.2 · p. 6, Figure 4 · p. 6
  • Full-Swingers outperforms Half-Swingers at smaller model sizes, while Half-Swingers scales better to ViT-L; stronger augmentation tends to help Full-Swingers across sizes.The size comparison uses the default 75% mask rate, which is suboptimal for Full-Swingers and may narrow its gap at larger sizes.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3 ANALYSIS, ABLATIONS, AND ALTERNATIVES

  • With longer pre-training, Full-Swingers’ IN-1K kNN accuracy falls when views are highly similar but rises when views are more dissimilar.The authors conclude that mutual information between views shapes the learned representations.Source: §4.3 · p. 7, Table 1 · p. 7
  • CLS effective rank rises with Omniglot accuracy, while IN-21K accuracy rises only up to a point and then declines.The authors suggest, tentatively, that additional directions may encode subtle stroke shapes that help distinguish Omniglot characters but eventually hurt IN-21K.Source: §4.3 · pp. 7–8, Figure 6 · p. 7
  • Sharing augmentations substantially hurts frozen-feature accuracy but can boost fine-tuning; exchanging mean-pooled patch tokens is also competitive with CLS exchange.The shared-augmentation ablation reduces IN-1K kNN accuracy by 25.2 pp for Full-Swingers and 38.0 pp for Half-Swingers.Source: §4.3 · p. 8

4.4 STATE PROBING FOR WORLD MODELING

  • Swingers reduces state-probing error by 64% relative to MAE/ColorMAE, averaged over all models and three tasks.It also reduces error relative to SimDINOv2 by 73% on Dino and 59% on Golf; the MotionJEPA comparison is not direct because its model and training setup differ.Source: §4.4 · p. 9, Table 4

5 RELATED WORK: A WALK AROUND THE NEIGHBOURHOOD

Related work includes cross-view autoencoders and other masked-autoencoder improvements, while masked latent modeling predicts representations rather than pixels. Masked Swingers differs from these methods through its use of augmentation and a single exchanged token.

Cross-View Autoencoding

  • CropMAE uses a mild crop and flip, masks nearly all of its second view, and decodes with tokens from both views; the authors’ ViT-S evaluation gives it 12% IN-1K kNN accuracy versus 40% for Swingers.CroCo and SiamMAE instead derive views from 3D scenes or neighboring video frames; their reported results use different data and training lengths and are not directly comparable.Source: §5 · p. 9

Other Masked Autoencoders

  • Masked-autoencoder work also explores masking strategies, image-derived targets, and decoder changes, which the authors describe as potentially complementary to Swingers.They study ColorMAE masking because it is data-independent, model-free, and intuitive.Source: §5 · p. 9

Masked Latent Modeling

  • Masked latent modeling predicts representations at masked locations rather than pixels, using teacher representations from an EMA of the student; the authors identify it as a possible direction for future Swingers work.Source: §5 · p. 9

6 SWINGING SHUT

The study is limited to natural-image pre-training, and Masked Swingers requires data augmentation unlike MAE. Controlled experiments report stronger performance than MAE, especially on fine-grained tasks, and identify independent augmentations as important for higher-level features.

Limitations and future work

  • The study pre-trains only on natural images, leaving experiments on 3D data and video to future work.Source: §6 · p. 9
  • Masked Swingers requires data augmentation to create different views, unlike MAE, though the authors characterize this limitation as minor in practice.The authors also note that Swingers can underperform state-of-the-art methods from other SSL frameworks, while retaining robustness to training settings and low device-memory requirements.Source: §6 · p. 9

Conclusion

  • Swingers significantly outperforms the MAE baseline in controlled experiments, especially on fine-grained tasks, and independent pre-training augmentations are crucial to extracting higher-level features.The conclusion also reports potential as a frame encoder in world modeling.Source: §6 · p. 9

A APPENDIX

The authors say code and models will be released soon and that the arXiv will be updated when they are ready.

Appendix

  • The authors plan to release all code and models soon and update the arXiv when they are ready.Source: A APPENDIX · p. 16

A.1 ADDITIONAL PRE-TRAINING DETAILS

Pre-training uses independently augmented and masked image views, with specified model, optimizer, and learning-rate settings. The appendix also details color-mask generation and fixed positional embeddings.

A.1 Additional Pre-Training Details

  • Each training step samples 1,024 ImageNet-1K images and makes two views per image; each view receives an independent random resized crop, horizontal flip, and RandAugment, with no color jitter or random erasing.Weak, medium, and strong RandAugment use 1, 2, and 4 operations at magnitudes 5, 9, and 15, respectively.Source: §A.1 · p. 16
  • Views are independently masked at a default 75% rate; color masking filters 14 × 14-grid Gaussian noise and keeps the lowest-scoring patches.Red uses Gaussian filtering with σ=2 patches, while green uses a difference of Gaussians with σ=1 and 4 patches.Source: §A.1 · p. 16
  • The encoder and decoder use fixed 2D sin-cos positional embeddings, and AdamW training uses a peak learning rate of 6×10^-4 with warmup followed by cosine decay.The decoder has 8 blocks; encoder and decoder widths vary by ViT size.Source: §A.1 · p. 16

A.2 ADDITIONAL FINE-TUNING DETAILS

Fine-tuning uses a 100-epoch IN-1K recipe with tuned learning rates and a specified augmentation pipeline, evaluated on the validation set.

A.2 ADDITIONAL FINE-TUNING DETAILS

  • Models are fine-tuned on IN-1K for 100 epochs with batch size 1,024, AdamW, layer-wise learning-rate decay, and drop path.The learning rate warms up linearly for 5 epochs and then cosine-decays to 10^-6.Source: §A.2 · p. 16
  • Augmentation combines random resized crops, horizontal flips, RandAugment, random erasing, and an equal-probability choice of Mixup or CutMix.Label smoothing is 0.1.Source: §A.2 · p. 16
  • Evaluation reports the best result across four base learning rates on 50K IN-1K validation images resized to 256 pixels and center-cropped to 224 × 224.The tested base rates are {2.5, 5, 10, 20}×10^-4, scaled by batch size.Source: §A.2 · p. 16

A.3 REFERENCE MODELS

Reference models are evaluated using public or author-provided pretrained checkpoints, with the main comparison set based on IN-1K-pretrained ViT-B/16 models.

A.3 REFERENCE MODELS

  • Reference models use ViT-B/16 checkpoints pretrained on IN-1K at 224 × 224 pixels, evaluated with the same frozen-feature protocols as the authors’ models.Bootleg uses its EMA target encoder, with four register tokens discarded before pooling.Source: §A.3 · p. 16
  • The reference set includes MAE, ColorMAE, LeJEPA, SimDINOv2, and Bootleg checkpoints, with checkpoint sources and pretraining lengths specified.LeJEPA uses a public reproduction because no official ViT-B/16 release is available.Source: §A.3 · p. 16
  • Additional comparisons use a ViT-S/16 CropMAE checkpoint and ViT-B CroCo and CroCo v2 checkpoints pretrained on image pairs of 3D scenes rather than IN-1K.These further checkpoint sets are used for comparisons in Sec. 5.Source: §A.3 · p. 16

A.4 FROZEN-FEATURE EVALUATION DETAILS

The appendix specifies frozen-feature evaluation procedures and presents result tables for the pretrained models and public checkpoints, with table headings identifying the reported tasks and metrics.

A.4 FROZEN-FEATURE EVALUATION DETAILS

  • For frozen-feature evaluation, images are center-cropped to 224 × 224, and both the final-layer CLS token and mean final-layer patch tokens are extracted.The crop follows resizing the shorter image side to 224 pixels.Source: §A.4 · p. 16
  • kNN features are standardized using training-set statistics and L2-normalized; retrieval, animal re-identification, and Omniglot use cosine similarity between normalized features without tuning.The kNN procedure jointly sweeps k and temperature, selecting them on the evaluation split; specified retrieval datasets use query bounding-box crops.Source: §A.4 · p. 17

A.5 ALL RESULTS

  • The appendix presents result tables covering all 81 pretrained models and evaluations of official or public checkpoints, with headings for retrieval, kNN, and state-information prediction results.The listed table headings include ROxford, RParis, ILIAS, SOP, Omniglot, FungiTastic, and Pong, Dino, and Golf prediction tasks.Source: §A.5 · pp. 18–19
Loading 2609.38278v1…