PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Frequency Modeling for Self-Supervised Visual Pre-Training

Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, Chen Change Loy

arXiv:2206.07706v2cs.CVcs.LG

TL;DR

Spatial patch masking confronts image redundancy, and spatial restoration corruptions obscure what information is lost. MFM masks and predicts image-frequency components, and the paper reports competitive downstream performance and robustness while studying restoration corruptions across architectures.

  • Problem

    Spatial-domain image patches have low information density and heavy redundancy, motivating alternatives to high-proportion masking for meaningful representation learning.

    Source: §1 · p. 1

  • Method

    MFM applies FFT, masks a portion of frequency components, converts the corrupted spectrum back with inverse FFT, and predicts masked frequencies with a lightweight decoder.

    Source: §1 · p. 2

  • Results

    MFM achieves competitive image-classification and semantic-segmentation performance and appealing robustness compared with recent MIM approaches, while low-level corruption tasks show architecture-dependent results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Frequency-domain masking can learn representations for both ViT and CNN without mask tokens or more complex designs.

    Source: §5 · p. 9

    Scaling behavior remains under-explored, evaluation focuses on ImageNet-1K classification and ADE20K segmentation, and frequency redundancy may remain.

    Source: D FURTHER DISCUSSION · p. 17

Abstract

from arXiv · show

We present Masked Frequency Modeling (MFM), a unified frequency-domain-based approach for self-supervised pre-training of visual models. Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain. Specifically, MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum. Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due to the heavy spatial redundancy. Our findings suggest that with the right configuration of mask-and-predict strategy, both the structural information within high-frequency components and the low-level statistics among low-frequency counterparts are useful in learning good representations. For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token. Experimental results on image classification and semantic segmentation, as well as several robustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches. Furthermore, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from a unified frequency perspective and reveal their intriguing relations with our MFM approach.

1 INTRODUCTION

Spatial patch masking contends with image redundancy, while spatial restoration corruptions do not reveal exactly what information is lost. MFM instead masks and predicts frequency components, and the paper studies its architectural flexibility and performance alongside low-level corruption tasks.

1 INTRODUCTION

  • Spatial patch masking faces heavy image redundancy, so MAE uses a very high masking proportion, such as 75%, to encourage meaningful features.Raw spatial pixel values have lower information density than human-generated language.Source: §1 · p. 1
  • Downsampling and blur mainly remove high-frequency details, whereas noise tends to corrupt the full frequency spectrum.The introduction motivates examining such corruptions in the frequency domain because spatial corruption obscures what information must be reconstructed.Source: §1 · p. 2, Figure 1 · p. 2
  • MFM masks selected frequency components and predicts their missing values, using FFT and inverse FFT without inserting mask tokens.A lightweight linear decoder reconstructs masked frequency values through a frequency loss.Source: §1 · p. 2
  • MFM is presented as architecture-agnostic, with demonstrated flexibility across ViT and CNN families.Its frequency-mask strategy is proposed as a self-supervised visual pre-training task.Source: §1 · p. 2
  • Low-level corruption tasks achieve comparable or better results than their supervised counterparts on ViT, but show no gains on CNN.The study examines super-resolution, deblurring, and denoising corruptions for self-supervised representation learning.Source: §1 · pp. 1–2
  • MFM achieves competitive performance on image classification and semantic segmentation, with appealing robustness compared with recent MIM approaches.The method uses neither mask tokens nor more complex designs.Source: §1 · p. 2

2 RELATED WORK

Related work covers masked prediction, contrastive learning, low-level restoration, and frequency-domain analysis. MFM differs by masking frequency components rather than spatial patches, while the paper also examines restoration corruptions for representation learning.

2 RELATED WORK

  • MFM masks in the frequency domain, unlike prior methods that randomly mask spatial image patches or predict tokens, pixels, or handcrafted features.The paper describes its approach as using no extra data, extra model, or mask token, and reports negligible computational cost for frequency-domain corruption.Source: §2 · p. 3
  • Masked image modeling follows the mask-and-predict pattern, with approaches varying in their prediction targets and architectures.The cited examples include discrete tokens, raw pixels, handcrafted features, autoregressive pixel prediction, and Siamese frameworks.Source: §2 · p. 3
  • Contrastive learning typically uses a Siamese framework and relies heavily on data augmentation, unlike mask-and-predict pretext tasks.The paper distinguishes these learning setups in its discussion of self-supervised approaches.Source: §2 · p. 3
  • Low-level restoration tasks recover high-fidelity images from inputs degraded by operations such as downsampling, blur, noise, and JPEG compression.The paper investigates these corruption operations for representation learning in light of MIM results.Source: §2 · p. 3
  • Prior frequency-domain studies report that phase largely captures high-level semantics, while amplitude mainly retains low-level statistics.The paper uses this frequency perspective to motivate masked-information modeling for images.Source: §2 · p. 3

3 APPROACH

MFM masks image frequencies and predicts the missing spectrum values, using filtered spatial images as encoder inputs. The paper also relates this approach to low-level image restoration tasks through a unified frequency perspective.

3 APPROACH

  • MFM masks image frequency components and predicts the missing spectrum values from filtered spatial images using a frequency loss.The pipeline applies FFT, a low- or high-pass filter, and inverse FFT before encoding.Source: §3 · p. 3, Figure 2 · p. 4

3.1 MASKED FREQUENCY MODELING

  • A thresholded mask separates low and high frequencies by radius; the default uses a circular mask, though other shapes are possible.The low- and high-pass filtered images are formed separately, then randomly selected as encoder inputs.Source: §3.1 · p. 4
  • MFM accepts spatial images without mask tokens, allowing use with ViT or CNN encoders while keeping the input domain consistent with fine-tuning.ViT receives patches with positional embeddings; the CNN receives the filtered spatial image directly.Source: §3.1 · pp. 4–5
  • A lightweight linear decoder supports frequency reconstruction, with distance measured between reconstructed and original frequency values using real and imaginary components.The default distance exponent γ is 1, and experiments found that computing loss only on masked spectrum positions is more accurate than using the full spectrum.Source: §3.1 · p. 5

3.2 RELATION WITH LOW-LEVEL IMAGE PROCESSING TASKS

  • MFM’s frequency view unifies corruptions used in super-resolution, deblurring, and denoising, and the experiments report better representation-learning performance than those tasks.Super-resolution and deblurring mainly remove high frequencies, while denoising alters both low and high frequencies.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments evaluate MFM’s masking choices, low-level restoration tasks, and downstream performance across classification, segmentation, and robustness benchmarks. Results show competitive performance, with MFM improving on supervised counterparts for both ViT and CNN architectures and offering a strong standard-accuracy/robustness trade-off.

4.1 IMPLEMENTATION DETAILS

  • Pre-training uses unlabeled ImageNet-1K data, and the study evaluates ViT-S/16, ViT-B/16, and ResNet-50 backbones with supervised fine-tuning on ImageNet-1K and ADE20K.The MFM ablations use ViT-B/16 with 300-epoch pre-training and typically 100-epoch fine-tuning; the default mask samples low- and high-pass filters, with radius 16 and a circle shape.Source: §4.1 · pp. 5–6, §4.2 · p. 6, Table 1 · p. 6

4.2 MAIN PROPERTIES

  • Masking either high or low frequencies beats reconstructing the original image, and randomly sampling both filters improves performance further; the best tested settings use a moderate radius, circle mask, and equal filter sampling.Predicting masked frequencies outperforms reconstructing the full spectrum (83.1% vs. 82.4%), and frequency loss performs better than spatial losses.Source: §4.2 · pp. 6–7

4.3 DIAGNOSIS OF LOW-LEVEL IMAGE PROCESSING TASKS

  • With task difficulty configured appropriately, super-resolution, deblurring, and denoising can match or exceed their supervised counterpart; denoising performs closer to MFM than super-resolution or deblurring.The best super-resolution setting uses ×8 downsampling, heavier than the usual setting within ×4; denoising intensifies the full frequency spectrum, unlike super-resolution and deblurring, which remove high frequencies.Source: §4.3 · pp. 7–8

4.4.1 IMAGE CLASSIFICATION

  • MFM is competitive with representative self-supervised learners on ViT classification and improves on supervised counterparts for both ViT and CNN; on ResNet-50 it improves the RSB A2 baseline by 0.3%.ViT comparisons use fewer pre-training epochs without extra data, an extra model, or mask tokens; under 300-epoch fine-tuning, MFM also surpasses several contrastive methods.Source: §4.4.1 · pp. 8–9, Table 4 · p. 8, Table 3 · p. 8

4.4.2 SEMANTIC SEGMENTATION

  • Corruption-based models achieve competitive ADE20K semantic-segmentation performance against representative self-supervised learners that are usually more expensive to compute.Models use UperNet and are fine-tuned for 160K iterations at 512×512 input resolution.Source: §4.4.2 · p. 9, Table 5 · p. 9

4.5 ROBUSTNESS EVALUATION

  • Across six robustness benchmarks, MFM ranks within the top two in robustness while achieving the best standard accuracy, the strongest reported trade-off.The evaluation covers adversarial, common-corruption, and out-of-distribution robustness without specialized fine-tuning on those validation sets.Source: §4.5 · p. 9, Table 6 · p. 9

5 CONCLUSION

The paper introduces a frequency-domain self-supervised pre-training framework and finds that a simple frequency-masking strategy achieves competitive performance for both ViT and CNN without mask tokens or more complex designs.

5 CONCLUSION

  • MFM performs frequency-domain image corruptions in a unified, flexible, and robust self-supervised pre-training framework.The framework does not rely on mask tokens or more complex designs.Source: §5 · p. 9
  • A simple mask-frequency strategy achieves competitive performance for both ViT and CNN.Source: §5 · p. 9
  • The authors hope their frequency perspective will motivate researchers to rethink low-level tasks for unsupervised representation learning.Source: §5 · p. 9

ETHICS STATEMENT

The authors note that the method learns training-dataset statistics and may reflect data biases; they also flag possible environmental impacts from deployment with large-scale models and data.

ETHICS STATEMENT

  • The method may reflect training-data biases and may have negative environmental impacts when deployed with large-scale models and data.The authors state that debiased measures are therefore needed.Source: ETHICS STATEMENT · p. 10

A MORE ABLATIONS

The ablations compare decoder choices and spatial versus frequency masking, with results favoring a linear decoder and frequency masking across architectures. The table evaluates models pre-trained for 300 epochs using top-1 fine-tuning accuracy.

A MORE ABLATIONS

  • The ablation table covers decoder depth and masking domain, with top-1 fine-tuning accuracy as its evaluation measure.Source: Table 7 · p. 15, A MORE ABLATIONS · p. 15
  • 83.1% vs 82.8% top-1 accuracy for ViT-B/16, and 78.5% vs 77.7% for ResNet-50, favor MFM over MIM.The table reports top-1 fine-tuning accuracy after 300 pre-training epochs.Source: A MORE ABLATIONS · p. 15, Table 7 · p. 15
  • A simple linear layer performs best among the tested decoders and costs less to train than a deeper decoder.Source: A MORE ABLATIONS · p. 15
  • MFM outperforms the supervised RSB baseline and MIM counterpart across architectures, while spatially masked ResNet-50 falls below the 78.1% supervised RSB A3 baseline.The comparison replaces frequency masking with random spatial patch masking to reduce interference from other design choices.Source: A MORE ABLATIONS · p. 15

B TRAINING TIME COMPARISON

MFM is a one-view method measured against other pre-training approaches, and is relatively efficient compared with MIM methods except MAE. Its architecture flexibility is contrasted with MAE’s restriction to ViT.

B TRAINING TIME COMPARISON

  • MFM is relatively efficient compared with other MIM methods such as BEiT, except for MAE.Pre-training time per epoch is measured on the same 8-GPU machine and batch size using ViT-B/16; values are relative to MFM.Source: B TRAINING TIME COMPARISON · p. 15, Table 8 · p. 15
  • MFM is a one-view method without view switching, while MoCo v3 and DINO require four and 14 forward passes, respectively.Source: B TRAINING TIME COMPARISON · p. 15
  • Unlike MAE, which is only applicable to ViT because visible-patch inputs break regular 2D image structure, MFM applies to both ViT and CNN.The authors consider MFM’s slightly higher time than MAE acceptable given its flexibility and universality.Source: B TRAINING TIME COMPARISON · p. 15

C COMBINATION OF LOW-LEVEL IMAGE PROCESSING TASKS

Randomly sampling low- and high-pass filters lets MFM use the full frequency spectrum and benefits performance. Combining SR, deblurring, and denoising does not provide the same benefit and degrades performance.

C COMBINATION OF LOW-LEVEL IMAGE PROCESSING TASKS

  • Randomly sampling low- and high-pass filters to predict both high and low frequencies benefits MFM by using the full frequency spectrum.The passage states this leads to better performance.Source: C COMBINATION OF LOW-LEVEL IMAGE… · p. 15
  • Combining SR, deblurring, and denoising does not bring similar benefits to MFM and even degrades performance.The comparison uses ViT-B/16 on ImageNet-1K with top-1 fine-tuning accuracy after 300-epoch pre-training.Source: C COMBINATION OF LOW-LEVEL IMAGE… · p. 16, Table 9 · p. 16
  • Table 9 evaluates combinations of SR, deblurring, and denoising using ViT-B/16 on ImageNet-1K after 300 epochs of pre-training.Evaluation is by top-1 fine-tuning accuracy.Source: Table 9 · p. 16

D FURTHER DISCUSSION

The discussion highlights MFM's architecture flexibility, contrasts it with hybrid and concurrent MIM approaches, and notes study limitations and future directions. The appendix pseudocode describes its frequency masking and loss, while a spectrum visualization identifies artifacts from spatial patch masking.

D FURTHER DISCUSSION

  • MFM masks frequencies without mask tokens, making it applicable to both ViT and CNN architectures.The authors attribute CNN incompatibility with inserted mask tokens to information leakage in convolutional dense sliding windows.Source: D FURTHER DISCUSSION · p. 16
  • Hybrid MIM approaches combine masking with contrastive learning but add techniques such as multi-crop augmentation or intermediate-layer losses.The authors focus comparisons on pure MIM to isolate other design factors; they note those additional techniques may also be incorporated into MFM.Source: D FURTHER DISCUSSION · p. 16, Table 10 · p. 16
  • Unlike the concurrent frequency-domain work, MFM explores frequency corruption beyond MIM and remains architecture-agnostic rather than relying on spatial masking and mask tokens.The concurrent work uses a more complex frequency decoder and losses in both spatial and frequency domains; MFM also studies low-level image-processing corruptions for representation learning.Source: D FURTHER DISCUSSION · pp. 16–17
  • The study leaves scaling underexplored, evaluates transfer on ImageNet-1K classification and ADE20K segmentation, and notes that frequency redundancy may remain.The authors identify broader downstream evaluation and more advanced information suppression as future research directions.Source: D FURTHER DISCUSSION · p. 17
  • Future directions include frequency-domain SSL across modalities, combining MFM with contrastive learning or MIM, robustness analysis and calibration, and reconstruction or synthesis tasks.Source: D FURTHER DISCUSSION · p. 17

E PSEUDOCODE

  • The pseudocode transforms images with FFT, applies sampled low- or high-pass masks, uses inverse FFT to form corrupted inputs, and predicts from those inputs.The loss is computed only over masked frequencies; the frequency-distance calculation uses the exponent gamma.Source: E PSEUDOCODE · p. 17

F FREQUENCY SPECTRUM OF MIM

G IMPLEMENTATION DETAILS

Implementation settings use shared pre-training configurations across architectures and describe separate fine-tuning setups for ViT, ResNet-50, and ADE20K segmentation.

G IMPLEMENTATION DETAILS

H VISUALIZATION

The visualizations cover corrupted and recovered images and their frequency spectra across ImageNet-1K and COCO. They report that moderate degradation works best for each task and that MFM uses both high- and low-frequency information for prediction.

H VISUALIZATION

  • Each of SR, deblurring, denoising, and MFM achieves its best performance at a moderate degradation intensity.Figure 4 visualizes corrupted ImageNet-1K samples and their frequency spectra at different degradation levels.Source: Figure 4 · p. 20
  • MFM can use both high- and low-frequency information for prediction, unlike the illustrated SR, deblurring, and denoising recovery results.Figure 5 shows ImageNet-1K validation examples using SR scale ×8, deblurring sigma 5, denoising sigma 75, and MFM mask radius 16.Source: Figure 5 · p. 21
  • The recovery visualizations also include COCO validation images using models pre-trained on ImageNet-1K.Figure 6 shows recovered images and their frequency spectra for SR, deblurring, denoising, and MFM.Source: H VISUALIZATION · p. 19, Figure 6 · p. 22

Related papers

Loading 2206.07706v2…