Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling

Keyu Tian; Yi Jiang; Qishuai Diao; Chen Lin; Liwei Wang; Zehuan Yuan

Source-linked AI summary

Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling

Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, Zehuan Yuan

arXiv:2301.03580v2Jan 9, 2023cs.CVcs.AIcs.LG

TL;DR

BERT-style masked pre-training struggles on convnets because convolution cannot directly process irregular masked images and single-scale modeling mismatches convnets’ hierarchy. SparK uses sparse convolution for visible patches and hierarchical decoding, outperforming contrastive and transformer-based masked modeling across downstream tasks, with gains up to +3.5%.

Problem
BERT-style masked pre-training faces irregular-input and single-scale obstacles when extended from transformers to hierarchical convolutional networks.
Method
SparK encodes unmasked patches as sparse voxels with sparse convolution and reconstructs images using a hierarchical decoder over multi-scale features.
Results
SparK surpasses contrastive learning and transformer-based masked modeling by around +1.0% across three downstream tasks, with improvements up to +3.5% on COCO tasks.
Takeaways & Limitations
Generative masked pre-training can transfer effectively to convnets without backbone modifications, including classical ResNets and modern ConvNeXts.

Abstract

from arXiv · show

We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, random-masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet's hierarchical structure. For (i), we treat unmasked pixels as sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop a hierarchical decoder to reconstruct images from multi-scale encoded features. Our method called Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%). Improvements on object detection and instance segmentation are more substantial (up to +3.5%), verifying the strong transferability of features learned. We also find its favorable scaling behavior by observing more gains on larger models. All this evidence reveals a promising future of generative pre-training on convnets. Codes and models are released at https://github.com/keyu-tian/SparK.

1 INTRODUCTION

SparK adapts BERT-style masked pre-training to convolutional networks by addressing irregular masked inputs and convnets’ hierarchical representations. It uses sparse convolution and hierarchical decoding, improving downstream performance across convnet families and tasks.

Applying BERT-style masking directly to convnets is difficult because convolution expects regular grids and sliding windows, while zeroing masked pixels creates distribution shift and other artifacts.The problem reflects differences between variable-length language processing and multi-scale raw-pixel vision processing.
SparK treats unmasked patches as sparse voxels and encodes them with sparse convolution, while a hierarchical decoder reconstructs images from multi-scale features.This removes masked information without forcing convnets to process irregular dense inputs and uses their hierarchical structure.
SparK outperforms contrastive learning and transformer-based masked modeling by around +1.0% across classification, detection, and instance segmentation, with gains up to +3.5% on COCO tasks.The larger COCO improvements indicate strong transferability of the learned representations.
SparK is designed as a general method usable directly with ResNets and ConvNeXts without backbone modifications.The paper presents sparse convolution for masked image modeling and hierarchical pre-training as its central design insights.
Convnets remain important for challenging and structural vision tasks despite recent attention shifting toward vision transformers.Their scale and translation equivariance, locality, weight sharing, and hardware friendliness support this continuing role.

2 RELATED WORK

Related work spans hierarchical visual representations, contrastive learning, masked image modeling, and sparse convolution. These strands motivate SparK’s combination of multi-scale processing, masking, and sparse computation.

Hierarchical visual representations help systems cope with objects appearing at varying scales and have long been central to handcrafted and learned vision systems.Examples include scale-space descriptors and feature-pyramid networks.
Contrastive learning formulated self-supervision as instance classification and developed methods to address mode collapse, later dominating visual unsupervised learning.
Masked image modeling transfers masked language modeling to visual pre-training, with vision transformers exploiting variable-length inputs and masked patches.
Sparse convolution skips empty voxels in highly sparse and irregular point clouds, reducing the cost of conventional convolution over dense 3D grids.

3 APPROACH

SparK adapts masked image modeling to convolutional networks by encoding visible patches sparsely and decoding hierarchical features into a dense reconstruction. It trains on masked-patch reconstruction while preserving convnet backbones for downstream use.

3.1 Sparse Masking Strategy: SparK gathers unmasked patches into a sparse image and applies sparse convolution, avoiding information leakage while enabling masked inputs for convnets.Sparse convolution computes only at unmasked positions and can be applied directly without backbone modifications.
3.2 Hierarchical Encoding and Decoding: The encoder produces multi-scale sparse feature maps from a hierarchical convnet, such as ResNet stages with resolutions H/4 × W/4 through H/32 × W/32.These features provide the inputs for hierarchical decoding.
3.1 Sparse Masking Strategy: Sparse convolution preserves the mask pattern by skipping masked positions, whereas repeated dense convolution would erode masked regions and dilate visible ones.This addresses the mask pattern vanishing problem caused by ordinary convolution on zeroed masked images.
3.2 Hierarchical Encoding and Decoding: A UNet-style decoder densifies each scale with scale-specific mask embeddings, upsamples through successive blocks, and reconstructs a dense image.Projection layers accommodate differing encoder and decoder widths before decoding.
3.3 Optimization Target and Transferring to Downstream: Pre-training uses per-patch normalized pixels with L2 loss computed only on masked positions, then discards the decoder and transfers the encoder downstream.The sparse encoder can process dense fine-tuning inputs because dense images are a special case in which every position is active.

4 EMPIRICAL RESULTS

SparK is evaluated across ImageNet and COCO using ResNet and ConvNeXt backbones, with comparisons against transformer-based masked modeling and contrastive learning. It improves downstream performance, transferability, scaling, and reconstruction quality while sparse and hierarchical designs are validated as important.

Performance vs. self-supervised transformers: SparK outperforms transformer-based self-supervised learning by +0.8, +0.8, and +0.7 on classification, detection, and segmentation, respectively.Table 2 evaluates ImageNet classification and COCO detection and segmentation under matched task settings.
Performance vs. self-supervised convnets: Against convolutional contrastive methods, SparK improves performance across all downstream tasks by +0.5~1.2 points.The comparison uses ResNet-50 results and identifies SparK as the first generative pre-training method for hierarchical convnets.
Feature transferability: SparK produces larger transfer gains on COCO than ImageNet, reaching +3.5 for ConvNet and +2.7 for ResNet over supervised baselines.The authors associate this pattern with COCO's higher resolution and multiple disorganized objects, describing the learned features as highly transferable.
Scaling up SparK: SparK shows favorable scaling: larger models benefit more, and all models except ResNet-50 match the performance of non-pretrained larger models.The scaling evaluation varies model size and training resolution on ImageNet.
Ablation study: Removing either sparse masking or hierarchical decoding impairs fine-tuning, while longer pre-training improves accuracy and positional embeddings are practically useless.The ablation study evaluates ConvNeXt-Small variants on ImageNet.

5 CONCLUSION

SparK adapts masked modeling to convnets by combining sparse convolution for masked images with hierarchical decoding. It works across convnet families and improves downstream performance.

SparK treats unmasked patches as sparse voxels and uses sparse convolution to encode masked images, while a hierarchical decoder exploits multi-scale features.

A DETAILS: DECODER ARCHITECTURE

The decoder is encoder-agnostic and uses a lightweight UNet-style architecture to reconstruct dense images from multi-scale features.

SparK allows any convolutional encoder and uses the same LightDecoder(768, 32) across ResNet and ConvNeXt variants.The decoder is the only component requiring a fixed implementation in the presented setup.
DecoderConv uses transposed convolution for 2× upsampling and convolutional blocks with batch normalization and ReLU6.
The LightDecoder stacks upsampling DecoderConv blocks and projects the final feature map to three output channels.

B ADDITIONAL RESULTS: LINEAR EVALUATION

Linear evaluation tests whether frozen SparK representations support classification through a learned projection head. SparK shows decent performance relative to other non-contrastive methods.

Linear evaluation freezes the pretrained backbone and fine-tunes only a linear projection head to probe representation separability.
SparK shows decent performance compared with other non-contrastive methods in the small-model linear evaluation results.The reported comparisons are summarized in Table 6.

C DETAILS: IMAGENET FINE-TUNING

ImageNet fine-tuning uses established official implementations and adjusts their supervised-training configurations for fine-tuning.

ResNet and ConvNeXt fine-tuning uses open-source or official implementations from Wightman et al. and Liu et al., respectively.
The authors modify some original hyperparameters because the baseline configurations were designed for supervised training from scratch.The detailed recipes are provided separately for ResNets and ConvNeXts.

D DETAILS: COCO FINE-TUNING

COCO fine-tuning evaluates ResNets and ConvNeXts using standard implementations and omits advanced testing or augmentation techniques for fairness. Mask R-CNN with FPN is used across the configurations, with ConvNeXts following a 3× schedule.

D DETAILS: COCO FINE-TUNING: COCO experiments use official MoCoV2 and ConvNeXt implementations for evaluating ResNets and ConvNeXts.The implementations are based on Detectron2 and MMDetection, respectively.
D DETAILS: COCO FINE-TUNING: Advanced techniques, including multi-scale testing, large-scale jittering augmentation, and soft-NMS, are omitted from all COCO experiments for fairness.
D DETAILS: COCO FINE-TUNING: ResNet fine-tuning uses Mask R-CNN with FPN under the standard MoCoV2 configuration.A 2× schedule corresponds to 24 training epochs with a 0.1-epoch warm-up.
D DETAILS: COCO FINE-TUNING: ConvNeXt fine-tuning uses 3× Mask R-CNN with FPN following the standard ConvNeXt implementation and self-supervised-learning convention.