Source-linked AI summary
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, Baining Guo
TL;DR
Existing masked image modeling targets can disagree with human perception because pixel-based objectives inadequately represent perceptual similarity. PeCo introduces perceptual visual tokens by adding feature-wise perceptual similarity to codebook learning, achieving stronger transfer performance across downstream tasks, including 84.5% ImageNet-1K Top-1 accuracy with ViT-B and +1.3% over BEiT.
Problem
Current masked image modeling prediction targets can disagree with human judgments of image similarity because per-pixel losses inadequately capture structured perceptual differences.
Method
PeCo learns discrete visual tokens by enforcing feature-wise perceptual similarity between original and reconstructed images during VQ-VAE codebook training.
Results
PeCo transfers better than BEiT across image classification, object detection, and semantic segmentation, reaching 84.5% Top-1 accuracy on ImageNet-1K with ViT-B and +1.3% over BEiT.
Takeaways & Limitations
Perceptually aligned visual tokens provide a stronger prediction target for BERT pre-training of vision transformers and support scalable variants such as PeCoMAE.
Abstract
from arXiv · showhide
This paper explores a better prediction target for BERT pre-training of vision transformers. We observe that current prediction targets disagree with human perception judgment.This contradiction motivates us to learn a perceptual prediction target. We argue that perceptually similar images should stay close to each other in the prediction target space. We surprisingly find one simple yet effective idea: enforcing perceptual similarity during the dVAE training. Moreover, we adopt a self-supervised transformer model for deep feature extraction and show that it works well for calculating perceptual similarity.We demonstrate that such learned visual tokens indeed exhibit better semantic meanings, and help pre-training achieve superior transfer performance in various downstream tasks. For example, we achieve $\textbf{84.5\%}$ Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming the competitive method BEiT by $\textbf{+1.3\%}$ under the same pre-training epochs. Our approach also gets significant improvement on object detection and segmentation on COCO and semantic segmentation on ADE20K. Equipped with a larger backbone ViT-H, we achieve the state-of-the-art ImageNet accuracy (\textbf{88.3\%}) among methods using only ImageNet-1K data.
Introduction
Current masked image modeling targets can disagree with human judgments of image similarity because pixel-based objectives inadequately capture perceptual differences. PeCo addresses this by learning perceptual visual tokens and improves transfer across several vision tasks.
- Current Prediction Targets: Masked image modeling predicts randomly masked image patches from visible patches, using targets that range from pixel-level predictions to discrete tokens.Different methods associate patches with positional embeddings and vary in their prediction targets.
- PeCo: PeCo agrees with human similarity judgments in Figure 1, whereas L2 and DALL-E prediction targets disagree.The figure compares distances from two views to a reference; smaller distances indicate greater similarity.
- Current Prediction Targets: Pixel-based objectives can misjudge perceptual similarity: blurring may cause large perceptual change with small pixel error, while shifting may do the opposite.Discrete tokens learned from reconstruction-loss VQ-VAE inherit this per-pixel limitation.
- PeCo: PeCo learns visual tokens by enforcing perceptual similarity during VQ-VAE training, using multi-scale deep features from a self-supervised Transformer.The feature-wise loss measures differences between original and reconstructed images across layers at different depths.
- Results: 84.5% Top-1 accuracy on ImageNet-1K with ViT-B surpasses BEiT by +1.3% under the same 800 pre-training epochs.The method also improves transfer on COCO object detection and segmentation and ADE20K semantic segmentation, while ViT-H reaches 88.3% ImageNet accuracy.
Related Works
Self-supervised learning in vision includes contrastive and generative approaches. Generative methods commonly reconstruct corrupted inputs, while computer vision methods have often used regression or, more recently, discrete visual-token classification.
- Self-supervised Learning: Self-supervised learning is broadly divided into contrastive and generative approaches.Contrastive methods compare similar and dissimilar samples through a contrastive loss.
- Generative Methods: Generative self-supervised methods train encoders and decoders to reconstruct corrupted or masked inputs.Autoregressive and denoising autoencoder objectives are identified as successful generative frameworks.
- Visual Prediction Targets: Computer vision generative methods often rely on regression because visual vocabulary is limited, while BEiT uses classification with discrete visual tokens.The passage contrasts regression-based methods such as iGPT with BEiT’s discrete-token prediction.
Method
PeCo learns discrete visual tokens by augmenting VQ-VAE reconstruction with perceptual feature similarity, then uses those tokens as semantic targets for masked image modeling. The pipeline trains a perceptual codebook, applies it in BERT-like pre-training, and transfers the resulting model to downstream vision tasks.
- Learning Discrete Codebook for Visual Content: VQ-VAE converts continuous images into discrete visual codewords using an encoder, nearest-neighbor quantizer, and decoder.The quantizer maps latent vectors to codebook entries, while the decoder reconstructs the input from the quantized codewords.
- Learning Perceptual Codebook for Visual Content: Pixel-wise reconstruction can misrepresent perceptual similarity because small shifts or rotations may incur large ℓ1/ℓ2 errors without perceptual change.The vanilla codebook is learned from element-wise pixel loss, which accounts only for individual pixel correctness.
- Learning Perceptual Codebook for Visual Content: PeCo adds a feature-wise perceptual loss to VQ-VAE, enforcing similarity between original and reconstructed images beyond pixel loss.The loss uses deep features from multiple layers of a self-supervised Vision Transformer to capture multi-scale perceptual information.
- Learning Perceptual Codebook for Visual Content: The combined objective preserves pixel reconstruction while weighting perceptual similarity through λ, and the trained encoder and quantizer become the downstream tokenizer.The perceptual codebook is trained before subsequent pre-training, with λ controlling the contribution of the perceptual loss.
- BERT Objective over Perceptual Codebook: BERT-like masked image modeling predicts discrete perceptual visual words for masked patches from visible image patches.A Transformer produces hidden states at masked positions, and a K-way classifier estimates the corresponding codeword probabilities.
- Downstream Evaluation: The pre-trained model is evaluated on ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation.These downstream tasks assess transfer from the perceptual-token masked image modeling pre-training.
Experiments
PeCo improves transfer performance by using perceptual visual tokens, with gains across ImageNet classification, ADE20K segmentation, and tokenizer efficiency comparisons. Ablations examine codeword semantics, perceptual-similarity architectures, implicit versus explicit learning, and adversarial loss.
- Image Classification: 84.5% Top-1 accuracy on ImageNet-1K is achieved by ViT-B PeCo after 800 pre-training epochs, exceeding BEiT by 1.3% and MAE by 0.9%.The comparison uses prior self-supervised pre-training models.
- Image Classification: 88.3% Top-1 accuracy is achieved by PeCo-H448 on ImageNet-1K without external data, outperforming MAE by 0.5%.The paper identifies this as a new state-of-the-art result using only ImageNet-1K data.
- Semantic Segmentation: 48.5 mIoU on ADE20K exceeds supervised methods by 1.1 mIoU, MoCo by 1.2 mIoU, BEiT by 1.4 mIoU, and MAE by 0.9 mIoU.The evaluation uses UperNet with ViT-B, 512 × 512 input, and 160K training iterations.
- Tokenizer Efficiency: PeColite halves the tokenizer channel number and reduces the tokenizer’s extra time cost by about 2× while retaining competitive performance.The tokenizer comparison is against BEiT.
- Perceptual Similarity: Self-supervised Transformer features and supervised VGG features achieve comparable performance for calculating perceptual similarity.The paper compares architectures used for the perceptual metric.
- Codeword Semantics: Perceptual codewords are evaluated through linear probing and reconstructed-image classification to test whether they contain more semantic information.The ablation uses codewords as image features and examines reconstructed ImageNet validation images.
- Loss Ablation: Adding adversarial loss with weight 0.4 does not improve transfer performance of pre-training.The adversarial-loss variant is evaluated as an alternative loss-function design.
Conclusion
The paper argues that prediction targets for masked image modeling should agree with human perception and presents perceptually discrete tokens that improve downstream performance.
- PeCo uses perceptually discrete tokens as prediction targets for masked image modeling.The strategy is presented as simple and effective, with extensive downstream comparisons.
More Experiments
Additional experiments show that PeCo improves efficiency and performance across accelerated pre-training, video recognition, perceptual-loss weighting, robustness, and tokenizer architectures.
- Accelerated BERT pre-training: PeCoMAE combines PeCo’s perceptual codebook with an asymmetric encoder-decoder framework to reduce computation while improving performance.The accelerated variant is evaluated on image classification, semantic segmentation, and object detection or instance segmentation.
- Video recognition: PeCo outperforms supervised DEiT and BEiT on video recognition with a large margin.The extension uses Kinetics-400 and Something-Something-v2.
- Loss weight of perceptual similarity: 84.1% ImageNet-1K accuracy with perceptual loss exceeds 82.9% without it, while larger loss weights reduce performance.The authors suggest that excessive perceptual weighting may sacrifice local details needed alongside semantic information.
- Adversarial robustness: PeCo performs best in the reported adversarial-robustness comparison, whereas MAE is the only method worse than the vanilla DEiT baseline.The analysis uses BIM and MIM attacks with threshold 2/255 and 20 iterations.
- VQ-VAE architectures: CNN-based VQ-VAE encoders and decoders achieve better results than vision Transformer architectures.The comparison evaluates multiple encoder and decoder variants on ImageNet-1K.
Experiment Details
The experiments use convolutional VQ-VAE components, multi-layer self-supervised Transformer features, and ViT-B/16 pre-training for 300 or 800 epochs.
- Perceptual codebook learning: The default perceptual codebook uses a convolutional VQ-VAE trained on ImageNet-1K with multi-scale features from a self-supervised MoCo v3 ViT-B model.Features are taken from the 3rd, 6th, 9th, and 12th layers.
- BERT pre-training: BERT pre-training uses ViT-B/16 with batch size 2048 for 300 or 800 epochs.AdamW is used as the optimizer.
More visual results
Visual reconstructions show that PeCo can predict reasonable objects from limited visible patches under increasingly extensive masking.
- PeCo reconstructs reasonable objects even when many image patches are masked.Figure 5 compares masking 45, 75, and 120 patches across original, masked, and reconstructed images.