PaperlayerView exact revision on arXiv

Source-linked AI summary

Momentum Contrast for Unsupervised Visual Representation Learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick

arXiv:1911.05722v3cs.CV

TL;DR

Unsupervised visual representation learning generally lags supervised pre-training, partly because visual dictionaries must handle continuous, high-dimensional signals. MoCo builds a large, consistent dynamic dictionary and surpasses supervised pre-training in seven detection and segmentation tasks.

  • Problem

    Unsupervised visual representation learning generally lags supervised pre-training, while effective dictionary construction remains challenging for continuous, high-dimensional visual signals.

  • Method

    MoCo uses a queue and momentum-updated key encoder to build a large, consistent dynamic dictionary for contrastive unsupervised learning.

  • Results

    MoCo unsupervised pre-training surpassed its supervised ImageNet counterpart in 7 downstream detection or segmentation tasks.

  • Takeaways & Limitations

    MoCo largely closes the gap between unsupervised and supervised representation learning across many computer vision tasks and can serve as an alternative in several applications.

  • Takeaways & Limitations

    MoCo’s improvement from IN-1M to IG-1B was consistently noticeable but relatively small, suggesting that larger-scale data may not be fully exploited.

Abstract

from arXiv · show

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

1. Introduction

MoCo builds large, consistent dynamic dictionaries for unsupervised visual representation learning using a queue and a momentum-updated key encoder. It achieves competitive ImageNet linear-classification results and surpasses supervised pre-training on seven detection or segmentation tasks.

  • Motivation: Recent contrastive approaches [61] [46] [36] [66] [35] [56] [2] view unsupervised visual learning as dictionary look-up over encoded data samples, contrasting with language methods such as GPT and BERT [12].The introduction attributes the continuing dominance of supervised pre-training in vision partly to differences between continuous visual and discrete language signal spaces.
  • Motivation: The approach addresses the hypothesis that effective contrastive dictionaries should be both large enough to sample visual space and consistent as training evolves.Existing contrastive methods can be limited in one of these two aspects.
  • Method: MoCo builds a large, consistent dictionary by queueing encoded mini-batches and using a slowly progressing momentum-updated encoder for keys.The queue decouples dictionary size from mini-batch size, while the slowly progressing encoder keeps comparisons to queries consistent.
  • Method: MoCo uses instance discrimination [61] [63] [2], matching views of the same image, and achieves competitive ImageNet linear-classification results.The method is presented as a mechanism for dynamic dictionaries that can support various pretext tasks.
  • Results: MoCo unsupervised pre-training surpasses its ImageNet supervised counterpart on 7 downstream detection or segmentation tasks, sometimes by nontrivial margins.Experiments use MoCo pre-trained on ImageNet or a one-billion Instagram image set.

2. Related Work

Unsupervised/self-supervised learning methods combine pretext tasks and loss functions, and MoCo focuses on the loss-function aspect. Related losses include fixed-target reconstruction or classification, contrastive objectives, and adversarial objectives, while contrastive learning also intersects with diverse pretext tasks.

  • MoCo focuses on the loss-function aspect of unsupervised/self-supervised learning, which commonly combines pretext tasks with loss functions.The paper treats loss functions and pretext tasks as aspects that can often be investigated independently.
  • Loss functions: Fixed-target losses reconstruct input pixels with L1 or L2 losses or classify inputs into predefined categories using cross-entropy or margin-based losses.Examples include auto-encoders, eight-position prediction [13], and color-bin prediction [64].
  • Loss functions: Contrastive losses compare sample similarities in representation space, allowing on-the-fly targets defined by network-computed representations.Contrastive learning is central to several recent unsupervised-learning works [61] [46] [36] [66] [35] [56] [2].
  • Loss functions: Adversarial losses compare probability distributions and support unsupervised data generation, while adversarial representation learning is explored in [15] [16].The paper also notes relations between generative adversarial networks and noise-contrastive estimation (NCE).
  • Pretext tasks: Pretext tasks span corruption recovery, exemplar transformations, patch ordering, video tracking or object segmentation, and feature clustering.Contrastive formulations can underlie these tasks: instance discrimination [61] relates to exemplar-based learning and NCE, CPC [46] to context auto-encoding [48], and CMC [56] to colorization [64].

3. Method

MoCo frames contrastive learning as dictionary lookup and builds a large, consistent dynamic dictionary using a queue and momentum-updated key encoder. It trains on augmented views of the same image as positive pairs, with queued keys supplying negatives.

  • Contrastive objective: InfoNCE trains an encoder to make query q similar to its positive key k+ and dissimilar to K negative keys using a temperature-scaled softmax loss.The loss is equivalent to log loss for a (K+1)-way classifier identifying k+; τ is the temperature hyper-parameter [61].
  • Design rationale: MoCo combines a large dictionary with a slowly evolving key encoder to seek rich negative coverage and consistency across keys.The method is presented as a general mechanism for contrastive losses, alongside end-to-end and memory-bank alternatives that differ in dictionary size and consistency.
  • Dictionary as a queue: MoCo maintains a queue of encoded keys, decoupling dictionary size from mini-batch size while progressively replacing the oldest, least-consistent mini-batch.The queue reuses keys from preceding mini-batches, supports a flexibly chosen dictionary size, and keeps maintenance computationally manageable.
  • Momentum update: The key encoder is updated by momentum rather than back-propagation, making queued representations more consistent as training evolves.Only the query encoder receives gradient updates; m = 0.999 works better than m = 0.9 in experiments.
  • Pretext task: MoCo uses two random augmented views of each image as a positive query-key pair, while keys from the queue serve as negative samples.This instance-discrimination pretext task follows [61], with related work including [63] [2].

4. Experiments

Across linear evaluation and downstream transfer, MoCo learns competitive representations, often surpassing supervised pre-training, while benefiting from large, consistent dictionaries and large-scale, uncurated data. Its gains extend across detection, segmentation, keypoint, and dense pose tasks, with IG-1B consistently improving over IN-1M.

  • Linear classification: MoCo achieves 60.6% ImageNet accuracy with standard R50 and 68.6% with R50w4×, outperforming similarly sized competitors without architecture-specific designs.The method uses a standard ResNet-50 rather than customized pretext-task architectures, facilitating transfer across visual tasks.
  • PASCAL VOC detection: On downstream detection, MoCo matches or surpasses supervised pre-training, with R50-C4 gains up to +0.9 AP50, +3.7 AP, and +4.9 AP75.For trainval07+12, IG-1B MoCo surpasses supervised pre-training with R50-dilated-C5, while IN-1M is comparable.
  • COCO detection: On COCO, MoCo exceeds ImageNet supervised pre-training in every metric for both FPN and C4 backbones under the 2× schedule.The 1× schedule heavily under-trains all models, with approximately 2-point gaps relative to 2× cases.
  • COCO keypoints and dense pose: MoCo outperforms supervised pre-training across COCO keypoint and dense pose estimation, including a 3.7-point APdp gain for dense pose.Supervised pre-training has no clear advantage over random initialization for keypoint detection, whereas MoCo improves all metrics.

5. Discussion and Conclusion

MoCo shows positive unsupervised-learning results across varied computer-vision tasks and datasets, while its modest improvement from IN-1M to IG-1B suggests larger-scale data is not fully exploited. The authors identify advanced pretext tasks, including masked auto-encoding, as a direction for improvement.

  • 5. Discussion and Conclusion: MoCo shows positive unsupervised-learning results across diverse computer-vision tasks and datasets.These include detection, segmentation, keypoint detection, dense pose, and semantic segmentation across VOC, COCO, LVIS, Cityscapes, and related datasets.
  • 5. Discussion and Conclusion: MoCo’s improvement from IN-1M to IG-1B is consistently noticeable but relatively small, suggesting the larger-scale data is not fully exploited.The authors hope an advanced pretext task will improve the use of larger-scale data.
  • 5. Discussion and Conclusion: Beyond instance discrimination [61], MoCo could be adapted to advanced pretext tasks such as masked auto-encoding.The paper presents this adaptation as an open question and possible direction for future improvement.

A. Appendix · A.1. Implementation: Object detection backbones

The object-detection backbones follow Detectron2-style designs, using distinct R50-dilated-C5 and R50-C4 configurations with corresponding box-prediction heads.

  • A.1. Implementation: Object detection backbones: The R50-dilated-C5 and R50-C4 backbones are similar to Detectron2 implementations.
  • A.1. Implementation: Object detection backbones: R50-dilated-C5 includes ResNet’s conv5 stage with dilation 2 and stride 1.
  • A.1. Implementation: Object detection backbones: R50-dilated-C5 applies a 3×3 convolution with BN to reduce the feature dimension to 512.
  • A.1. Implementation: Object detection backbones: The R50-dilated-C5 box-prediction head contains two hidden fully connected layers.
  • A.1. Implementation: Object detection backbones: R50-C4 ends with the ResNet conv4 stage.
  • A.1. Implementation: Object detection backbones: The R50-C4 box-prediction head uses the conv5 stage, including global pooling, followed by a BN layer.

A.2. Implementation: COCO keypoint detection

The COCO keypoint detection implementation uses Mask R-CNN with an R50-FPN backbone, fine-tuned on COCO train2017 and evaluated on val2017 with a 2× schedule.

  • A.2. Implementation: COCO keypoint detection: Mask R-CNN keypoint detection with R50-FPN is fine-tuned on COCO train2017 and evaluated on val2017 using a 2× schedule.The implementation follows.

A.3. Implementation: COCO dense pose estimation

The COCO dense pose estimation experiments use DensePose R-CNN with an R50-FPN, fine-tuned on COCO train2017 and evaluated on val2017 under the “s1×” schedule.

  • A.3. Implementation: COCO dense pose estimation: Dense pose estimation uses DensePose R-CNN with R50-FPN, implemented in, and is fine-tuned on COCO train2017 before evaluation on val2017.The training schedule is “s1×”.

A.4. Implementation: LVIS instance segmentation

For LVIS instance segmentation, the study uses Mask R-CNN with R50-FPN following the LVIS baseline [27] and evaluates supervised and MoCo pre-training under ablated batch-normalization settings. MoCo reaches 24.1 APmk with IN-1M and 24.9 APmk with IG-1B, outperforming the supervised counterpart under the same tunable-BN setting and achieving 24.9 versus 24.4 APmk under the best individual settings.

  • A.4. Implementation: LVIS instance segmentation: The experiments use Mask R-CNN with R50-FPN, fine-tuned on LVIS train v0.5 and evaluated on val v0.5 following the baseline in [27].The baseline follows arXiv v3 Appendix B of [27].
  • A.4. Implementation: LVIS instance segmentation: The LVIS ablations average results over 5 trials because LVIS is a new dataset with model designs still being explored.The table includes the relevant ablations.
  • A.4. Implementation: LVIS instance segmentation: 24.9 versus 24.4 APmk: MoCo with IG-1B outperforms supervised pre-training under the best individual settings.MoCo also reaches 24.1 APmk with IN-1M; both MoCo variants outperform the supervised counterpart under the same tunable-BN setting.
  • A.4. Implementation: LVIS instance segmentation: The supervised baseline achieves 24.4 APmk with end-to-end tuning and frozen batch normalization, while tuning BN causes worse results and overfitting.This differs from COCO/VOC, where tuning BN gives better or comparable accuracy.

A.5. Implementation: Semantic segmentation

The semantic-segmentation implementation uses an FCN-16s architecture built on an R50 convolutional backbone with dilated convolutions and per-pixel classification. Training applies scale and crop augmentation, horizontal flipping, and dataset-specific optimization settings for VOC and Cityscapes.

  • A.5. Implementation: Semantic segmentation: The model uses an FCN-16s [43] structure with an R50 convolutional backbone, dilated conv5 blocks, two extra 3×3 convolutions, and 1×1 per-pixel classification.The total stride is 16; the extra convolutions use 256 channels, BN, ReLU, and dilation 6 following the large-field-of-view design in [6].
  • A.5. Implementation: Semantic segmentation: Training uses random scaling in [0.5, 2.0], cropping, horizontal flipping, and inference at the original image size.Crop sizes are 513 on VOC and 769 on Cityscapes [6], with mini-batch size 16 and weight decay 0.0001.
  • A.5. Implementation: Semantic segmentation: VOC training uses learning rate 0.003 for 30k iterations on the 10,582-image train aug2012 set, with evaluation on val2012; Cityscapes uses learning rate 0.01.The learning rate is multiplied by 0.1 at the 70th and 90th percentiles of training.

A.6. iNaturalist fine-grained classification … A.9. Ablation on Shuffling BN

Across downstream classification and detection, MoCo pre-training improves over random initialization and remains competitive with supervised pre-training, with larger advantages emerging under longer COCO fine-tuning. Shuffling BN is necessary to prevent pretext-task overfitting caused by batch-statistics leakage.

  • A.6. iNaturalist fine-grained classification: 61.8% from random initialization rises to 65.8% with MoCoIG-1B on iNaturalist 2018, approximately 4% better and closely comparable to the 66.1% supervised ImageNet baseline.The experiment fine-tunes ResNet models end-to-end on approximately 437k images across 8,142 classes.
  • A.7. Fine-tuning in ImageNet: 0.8%: MoCo pre-training on the separate unlabeled IG-1B dataset improves end-to-end ImageNet accuracy from 76.5% with random initialization to 77.3%.MoCo pre-training on IN-1M is not a separate-dataset scenario; its reported fine-tuned accuracy is 77.0%.
  • A.8. COCO longer fine-tuning: 42.8 APbb with MoCo exceeds 41.9 APbb with ImageNet-supervised pre-training and 41.4 APbb from scratch under longer COCO fine-tuning, with a +0.9 APbb gap at 6× versus +0.5 at 2×.The results suggest MoCo features can have larger advantages than ImageNet-supervised features when fine-tuning longer.
  • A.8. COCO longer fine-tuning: The COCO comparison extends prior 1× and 2× evaluations with a 6× schedule of approximately 72 epochs to test whether longer fine-tuning changes the relative advantages of pre-training methods.The original schedules were inherited from Mask R-CNN and may be suboptimal given later advances.
  • A.9. Ablation on Shuffling BN: Without shuffling BN, sub-batch statistics provide a signature revealing which sub-batch contains the positive key, enabling cheating on the pretext task.Shuffling BN removes this signature and prevents the shortcut.
  • A.9. Ablation on Shuffling BN: Removing shuffling BN causes pretext-task training accuracy to exceed 99.9% while kNN validation accuracy soon drops, indicating overfitting in both MoCo and end-to-end variants.The memory-bank variant avoids this issue because q and k implicitly use different statistics; the plotted monitor is kNN-based rather than a linear classifier.
Loading 1911.05722v3…