Improved Baselines with Momentum Contrastive Learning

Xinlei Chen; Haoqi Fan; Ross Girshick; Kaiming He

Source-linked AI summary

Improved Baselines with Momentum Contrastive Learning

Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He

arXiv:2003.04297v1Mar 9, 2020cs.CV

TL;DR

It remained unclear whether SimCLR’s design improvements would strengthen MoCo while preserving its efficient use of negative samples. This paper integrates an MLP projection head and stronger data augmentation into MoCo, producing MoCo v2 baselines that outperform SimCLR without large training batches.

Problem
The effectiveness and feasibility of applying SimCLR’s design improvements within the MoCo framework required direct evaluation.
Source: §1 · p. 1
Method
The paper evaluates MoCo with an MLP projection head and stronger data augmentation for unsupervised representation learning and transfer.
Source: §1 · p. 1, §3 · p. 2
Results
67.5% ImageNet accuracy after 200 epochs and batch size 256 is 5.6% higher than SimCLR under the same settings and exceeds SimCLR’s 66.6% large-batch result.
Source: §3 · p. 2
Takeaways & Limitations
MoCo v2 provides stronger unsupervised-learning baselines that do not require large training batches and can run on a typical 8-GPU machine.
Source: §1 · p. 1, §3 · p. 2

Abstract

from arXiv · show

Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.

1. Introduction

The note builds stronger, more feasible unsupervised-learning baselines within MoCo by adopting SimCLR’s MLP projection head and stronger data augmentation. These modifications improve image-classification and object-detection transfer results while MoCo handles many negative samples without large training batches.

Background

Contrastive learning is emerging as a central concept in unsupervised image representation learning.Recent studies report promising progress, including MoCo and SimCLR.Source: §1 · p. 1
MoCo shows that unsupervised pre-training can surpass ImageNet-supervised pre-training across multiple detection and segmentation tasks.Source: §1 · p. 1

Contribution

An MLP projection head and stronger data augmentation are orthogonal to MoCo and SimCLR and improve image classification and object detection transfer results when used with MoCo.Source: §1 · p. 1
MoCo processes a large set of negative samples without requiring large training batches.This property is illustrated in Fig. 1.Source: §1 · p. 1

2. Background

Contrastive learning learns representations from similar and dissimilar pairs, with instance discrimination defining positives as augmented views of the same image and negatives otherwise. SimCLR and MoCo differ in how negative keys are maintained, motivating study of SimCLR’s MLP-head and augmentation improvements within MoCo.

Contrastive learning

Contrastive learning learns similar and dissimilar representations from data organized into similar and dissimilar pairs.The framework can be formulated as a dictionary look-up problem.Source: §2 · p. 1
In instance discrimination, augmented versions of the same image form a positive pair, while pairs from different images are negative.This pretext task is used by both MoCo and SimCLR.Source: §2 · p. 1
SimCLR draws negative keys from the same batch and requires a large batch, whereas MoCo maintains negatives in a queue.The two mechanisms differ in how keys are maintained during contrastive-loss optimization.Source: §2 · p. 1

Improved designs

SimCLR improves instance discrimination with a 4k or 8k batch, an MLP projection head, and stronger data augmentation.The larger batch provides more negative samples, while the other two changes modify the representation-learning design.Source: §2 · p. 2
MoCo already provides many negative samples, making the MLP head and stronger augmentation orthogonal to its contrastive-learning mechanism.The paper therefore studies these two SimCLR improvements within the MoCo framework.Source: §2 · p. 2

3. Experiments

Experiments show that modest MoCo changes—an MLP projection head and stronger augmentation—substantially improve ImageNet linear accuracy, while MoCo v2 outperforms SimCLR without large batches. The implementation also reduces computational demands and supports more accessible state-of-the-art research.

Settings

Experiments evaluate frozen-feature ImageNet linear classification and end-to-end VOC object-detection transfer using Faster R-CNN.ImageNet uses 1-crop 224×224 top-1 validation accuracy; detection uses VOC 07+12 trainval and VOC 07 test sets.Source: §3 · p. 2

MLP head

66.2% ImageNet accuracy results from replacing MoCo’s fc head with a 2-layer, 2048-dimensional ReLU MLP and tuning τ to 0.2.With default τ = 0.07, the MLP improves accuracy from 60.6% to 62.9%.Source: §3 · p. 2

Augmentation

2.8% extra ImageNet accuracy comes from adding blur augmentation alone, reaching 63.4%, while detection transfer exceeds the MLP-only variant despite lower linear accuracy.The comparison is 63.4% versus 66.2%, indicating that linear classification accuracy is not monotonically related to detection transfer performance.Source: §3 · p. 2

Comparison with SimCLR

67.5% ImageNet accuracy is achieved by MoCo v2 after 200 epochs with batch size 256, surpassing SimCLR by 5.6% and its 66.6% large-batch result.With 800-epoch pre-training, MoCo v2 reaches 71.1%.Source: §3 · p. 2

Computational cost

The 4k batch size is intractable on an 8-GPU machine, whereas MoCo is cheaper than end-to-end training at batch size 256 because it back-propagates only through the q encoder.These results suggest large batches are unnecessary for good accuracy and make state-of-the-art research more accessible with only a few code changes.Source: §3 · p. 2