With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

1. Introduction

• Same-image augmentations cannot provide positives across viewpoints, deformations, or other instances in a semantic class, leaving generalization heavily dependent on the augmentation pipeline.Source: §1 · pp. 1–2
• Nearest-neighbor positives extend instance discrimination without clustering, which can cause early over-generalization when entire clusters or prototypes are treated as positives.The introduction notes that other approaches to cross-instance positives require class labels or multiple modalities.Source: §1 · p. 2
• NNCLR uses nearest neighbors in learned representation space as positives alongside different views of the same sample, retrieving cross-sample positives from a replenished support set.Unlike memory banks and queues, the support set is used for nearest-neighbor search rather than storing negatives.Source: Figure 1 · p. 1, §1 · p. 2
• NNCLR outperforms state-of-the-art self-supervised and supervised ImageNet-pretrained features on 8 out of 12 transfer learning tasks and achieves state-of-the-art ImageNet linear-evaluation and limited-label semi-supervised results.The introduction also reports a ∼3.8% performance increase for contrastive learning methods such as SimCLR.Source: §1 · p. 2
• 73.3% ImageNet accuracy is achieved using nearest neighbors as positives with only random-crop augmentations, reducing reliance on augmentation strategies.Source: §1 · p. 2

Self-supervised Learning

• Earlier self-supervised methods trained encoders on pretext tasks, including image rotation [27], patch-position prediction [17], and video-patch tracking [49] [58].Such representations are intended to support downstream tasks that require annotations.Source: §2 · p. 2
• NNCLR extends instance discrimination by using nearest neighbors from different images as positives, rather than only augmented views of the same image.Prototype- and cluster-based methods [1] [3] [6] [7] [8] [26] [36] [59] [62] [63] [64] [66] also associate multiple samples, but NNCLR retrieves positives from a support set instead.Source: §2 · p. 2

Queues and Memory Banks

• Unlike MoCo [33], which uses queue elements as negatives, NNCLR finds positives from its queue; unlike, it does not maintain running-average embeddings.Its support-set size is fixed independently of dataset size and the set is refreshed with new embeddings, without aggregation or clustering [8] [66].Source: §2 · pp. 2–3

Nearest Neighbors in Computer Vision

• Nearest-neighbor methods have supported retrieval and unsupervised learning; NNCLR instead targets transferable features across many object classes using RGB-only retrieval and prior embeddings.This contrasts with [30], which retrieves across video modalities, and with alignment work using neighbors from images of the same object [54] or action.Source: §2 · p. 3
• Concurrent work [2] adds a nearest-neighbor term to BYOL [28], whereas NNCLR uses the nearest neighbor in all loss terms.Source: §2 · p. 3

3.1. Contrastive instance discrimination

• SimCLR forms positives from two augmented views of the same image and negatives from other mini-batch embeddings, pulling positives together and separating negatives.Because it relies on predefined transformations of a single sample, SimCLR cannot link samples that may share a semantic class, potentially limiting invariance to intra-class variation.Source: §3.1 · pp. 3–4

3.2. Nearest-Neighbor CLR (NNCLR)

• NNCLR uses an embedding’s nearest neighbor in a support set as its positive, while retaining mini-batch negatives in a variant of the InfoNCE loss.The support set is intended to represent the full data distribution and provide more diverse positive pairs than single-instance positives.Source: §3.2 · p. 4
• NNCLR maintains its support set as a FIFO queue, replacing the oldest batch of embeddings with current-step embeddings from one view.The queue is sized to approximate the full dataset distribution in embedding space; using both views to update it made no significant downstream-performance difference.Source: §3.2 · p. 4
• A prediction MLP applied to the positive embedding adds a small performance boost, while the symmetric-loss addition does not affect performance empirically.The prediction head is inspired by BYOL [28].Source: §3.2 · p. 4

4.1. Implementation details

• The experiments use an unlabeled ImageNet-2012 training set, ResNet-50 embeddings, and a 256-dimensional projection embedding by default.Training lasts 1,000 epochs with 10 warm-up epochs, LARS, BYOL augmentations, temperature 0.1, and a 98,304-entry queue; the encoder produces 2,048-dimensional embeddings.Source: §4.1 · pp. 4–5

4.2. ImageNet evaluations

• NNCLR outperforms all compared methods on ImageNet 1% semi-supervised learning, but SwAV multi-crop outperforms it on ImageNet 10%.On the 10% subset, NNCLR outperforms SimCLR and other methods.Source: §4.2 · p. 5, Table 2 · p. 5
• NNCLR outperforms other two-view ResNet-50 methods on ImageNet linear evaluation, improving Top-1 by more than 3.6% over MoCo v2 and SimCLR v2, more than 2% over InfoMin Aug., and more than 1% over BYOL.It also improves Top-1 by 3.6% over SwAV in the two-view setting and by 0.3% in an 8-view multi-crop comparison.Source: Table 1 · p. 5, §4.2 · p. 5

4.3. Transfer learning evaluations

• NNCLR outperforms supervised ImageNet features on 11 of 12 transfer datasets and BYOL and SimCLR on 8 of 12.The benchmark reports Top-1 accuracy except for Aircraft, Caltech-101, Pets, and Flowers (mean per-class accuracy) and VOC2007 (11-point MAP).Source: §4.3 · p. 6, Table 3 · p. 6

4.4. Ablations

• With random crops alone, NNCLR reaches 68.2% ImageNet linear-evaluation Top-1 after 300 epochs, a 4.7% drop, versus drops of 27.6% for SimCLR and 13.1% for BYOL.At 1,000 epochs, NNCLR scores 73.3% with crops alone versus 74.9% with all augmentations, narrowing the gap to 1.6%; the method is therefore less reliant on color jitter and blur.Source: Table 5 · p. 6, §4.4 · p. 6
• Larger support sets generally improve performance, while the top-1 neighbor performs best and increasing top-k beyond 1 degrades results slightly.Performance peaks at batch size 4,096 and is robust across embedding sizes 128–2,048; a prediction head adds 0.4% Top-1. Support sets above 98,304 add no significant gain; FIFO updates beat random updates by more than 2% Top-1, and a 98k queue adds 100 MB of memory.Source: §4.4 · p. 7

4.5. Discussion

• The retrieved neighbor matches the query’s ImageNet class about 57% of the time near training’s end; an oracle same-class neighbor yields 75.8% accuracy after 300 epochs versus 72.9% with the top-1 retrieved neighbor.The authors suggest background-focused random crops may explain some mismatches and note that better unsupervised neighbor selection could improve performance.Source: §4.5 · p. 7
• NNCLR begins to outperform SimCLR after about 120 epochs and remains ahead through 1,000 epochs; its retrieved positives can share semantic or visual similarities such as texture.Examples include underwater scenes and images containing cages, illustrating that retrieved neighbors are not always from the same class.Source: §4.5 · p. 8

5. Conclusion

• NNCLR uses nearest neighbors from a support set as positives to increase diversity in contrastive self-supervised learning.Figure 5 illustrates the increased diversity of NNCLR positives.Source: §5 · p. 8, Figure 5 · p. 9
• NNCLR achieves state-of-the-art performance on multiple datasets.Source: §5 · p. 8
• NNCLR drastically reduces reliance on data augmentation techniques.Source: §5 · p. 8

• For each minibatch, NNCLR augments each sample twice, then applies the backbone-plus-projection encoder and prediction MLP to both views.Source: A. Pseudo-code · p. 11
• It finds each view’s top-1 neighbor in the queue by L2-normalizing embeddings and queue entries, computing similarities, and returning the highest-scoring queue item.Source: A. Pseudo-code · p. 11
• Training back-propagates the loss, updates the encoder and prediction MLP with SGD, and updates the queue with the latest z1 embeddings.Source: A. Pseudo-code · p. 11
• For each neighbor-prediction pair, the loss function forms n-by-n logits by matrix multiplication and scales them by temperature 0.1.Source: A. Pseudo-code · p. 11
• NNCLR can also use a momentum encoder; its corresponding pseudo-code is presented in Algorithm 2.Source: A. Pseudo-code · p. 11

B. Evolution of Nearest-Neighbors

• As training proceeds, nearest neighbors shift from matches based on color and texture to examples from similar semantic classes.This transition occurs as the encoder becomes better at recognizing classes.Source: B. Evolution of Nearest-Neighbors · p. 11

C. SimSiam with Nearest-neighbor as Positive

• NNSiam replaces SimSiam’s stop-gradient branch with its nearest neighbor from the support set; hard nearest-neighbor selection provides an implicit stop-gradient, using a 2048-dimensional embedding.The experiment uses the self-supervised framework SimSiam [14].Source: C. SimSiam with Nearest-neighbor as… · pp. 11–12
• NNSiam improves accuracy by 1.3% under the ImageNet linear evaluation protocol using a non-contrastive loss with the nearest neighbor as positive.The authors interpret this as evidence that harder positives can improve performance beyond the InfoNCE loss.Source: C. SimSiam with Nearest-neighbor as… · p. 12, Table 10 · p. 13

D. Experiments with Vision Transformers

• ViT training used Adam for 1000 epochs with two crops, a 3 × 10−4 base learning rate, 10-epoch warmup, cosine decay, 0.05 weight decay, and 0.1 stochastic depth dropout.The output corresponding to the final-layer [CLS] token served as the loss embedding and the linear-classifier representation.Source: D. Experiments with Vision Transformers · p. 13
• The final-layer [CLS] token output supplied both the loss embedding and the representation used by the linear classifier.Source: D. Experiments with Vision Transformers · p. 13
• NNCLR outperformed the supervised model trained without DeIT augmentation by 4.4% and SimCLR by 2% on ViT ImageNet linear-protocol Top-1 accuracy.Table 11 reports the metric; the passage does not provide the models’ individual accuracy values.Source: Table 11 · p. 13, D. Experiments with Vision Transformers · p. 13

E. Self-supervised Learning as a Pre-training Step for Supervised Learning

• The authors present self-supervised pretraining as a way to boost final-model performance without extra data or complex regularization techniques.They contrast this use with the prior belief that supervised learning on a dataset would always outperform self-supervised learning on that dataset, noting BYOL [28] as a counterexample.Source: E. Self-supervised Learning as a… · p. 13
• ViT-B/16 pre-trained with NNCLR outperformed DeIT [57] but trailed the same architecture trained with JFT-300M by 0.5%.The passage identifies ViT as the newly proposed architecture [19] and notes that it requires additional data or regularization to achieve good performance.Source: E. Self-supervised Learning as a… · p. 13
• The experiment used NNCLR pretraining for 1000 epochs on ImageNet 2012, followed by 100 epochs of supervised training with 100% of labels available.This setup tests self-supervised pretraining as an initialization for fully supervised learning.Source: E. Self-supervised Learning as a… · p. 13
• ResNet50 ImageNet accuracy rose from 76.2% to 79.1% with NNCLR initialization, comparable to JFT-300M [40] despite its considerably larger data.The comparison follows 1000 epochs of NNCLR pretraining on ImageNet 2012 with 100 epochs of supervised training.Source: Table 12 · p. 13, E. Self-supervised Learning as a… · p. 13

F. Transfer Learning

• ViT-B/16 trained with NNCLR outperforms same-loss ResNet50 on 8 of 12 transfer datasets, while supervised ViT-B/16 trails supervised ResNet50 on 10 of 12.NNCLR also improves ViT-B/16 over its supervised-only counterpart on all 12 datasets.Source: F. Transfer Learning · p. 14
• ViT-B/8 outperforms ViT-B/16 on all transfer datasets, and NNCLR improves ViT-B/8 over supervised-only training.The reported NNCLR gains include approximately 9.8% on Birdsnap, 3.9% on Sun397, 10.5% on Aircraft, 7.1% on DTD, and 5.7% on Flowers.Source: F. Transfer Learning · p. 14
• Fine-tuning NNCLR-pretrained ViT-B/8 with supervised loss boosts results on six datasets but reduces them on DTD and Flowers, while remaining better than supervised-only training everywhere.Reported gains are 8.3% on Food, 5.3% on Birdsnap, 2.9% on Sun397, 15.8% on Cars, 2.2% on VOC2007, and 2.1% on Pets; DTD and Flowers each decline 1.1%.Source: F. Transfer Learning · p. 14

G.1. Attention in NNCLR ResNet

• NNCLR-trained ResNet50 visualizations exhibit objectness and part-localization properties also observed in self-supervised vision transformers.The authors visualize attention by convolving query embeddings across feature maps, showing queries on COCO images.Source: §G.1 · p. 14

G.2. Cross-image Attention with NNCLR ResNets

• Cross-image attention localizes same-category objects across images, including multiple giraffes and small objects such as balls and frisbees.Stop-sign queries also match differently colored street-sign types, while not activating on the stop-sign text, indicating similarity beyond color.Source: §G.2 · p. 14

G.3. Comparison of Attention in ResNets vs ViT

• The experiment compares ResNet-50 and ViT-B/16 with random, ImageNet-supervised, and ImageNet-NNCLR weights, using each architecture’s global embedding as the query.For ViT, the query is the final-layer [CLS] token’s average self-attention over heads; for ResNet-50, it is the average-pooled embedding.Source: §G.3 · pp. 14–15
• Randomly initialized ResNets and ViTs can sometimes localize objects through color and texture similarity, so well-delineated attention does not establish semantic features.The authors recommend combining self-attention with cross-attention maps for more robust interpretation.Source: §G.3 · p. 15
• In Figure 10, attention maps delineate salient objects for both supervised- and self-supervised-trained ResNet-50s, whereas ViTs trained with only supervised learning do not highlight objects in their attention maps.The comparison includes random initialization, supervised-loss weights, and weights trained with the NNCLR objective.Source: §G.3 · p. 15, Figure 10 · p. 17