PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Visual Pretraining with Contrastive Detection

Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, João Carreira

arXiv:2103.10957v2cs.CV

TL;DR

Self-supervised pretraining offers transferable representations but is computationally expensive. The paper introduces contrastive detection, which learns object-level features across augmentations, achieving strong downstream transfer with up to 10× less pretraining. Its strongest ImageNet model matches SEER despite using 1000× less pretraining data, and COCO pretraining closes the gap with supervised transfer.

  • Problem

    State-of-the-art self-supervised pretraining requires an order of magnitude more computation than supervised pretraining, creating a critical bottleneck.

  • Method

    Contrastive detection maximizes similarity across augmentations for local features representing the same object, using regions from unsupervised segmentation.

  • Results

    The method achieves strong transfer across downstream vision tasks, requires up to 10× less computation to match supervised ImageNet transfer, and its best model matches SEER trained on 1000× more images.

  • Takeaways & Limitations

    DetCon accelerates pretraining while improving transfer accuracy across tasks and supports pretraining on complex scenes such as COCO.

  • Takeaways & Limitations

    DetCon's power strongly correlates with how well its masks align with object boundaries, and higher-quality segmentations improve representational quality.

Abstract

from arXiv · show

Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-of-the-art methods requiring an order of magnitude more computation than supervised pretraining. We tackle this computational bottleneck by introducing a new self-supervised objective, contrastive detection, which tasks representations with identifying object-level features across augmentations. This objective extracts a rich learning signal per image, leading to state-of-the-art transfer accuracy on a variety of downstream tasks, while requiring up to 10x less pretraining. In particular, our strongest ImageNet-pretrained model performs on par with SEER, one of the largest self-supervised systems to date, which uses 1000x more pretraining data. Finally, our objective seamlessly handles pretraining on more complex images such as those in COCO, closing the gap with supervised transfer learning from COCO to PASCAL.

1. Introduction

The paper introduces contrastive detection to reduce self-supervised pretraining cost by learning object-level features across augmentations. DetCon produces strong transfer representations across vision tasks and complex scenes while requiring substantially less computation.

  • State-of-the-art self-supervised methods require an order of magnitude more computation than supervised pretraining, making cost a critical bottleneck.
  • Contrastive detection maximizes similarity between object-level features across augmentations using regions from unsupervised segmentation.The method groups local feature vectors by approximate object regions before applying the contrastive objective.
  • Object-level contrast provides separate learning signals for all objects, more diverse negatives, and better support for complex scenes.These benefits enrich each image's training signal and accelerate learning.
  • DetCon representations transfer to COCO detection and instance segmentation, PASCAL and Cityscapes semantic segmentation, and NYU depth estimation.The representations are reported as more accurate than recent self-supervised alternatives and obtainable with less training time.
  • Up to 10× less computation is required to match supervised ImageNet transfer, while longer schedules achieve state-of-the-art COCO transfer and match SEER trained on 1000× more images.
  • When transferring from complex scene datasets such as COCO, DetCon closes the gap with a supervised model trained on human-annotated segmentations.

2. Related work

Related work has shifted from whole-image self-supervision toward contrastive objectives and local descriptors for downstream vision tasks. DetCon builds on this direction by applying contrastive learning to object-level local features.

  • Transfer learning remains central in computer vision, while recent self-supervised work increasingly focuses on pretraining objectives and tasks.
  • Contrastive objectives maximize similarity across views while minimizing similarity to distracting negative samples.Prior views include crops, input channels, and augmented versions of images.
  • Recent work also learns local descriptors for detection and segmentation using auxiliary losses, architectural components, or both.
  • DetCon extends perceptual grouping with contrastive learning by matching local features that represent the same object across views.

3. Method

DetCon applies contrastive learning to object-level regions identified by unsupervised segmentation, aligning corresponding regions across augmented views while contrasting different regions and images. Its design reuses established contrastive frameworks and adds mask-based representations with modest computational overhead.

  • Contrastive detection framework: DetCon maximizes similarity across views for local features representing the same object, using object regions supplied by simple unsupervised heuristics.DetConS and DetConB adapt the SimCLR and BYOL frameworks respectively.
  • Contrastive detection framework: The objective pulls together paired mask features across views and contrasts them with features from different masks and images.Negative samples may come from other masks in the same image or from other images, and overlapping negative masks are allowed.
  • Contrastive detection framework: Object-based masks are transformed alongside two augmented image views, producing aligned mask sets for feature extraction.The masks undergo the same cropping and resizing as the corresponding RGB images.
  • Contrastive detection framework: Mask-pooled feature vectors are extracted from convolutional spatial maps and transformed into latent representations for contrastive learning.The feature extractor produces a spatial hidden-vector map, while a projection network transforms each mask-pooled vector.
  • Contrastive detection framework: Each iteration samples 16 masks and densely evaluates mask pairs across images to provide diverse negative samples, while masking the loss for paired locations.This handles cases where a mask appears in one augmented view but not the other.
  • Computational cost: The added computation is 5.3% for DetConS and 11.6% for DetConB relative to the baseline, while segmentation costs are negligible and reusable.The additional projection and contrastive-loss computation contributes only a small fraction of total cost.
  • Unsupervised mask generation: DetCon supports spatial, FH, MCG, and human-annotated masks, with a single global mask reducing DetConS to SimCLR.MCG supplies overlapping object proposals, while human-annotated masks assess potential gains from higher-quality segmentations.

4. Experiments

DetCon improves transfer-learning efficiency and accuracy across tasks, remains effective with larger models, and benefits from object-aligned masks and dense regional sampling.

  • Pretraining efficiency: Up to 10× less pretraining computation is required to match supervised ImageNet transfer, with gains spanning COCO, PASCAL, Cityscapes, and NYU tasks.DetConS gives a 5× COCO efficiency gain, 2× on PASCAL semantic segmentation, and 10× on Cityscapes semantic segmentation and NYU depth prediction.
  • Pretraining efficiency: DetConB outperforms BYOL across pretraining budgets and downstream tasks, yielding 3× efficiency gains on COCO, PASCAL, and Cityscapes and 10× on NYU depth.
  • Scaling model capacity: DetCon continues to outperform prior methods with ResNet-101, ResNet-152, and ResNet-200 feature extractors, showing that its gains persist at higher capacity.
  • Scaling model capacity: A 250M-parameter ResNet-200 DetCon model matches SEER transfer performance despite using ImageNet rather than 1000× more pretraining data.The comparison uses different architectures and is imperfect because large-scale data is necessarily noisier.
  • Complex-scene transfer: DetCon pretraining on COCO surpasses a fully supervised COCO baseline when transferring to PASCAL segmentation.
  • Mask ablations: A 5×5 spatial grid is optimal among tested grids, while FH and MCG masks outperform spatial heuristics, with MCG slightly better.Representation quality correlates with overlap between pretraining masks and ground-truth objects.
  • Objective ablations: Dense sampling of object regions improves detection and segmentation by learning from the entire image and supplying diverse positive and negative samples.Sampling one FH mask per image slightly deteriorates performance, whereas duplicating a global mask does not change representation quality.
  • Objective ablations: High-quality masks make DetCon less dependent on large negative-sample pools, whereas reducing negatives harms DetCon with approximate FH masks.DetConS with ground-truth masks improves despite the reduction in negative samples.

5. Discussion

DetCon accelerates self-supervised pretraining by organizing images into object-like entities, while its effectiveness depends strongly on mask alignment with object boundaries.

  • DetCon uses low-level cues to organize images into objects and background regions, accelerating pretraining while improving accuracy across downstream tasks.
  • DetCon achieves state-of-the-art self-supervised ImageNet transfer and matches methods trained with much larger models on much larger datasets.
  • DetCon power strongly correlates with mask alignment to object boundaries because distinct regions provide independent signals and diverse negatives.
  • The results motivate jointly discovering objects and learning representations, potentially enabling iterative improvement of unsupervised scene understanding.

A.1. Implementation: data augmentation

The method uses paired image augmentations built from spatial and appearance transformations, with separate augmentation settings for DetConS and DetConB. Transfer experiments additionally apply task-specific resizing, cropping, flipping, and scaling.

  • A.1. Implementation: data augmentation: Each image is randomly augmented twice to produce two views, x and x′.
  • A.1. Implementation: data augmentation: Random cropping samples patch area from [0.08 · A, A] and aspect ratio from [3/4, 4/3], then resizes the patch to 224 ×224 pixels.
  • A.1. Implementation: data augmentation: Appearance transformations include color jittering, grayscale conversion, Gaussian blurring, and solarization.
  • A.1. Implementation: data augmentation: DetConS and DetConB use the SimCLR and BYOL augmentation pipelines, respectively, with potentially different transformation probabilities and magnitudes across views.
  • A.1. Implementation: data augmentation: Transfer preprocessing is task-specific: COCO uses resizing and flipping, while PASCAL, Cityscapes, and NYU-Depth use scaling, cropping, flipping, or color jittering.

A.2. Implementation: architecture

The architecture encodes images with convolutional feature extractors, pools spatial features within object masks, and projects the resulting vectors before contrastive learning. DetConS and DetConB add different projection designs and incur modest computational overhead.

  • A.2. Implementation: architecture: The default feature extractor is a ResNet-50, with experiments also using deeper and wider variants.
  • A.2. Implementation: architecture: Hidden vectors are pooled within masks into object-level vectors, then transformed by projection and optionally prediction heads before the contrastive loss.
  • A.2. Implementation: architecture: DetConS uses a two-layer MLP with hidden and output dimensions 2048 and 128, while sharing learned parameters across both views.
  • A.2. Implementation: architecture: DetConB uses 4096- and 256-dimensional projection layers, an exponential-moving-average encoder for one view, and a prediction head for the first view.
  • A.2. Implementation: architecture: DetConS adds 201M FLOPS and DetConB 441M FLOPS, corresponding to 5.3% and 11.6% of backbone evaluation cost.

A.3. Implementation: optimization

Pretraining uses large-batch LARS optimization with schedules adapted to the dataset and baseline framework, while downstream fine-tuning uses task-specific stochastic-gradient procedures. COCO comparisons use matched 1× and 2× schedules.

  • A.3. Implementation: optimization: ImageNet pretraining uses LARS with batch size 4096 across 128 Cloud TPU v3 workers and cosine learning-rate decay.
  • A.3. Implementation: optimization: COCO pretraining replaces cosine decay with piecewise-constant drops at the 96th and 98th percentiles to alleviate overfitting.
  • A.3. Implementation: optimization: COCO fine-tuning uses stochastic gradient descent with linear warmup, two tenfold learning-rate drops, momentum 0.9, and batch size 64.
  • A.3. Implementation: optimization: PASCAL fine-tuning runs for 45 epochs, Cityscapes for 160 epochs, and NYU-Depth v2 for 7500 steps, each with task-specific schedules.
  • A.3. Implementation: optimization: COCO comparisons evaluate ImageNet-pretrained methods after 12-epoch 1× or 24-epoch 2× fine-tuning schedules.

A.4. Results: larger models

DetCon remains competitive when model capacity increases, continuing to outperform prior self-supervised methods with deeper ResNet architectures.

  • A.4. Results: larger models: DetCon continues to outperform prior self-supervised work when transferred to COCO with larger ResNet-101, ResNet-152, and ResNet-200 models.
Loading 2103.10957v2…