PaperlayerView exact revision on arXiv

Source-linked AI summary

Region Similarity Representation Learning

Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, Trevor Darrell

arXiv:2103.12902v2cs.CV

TL;DR

Global image-level self-supervision does not enforce spatial consistency in convolutional features used by localization tasks. ReSim adds similarity learning between corresponding regions across views while retaining semantic image-level learning, and reports improvements over MoCo-v2 on localization-based benchmarks.

  • Problem

    Global instance-discrimination features are pooled at the image level and do not enforce spatial consistency in convolutional maps used for object localization.

    Source: §1 · pp. 1–2

  • Method

    ReSim combines global image-level learning with similarity learning on aligned regions across views, extending regional learning to feature pyramids.

    Source: §1 · p. 2

  • Results

    ReSim improves localization and classification over MoCo-v2; ReSim-C4 and ReSim-FPN gains include 2.7 and 2.3 AP75, respectively.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The authors report that maximizing similarity between convolutional regions improves localization for downstream detection and instance segmentation.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present Region Similarity Representation Learning (ReSim), a new approach to self-supervised representation learning for localization-based tasks such as object detection and segmentation. While existing work has largely focused on solely learning global representations for an entire image, ReSim learns both regional representations for localization as well as semantic image-level representations. ReSim operates by sliding a fixed-sized window across the overlapping area between two views (e.g., image crops), aligning these areas with their corresponding convolutional feature map regions, and then maximizing the feature similarity across views. As a result, ReSim learns spatially and semantically consistent feature representation throughout the convolutional feature maps of a neural network. A shift or scale of an image region, e.g., a shift or scale of an object, has a corresponding change in the feature maps; this allows downstream tasks to leverage these representations for localization. Through object detection, instance segmentation, and dense pose estimation experiments, we illustrate how ReSim learns representations which significantly improve the localization and classification performance compared to a competitive MoCo-v2 baseline: $+2.7$ AP$^{\text{bb}}_{75}$ VOC, $+1.1$ AP$^{\text{bb}}_{75}$ COCO, and $+1.9$ AP$^{\text{mk}}$ Cityscapes. Code and pre-trained models are released at: \url{https://github.com/Tete-Xiao/ReSim}

75 VOC, +1.1 APbb

The reported results include a +1.9 APmk result on Cityscapes.

1. Introduction

Global instance-discrimination objectives do not enforce spatial consistency in convolutional features, which localization tasks use. ReSim adds aligned regional similarity learning across convolutional layers while retaining global learning, and extends region learning to feature pyramids.

1. Introduction

  • Instance discrimination optimizes globally pooled image features but does not ensure that shifted or scaled objects correspond to spatially shifted convolutional features.This matters for detection, which uses convolutional spatial information for localization; Figure 1 contrasts global-view matching with ReSim’s region alignment.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • ReSim’s regional objective is motivated by the need for convolutional features that are both spatially consistent and semantically sensitive for localization and classification.Source: §1 · p. 2
  • ReSim slides a fixed-size window across the overlapping area of two views, aligns corresponding convolutional regions, and maximizes their similarity alongside the global objective.Source: §1 · p. 2
  • ReSim extends region learning across feature-map scales using multiple window sizes and FPN top-down features, with region-level learning training pyramid layers without labels.The top-down path propagates stronger semantic features to earlier convolutional layers.Source: §1 · p. 2
  • Experiments on PASCAL VOC, COCO, and Cityscapes cover object detection, instance segmentation, and dense pose estimation, with improvements over MoCo-v2 in classification and localization.Source: §1 · p. 2
  • +1.1 APbb75 on COCO and +1.9 APmk on Cityscapes are reported against a MoCo-v2 baseline.Source: §1 · p. 2

2. Related work

Prior self-supervised methods largely learn global image-level representations, while localization tasks use regional convolutional features. ReSim combines image-level semantic learning with region-level features across scales, shifts, and resolutions.

2. Related work

  • Instance-discrimination methods such as SimCLR and MoCo improved self-supervised representation learning, while SwAV and BYOL did not tend to improve localization tasks such as detection and segmentation.The cited work associates this pattern with strictly optimizing global image-level representations.Source: §2 · pp. 2–3
  • Unlike earlier pixel- or region-level pretexts, ReSim builds on instance discrimination to learn semantic image features and regional features across multiple scales, shifts, and resolutions.Source: §2 · p. 3
  • Detection and instance segmentation use convolutional region features whose spatial sensitivity matters for regression and semantic sensitivity for classification.Source: §2 · p. 3
  • Wang et al.’s grid-based method [50] maximizes the most similar pair among feature vectors, whereas ReSim compares region features; the paper reports superior performance over these models.Source: §2 · p. 3

3. Region Similarity Representation Learning

ReSim adds region-level similarity learning to image-level self-supervision, aligning corresponding regions across views to learn spatially and semantically consistent features. Its framework extracts features from mapped regions and combines regional learning with global image-level similarity.

3.1. Preliminaries

  • For localization-sensitive tasks, corresponding regions across views should map to consistent representations, while different regions should map to different representations.Small feature-map shifts can change pixel-level classification and regression targets, so region features need both spatial and semantic sensitivity.Source: §3.1 · pp. 3–4

3.2. Framework Overview

  • ReSim slides fixed-size windows only across overlapping query and key areas, then uses Precise RoI Pooling to extract vectors from corresponding feature-map regions.Earlier layers use smaller windows because they have higher spatial resolution.Source: §3.2 · p. 4, §3.3 · p. 4

3.3. Region-level Similarity

  • ReSim maps query-region coordinates to the key view using the views' Random Resized Crop transforms, with horizontal flips matched between views.Only spatial transformations affecting coordinates are considered; the correspondence is computed from the crop translations.Source: §3.3 · p. 4
  • ReSim-FPN adds top-down connections that propagate semantically strong features to earlier layers, training region-similarity features across pyramid levels.Lateral connections form P3, P4, and P5; nearest-neighbor sampling and 1×1 convolutions match spatial and channel dimensions.Source: §3.3 · p. 5
  • The regional objective contrasts aligned region features, while the global image-level objective uses global average pooling; λ weights the two objectives.Regional negatives can include nonmatching regions from the same image and regions from other images.Source: §3.3 · p. 5, §3.4 · p. 5

3.4. Framework Configurations

  • ReSim-C4 applies region similarity on C4 without FPN, whereas ReSim-FPN applies it on P3 and P4, using level-specific window sizes and strides.C4/P4 uses a 48-pixel window with a 32-pixel stride; P3 uses a 32-pixel window with a 24-pixel stride.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate ReSim on IN-100 and IN-1K pretraining, then transfer the learned representations to localization-dependent downstream tasks. The reported results cover window and feature-extraction choices, alternative self-supervised frameworks, and object detection, instance segmentation, and dense pose estimation.

4.1. IN-100 Study

  • On VOC, ReSim-C4 improves the MoCo-v2 baseline by 1.5, 1.2, and 2.1 at AP, AP50, and AP75, with a larger gap at AP75 than AP50.The frozen-backbone setup freezes C1–C4 and finetunes the RPN and C5 classifier head to assess the pretrained features.Source: §4.1 · pp. 6–7
  • ReSim built on SimSiam improves its baseline by 1.1, 0.6, and 1.2 at AP, AP50, and AP75, respectively, and is comparable to ReSim built on MoCo.The authors choose MoCo as the backbone framework because SimSiam requires more pretraining time.Source: Table 2 · p. 6, §4.1 · pp. 6–7
  • In the frozen-backbone setting, ReSim improves the MoCo-v2 baseline by 2.9, 1.8, and 4.4 at AP, AP50, and AP75, respectively.The improvement is larger than in full finetuning, particularly at AP75.Source: Table 3 · p. 6, §4.1 · p. 7

4.2. IN-1K Results

  • ReSim-C4 improves over MoCo-v2 by 1.7/0.7/2.7 at AP/AP50/AP75, while ReSim-FPN improves by 2.2/0.5/2.3 on PASCAL VOC.Both methods also achieve state-of-the-art results over DenseCL, according to the comparison.Source: Table 4 · p. 7, §4.2 · p. 8

4.1. IN-100 Study

  • W48-S32 is selected as the default ReSim-C4 window configuration after considering frozen-backbone and full-finetuning performance.“Wl-Sk” denotes a window of size l×l and stride k.Source: §4.1 · p. 7
  • RoI Align and Precise RoI Pooling both work well, but RoI Align performs worse in the frozen-backbone setting; the authors conjecture that Precise RoI Pooling enables more robust, simpler optimization.For RoI Align, VOC AP/AP50/AP75 are 55.7/81.0/62.0 with all parameters finetuned and 48.9/76.6/53.0 with a frozen backbone.Source: §4.1 · p. 7

4.2. IN-1K Results

  • All ReSim variations improve over their MoCo-v2 counterparts on COCO box and mask metrics, and transferring FPN weights improves over ReSim-FPN.Performance continues to increase after 400 pretraining epochs.Source: Table 5 · p. 8, §4.2 · p. 8
  • ReSim variants improve over the supervised baseline by 3.3–3.6 points under APmk on Cityscapes, while ReSim-FPN improves by 2.2 points under APgpsm on DensePose.The Cityscapes and DensePose comparisons are reported in Table 6.Source: Table 6 · p. 8, §4.2 · pp. 8–9

5. Conclusion

ReSim combines global and region-level contrastive learning to learn semantic and spatial feature representations. The paper reports improvements across localization-based tasks and describes feature-pyramid pretraining and transfer.

5. Conclusion

  • ReSim learns semantic image-level representations alongside spatially consistent representations on convolutional feature-map regions.The method uses both global and region-level contrastive learning.Source: §5 · p. 9
  • ReSim incorporates feature pyramids to learn hierarchical representations and transfers those parameters to downstream FPN-based frameworks.The top-down path propagates stronger semantic features to earlier layers.Source: §5 · p. 9
  • The paper reports significant improvements for object detection, instance segmentation, and dense pose estimation.Source: §5 · p. 9

A.1. Experimentation Details

The appendix specifies pretraining parameters and region-level negative-sample construction for ReSim-C4 and ReSim-FPN.

A.1. Experimentation Details

  • IN-1K pretraining uses 200 epochs and a queue of 65,536, while IN-100 uses 500 epochs and a queue of 16,384.The smaller queue for IN-100 reflects its smaller image set.Source: §A.1 · p. 9, Table 7 · p. 9
  • Region-level negatives come from stride-one average pooling of key-view feature maps, with pooling kernels matched to feature-map downsampling and window size.For C4/P4, the 48×48 input window and 16× downsampling correspond to a 3×3 average-pooling kernel.Source: §A.1 · p. 9
  • A queue is unnecessary for region-level negatives because each batch supplies many; synchronized C4/P4 features yield 36,864 negatives, while per-GPU P3 features yield 21,632.The contrastive temperature is unchanged because the negative counts are roughly comparable to those from the image-level momentum queue.Source: §A.1 · p. 9

A.2. Extended Experimental Results

ReSim-C4’s ImageNet linear-probe accuracy fell from 67.5% to 66.1% after 200 epochs, despite improved object-detection transfer; ImageNet classification accuracy does not necessarily track region-level transfer.

A.2. Extended Experimental Results

  • 67.5% to 66.1%: ReSim-C4’s ImageNet linear-probe accuracy dropped after 200 epochs, despite improved object-detection transfer versus MoCo-v2.The passage notes that ImageNet classification accuracy does not necessarily indicate improved region-level transfer, echoing Chen et al. [6].Source: §A.2 · p. 10
Loading 2103.12902v2…