PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-aware Contrastive Learning for Debiased Scene Representation

Sangwoo Mo, Hyunwoo Kang, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin

arXiv:2108.00049v2cs.CVcs.LG

TL;DR

Contrastive learning can entangle co-occurring objects and backgrounds, potentially harming downstream generalization. The paper uses ContraCAM to localize salient regions and guide object-aware augmentations; experiments report gains on multi-object images and background and distribution shifts.

  • Problem

    Contrastive learning can entangle co-occurring objects or object-background pairs, potentially harming generalization on downstream tasks.

    Source: §1 · pp. 1–2

  • Method

    The framework uses ContraCAM to localize discriminative regions, then applies object-aware random crop and background mixup to reduce contextual and background biases.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    The framework improves representation learning under multi-object images and generalizability on background, shape-biased, and corrupted distribution shifts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors describe the approach as a step toward strong self-supervised learning under real-world scenarios, where scene bias is more severe in uncurated images.

    Source: §5 · p. 10

    ContraCAM may detect salient regions that differ from users’ desired objects and has difficulty separating occluded objects.

    Source: §5 · p. 10

Abstract

from arXiv · show

Contrastive self-supervised learning has shown impressive results in learning visual representations from unlabeled images by enforcing invariance against different data augmentations. However, the learned representations are often contextually biased to the spurious scene correlations of different objects or object and background, which may harm their generalization on the downstream tasks. To tackle the issue, we develop a novel object-aware contrastive learning framework that first (a) localizes objects in a self-supervised manner and then (b) debias scene correlations via appropriate data augmentations considering the inferred object locations. For (a), we propose the contrastive class activation map (ContraCAM), which finds the most discriminative regions (e.g., objects) in the image compared to the other images using the contrastively trained models. We further improve the ContraCAM to detect multiple objects and entire shapes via an iterative refinement procedure. For (b), we introduce two data augmentations based on ContraCAM, object-aware random crop and background mixup, which reduce contextual and background biases during contrastive self-supervised learning, respectively. Our experiments demonstrate the effectiveness of our representation learning framework, particularly when trained under multi-object images or evaluated under the background (and distribution) shifted images.

1 Introduction

Contrastive learning can entangle co-occurring objects and object-background pairs, weakening generalization. The paper proposes object-aware contrastive learning that uses ContraCAM to locate salient regions and guide augmentations that reduce these biases.

1 Introduction

  • ContraCAM locates discriminative image regions by contrasting other images; iterative refinement extends it to multiple objects and entire shapes.The framework uses object-aware random crop and background mixup for positive pairs, and masked images as effective negatives.Source: §1 · p. 2
  • The framework improves MoCov2 and BYOL by reducing contextual and background biases and learning object-centric representations.Its approach combines ContraCAM-based augmentations with object localization.Source: §1 · p. 2
  • On COCO multi-object images, the method boosts downstream classification and detection performance.The paper identifies object-aware random crop as reducing contextual bias during training.Source: §1 · p. 2
  • On the Background Challenge, the method improves generalizability when objects appear in unusual backgrounds.Background mixup is introduced to reduce background bias.Source: §1 · p. 2
  • The method improves generalizability on shape-biased and corrupted distribution shifts.Source: §1 · p. 2
  • ContraCAM achieves localization results comparable to state-of-the-art unsupervised localization and supervised classifier CAM while remaining simple.Source: §1 · p. 2

2 Object-aware Contrastive Learning

The section addresses scene bias in contrastive learning by using ContraCAM to localize objects and guide augmentations that reduce contextual and background correlations. It develops iterative localization and two object-aware augmentations for learning debiased representations.

2.1 Contrastive learning

  • Contrastive learning attracts augmented views as positives and separates other images, but random crops can entangle co-occurring objects or object-background pairs, harming shifted generalization.Source: §2.1 · pp. 2–3

2.2 ContraCAM: Unsupervised object localization via contrastive learning

  • ContraCAM extends CAM by scoring contrastive similarity instead of classifier probability and discarding negative gradient signals that hinder localization.The method seeks discriminative image regions without class labels.Source: §2.2 · pp. 3–4
  • Iterative refinement masks current salient regions, recalculates CAMs using masked images as negatives and the original as positive, then aggregates masks to detect multiple objects and entire shapes.The CAM converges to a stationary value when the iteration count is sufficiently large.Source: §2.2 · pp. 3–4

2.3 Object-aware augmentations for debiased contrastive learning

  • Object-aware random crop selects a ContraCAM bounding box and generates positives within it, avoiding attraction between different objects; ground-truth boxes can hinder contrastive optimization for small objects.The passage attributes the difficulty to small objects being hard to discriminate as negatives.Source: §2.3 · pp. 4–5
  • Background mixup replaces an image's background with a ContraCAM-inferred background patch from another image, so positives share foreground objects but differ in background.The patch is formed by tiling the largest rectangle in the other image's binarized CAM background.Source: §2.3 · p. 5
  • Background mixup reportedly often outperforms Mixup [41] and CutMix [42] for contrastive learning [43], and improves generalization on shape-biased and corrupted datasets.The method is described as improving generalization on distribution shifts.Source: §2.3 · p. 5

3 Experiments

Experiments assess ContraCAM localization and whether object-aware crops and background mixup improve representations across multi-object, background-shift, and distribution-shift settings. ContraCAM localizes comparably to prior unsupervised methods, while the augmentations improve downstream performance and shifted-image generalization.

3.1 Unsupervised object localization

  • ContraCAM detects multiple objects and has mask mIoU comparable to ReDo; it also outperforms classifier CAM on CUB and Flowers and often on unseen transfer images.The reported localization comparisons include same-dataset and ImageNet-trained transfer settings.Source: Table 1 · p. 6, §3.1 · p. 7, Table 2 · p. 7
  • ContraCAM localization benefits from negative-signal removal, while too few refinement iterations often detect only subregions.The method converges with sufficient iterations, and the experiments use 10 iterations across datasets.Source: §3.1 · p. 7

3.2 Reducing contextual bias: Representation learning from multi-object images

  • OA-Crop significantly improves linear evaluation across tested classification tasks; ContraCAM boxes even outperform ground-truth boxes for MoCov2 with ResNet-50.The reported tasks include COCO-Crop, CIFAR-10/100, CUB, Flowers, Food, and Pets.Source: §3.2 · p. 7, Table 4 · p. 8
  • OA-Crop with ContraCAM boxes outperforms baselines on COCO detection and segmentation, while ground-truth boxes are on par or worse.The comparison uses fine-tuned MoCov2 and BYOL models with ResNet-50.Source: §3.2 · p. 8, Table 5 · p. 8
  • OA-Crop performs on par with vanilla MoCov2 and BYOL on original ImageNet-9 but better on distribution-shifted datasets.The shifted evaluations include shape-biased and corrupted images.Source: §3.2 · p. 8, Table 6 · p. 8

3.3 Reducing background bias: Generalization on background and distribution shifts

  • BG-Mixup improves predictions on object-focused background-shift datasets, regularizes background-focused datasets, and mostly outperforms Mixup and CutMix on distribution shifts.The reported advantage spans both background shifts and distribution shifts.Source: §3.3 · p. 9, Table 7 · p. 9, Table 8 · p. 9

4 Related work

Related work frames scene bias as a weakness of contrastive representation learning and distinguishes this paper's fully self-supervised localization and debiasing approach from methods requiring object annotations or pretrained models. It also positions ContraCAM as an unsupervised extension of class activation maps using contrastive rather than classifier scores.

Contrastive learning

  • Prior scene-bias approaches for contrastive learning assumed ground-truth object annotations or pretrained segmentation models, whereas this work proposes fully self-supervised localization and debiasing.Other object-aware video methods cited attract objects across temporal views and require a pretrained detector.Source: §4 · p. 10

Bias in visual representation

  • This work addresses scene bias in self-supervised representation learning through unsupervised debiasing, as part of broader efforts to remove visual shortcuts.The passage describes an aim toward unbiased, robust visual representations.Source: §4 · p. 10

Unsupervised object localization

  • Unsupervised localization work includes generative, noisy-ensemble, and salient-direction approaches; scene decomposition targets fuller object-background separation but does not yet scale to complex images.The passage identifies generative approaches as state of the art for fully unsupervised scenarios.Source: §4 · p. 10

Class activation map

  • ContraCAM extends class activation maps from weakly supervised localization to unsupervised localization by using a contrastive score instead of classifier scores.Classifier CAM finds regions salient to the classifier score.Source: §4 · p. 10

5 Conclusion and Discussion

The paper presents ContraCAM as self-supervised object localization and uses it to build augmentations that reduce scene bias and improve learned representations. The supplied implementation details describe localization, contextual-bias, and transfer evaluations.

5 Conclusion and Discussion

  • ContraCAM-based augmentations reduce scene bias and improve the quality of learned contrastive representations.The conclusion also notes that scene bias is more severe in uncurated images.Source: §5 · p. 10

A Algorithms

  • Iterative ContraCAM computes feature-based scores and CAMs on masked images, unions the masks, and uses soft masking to reveal additional regions.The localization experiments use ten iterations and report mask mean intersection-over-union.Source: A Algorithms · p. 15, §B.1 · p. 16

B.1 Implementation details for object localization results

  • Object-localization experiments train MoCov2 with ResNet-18 on CUB, Flowers, COCO, and ImageNet, using 2,000 epochs for CUB and Flowers and 800 for COCO and ImageNet.The experiments use batch size 256; the reported mask metric is mIoU.Source: §B.1 · p. 16

B.2 Implementation details for contextual bias results

  • For contextual-bias training, MoCov2-derived CAM boxes are reused for MoCov2 and BYOL, with 20% margins and removal of boxes smaller than 1% of image size.The box-based training setup is described for ResNet-18 and ResNet-50.Source: §B.2 · p. 16
  • Transfer is evaluated with linear classifiers on COCO-Crop and six other datasets, while detection is fine-tuned and evaluated on COCO.The classification evaluations include CIFAR-10, CIFAR-100, CUB, Flowers, Food, and Pets.Source: §B.2 · p. 16

B.3 Implementation details for background bias results

The background- and distribution-shift evaluations use ImageNet-9 and its corresponding shifted subsets. The supplied visualization reports that removing negative signals improves ContraCAM localization.

B.3 Implementation details for background bias results

  • The appendix provides visual examples of the Background Challenge and dog-class distribution-shifted ImageNet datasets.These examples appear in Figures 5 and 6.Source: Figure 5 · p. 17, Figure 6 · p. 17
  • The shift experiments train on the ORIGINAL Background Challenge dataset, a nine-superclass subset of ImageNet, and evaluate on the Background Challenge and shifted ImageNet-9 subsets.The shifted sets include ImageNet-Sketch-9, Stylized-ImageNet-9, ImageNet-R-9, and ImageNet-C-9.Source: §B.3 · p. 17

C.1 Visualization of ContraCAM without negative signal removal

  • Removing negative signals improves localization because similar-object negatives can cancel positive signals and scatter activation across random locations.Figure 7 illustrates ContraCAM without negative signal removal.Source: §C.1 · p. 18, Figure 7 · p. 18

C.2 Ablation study on the number of iterations

ContraCAM's iterative refinement needs enough passes to recover broader object regions, while larger iteration counts converge but may reduce the best mask score.

C.2 Ablation study on the number of iterations

  • Too few iterations often detect only subregions or miss objects; about five iterations are noted as sufficient.The ablation reports mask mIoU across iteration counts.Source: §C.2 · p. 18, Table 9 · p. 18
  • At larger iteration counts, ContraCAM stabilizes and converges toward stationary values.The passage gives 20 iterations as an example of a large count.Source: §C.2 · p. 18
  • Twenty iterations slightly harms the best reported value, which occurs at 10 iterations.Source: §C.2 · p. 18

C.3 Ablation study on the choice of negative batch

The negative-batch ablation finds that small random batches nearly match larger ones, while ContraCAM can outperform classifier CAM in ImageNet-to-CUB transfer.

C.3 Ablation study on the choice of negative batch

  • A pathological negative batch of similar objects can make ContraCAM highlight background, but such selection is described as rare in practice.The method assumes negatives have similar backgrounds but different objects.Source: §C.3 · p. 19, Figure 8 · p. 19
  • A random negative batch of size 4 almost matches the performance of size 64; the experiments use size 64.Table 10 reports mask mIoU across negative batch sizes.Source: §C.3 · p. 19, Table 10 · p. 19

C.4 Comparison with the Classifier CAM

  • ContraCAM on the MoCov2 backbone outperforms classifier CAM on that same backbone for ImageNet-to-CUB transfer.The comparison also reports that ContraCAM often outperforms the supervised classifier CAM in transfer settings, including CUB.Source: §C.4 · p. 20, Figure 9 · p. 20
  • The double expansion trick helps MoCov2 but degrades the supervised classifier, so it is applied only to MoCov2 results in Table 2.The authors attribute MoCov2's robustness to stronger training augmentations.Source: §C.4 · p. 20, Table 11 · p. 20

C.5 Comparison with the gradient-based saliency methods

The comparison finds that ContraCAM produces smooth, interpretable maps, while gradient-based saliencies produce sparse points and larger regions. In MoCov2, ground-truth-box OA-Crop is unstable and often underperforms the original images, whereas ContraCAM boxes are more stable.

C.5 Comparison with the gradient-based saliency methods

  • Integrated Gradients averages gradients along image-to-zero interpolations, while SmoothGrad averages gradients from Gaussian-noise-perturbed images.Both methods average ten gradients and use the same contrastive score as ContraCAM.Source: §C.5 · p. 21
  • ContraCAM maps are smooth and interpretable, while gradient-based saliencies yield sparse points that are harder to aggregate into segmentation masks.Figure 10 visualizes the methods; Table 12 reports their MaxBoxAccV2 results, but the supplied passage gives no values or winner.Source: §C.5 · p. 21, Figure 10 · p. 21, Table 12 · p. 21

D.1 Hard negative issue in MoCov2

  • Ground-truth-box OA-Crop often underperforms the baseline and is unstable, while ContraCAM-box OA-Crop is more stable.The OA-Crop (GT) training loss curve illustrates the optimization instability.Source: §D.1 · p. 22, Figure 11 · p. 22
  • Ground-truth boxes can include hard-to-distinguish objects; ContraCAM instead selects the most discriminative 1–3 objects.Figures 12 and 13 show the box-count histogram and a visual comparison with ground-truth boxes.Source: §D.1 · p. 22, Figure 12 · p. 22, Figure 13 · p. 22

D.2 Analysis on the contextual bias

The contextual-bias analyses find less entanglement and greater embedding distance for giraffe and zebra under OA-Crop (CAM), while vanilla MoCov2’s binary classification error rises over later epochs. The accompanying comparisons cover supervised models, class-wise CIFAR-10 accuracy, training datasets, a second ContraCAM iteration, and background mixup versus copy-and-paste.

D.2 Analysis on the contextual bias

  • OA-Crop (CAM) less entangles giraffe and zebra embeddings and yields a larger average minimum distance than original MoCov2.The t-SNE visualization and Table 13 correspond to these embedding comparisons.Source: §D.2 · p. 23, Figure 14 · p. 23, Table 13 · p. 23
  • Vanilla MoCov2’s giraffe-versus-zebra classification error rises in later epochs, while OA-Crop shows consistent results.The reported error is a five-seed average of 1-shot binary classification error on fixed representations.Source: §D.2 · p. 23, Table 14 · p. 23

D.3 Comparison with supervised models

  • Supervised Faster R-CNN and Mask R-CNN learn better representations than the self-supervised models, while BYOL with ground-truth boxes matches their linear-evaluation performance.Tables 15 and 16 cover linear evaluation and COCO detection/segmentation, respectively.Source: §D.3 · p. 24, Table 15 · p. 24, Table 16 · p. 24

D.4 Class-wise accuracy on CIFAR-10

  • OA-Crop does not degrade class-wise CIFAR-10 performance on particular classes in the considered transfer scenario.Table 17 reports class-wise linear evaluation accuracy.Source: §D.4 · p. 25, Table 17 · p. 25

D.5 Comparison with the ImageNet-trained models

  • ImageNet-trained models outperform COCO-trained models on the compared tasks overall, although COCO-trained models perform better on COCO-Crop.The comparison uses a 10% ImageNet subset of roughly 100,000 samples and ResNet-18 models.Source: §D.5 · p. 25, Table 18 · p. 25

D.6 Second iteration using the CAM from the debiased models

  • Using ContraCAM inferred from debiased models in a second iteration provides no additional gain over the original-model iteration.Table 19 compares the two iterations.Source: §D.6 · p. 25, Table 19 · p. 25

E.1 Comparison with the copy-and-paste augmentation

  • BG-Mixup (CAM) has better accuracy and generalization than binarized-CAM copy-and-paste BG-HardMix in the reported comparisons.The authors attribute this to soft blending, which accounts for uncertain masks and adds mixup regularization.Source: §E.1 · p. 26, Table 20 · p. 26

E.2 Ablation study on the mixup probability

BG-Mixup improves performance on background and distribution shifts, with results varying by comparison and benchmark. Its gains are consistent across the tested application probabilities, and its soft blending compares favorably with copy-and-paste in the reported settings.

E.2 Ablation study on the mixup probability

  • BG-Mixup gives consistent gains across the tested probabilities; the selected values are 0.4 for MoCov2 and 0.3 for BYOL.These probabilities performed best for most datasets in the Background Challenge; Table 21 reports the probability ablation.Source: §E.2 · p. 26, Table 21 · p. 26

E.3 ContraCAM vs. GT masks on the distribution shifts

  • BG-Mixup (CAM) performs better than BG-HardMix (GT) on distribution shifts, while BG-HardMix (GT) performs better for background shifts.BG-Mixup’s reported advantage is attributed to object-centric learning and input interpolation; Table 22 covers distribution-shift test accuracy.Source: §E.3 · p. 27, Table 22 · p. 27

E.4 Mixup and CutMix on the background shifts

  • BG-Mixup is effective on both background and distribution shifts, whereas Mixup and CutMix are not effective on the Background Challenge benchmarks.Table 23 compares test accuracy on background shifts.Source: §E.4 · p. 27, Table 23 · p. 27

E.5 Corruption-wise results on ImageNet-C-9

BG-Mixup (CAM) has the strongest overall reported ImageNet-C-9 performance, with especially strong results on weather and digital corruptions. Additional evaluations report advantages over vanilla MoCov2/BYOL on ObjectNet-9 and SI-Score-9, with different outcomes against BG-HardMix across those datasets.

E.5 Corruption-wise results on ImageNet-C-9

  • 54.30% (+29.89%) for BG-Mixup on weather and digital corruptions, up from the 24.41% baseline; BG-Mixup (CAM) has the overall best ImageNet-C-9 performance.It performs less well on noise corruptions; the authors note weather and digital corruptions require more object understanding than noise.Source: §E.5 · p. 27, Table 24 · p. 28

E.6 Results on additional datasets

  • BG-Mixup outperforms vanilla MoCov2/BYOL on ObjectNet-9 and SI-Score-9, but BG-Mixup (CAM) is better than BG-HardMix (GT) on ObjectNet-9 and less effective on SI-Score-9.ObjectNet-9 represents distribution shift and SI-Score-9 background shift in the reported evaluation.Source: §E.6 · p. 29, Table 25 · p. 29
Loading 2108.00049v2…