PaperlayerView exact revision on arXiv

Source-linked AI summary

What Should Not Be Contrastive in Contrastive Learning

Tete Xiao, Xiaolong Wang, Alexei A. Efros, Trevor Darrell

arXiv:2008.05659v2cs.CV

TL;DR

Augmentation-derived invariances can discard information needed by downstream tasks, and current contrastive methods assume those invariances in advance. The paper introduces Leave-one-out Contrastive Learning, which separates augmentation-varying and invariant information across embedding spaces. Its shared representation and concatenated task-specific heads outperform baselines across coarse-grained, fine-grained, few-shot, and corrupted-data classification, with the concatenation performing best overall.

  • Problem

    Current contrastive methods assume downstream task invariances, but augmentation-induced invariance can harm tasks that require the discarded information.

  • Method

    Leave-one-out Contrastive Learning constructs multiple embedding spaces, each sensitive to one augmentation and invariant to the others, using a shared backbone and multiple heads.

  • Results

    The shared representation and concatenated task-specific heads outperform all baselines, with concatenation providing the greatest overall performance across evaluated tasks.

  • Takeaways & Limitations

    Representations can preserve augmentation-varying and invariant factors for transfer across coarse-grained, fine-grained, few-shot, and corrupted-data classification tasks.

  • Takeaways & Limitations

    The general embedding space must preserve information about all augmentations to optimize the combined learning objectives.

Abstract

from arXiv · show

Recent self-supervised contrastive methods have been able to produce impressive transferable visual representations by learning to be invariant to different data augmentations. However, these methods implicitly assume a particular set of representational invariances (e.g., invariance to color), and can perform poorly when a downstream task violates this assumption (e.g., distinguishing red vs. yellow cars). We introduce a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances. Our model learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but one augmentation. We use a multi-head network with a shared backbone which captures information across each augmentation and alone outperforms all baselines on downstream tasks. We further find that the concatenation of the invariant and varying spaces performs best across all tasks we investigate, including coarse-grained, fine-grained, and few-shot downstream classification tasks, and various data corruptions.

1 INTRODUCTION

Augmentation-based invariances can help some downstream tasks while harming others, motivating representations that preserve factors of variation without assuming task-specific invariances. The proposed model uses shared and separate embedding spaces, with strong transfer results across diverse tasks.

  • Motivation: Augmentation-induced invariance can benefit some tasks but harm others when discarded information is task-essential.Rotation may help view-independent aerial recognition but hurt determining which way is up; color can matter in fine-grained bird classification.
  • Approach: The model learns separate sub-spaces that remain sensitive to specific augmentations while invariant to the others.A multi-head architecture with a shared backbone jointly preserves augmentation-varying and invariant information.
  • Results: Both the shared backbone representation and concatenated task-specific heads outperform all baselines, with concatenated heads providing the greatest overall performance.The shared representation uses the same embedding dimensions as typical baselines.
  • Results: The method shows consistent gains as augmentations increase and transfers across large-scale, fine-grained, few-shot, and corrupted-data classification.It also avoids hand-selection of augmentation strategies and demonstrates transferability, generalizability, and robustness across tasks and categories.

2 BACKGROUND: CONTRASTIVE LEARNING FRAMEWORK

Contrastive learning matches augmented views of the same image while separating views from different images, but augmentation choices impose invariances that can discard task-relevant information. The paper motivates separate embedding spaces to retain such information selectively.

  • Contrastive objective: Contrastive learning maximizes similarity for similar pairs and dissimilarity for dissimilar pairs.A reference image produces query and key views, with the query matched against its designated positive key and sampled negatives.
  • Augmentations: The augmentation module applies n atomic operators, sampling parameters to transform an image into random views.Examples include random cropping, color jittering, and random flipping.
  • Contrastive objective: The temperature parameter τ scales the distribution of distances in the contrastive objective.This parameter controls the scaling applied to distance-based comparisons.
  • Proposed direction: Leave-one-out contrastive learning addresses this issue by using separate spaces that are invariant to all augmentations or all but one.The general embedding V and concatenated sub-spaces Z are proposed alternatives that avoid decreased performance from task-mismatched augmentations.
  • Inductive bias: Augmentation choices impose invariance to transformations, which may harm generalizability when discarded information is essential.The passage highlights color for fine-grained bird classification and notes that adding augmentations can help some classes while harming others.

3 LOOC: LEAVE-ONE-OUT CONTRASTIVE LEARNING

LooC preserves augmentation-specific information by learning multiple embedding spaces with distinct invariance patterns, while sharing a general representation across them. Its contrastive training uses matched augmentation parameters to make each sub-space sensitive to one augmentation and invariant to the others.

  • Contrastive Embedding Space: LooC projects images into multiple embedding spaces, each sensitive to one augmentation while invariant to the others.The shared layers retain both augmentation-varying and invariant information.
  • View Generation: Additional key views copy the query’s parameter for one augmentation while independently sampling the remaining augmentation parameters.This construction isolates which augmentation each embedding space should preserve.
  • Contrastive Embedding Space: The model creates a general space invariant to all augmentations and additional spaces specialized to individual augmentations.The general space is denoted V, while the sub-spaces are Z0, Z1, …, Zn.
  • Contrastive Embedding Space: Contrastive objectives treat views sharing the selected augmentation as positives and other views from the same instance as negatives in that specialized space.This trains each space to be sensitive to its designated augmentation and insensitive to the others.
  • Learnt representations: The downstream representation can use the shared general space alone or concatenate it with all specialized embedding sub-spaces.The concatenated implementation is termed LooC++.

4 EXPERIMENTS

Experiments evaluate LooC across downstream classification, few-shot recognition, corruption robustness, retrieval, and ablations, showing that separating augmentation-dependent and invariant information improves transfer across varied tasks.

  • Experimental setup: LooC is evaluated with color jittering, random rotation, and texture randomization across large-scale, fine-grained, few-shot, and corrupted-data classification tasks.Random-resized cropping, horizontal flipping, and Gaussian blur are used without designated embedding spaces.
  • Downstream transfer: LooC outperforms or matches corresponding MoCo baselines across diverse downstream tasks, with gains increasing as additional augmentations are included.The method trails original MoCo on none of the reported fine-grained datasets and improves performance when multiple augmentations are learned jointly.
  • Robustness: Rotation augmentation benefits ON-13 but harms IN-C-100 robustness, whereas texture randomization improves IN-C-100, especially for Blur and Weather at severity levels at least 3.Combining rotation and texture improves both datasets, and LooC++ further improves IN-C-100 performance.
  • Representation preservation: LooC better preserves information discarded by invariant contrastive training, including color, rotation, and texture-dependent factors.Nearest-neighbor retrieval and downstream evaluations support preservation of augmentation-dependent information.
  • Ablations: Multiple heads improve baseline MoCo, while LooC remains better or comparable and avoids the catastrophic rotation-information loss observed for standard invariant training.Explicit feature fusion provides additional gains, particularly on iNat-1k.
  • Feature fusion: Concatenating invariant, color-, rotation-, and texture-dependent features improves IN-100, iNat-1k, and Flowers-102, but texture features reduce IN-C-100 performance.Different downstream tasks prefer different mixtures of augmentation-dependent and invariant representations.

5 RELATED WORK

Prior work combines visual cues, uses multiple pretext tasks, or learns invariance to augmentation-derived transformations. Multi-task self-supervision can suffer from conflicting objectives, while task-specific weighting may require labels and fail to generalize.

  • Feature design combined cues such as brightness, color, and texture for image retrieval and recognition.
  • Multi-Task Self-Supervised Learning: Jointly optimizing multiple self-supervised tasks can hurt performance when their objectives conflict.
  • Multi-Task Self-Supervised Learning: Learning task-specific weights can address conflicting objectives but typically requires labels, is time-consuming, and may not generalize across tasks.
  • Contrastive learning trains representations to be invariant to transformations induced by corresponding augmentations.

6 CONCLUSIONS

Existing contrastive methods may presume unsuitable transformation invariances for downstream tasks. The proposed framework learns both transformation-dependent and invariant representations through multiple embeddings and reports improvements across diverse classification and corruption settings.

  • The framework constructs multiple embeddings, each non-contrastive to one transformation, to retain both transformation-dependent and invariant information.
  • The method outperforms a baseline contrastive method on coarse-grained, fine-grained, and few-shot classification tasks.
  • The framework demonstrates better robustness to real-world data corruptions than the baseline contrastive method.

A AUGMENTATION DETAILS

The augmentation policy uses specified probabilities and strengths for color jittering, random scale, rotation, and texture randomization.

  • Color jittering is applied with probability 0.8 using brightness, contrast, saturation, and hue strengths of 0.4, 0.4, 0.4, and 0.1.
  • Random scale is applied with probability 0.2.
  • Random rotation and texture randomization are each applied with probability 0.5.

B DATASETS

The study evaluates datasets spanning large-scale, fine-grained, few-shot, viewpoint-shifted, and corruption-focused recognition settings.

  • iNat-1k contains 1,010 species and 268,243 combined training and validation images.
  • CUB-200 contains 5,994 training images and 5,794 testing images covering 200 bird species.
  • Flowers-102 contains 102 flower categories with between 40 and 258 images per category.
  • ObjectNet tests objects from new viewpoints, new backgrounds, and different rotations, using 13 categories overlapping with IN-100.
  • ImageNet-C applies 15 diverse corruption types to ImageNet validation images.

C LINEAR CLASSIFICATION

Linear-classification training uses dataset-specific schedules and optimization settings. ImageNet-100 and CUB-200 use momentum SGD, while Flowers-102 uses Adam.

  • ImageNet-100 and CUB-200 linear layers are trained for 200 epochs with momentum SGD.The learning rate is 30, reduced by 0.1 at 60% and 80% of training.
  • iNat-1k linear-layer training lasts 100 epochs with momentum SGD at learning rate 30.The learning rate is reduced by 0.1 at 60% and 80% of the schedule.
  • Flowers-102 uses Adam for 250 iterations with learning rate 0.03.

D LEAVE-ONE-OUT VS. ADD-ONE AUGMENTATION

The section compares leave-one-out and add-one augmentation strategies. Add-one keeps the query image unaugmented while applying a designated augmentation to each augmentation-specific view.

  • Table 6 compares leave-one-out and add-one augmentation strategies, with the default strategy marked by an asterisk.
  • In add-one augmentation, the query image remains unaugmented while each augmentation-specific view receives its designated augmentation.
  • Add-one augmentation oversimplifies instance discrimination, such as when leaving color augmentation out of the query view makes the task very easy.

E IMAGENET-1K EXPERIMENTS

The experiments train models on 1000-category ImageNet and fine-tune them on iNat-1k using linear classification. The compared representations are dimension-matched, and the authors report strong baseline behavior on the ImageNet-100 subset.

  • Models are trained self-supervised on 1000-category ImageNet and fine-tuned on iNat-1k following a linear-classification protocol.The model uses color, rotation, and texture augmentations, with results reported in Table 7.
  • MoCo and LooC representations each have dimension 2048, while MoCo++ and LooC++ each have dimension 2048 * # augmentations for fair comparison.
  • The baseline was first made as strong as possible before applying the same hyperparameters to the proposed method, and the ImageNet-100 subset behaved similarly to ImageNet-1k.The baseline is reported to outperform the best prior method on the same subset, at 75.8% CMC versus 81.0% top1 [our.
Loading 2008.05659v2…