PaperlayerView exact revision on arXiv

Source-linked AI summary

Can contrastive learning avoid shortcut solutions?

Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, Suvrit Sra

arXiv:2106.11230v3cs.LG

TL;DR

Contrastive learning may suppress predictive features through shortcut solutions because InfoNCE does not fully determine which features are learned. The paper studies how instance-discrimination difficulty shapes feature learning and proposes implicit feature modification, which reduces suppression and improves downstream performance across vision and medical imaging tasks.

  • Problem

    Contrastive learning can suppress predictive features through shortcut solutions, while the instance-discrimination task influences which features become discriminative and learned.

  • Method

    Implicit feature modification alters contrastive representations by removing well-represented discriminative features, encouraging encoders to use additional input features.

  • Results

    Implicit feature modification reduces feature suppression and improves generalization across vision and medical imaging tasks.

  • Takeaways & Limitations

    Changing the instance-discrimination task can control feature learning, but IFM is designed to broaden represented predictive features without the usual trade-offs.

  • Takeaways & Limitations

    The study focuses on contrastive learning; whether other successful self-supervised methods exhibit different feature-learning biases remains open.

Abstract

from arXiv · show

The generalization of representations learned via contrastive learning depends crucially on what features of the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, a behavior that can negatively impact the performance on downstream tasks via "shortcuts", i.e., by inadvertently suppressing important predictive features. We find that feature extraction is influenced by the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation of some features, the improvement comes at the cost of suppressing previously well represented features. In response, we propose implicit feature modification (IFM), a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks. The code is available at: \url{https://github.com/joshr17/IFM}.

1 Introduction

Contrastive learning can learn shortcut representations that suppress predictive features, and changing instance-discrimination difficulty may trade one represented feature for another. The paper proposes implicit feature modification to encourage multiple features and improve generalization.

  • Motivation: Contrastive encoders discriminate similar from dissimilar pairs, but training-discriminative features do not necessarily support generalization.Shortcut features can produce unpredictable behavior under perturbations and failure outside the training distribution.
  • Problem: InfoNCE optimization does not guarantee avoidance of shortcut solutions that suppress certain input features.The paper analyzes why feature suppression can occur despite optimizing the contrastive objective.
  • Problem: Adjusting instance-discrimination difficulty changes which features are learned, but improving one feature often harms another.These trade-offs leave feature suppression prevalent.
  • Approach: Implicit feature modification encourages encoders to discriminate instances using multiple input features without computational overhead.The method is presented as a response to trade-offs caused by simpler difficulty adjustments.
  • Outcome: The paper reports that implicit feature modification reduces feature suppression and improves generalization on downstream tasks.The introduction frames these improvements across vision and medical imaging applications.

2 Feature suppression in contrastive learning

Contrastive learning can suppress predictive features because InfoNCE optimization does not uniquely determine which features encoders retain. Instance-discrimination difficulty shifts feature emphasis, but harder discrimination can trade representation of one feature for another.

  • Feature suppression: Feature suppression occurs when a model uses only a subset of multiple predictive input features, often favoring simpler features such as color over shape.Such shortcut features can harm generalization and robustness under data shifts.
  • InfoNCE and shortcuts: InfoNCE has optimal encoders that either suppress or distinguish a feature, so minimizing the loss alone does not guarantee avoidance of shortcuts.This result holds under the stated uniform-marginal assumption.
  • InfoNCE and shortcuts: Lower InfoNCE loss improves color and MNIST-digit prediction but can increase shape, texture, and STL10 error, depending on which shortcuts are used.Downstream task errors can therefore be negatively correlated across tasks.
  • Controlling feature learning: Easier instance discrimination favors color, whereas harder discrimination favors shape and texture, with the latter occurring at the expense of color.The relevant controls are temperature τ and hardness concentration β.
  • Controlling feature learning: Hard examples expose features absent from the current representation, but concentrating on them can suppress the features previously used by the encoder.This trade-off motivates implicit feature modification.

3 Implicit feature modification for reducing feature suppression

IFM modifies encoded positive and negative samples to remove currently exploited features, encouraging contrastive encoders to learn complementary predictive features without feature-suppression trade-offs. Its analytic embedding-space perturbations can alter high-level semantics while retaining efficient computation.

  • Motivation: Difficulty adjustments can change which features contrastive models represent, but features held constant across pairs become suppressed, creating trade-offs.Harder or easier instance discrimination shifts the learned feature balance rather than reliably broadening representation.
  • Method: IFM adaptively removes features used to discriminate each pair, then trains on both modified and original representations to encourage multiple-feature discrimination.The method optimizes the combined objective (L + αLε)/2, with experiments generally setting α = 1.
  • Method: Embedding-space perturbations modify high-level semantic features, whereas input-space adversarial methods focus on lower-level features and require costly inner-loop optimization.The paper describes IFM as analytically and efficiently computable, avoiding the optimization overhead associated with PGD-style approaches.
  • Objective: IFM modifies contrastive logits by reducing the positive logit by ε+/τ and increasing negative logits by ε_i/τ.The resulting objective reweights positive samples by e^(-ε+/τ) and each negative sample by e^(ε_i/τ).
  • Visualization: For positive samples, increasing ε moves semantics away from the anchor; for negative samples, it moves them toward the anchor in the illustrated vehicle examples.The visualized changes include vehicle color, pose, orientation, and type, demonstrating semantic alteration through embedding perturbations.

4 Experimental results

Experiments evaluate IFM on synthetic feature-suppression settings, object classification, and medical imaging using established contrastive frameworks. Across these settings, IFM improves representation of multiple features and downstream performance, although some medical-imaging features are not statistically significant.

  • Feature suppression: IFM improves linear readout accuracy across all three Trifeature attributes for every tested temperature, unlike temperature tuning or hard negatives, which trade off represented features.The comparison uses ResNet-18 encoders trained for 200 epochs, with τ ∈ {0.05, 0.2, 0.5} and IFM ε = 0.1.
  • Classification: Across eight classification runs, Lε averages 76.0%, the standard contrastive loss averages 75.9%, and the combined IFM loss performs best.The results suggest that the standard and perturbed objectives learn complementary features; IFM also improves ImageNet100 performance over MoCo-v2 by 0.9%.
  • Medical imaging: IFM improves all COPDGene phenotype predictions, with the largest gain on spirometry logFEV1pp: 8.7% at ε = 0.1.The table reports five-fold cross-validation averages with standard deviations, and the authors note that not all features improve significantly.
  • Robust features: On standard test data, IFM generalizes better than SimCLR by representing robust features more effectively.The robust and non-robust datasets are constructed from encoder-specific FGSM perturbations, and the test set itself is unperturbed.

5 Discussion

The paper studies how contrastive instance discrimination shapes feature learning, while noting that extending this analysis beyond contrastive learning remains an open direction.

  • The analysis is specific to contrastive learning and does not examine feature learning in other self-supervised methods.

A Proofs for Section 2

The proofs establish that InfoNCE can have optimal encoders that suppress features, and that conditioning instance discrimination on shared features forces their suppression. These results motivate IFM, which modifies embeddings to encourage learning previously suppressed features.

  • Definitions: Suppression means the encoder’s output distribution is invariant to the value of the suppressed feature, whereas distinction assigns different values disjoint output regions.
  • Theoretical results: InfoNCE can have equally optimal encoders that either suppress or discriminate any given feature.
  • Implicit feature modification: IFM removes information used by the current embedding for discrimination, then asks the model to discriminate using both original and modified embeddings.
  • Theoretical results: A feature set shared across positive and negative pairs cannot help instance discrimination, so InfoNCE minimizers suppress it under the proposition’s conditions.
  • Implicit feature modification: Analytic InfoNCE gradients enable IFM’s adversarial embedding updates without additional memory or runtime overhead.
  • Alternative formulations: Alternative normalized IFM formulations remain computationally cheap, but their pre-normalization variant requires iterative approximation because its gradients have additional dependencies.

C.2 Feature suppression experiments

The feature-suppression experiments use synthetic datasets with controlled semantic features, enabling direct study of whether contrastive representations retain or suppress those features.

  • STL-digits: Sample STL-digits images contain two features—object class and MNIST digit—with 10 possible values for each.
  • Trifeature: Trifeature images combine color, shape, and texture, each taking 10 possible values, across 1000 feature triplets and 100 examples per triplet.
  • STL-digits: STL-digits combines MNIST digits with STL10 objects by placing digit copies across a 3 × 3 grid over each STL10 image.

C.2.2 Experimental protocols

The protocols train ResNet-18 encoders with SimCLR and evaluate their representations using linear classifiers under specified augmentation and optimization settings.

  • Training: ResNet-18 encoders are trained with SimCLR using batch size 512, Adam, learning rate 1 × 10^-3, weight decay 1 × 10^-6, and temperature 0.5.
  • Training: Trifeature training removes grayscaling and color jittering to avoid corrupting color features, while retaining standard SimCLR augmentations otherwise.
  • Linear evaluation: Linear evaluation extracts encoder features and trains a scikit-learn logistic-regression classifier with L-BFGS for at most 500 iterations.

C.2.3 Details on results

On STL-digits, IFM improves suppressed STL10 features without harming MNIST performance, unlike harder instance-discrimination settings that trade one feature for another.

  • IFM improves suppressed STL10 performance while only slightly increasing it, and does not harm MNIST performance in the process.This contrasts with hard instance-discrimination methods, which improve STL10 at MNIST's expense.
  • The STL-digits experiments use one training run per hyperparameter setting because performance is more stable than on Trifeature.Figure 12 shows that color-prediction linear readout is particularly noisy for Trifeature.

C.3 Comparing IFM and ACL(DS)

The experiments compare IFM with ACL(DS) using matched training settings and evaluate standard and robust linear readout. IFM improves standard accuracy without added memory or computation, whereas ACL(DS) improves adversarial performance at a substantial cost to standard performance and training resources.

  • STL-digits: On STL-digits, harder instance discrimination improves STL10 performance at the expense of MNIST, while IFM enhances both features simultaneously.When discrimination is too easy, STL10 features can be suppressed below their initialization readout.
  • Evaluation: The CIFAR10 and CIFAR100 evaluation protocols report both standard accuracy and robust accuracy using clean-image linear evaluation and adversarial finetuning.Training uses SimCLR with a ResNet-18 backbone for 1000 epochs; ACL(DS) adds five PGD steps during pretraining.
  • Results: IFM improves standard linear readout accuracy with zero memory or compute-time cost increase, whereas ACL(DS) improves adversarial readout but worsens standard readout.The comparison uses the official ACL implementation while changing only the loss function for IFM, with shared hyperparameters.
  • Results: ACL(DS) requires 2× memory and 6× time per epoch, while IFM does not increase these costs.This supports the paper's distinction between the problems addressed by the two methods.

C.4 Object classification experiments

The object-classification experiments evaluate IFM with SimCLR, MoCo-v2, and AdCo under fixed representation-learning and linear-evaluation protocols. On ImageNet100, IFM raises MoCo-v2 top-1 linear readout from 80.5% to 81.4%.

  • Experimental protocol: The study evaluates CIFAR10, CIFAR100, STL10, and tinyImageNet using fixed objective weighting and repeated validation runs for several datasets.MoCo-v2 uses 5-fold cross-validation on CIFAR10/CIFAR100 and three replicated runs on tinyImageNet/STL10.
  • Training: Encoders use ResNet-50 backbones trained for 400 epochs, with SimCLR and MoCo-v2 differing in temperature and negative-sample construction.SimCLR uses temperature τ = 0.5 and batch negatives, whereas MoCo-v2 uses τ = 0.1 and a queue.
  • Linear evaluation: Representation quality is measured by test performance from a linear classifier trained for 100 epochs while keeping the embedding model fixed.SimCLR and MoCo-v2 use separate optimizer, batch-size, normalization, and augmentation settings for linear evaluation.
  • Baselines: AdCo comparisons use official code, the same training hyperparameters as MoCo-v2, and grid search over negative learning rate and temperature.The search includes lrneg ∈ {1, 2, 3, 4} and τneg ∈ {0.02, 0.1}.
  • Results: 81.4% top-1 linear readout is achieved by IFM-MoCo-v2 versus 80.5% for standard MoCo-v2 on ImageNet100.The reported standard MoCo-v2 result uses 200-epoch training and is described as state of the art for that setting.

C.6 Further discussion of feature robustness experiments (Sec. 4.3)

IFM differs from supervised adversarial training by operating in feature space rather than using input-space attacks. The reported behavior suggests improved generalization is associated with stronger robust-feature extraction, while IFM has little effect on non-robust features.

  • Comparison: IFM operates in feature space, unlike supervised adversarial training, which uses input-space adversarial attacks.The paper links this conceptual difference to different behavior in feature learning.
  • Feature robustness: IFM enhances robust features rather than suppressing non-robust features, suggesting its improved generalization is associated with features aligned with human semantics.The paper explicitly notes that IFM has no significant effect on learning non-robust features.
  • Scope: The paper discusses combining IFM with adversarial training to potentially obtain complementary benefits.This proposal follows the observation that IFM does not significantly affect non-robust feature learning.

D Discussion of limitations and possible extensions

The proposed method has two stated limitations: it targets high-level semantic features rather than pixel-level shortcuts, and it does not immediately reveal which features are removed during modification.

  • Scope boundary: Because implicit feature modification acts on embedded points instead of raw inputs, it is not well suited to ℓp robustness or removing pixel-level shortcut solutions.The method instead focuses on high-level semantic features.
  • Possible extension: The authors identify combining their high-level method with existing pixel-level methods as a valuable direction for future study.
  • Interpretability limitation: Implicit feature modification does not immediately determine which features are removed during modification, meaning the features used for instance discrimination remain unclear.The authors suggest manually studying examples with their visualization technique as one option.
Loading 2106.11230v3…