PaperlayerView exact revision on arXiv

Source-linked AI summary

Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations

Michael Zhang, Nimit S. Sohoni, Hongyang R. Zhang, Chelsea Finn, Christopher Ré

arXiv:2203.01517v2cs.LG

TL;DR

Spurious correlations can make ERM models rely on attributes that fail across groups, while group labels needed by oracle methods may be unavailable. CNC infers spurious differences from ERM outputs and uses contrastive learning to align representations of same-class samples. It achieves state-of-the-art or near-state-of-the-art worst-group accuracy, including a 3.6% average absolute lift, while approaching methods that require group labels.

  • Problem

    Spurious correlations can cause ERM models to perform well on average but poorly on groups lacking those correlations, especially when spurious-attribute labels are unavailable.

  • Method

    CNC uses ERM predictions to identify same-class samples with different inferred spurious features and contrastively aligns their representations while separating different-class samples.

  • Results

    CnC achieves state-of-the-art or near-state-of-the-art worst-group accuracy across benchmarks, with a 3.6% average absolute lift over the second-best method.

  • Takeaways & Limitations

    Representation alignment is linked to worst-group robustness, and CNC improves class dependence while reducing reliance on spurious attributes without training group labels.

  • Takeaways & Limitations

    The setup assumes at least one observed training example from each class–spurious-attribute group, and regularization is used to prevent ERM memorization when inferring attributes.

Abstract

from arXiv · show

Spurious correlations pose a major challenge for robust machine learning. Models trained with empirical risk minimization (ERM) may learn to rely on correlations between class labels and spurious attributes, leading to poor performance on data groups without these correlations. This is particularly challenging to address when spurious attribute labels are unavailable. To improve worst-group performance on spuriously correlated data without training attribute labels, we propose Correct-N-Contrast (CNC), a contrastive approach to directly learn representations robust to spurious correlations. As ERM models can be good spurious attribute predictors, CNC works by (1) using a trained ERM model's outputs to identify samples with the same class but dissimilar spurious features, and (2) training a robust model with contrastive learning to learn similar representations for same-class samples. To support CNC, we introduce new connections between worst-group error and a representation alignment loss that CNC aims to minimize. We empirically observe that worst-group error closely tracks with alignment loss, and prove that the alignment loss over a class helps upper-bound the class's worst-group vs. average error gap. On popular benchmarks, CNC reduces alignment loss drastically, and achieves state-of-the-art worst-group accuracy by 3.6% average absolute lift. CNC is also competitive with oracle methods that require group labels.

1 Introduction

Spurious correlations can make ERM models rely on group-specific features, producing high worst-group error despite strong average performance. CnC addresses this without group labels by aligning same-class representations across inferred spurious differences and achieves strong benchmark performance.

  • Motivation: Spurious background features can let ERM achieve low average error while misclassifying minority groups.The cow-versus-camel example illustrates how background correlations can replace the desired class features.
  • Motivation: Prior label-free methods improve over ERM but remain weaker than group-label methods such as GDRO.
  • Representation alignment: Alignment between same-class representations across spurious attributes tracks worst-group performance, and class-level alignment loss helps upper-bound the worst-group versus average error gap.
  • Approach: CnC uses ERM predictions to find same-class samples with different inferred spurious attributes, then contrasts them as hard positives while separating different-class hard negatives.This two-stage strategy improves intra-class alignment while preserving inter-class separability.
  • Results: 7.7% absolute lift raises CelebA worst-group accuracy from 81.1% to 88.8%, while the average lift over the second-best method is 3.6%.CnC also falls only 0.9 points short of GDRO on average, despite not requiring training group labels.
  • Results: CnC achieves state-of-the-art worst-group accuracy across three benchmarks and learns representations more dependent on class labels than spurious features.Ablations also report robustness to noisy ERM predictions and improvements from the proposed sampling strategy.

2 Preliminaries

The preliminaries formalize worst-group robustness when each example has an unobserved spurious attribute and review ERM, GDRO, and supervised contrastive learning. The setup motivates methods that improve minority-group performance without training group labels.

  • Problem setup: Each example has an observed feature vector, class label, and unobserved spurious attribute; groups are class–attribute combinations.The framework assumes at least one training example from every group.
  • ERM and GDRO: ERM minimizes average training loss, but spurious correlations can produce high minority-group error despite low average error.
  • ERM and GDRO: GDRO directly targets worst-group error when training group labels are available, whereas this paper studies the harder label-free setting.
  • Contrastive learning: Supervised contrastive learning trains an encoder by bringing sampled positives closer than negatives in representation space.The objective uses normalized encoder outputs and a temperature hyperparameter.

3 Spurious correlations’ impact on learned data representations

The paper links worst-group performance to whether learned representations encode class labels rather than spurious attributes. Across ERM and Jtt, better worst-group accuracy tracks lower alignment loss and stronger class dependence.

  • Worst-group accuracy correlates inversely with class-specific representation alignment loss.The paper theoretically connects lower alignment loss with lower worst-group loss.
  • The alignment loss measures Euclidean proximity between representations of same-class samples from different spurious-attribute groups.Lower alignment loss means better within-class alignment.
  • ERM models trained with stronger spurious correlations show lower worst-group accuracy and higher alignment loss.Their representations also become more dependent on spurious attributes.
  • As spurious correlation increases, estimated I(A; Z) rises while estimated I(Y; Z) decreases.Worst-group accuracy drops especially when spurious-attribute dependence exceeds class-label dependence.
  • Jtt’s higher worst-group accuracy coincides with high I(Y; Z) and low I(A; Z).This suggests the representation-metric relationship also holds for an upsampling-based robust-training method.
  • The broader implication is that improving within-class alignment can reduce the gap between worst-group and average-group losses.Both representation metrics therefore motivate alignment-focused robustness methods.

4 Our approach: Correct-n-Contrast (CnC)

Correct-n-Contrast uses a regularized ERM model to infer pseudo-group information, then trains a robust model with contrastive learning. Its sampling scheme brings together same-class samples with different inferred spurious attributes while separating contrasting examples.

  • 4 Our approach: Correct-n-Contrast (CnC): CnC is a two-stage contrastive method designed to improve representation alignment and robustness without training group labels.Stage 1 infers spurious attributes with a regularized ERM model; Stage 2 trains the robust model.
  • Stage 2: Supervised contrastive learning: Stage 2 treats same-class samples with different inferred spurious attributes as positive pairs, encouraging aligned representations.The method samples these examples as distinct views of the same class.
  • 4 Our approach: Correct-n-Contrast (CnC): CnC reduces alignment loss and substantially improves worst-group accuracy, while alternative sampling strategies degrade performance.The method is also reported to tolerate noisy ERM predictions without requiring extremely accurate spurious-attribute predictions.
  • Stage 1: Inferring pseudo group labels: Stage 1 saves ERM predictions as pseudo-group labels for the training datapoints.CnC can use either argmax predictions or clusters of last-hidden-layer outputs; clustering performs better.
  • Optimization objective and updating procedure: CnC jointly optimizes supervised contrastive loss and cross-entropy loss while updating the encoder and full classifier.The balancing hyperparameter λ lies in [0, 1].

5 Experimental results

CnC improves worst-group accuracy across several spurious-correlation benchmarks and learns representations with lower spurious-attribute dependence. Its robustness persists under noisy ERM predictions, while ablations show that its sampling strategy is important.

  • Benchmark comparison: CnC achieves the highest worst-group accuracy among methods without group labels on CMNIST∗, Waterbirds, and CelebA, and near-SoTA accuracy on CivilComments-WILDS.The comparison includes ERM, GDRO, and recent methods that do not require group labels.
  • Benchmark comparison: CnC uniquely uses ERM predictions to learn robust representations through contrastive learning rather than only reweighting, upweighting, or invariance objectives.Its contrastive scheme encourages ignoring spurious attributes through positive and negative comparisons.
  • Representation analysis: CnC learns significantly higher alignment and lower mutual information with spurious attributes than ERM and Jtt, while retaining comparable mutual information with class labels.These representation properties correspond to the highest worst-group accuracy on Waterbirds and CelebA.
  • Representation analysis: As CMNIST∗ spurious correlation strengthens, CnC consistently achieves the lowest alignment loss and performs better in settings with pcorr > 0.99.CnC and Jtt maintain high worst-group accuracy where ERM fails.
  • Ablation studies: CnC’s accuracy degrades less than Jtt’s as noise increases in Stage 1 predictions, demonstrating greater robustness to imperfect spurious-attribute proxies.On Waterbirds and CelebA, Stage 1 ERM predicts the spurious attribute with 94.7% and 84.0% accuracy respectively.
  • Ablation studies: Removing hard positive or negative sampling lowers worst-group accuracy and produces less desirable representation metrics, generally including higher Lalign and greater spurious-attribute dependence.With spurious labels, CnC also improves worst-group accuracy over GDRO by 0.9% absolute on average.

6 Related work

Related work addresses spurious correlations either with known group labels or by inferring groups from an ERM model. CnC belongs to the latter line while introducing contrastive representation learning as its distinctive approach.

  • Methods using group labels: When group labels are known, prior methods balance groups through class balancing, importance weighting, or robust optimization.These approaches include GDRO-style robust optimization.
  • Methods without group labels: Methods without group labels commonly train an ERM model first, then use its predictions or representations to train a second robust model.Examples include clustering ERM representations, upweighting misclassified points, and inferring groups that violate invariant risk minimization.
  • Contrastive learning: The paper positions CnC as distinct because it uses ERM predictions to learn desirable representations through contrastive learning.This contrasts with prior invariance, input-transformation, and upweighting approaches.
  • Contrastive learning: Contrastive learning typically specifies similar positives and dissimilar negatives, with negatives often sampled uniformly and positives representing different views of an input.CnC adapts this framework to spurious-correlation robustness.

7 Conclusion

The paper presents CnC as a two-stage contrastive method for robust representations and connects representation alignment with worst-group performance. Across benchmarks, it reports strong worst-group accuracy and reduced spurious dependence.

  • Conclusion: CnC is a two-stage contrastive learning approach for representations robust to spurious correlations.The method is designed to improve robustness without requiring group labels.
  • Conclusion: The paper empirically observes and theoretically analyzes a connection between alignment and worst-group versus average-group losses.CnC improves representations by increasing class dependence and reducing spurious-attribute dependence.
  • Conclusion: CnC achieves state-of-the-art or near-state-of-the-art worst-group accuracy across several benchmarks.The conclusion summarizes the method’s reported benchmark performance without requiring group-label supervision.

A.1 Training setup

Correct-n-Contrast uses a two-stage procedure: an ERM model supplies predictions for contrastive batch construction, then a new model jointly optimizes supervised contrastive and cross-entropy objectives. Its sampling pairs same-class examples with differing ERM predictions and contrasts examples differing in class or inferred spurious attributes.

  • Two-stage training: Stage 1 trains an ERM model with cross-entropy, while Stage 2 trains a same-architecture model using supervised contrastive batches sampled from ERM predictions.The classifier also receives a cross-entropy loss during Stage 2.
  • Contrastive sampling: CnC samples same-class examples with different ERM predictions as hard positives and examples differing in class or inferred spurious attribute as negatives.The procedure infers likely group membership from whether the ERM model predicts an example correctly or incorrectly.
  • Contrastive sampling: Two-sided sampling swaps anchor and positive roles, increasing pairwise comparisons and providing additional hard-negative signal for class separation.Both directions push same-class positives together while contrasting them against negatives.
  • Objective: The method is designed to align representations within classes while separating classes and reducing reliance on spurious features.Its batches naturally provide hard positives and negatives based on ERM predictions.
  • Optimization: CnC jointly trains encoder and classifier layers with cross-entropy while balancing encoder contrastive learning using λ.The contrastive loss is computed directly on encoder outputs rather than an additional projection network.

B Omitted Proofs from Section 3.2

The theory connects within-class representation alignment to gaps between worst-group and average loss, while experiments link lower alignment loss and class-focused representations to stronger worst-group accuracy. CnC’s contrastive objective outperforms directly minimizing alignment loss because it also separates hard negatives.

  • Theoretical connection: Theorem 3.1 bounds the within-class worst-group versus average loss gap by alignment loss times a Lipschitz constant plus a concentration error term.The proof compares groups sharing a class label but having different spurious attributes.
  • Theoretical connection: The cross-class extension generalizes this relationship to worst-group loss across labels, with class-prior factors appearing in the bound.A balanced-class example shows the multiplicative factor can be tight asymptotically.
  • Empirical analysis: CnC’s default contrastive loss outperforms direct alignment-loss minimization on worst-group and average accuracy.Direct alignment minimization produces lower alignment but does not encourage separating hard negatives from anchors.
  • Representation analysis: CnC’s improved worst-group accuracy coincides with lower alignment loss, higher class mutual information, and lower spurious-attribute mutual information.Figure 10 reports these representation metrics alongside worst-group accuracy on increasingly spurious CMNIST* settings.
  • Representation analysis: In the 99.9% spurious-correlation regime, CnC uniquely learns representations with higher mutual information for class labels than for spurious attributes.Compared with Jtt, CnC has much lower dependence on the spurious attribute despite slightly lower estimated class mutual information in other settings.

C.3 Additional design choice ablations

Ablations show that CnC’s default design choices consistently improve performance, while domain-level alignment alone is insufficient for spuriously correlated benchmarks. The contrastive-loss balance remains effective across a broad λ range.

  • Design components: Default CnC achieves the highest worst-group and average accuracy across Waterbirds design ablations.The ablations vary projection heads, contrastive batch direction, and encoder-classifier training strategy.
  • Design components: One-sided contrastive comparisons reduce worst-group accuracy relative to two-sided sampling.Swapping anchor and positive roles increases the number of comparisons used during training.
  • Design components: Separating encoder contrastive training from later classifier fine-tuning causes a noticeable performance drop compared with joint training.If the encoder is not frozen during subsequent fine-tuning, spurious-correlation training can revert the contrastive representation and recreate an ERM-like error gap.
  • Hyperparameter: CnC maintains high worst-group accuracy across λ ∈[0.4, 0.9] on CMNIST*; GEORGE and JTT obtain 76.4% and 74.5% worst-group accuracy, respectively.Here λ balances the cross-entropy and contrastive loss components.
  • Domain-alignment comparison: DANN’s domain alignment is insufficient to improve worst-group accuracy compared with CnC on Waterbirds and CelebA.The paper hypothesizes that DANN aligns representations without accounting for class differences within domains.

D.1 Improving robustness to spurious correlations

The paper targets robustness to group shifts caused by spurious correlations, especially when training group labels are unavailable. Correct-N-Contrast uses ERM predictions to construct contrastive pairs that align same-class representations across inferred spurious groups.

  • ERM can exploit correlations between class labels and spurious attributes, producing variable performance across subpopulations.
  • Methods with group information directly optimize worst-group performance, while related approaches without training group labels often require validation spurious labels or do not directly prevent spurious dependencies.
  • Unlike pixel-level methods, CnC aligns hidden-layer representations and can apply beyond image modalities.
  • CnC uses ERM predictions to identify same-class points with different inferred spurious features and trains a contrastive model to learn invariant features between them.
  • CnC contrasts same-class samples with different ERM predictions against different-class samples sharing the anchor’s ERM prediction.
  • CnC aims to align same-class samples across inferred groups while separating different-class samples within the same inferred group.

E.1 Dataset details

The evaluation covers four spurious-correlation benchmarks and reports dataset configurations, representation metrics, implementation details, and comparison constraints. The experiments include Colored MNIST, Waterbirds, CelebA, and CivilComments-WILDS.

  • Dataset details: Colored MNIST classifies five digit pairs while treating color as the spurious attribute; high class-color correlation causes ERM failures on differently colored digits.
  • Dataset details: Waterbirds classifies waterbirds versus landbirds with water or land backgrounds as spurious attributes, and 95% of each bird class matches its associated background during training.
  • Dataset details: CelebA classifies blond versus non-blond hair with gender as the spurious attribute; only 6% of blond celebrities are male, a group on which ERM performs poorly.
  • Dataset details: CivilComments-WILDS classifies toxic versus non-toxic comments across 16 overlapping identity-based groups.
  • Evaluation: The study reports averaged worst-group accuracy over three seeds for three image benchmarks, but only one CivilComments-WILDS run because of time and compute constraints.
  • Evaluation: Representation dependence is quantified using alignment and estimated mutual information with class labels and spurious attributes.

F Visualization of learned data representations

UMAP visualizations compare ERM, Jtt, and CnC representations across Colored MNIST, Waterbirds, and CelebA. CnC most consistently organizes representations by class rather than spurious attributes.

  • Compared with ERM and Jtt, CnC more clearly depicts dependencies only on class labels, especially for Waterbirds and CelebA.
  • CnC representations show class-label dependence and separability across datasets while showing little organization by spurious attributes.

G Additional GradCAM visualizations

GradCAM visualizations compare salient image regions used by ERM, Jtt, and CnC on Waterbirds and CelebA. CnC more consistently focuses on class-relevant pixels rather than spurious backgrounds or gender-associated regions.

  • Waterbirds: On Waterbirds, ERM often emphasizes spurious background pixels, whereas Jtt and CnC correct this tendency and CnC focuses more exclusively on bird pixels.
  • CelebA: On CelebA, ERM often ignores hair pixels in favor of regions associated with gender, while Jtt and CnC usually include hair regions.
Loading 2203.01517v2…