PaperlayerView exact revision on arXiv

Source-linked AI summary

The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning

Wenqian Ye, Luyang Jiang, Eric Xie, Guangtao Zheng, Yunsheng Ma, Xu Cao, Dongliang Guo, Daiqing Qi, Zeyu He, Yijun Tian, Megan Coffee, Zhe Zeng, Sheng Li, Ting-hao, Huang, Ziran Wang, James M. Rehg, Henry Kautz, Aidong Zhang

arXiv:2402.12715v4cs.LG

TL;DR

Spurious correlations can make models rely on superficial features that may not remain predictive under distribution shifts, creating robustness concerns. This paper surveys their definitions, mitigation methods, datasets, benchmarks, and metrics, and synthesizes challenges for future work. It concludes that spurious correlations remain a broad machine-learning challenge, especially for brittleness and poor domain-shift performance.

  • Problem

    Spurious correlations between superficial input features and labels can undermine model robustness when those correlations no longer hold, while prior work lacks a comprehensive formal definition and systematic survey.

    Source: §1 · p. 1

  • Method

    The paper surveys spurious correlations through a formal definition, a taxonomy of mitigation methods, and an overview of datasets, benchmarks, metrics, and research challenges.

    Source: §1 · p. 2, §7.4 · p. 19

  • Results

    The survey characterizes spurious correlations as a broad challenge associated with model brittleness and poor performance under domain shifts, and reviews methods, datasets, and metrics for robustness evaluation.

    Source: §7.4 · p. 19

  • Takeaways & Limitations

    The survey lays groundwork for further progress on spurious correlations, including in the era of generative AI and larger foundational models.

    Source: §7.4 · p. 19

    The field lacks consistent methods for determining why models depend on particular features, how interventions affect outcomes, and whether mitigation succeeds for the right reasons.

    Source: §7.2 · p. 18

Abstract

from arXiv · show

Back in the early 20th century, a horse named Hans appeared to perform arithmetic and other intellectual tasks during exhibitions in Germany, while it actually relied solely on involuntary cues in the body language from the human trainer. Modern machine learning models are no different. These models are known to be sensitive to spurious correlations between non-essential features of the inputs (e.g., background, texture, and secondary objects) and the corresponding labels. Such features and their correlations with the labels are known as "spurious" because they tend to change with shifts in real-world data distributions, which can negatively impact the model's generalization and robustness. In this paper, we provide a comprehensive survey of this emerging issue, along with a fine-grained taxonomy of existing state-of-the-art methods for addressing spurious correlations in machine learning models. Additionally, we summarize existing datasets, benchmarks, and metrics to facilitate future research. The paper concludes with a discussion of the broader impacts, the recent advancements, and future challenges in the era of generative AI, aiming to provide valuable insights for researchers in the related domains of the machine learning community.

1 Introduction

Spurious correlations can undermine generalization and robustness when superficial input features stop aligning with labels, with potential negative social impact in critical domains. Despite progress across several fields, the survey identifies a gap in comprehensive definitions and systematic reviews of mitigation methods and challenges.

1 Introduction

  • When training correlations fail to hold at test time, reliance on superficial features can degrade model performance and robustness, including in healthcare.The introduction gives hospital-specific metal tokens in chest X-rays as an example of an extraneous cue.Source: §1 · p. 1
  • Prior work spans computer vision, natural language processing, and healthcare, but the paper identifies no comprehensive survey that formally defines spurious correlations and systematically reviews methods and challenges.Source: §1 · p. 1
  • The paper presents a survey with a formal definition, a taxonomy of state-of-the-art mitigation methods, and an overview of datasets, benchmarks, and evaluation metrics.Source: §1 · p. 2

2 Spurious Correlations

Spurious correlations are label-associated attributes that may be non-predictive, and reliance on them can fail when environments change. The section discusses how such patterns arise, why learning procedures may favor them, and how they can persist in learned representations.

2 Spurious Correlations

  • A spurious correlation associates a class label with a non-predictive attribute, and reliance on that attribute can produce incorrect predictions when the association changes across environments.The Clever Hans analogy frames this as exploiting cues rather than learning underlying concepts.Source: §2 · p. 2

2.1 Where do Spurious Correlations Come From?

  • Selection bias, underspecified datasets, simplicity bias, imbalanced groups, and sampling noise are described as sources of spurious correlations.The section illustrates environment shift with a background changing from grass to desert while the cow label remains unchanged.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 Why Are Machine Learning Models Sensitive to Spurious Correlations?

  • Inductive biases and ERM can favor spurious patterns: the objective omits the spurious attribute, so a strongly correlated attribute may be used for prediction and fail when the correlation breaks.Group DRO is described as mitigating this effect by minimizing training loss within the worst-performing groups.Source: §2.2 · p. 3

2.3 Theoretical Insights

  • Models may embed spurious features early because they are simple or strongly correlated with labels, delaying core-feature learning while retaining spurious traces in representations.Late-stage learning may also reinforce simplified representations that encode spurious patterns.Source: §2.3 · p. 4
  • Theoretical work characterizes spurious correlations as persistent representation structures, while also noting that rationale generators can introduce cues and regularizers can suppress useful signals when concepts are entangled.Feature alignment with training samples is described as a geometric characterization of spurious-correlation memorization.Source: §2.3 · p. 4

3 Related Areas

The section connects spurious correlations to domain generalization, group robustness, shortcut learning, and simplicity bias. Across these perspectives, models may rely on easy or group-specific cues that fail when distributions change.

Domain Generalization

  • Domain generalization can fail when source-domain shortcuts become unreliable in unseen targets, a risk heightened when training data span only one domain.A feature correlated with labels across some source domains may not remain correlated in the target domain.Source: §3 · p. 4

Group Robustness

  • Group robustness targets consistent performance across subpopulations, where majority-group shortcuts can produce sharp accuracy drops for minority groups.Group DRO minimizes worst-case loss across groups, and benchmarks such as WILDS evaluate worst-group accuracy.Source: §3 · pp. 4–5

Shortcut Learning

  • Shortcut learning is reliance on easy spurious cues instead of relevant features, which can undermine performance when context changes.Examples include texture or background cues in vision and keywords in NLP.Source: §3 · p. 5

Simplicity Bias

  • Simplicity bias favors easy-to-fit patterns, which can be spurious and less reliable than complex features when label correlations shift.The bias may help fit training data while undermining robustness to distribution shifts.Source: §3 · pp. 5–6

4 Methods

The methods section organizes approaches to spurious correlations by where they act in the machine-learning pipeline. Its taxonomy distinguishes interventions on data, learned representations, trained models, and specialized settings.

4 Methods

  • The survey groups methods by their stage in the machine-learning training pipeline to provide a framework for comparing approaches to spurious correlations.The taxonomy is illustrated in Figure 3.Source: §4 · pp. 6–7
  • Data-centric methods modify training data, while representation-learning methods change the objective or architecture to learn more robust features.These categories correspond to Sections 4.1 and 4.2.Source: §4 · p. 7
  • The taxonomy also includes post-hoc methods for refining trained models and specialized methods for non-standard settings.Source: §4 · p. 7

4.1 Data-Centric Methods

Data-centric methods modify training data to break harmful associations between spurious attributes and labels. The survey covers augmentation, balancing, and concept or pseudo-label discovery.

4.1 Data-Centric Methods

  • Data-centric methods alter training data before or during learning, using augmentation, balancing, or concept and pseudo-label discovery to manage spurious attributes.Source: §4.1 · p. 7

4.1.1 Data Augmentation

  • Data augmentation diversifies training examples, including through counterfactual generation that separates causal from non-causal factors.Methods also use annotations, semantic corruption, mixup, or graph perturbations to weaken spurious cues.Source: §4.1.1 · p. 7

4.1.2 Data Balancing

  • Data balancing methods reweight or subsample data, but their effects can vary by method, group structure, and training epoch.A combination of mini-batch upsampling and loss upweighting can achieve higher worst-group accuracy than either alone in the cited work.Source: §4.1.2 · p. 7

4.1.3 Concept and Pseudo-label Discovery

  • Concept and pseudo-label discovery uses predefined, clustered, or language-model-generated concepts to identify spurious attributes and support robust training.Some approaches use limited annotations, while others seek latent concepts without human labeling.Source: §4.1.3 · p. 8

4.2 Representation Learning

Representation-learning methods target the model’s internal features through training objectives or architecture. The section surveys causal intervention, invariance, disentanglement, and contrastive learning.

4.2 Representation Learning

  • Representation-learning methods modify objectives or architectures to favor robust, causally linked features over features that are merely statistically correlated.Source: §4.2 · p. 8

4.2.1 Causal Intervention

  • Causal intervention methods model relationships among inputs, labels, and spurious attributes to reduce reliance on spurious features.Examples include causal attention, semantic editing, embedding translations, causal graphs, and multi-task causal regularization.Source: §4.2.1 · p. 8

4.2.2 Invariant Learning

  • Invariant learning seeks features stable across environments, with methods enforcing consistency through risk objectives, environment inference, or feature-level penalties.Some approaches also target shortcut features in continual learning or hidden shortcuts in language models.Source: §4.2.2 · pp. 8–9

4.2.3 Feature Disentanglement

  • Feature disentanglement separates spurious from general representations, often using dual branches, projections, or targeted neuron removal.Source: §4.2.3 · p. 9

4.2.4 Contrastive Learning

  • Contrastive learning trains representations to be similar for same-class samples that differ in spurious features, helping distinguish essential from non-essential cues.Source: §4.2.4 · p. 9

4.3 Post-hoc Methods

Post-hoc methods address spurious correlations by adapting already-trained models or using a secondary training stage. The survey groups them by how they identify bias, refine models, change objectives, combine models, or use adversarial training.

4.3 Post-hoc Methods

  • Post-hoc methods adapt pre-existing models or use a secondary training stage rather than building a robust model from scratch.The survey identifies five approaches: identification then mitigation, fine-tuning, optimization, ensembles, and adversarial training.Source: §4.3 · p. 9

4.3.1 Identification then Mitigation

  • Identification-then-mitigation methods detect spurious correlations or affected samples, then reduce their influence through reweighting, tuning, pruning, or resampling.Examples include methods that probe representations, infer group membership, or identify spurious indicators.Source: §4.3.1 · pp. 9–10

4.3.2 Finetuning

  • Fine-tuning methods selectively adjust a pre-trained model or its training data; last-layer approaches have improved worst-group performance with minimal group annotations.The survey also describes domain-specific approaches and reports that fine-tuning techniques vary in performance across task types.Source: §4.3.2 · p. 10

4.3.3 Optimization-based Methods

  • Optimization-based methods alter objectives or sample weights to improve group or worst-case performance, with approaches including Group DRO, CVaR DRO, and JTT.Other methods adjust logits, penalize later-layer activations, or use teacher–student feedback.Source: §4.3.3 · p. 10

4.3.4 Ensemble Learning

  • Ensemble methods combine biased models or representations to identify bias-conflicting examples and encourage more robust predictions.The survey also describes methods that train models on diverging subgraphs or use two-stage ensembles.Source: §4.3.4 · pp. 10–11

4.3.5 Adversarial Training

  • Adversarial training methods address spurious correlations associated with backdoor poisoning by mitigating watermarked, mislabeled training examples.The survey frames these methods as strengthening models against adversarial inputs.Source: §4.3.5 · p. 11

4.4 Specialized Methods

Specialized methods address particular settings, including test-time label shifts, reinforcement learning, and multimodal separation of spurious attributes.

4.4 Specialized Methods

  • Test-Time Label-Shift Adaptation addresses label shifts at test time, while RSC-MDP tackles spurious dependencies in reinforcement learning.Source: §4.4 · p. 11
  • Other work explores multimodal models to explicitly separate spurious attributes from the main class.Source: §4.4 · p. 11
  • The surveyed approaches target distinct settings rather than one shared adaptation procedure.Source: §4.4 · p. 11

5 Datasets and Metrics

The survey reviews datasets for studying spurious correlations across five domains and describes synthetic and realistic construction strategies. It also outlines complementary metrics for assessing group-level performance and robustness.

5.1 Datasets

  • Benchmarks span vision, text, graph, multimodal, and health domains, with training label–attribute correlations not guaranteed to hold at test time.The reviewed datasets include both realistic and synthetic data.Source: §5.1 · p. 11

5.1.1 Synthetic Datasets

  • Synthetic datasets enable controlled analysis by composing data, manipulating pixels, injecting noise, or transferring style to create spurious cues.Examples include combining foregrounds and backgrounds, coloring digits, adding corruptions, and stylizing images.Source: §5.1.1 · pp. 11–12, Table 1 · p. 12

5.1.2 Realistic Datasets

  • Realistic datasets capture natural, often subtle correlations through data fusion, re-annotation, collection from scratch, or mining challenging examples.They offer less granular control than synthetic datasets but are important for evaluating performance on authentic distributions.Source: §5.1.2 · pp. 12–13

5.2 Metrics

  • Worst-group, average-group, bias-conflicting, and overall accuracy provide complementary views of group robustness and general utility.Worst-group accuracy measures the minimum across groups; average-group accuracy is less sensitive to the hardest subgroup, while bias-conflicting accuracy isolates groups where the spurious attribute conflicts with the training majority.Source: §5.2 · pp. 13–14

6 Broader Impacts of Spurious Correlations

The paper traces how spurious correlations affect LLM alignment, clinical AI, and embodied agents, where shortcuts can shape outputs and actions. Across these settings, it describes mitigation approaches alongside unresolved challenges in identifying and addressing the biases.

6.1 LLM Alignment

6.2 Healthcare

  • Clinical shortcuts can arise from dataset collection and design, including hospital signatures, demographic attributes, and uneven skin-tone representation that contribute to biased accuracy or subgroup underperformance.Examples include hospital-specific radiology artifacts, cost-based risk models disadvantaging minorities, and dermatology performance gaps (Daneshjou et al., 2022).Source: §6.2 · p. 15
  • Healthcare mitigations include holding out collection sites during training and using spatial annotations, though annotations can be costly and external test accuracy may conceal spurious cues.The paper also calls for auditing with domain knowledge, technical expertise, and explainable AI.Source: §6.2 · pp. 15–16

6.3 Embodied AI

  • Embodied AI agents can rely on scene lighting, object positions, brightness, or room appearance, disrupting robotics in unseen settings or driving when learned correlations fail.The paper defines embodied AI as AI agents that interact with the physical world through components including perception and action.Source: §6.3 · p. 16
  • In ProcTHOR, changing other objects’ colors and textures while sofa color stays fixed could mislead navigation; dataset shortcuts also affect autonomous-driving models, while ensembles and robust datasets are proposed mitigations.The paper cites WEDGE (Marathe et al., 2023) as an example of a more robust dataset.Source: §6.3 · p. 16, Figure 6 · p. 16
  • Vision shortcuts can mislead fine-tuned VLMs, while VLA methods include InSpire for spatial reasoning and PCD for shifting attention toward task-relevant features.RaVL uses local image features to identify spurious correlations and train VLMs to focus on other regions; VLA shortcuts can produce actions unrelated to instructions.Source: §6.3 · pp. 16–17
  • Embodied-AI methods emphasize dataset, input, or inference-time changes; training-stage mitigation needs further study, and spurious behavior can remain misaligned with human demands.The paper also notes limited research on spurious correlations in modalities such as audio.Source: §6.3 · p. 17

7 Discussion

The discussion frames spurious-correlation mitigation as unresolved, identifying practical and evaluation constraints while outlining research directions and foundation-model questions.

7.1 Prospective in the Era of Generative AI

  • Foundation models may help detect or generate diverse data, but their overparameterization may exacerbate spurious associations and hallucinations.The discussion also proposes structured knowledge, such as knowledge graphs, as a possible mitigation pathway.Source: §7.1 · p. 17

7.2 Unresolved Challenges in Spurious Correlation Mitigation

  • Many mitigation approaches rely on group annotations or human-defined attributes, which can be unavailable or incomplete in practice, especially in healthcare.Defining subgroup boundaries there may require expert knowledge and labor-intensive labeling.Source: §7.2 · p. 17
  • Group-aware objectives can improve fairness across subpopulations, but optimizing worst-group accuracy can degrade overall accuracy or increase variance.This tension complicates deployment decisions and motivates methods that navigate performance tradeoffs.Source: §7.2 · pp. 17–18
  • Narrow predefined benchmarks may mask other shortcuts, while manually identifying, labeling, or simulating every correlation is infeasible at modern scale.The limited protocols also make cross-modal and cross-domain generalization difficult to assess.Source: §7.2 · p. 18
  • The field lacks consistent methods to explain feature reliance, assess intervention effects, and determine whether mitigation succeeds for the right reasons.Source: §7.2 · p. 18

7.3 Open Questions and Future Directions

  • Future work calls for group-label-free mitigation, automated detection without human-defined attributes, and methods that generalize across modalities and domains.Automated detection also needs to scale to foundation models, and absent ground-truth groups complicate robustness validation.Source: §7.3 · pp. 18–19
  • Future directions include balancing robustness with utility, broadening evaluation beyond predefined biases, and using interpretability and causal modeling to understand mitigation.The discussion notes that discarding all correlations may unnecessarily limit model capacity because some spurious features can provide useful context.Source: §7.3 · p. 18

7.4 Final Remarks

  • The survey reviews methods, datasets, and metrics, and highlights future challenges as spurious correlations contribute to brittleness and poor performance under domain shifts.Source: §7.4 · p. 19
Loading 2402.12715v4…