Source-linked AI summary
Shortcut Learning in Deep Neural Networks
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, Felix A. Wichmann
TL;DR
Deep learning often lacks understanding of the principles governing its behavior, while standard i.i.d. benchmarks cannot distinguish intended solutions from shortcuts. This perspective connects these problems under shortcut learning and recommends out-of-distribution testing to improve robustness and transferability.
Problem
Deep learning lacks detailed understanding of its underlying principles, while i.i.d. benchmarks cannot distinguish intended solutions from shortcuts.
Method
The perspective develops a taxonomy of shortcuts, examines their origins and effects across learning domains, and identifies strategies for diagnosing and overcoming them.
Results
Shortcut learning appears ubiquitous across biological and artificial systems and connects many deep learning problems, including unexpected generalisation failures.
Takeaways & Limitations
Out-of-distribution generalisation tests should become standard because i.i.d. testing is insufficient to distinguish intended from shortcut solutions.
Takeaways & Limitations
Convolutional network design can prevent the specific shortcut illustrated in the taxonomy, limiting that example’s applicability across architectures.
Abstract
from arXiv · showhide
Deep learning has triggered the current rise of artificial intelligence and is the workhorse of today's machine intelligence. Numerous success stories have rapidly spread all over science, industry and society, but its limitations have only recently come into focus. In this perspective we seek to distill how many of deep learning's problems can be seen as different symptoms of the same underlying problem: shortcut learning. Shortcuts are decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios. Related issues are known in Comparative Psychology, Education and Linguistics, suggesting that shortcut learning may be a common characteristic of learning systems, biological and artificial alike. Based on these observations, we develop a set of recommendations for model interpretation and benchmarking, highlighting recent advances in machine learning to improve robustness and transferability from the lab to real-world applications.
1 Introduction
Deep learning’s remarkable successes coexist with failures that expose limited understanding of DNN behaviour and limitations. The perspective argues that many such failures reflect shortcut strategies that succeed superficially but fail under changed circumstances.
- 1 Introduction: DNNs have achieved reported successes across object classification, games, medical imaging, translation, climate change, and scientific progress.These achievements have given deep learning strong influence across science, industry, and society.
- 1 Introduction: Super-human performance can coexist with failures caused by invisible perturbations, changed backgrounds, or ignoring relevant image content.The passages describe object-recognition derailment, implausible image captions, and other unintuitive failures.
- 1 Introduction: Many apparently independent failures are connected because DNNs use unintended shortcut strategies that perform well superficially but fail under slightly different circumstances.A pneumonia model could predict reasonably well from hospital prevalence without learning much about pneumonia.
- 1 Introduction: Shortcut learning has longstanding connections to concepts including covariate shift, anti-causal learning, dataset bias, the tank legend, and the Clever Hans effect.The perspective places shortcut learning within machine learning’s broader effort to formally understand such behaviour.
- 1 Introduction: The perspective introduces shortcut learning in biological neural networks, develops a taxonomy and examines shortcut origins, then considers effects across major deep-learning areas.The applications include Computer Vision, Natural Language Processing, Agent-based Learning, and Fairness.
2 Shortcut learning in biological neural networks
Biological neural networks, like artificial ones, can achieve apparent task success through unintended shortcuts rather than the intended ability. Examples from rats and students show that these strategies perform well under narrow testing conditions but fail when the intended capability is tested more directly.
- Unintended cue learning: Rats navigated a complex maze by discriminating wall-paint colours through odour rather than using visual colour perception.The rats’ visual system was not involved, despite the experiment appearing to test colour discrimination.
- Unintended cue learning: Unintended cue learning in animals creates a discrepancy between performance in an experimental paradigm and the mental ability being investigated.This parallels the discrepancy between intended and actual learning outcomes in machine learning.
- Surface learning: Students’ surface learning relies on narrow testing conditions and simple discriminative generalisation, producing good performance that fails under more general tests.Surface learning can improve performance on typical multiple-choice exams despite failing to reflect broader understanding.
3 Shortcuts defined: a taxonomy of decision rules
The paper defines shortcuts as decision rules that perform well on training and i.i.d. test data but fail on out-of-distribution tests. Its taxonomy progresses from all possible rules to training solutions, i.i.d. solutions, shortcuts, and intended solutions that generalize robustly.
- Taxonomy of decision rules: Neural networks implement decision rules mapping inputs to outputs, and shortcuts are a particular group distinguished through a progressively constrained taxonomy.The taxonomy is visualized in Figure 3 and begins with all possible decision rules before adding constraints.
- Taxonomy of decision rules: The taxonomy first excludes non-solutions, such as predicting “star” whenever an image contains a white pixel, because they perform poorly on training data.White pixels occur in both moon and star images, making this feature uninformative for the classification task.
- Taxonomy of decision rules: Training solutions can include overfitting rules, while random training-test splits evaluate performance on similar, previously unseen data.The training set guides decision-rule selection, and the test set checks performance on similar data.
- Taxonomy of decision rules: i.i.d. test solutions may score highly on standard benchmarks yet rely on shape, white-pixel count, or location, with location representing a shortcut.In the toy task, location correlates with category in the training and i.i.d. test sets but is not the intended feature.
- Taxonomy of decision rules: Intended solutions use task-relevant features and generalize to out-of-distribution tests, whereas shortcut solutions fail when the data change.In the toy example, shape-based rules continue to work when object location or size changes.
- Taxonomy of decision rules: A convolutional network would be prevented by design from taking the location-based shortcut illustrated in the toy example.The architectural constraint is noted specifically for convolutional rather than fully connected networks.
4 Shortcuts: where do they come from?
Shortcuts arise when models exploit predictive but unintended cues, such as background, texture, or local patterns, rather than the intended object properties. These rules can succeed on training and i.i.d. data yet fail under o.o.d. conditions, exposing a mismatch between learned and intended solutions.
- Defining shortcuts: Shortcuts are decision rules that perform well on i.i.d. test data but fail on o.o.d. tests, revealing a mismatch between intended and learned solutions.Their defining concern is whether learned rules transfer beyond the conditions represented by the standard test distribution.
- Contextual cues: Systematic relationships between objects and backgrounds or contexts can make contextual cues sufficient for recognition, causing models to misclassify objects in unexpected settings.DNNs may reject a cow on a beach or label an animal-free landscape as a herd of grazing sheep.
- Local features: Standard DNNs strongly rely on texture and other local structures for object classification, largely ignoring global object shape.Texture cues can be sufficient for object categorisation on standard datasets, even though texture is only one object attribute.
- Representation learning: End-to-end discriminative learning can bias models toward overly simplistic, non-human-interpretable features that generalise only under a dataset’s specific design.Standard DNNs impose no human-interpretability requirements on intermediate image representations, allowing such feature extraction biases.
- Generalisation: A novel guitar-like pattern can trigger highly confident “guitar” predictions, showing that models may detect patterns such as body curvature or strings instead of guitars.This strategy succeeds on training and i.i.d. test data but produces unintended generalisation on o.o.d. data, linking shortcut learning to generalisation.
5 Shortcut learning across deep learning
Shortcut learning appears across deep learning as reliance on dataset opportunities and discriminative features that produce failures to generalize as intended. The section illustrates this pattern in computer vision, natural language processing, agent-based learning, and algorithmic decision-making.
- Cross-domain pattern: Across application areas, dataset shortcut opportunities and discriminative feature learning can cause models to fail to generalize as intended.The pattern is examined in computer vision, natural language processing, agent-based learning, and fairness or algorithmic decision-making.
- Computer Vision: Deep neural networks can change predictions after innocuous image transformations, including small shifts, rotations, noise, blur, background changes, or texture changes.These transformations can preserve the depicted object while altering the model’s prediction.
- Natural Language Processing: BERT can use superficial cue words such as “not” to identify argument conclusions above chance without understanding sentence content.This shortcut was observed in a dataset of natural language arguments.
- Agent-based (Reinforcement) Learning: In agent-based learning, an algorithm learned to pause Tetris to avoid losing rather than learning how to play the game.Reinforcement-learning systems in robotics also face a reality gap when they adapt to narrow details of simulated training environments; varying colour, size, texture, and lighting helps close it.
- Fairness & algorithmic decision-making: Amazon’s résumé-screening tool preferred men because gender was a strong predictor in prior human decisions, inferring gender even from all-woman college names after names were removed.The example shows how models can exploit indirect demographic cues in algorithmic decision-making.
6 Diagnosing and understanding shortcut learning
The paper proposes three actionable steps for diagnosing shortcut learning: distinguish benchmark performance from intended abilities, test strong shortcut-based baselines, and standardize out-of-distribution evaluation. It further recommends designing distribution-shift tests that expose misleading i.i.d. performance and unintended decision rules.
- Distinguishing datasets and underlying abilities: Shortcut learning is deceptive because i.i.d. benchmarks may measure unintended cues rather than the underlying ability of interest.ImageNet was intended to measure object recognition, but deep neural networks may instead rely mainly on counting texture patches.
- Testing (surprisingly) strong baselines: Testing strong baselines that use unintended features can reveal whether benchmark success is explainable without the intended capability.Examples include nearest neighbours for scene completion and geolocation, local features for object recognition, and single cue words for reasoning.
- Making o.o.d. generalisation tests a standard practice: Out-of-distribution evaluation should become standard because real-world data rarely satisfy the i.i.d. assumption, making i.i.d. performance potentially misleading.The paper compares this problem with surface learning, where multiple-choice success can fail to transfer while essay questions encourage deeper learning strategies.
- Designing good o.o.d. tests: Good o.o.d. tests require a clear distribution shift, which can be detected by training a classifier to distinguish samples from two datasets.The paper states that distribution shifts have a clear mathematical definition but can be difficult to detect in practice.
- Designing good o.o.d. tests: Adversarial attacks provide a diagnostic for shortcut learning when they change predictions without changing semantic content.The paper also highlights shortcut-removing datasets, cue-conflict stimuli, worst-case natural images, corruptions, controlled domain shifts, and clean-test accuracy drops as diagnostic designs.
7 Beyond shortcut learning
Research beyond shortcut learning targets out-of-distribution generalisation through diverse approaches, including prior-informed architectures, robustness analysis, domain-based methods, fairness, meta-learning, and disentangled generative representations.
- Domain-specific prior knowledge: Domain-specific priors reduce unintended cues through architectures and data augmentation, including rotation invariance when object orientation is irrelevant to category.The strategy extends to well-understood input transformations and is exemplified by auto-augment.
- Adversarial examples and robustness: Adversarial examples analyze worst-case generalisation as minimal counterfactual changes, linking adversarial robustness to explanations aligned with human intention and causality.Adversarial attacks provide a tool for studying how small input changes alter predictions.
- Domain adaptation, -generalisation and -randomisation: Domain adaptation, domain generalisation, and domain randomisation explicitly seek out-of-distribution generalisation by training across distributions, domains, or randomized simulation conditions.Under certain assumptions, multiple environments can reveal intended or causal solutions.
- Fairness and meta-learning: Fairness connects generalisation and causality by treating sensitive group membership as a domain indicator, while meta-learning seeks representations that adapt quickly to new conditions.Individual fairness treats similar individuals similarly; group fairness treats subgroups no differently from the rest of the population.
- Generative modelling and disentanglement: Generative modeling captures training-data variation, but disentanglement is needed to learn well-structured latent representations useful for downstream tasks and potentially out-of-distribution generalisation.Modeling every observed variation alone does not guarantee useful downstream representations.
8 Conclusion
The conclusion frames shortcut learning as a ubiquitous source of deep learning’s generalisation failures and a central obstacle to understanding machine learning. It recommends careful interpretation, routine out-of-distribution testing, and analysis of the factors that make solutions easy to learn.
- Conclusion: Shortcut learning is ubiquitous across biological and artificial systems, linking dataset shortcuts and selective feature use to unexpected generalisation failures.These connections may unify many problems in deep learning.
- Conclusion: Discovering a shortcut can expose an easy explanation for apparently complex datasets, so high-level abilities such as object recognition or language understanding require careful interpretation.The conclusion cautions that simpler explanations may account for observed behavior.
- Conclusion: Out-of-distribution generalisation tests should become standard because i.i.d. benchmarks cannot reliably distinguish intended from unintended shortcut solutions.The conclusion argues that current benchmark practice is insufficient for this distinction.
- Conclusion: DNNs learn the easiest available solution, whose learnability depends on interactions among architecture, training data, loss function, and optimisation.Understanding these interactions is necessary to determine which solutions are likely to be learned.
- Conclusion: Shortcut learning is a key roadblock to fair, robust, deployable, and trustworthy machine learning, but mitigating it can better align learned and intended solutions.The conclusion links such progress to more reliable behavior in complex settings far from training experience.
Appendix · A Toy example: method details
The toy example uses star and moon shapes on a 200×200 2D canvas to study classifier behavior. Two neural classifiers are trained for five epochs with specified optimization settings and evaluated on a test set.
- A Toy example: method details: The reproducible toy example places two easily distinguishable shapes, a star and a moon, on a 200×200 dimensional 2D canvas.Code is available from the linked GitHub repository.
- A Toy example: method details: The training set contains 4000 images: 2000 with a star shape and 2000 with a moon shape.
- A Toy example: method details: Star shapes are randomly placed in the canvas’s top right and bottom left quarters.
- A Toy example: method details: Moon shapes are randomly placed in the canvas’s top left and bottom right quarters.
- A Toy example: method details: The study trains two classifiers: a fully connected network and a convolutional network.Both classifiers are trained on the constructed dataset and evaluated on the test set.
- A Toy example: method details: Both classifiers are trained for five epochs with a batch size of 100, using standard crossentropy loss and Adam.The learning rate is 0.00001, with β1 = 0.9, β2 = 0.999 and ε = 1e−08.
- A Toy example: method details: The fully connected classifier is a three-layer ReLU MLP with 1024 units in each layer and two output units.
B Image rights & attribution
The section documents image sources, permissions, licenses, and adaptations for Figures 1 and 4 and images used in Sections 4.1–4.3. It identifies reused material and notes when images were cropped or adapted from original figures.
- Figure 1: Figure 1 combines four images: one used with the author’s permission, one generated by the authors, and one adapted from a CC BY 4.0 source.The first image came from aiweirdness.com, the second was generated by the authors, and the third came from ref..
- Sections 4.1–4.3: Images in Sections 4.1, 4.2, and 4.3 were adapted from cited figures with permission from their authors.Each image was cropped from its original figure by the paper’s authors.
- Figure 4: Figure 4 uses images from multiple sources, including ref. [8] under a CC BY 3.0 license and other images reproduced or adapted with author permission.The figure identifies corresponding publication first authors, and the authors cropped images from original figures.