PaperlayerView exact revision on arXiv

Source-linked AI summary

Shortcut Learning of Large Language Models in Natural Language Understanding

Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, Xia Hu

arXiv:2208.11857v2cs.CLcs.LG

TL;DR

LLMs achieve strong benchmark performance, but shortcut reliance on dataset biases and artifacts raises concerns about robustness and generalization. This paper reviews shortcut learning in language models, including its detection, causes, mitigation, and research challenges. It concludes that data-driven training alone is insufficient for high-level NLU and should be combined with domain knowledge.

  • Problem

    LLMs may rely on dataset biases and artifacts as shortcuts, while their generalization and adversarial robustness remain concerns.

    Source: §1 · p. 1

  • Method

    The paper surveys shortcut learning in language models, covering detection, causes, mitigation approaches, and future directions, with emphasis on medium-sized LLMs.

    Source: §1 · p. 1

  • Results

    The survey identifies skewed datasets, model architecture, and learning dynamics as causes of shortcut learning, and concludes that pure data-driven training is insufficient for high-level NLU.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The paper argues that future data-driven training should be combined with domain knowledge at every stage.

    Source: §8 · p. 8

    Human-prior-based mitigation can identify only limited biases, so models may continue to use other shortcuts.

    Source: §6.1 · p. 6

Abstract

from arXiv · show

Large language models (LLMs) have achieved state-of-the-art performance on a series of natural language understanding tasks. However, these LLMs might rely on dataset bias and artifacts as shortcuts for prediction. This has significantly affected their generalizability and adversarial robustness. In this paper, we provide a review of recent developments that address the shortcut learning and robustness challenge of LLMs. We first introduce the concepts of shortcut learning of language models. We then introduce methods to identify shortcut learning behavior in language models, characterize the reasons for shortcut learning, as well as introduce mitigation solutions. Finally, we discuss key research challenges and potential research directions in order to advance the field of LLMs.

1 INTRODUCTION

LLMs can score highly on in-distribution NLU benchmarks yet generalize poorly when input distributions change, with shortcut learning identified as a major reason. The paper reviews how shortcut behavior is detected and explained, why it occurs, and how it may be mitigated, with emphasis on medium-sized models and the pre-training/fine-tuning paradigm.

1 INTRODUCTION

  • Shortcut learning links dataset artifacts and biases to class labels, letting models predict from spurious cues instead of the underlying NLU task.Examples include BERT-like NLI models relying on words such as “not” and reading-comprehension models relying on lexical overlap [13] [24].Source: §1 · p. 1
  • The review covers shortcut concepts, detection, causes, mitigation, future directions, and prompt-based challenges, focusing mainly on medium-sized LLMs under pre-training and fine-tuning.It also briefly considers prompt-based methods for massive models such as GPT-3 and T5.Source: §1 · p. 1
  • LLMs can outperform humans on IID NLI benchmarks yet perform much worse OOD, where lexical bias and other shortcuts are a key reason.Figure 1 uses premise–hypothesis inference among entailment, contradiction, and neutral as its example.Source: Figure 1 · p. 2

2 SHORTCUT LEARNING PHENOMENA

Shortcut learning is reliance on non-robust features rather than robust semantic understanding; those shortcuts can support performance on matched-distribution data but fail under distribution shifts and adversarial attacks.

2.1 What is Shortcut Learning?

  • Shortcut learning describes LLMs’ reliance on non-robust features instead of robust features that capture high-level semantic understanding and reasoning.Non-robust features arise from training-data biases; robust features remain stable under input changes.Source: §2.1 · p. 2
  • Lexical, overlap, and positional biases supply shortcuts, while style bias lets models use semantics-independent text patterns rather than underlying meaning.Examples include negation words prompting contradiction, question–passage overlap guiding answers, and answers appearing in a fixed sentence position.Source: §2.1 · p. 2

2.2 Generalization and Robustness Challenge

  • When deployment data shift away from training distributions, they may lack the same biases and artifacts, weakening shortcut-based generalization.For BERT-base on NLU tasks, OOD accuracy is reduced by more than 20% relative to in-distribution accuracy. Distracting information can also sharply reduce performance on multiple-choice reading comprehension.Source: §2.2 · p. 2

3 SHORTCUT LEARNING DETECTION

The section presents complementary ways to detect shortcut learning: test beyond IID benchmarks, probe model sensitivity through attacks and ablations, and inspect explanations for spurious cues.

3.1 Comprehensive Performance Testing

  • IID test sets can share training-data biases, allowing superficial-pattern memorization to look successful and motivating evaluation beyond IID testing.Figure 2 distinguishes useless, robust, and non-robust features; the latter represent biases that do not hold in OOD settings.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
  • OOD evaluations such as HANS test NLI syntactic heuristics, and such tests have exposed substantial performance degradation in state-of-the-art LLMs.HANS targets lexical-overlap, subsequence, and constituent heuristics; symmetric tests apply a similar approach to fact verification [21] [34].Source: §3.1 · p. 3
  • Adversarial attacks test robustness by perturbing inputs, and reported examples reveal non-robust features and failure to generalize to the associated task.Examples include attacks on machine reading comprehension and TextFooler examples for commonsense reasoning.Source: §3.1 · p. 3
  • Randomization ablations indicate that pre-trained models can retain high downstream accuracy after word order is shuffled, including across GLUE tasks.These findings suggest syntax may be ignored and performance explained by higher-order word co-occurrence statistics [40] [27].Source: §3.1 · p. 3

3.2 Explainability Analysis

  • Feature attribution scores token contributions and has exposed reliance on hypothesis artifacts; models concentrate on frequent shortcut tokens while poorly learning informative tail tokens.Integrated Gradients [43] was used to analyze BERT-based models [9].Source: §3.2 · p. 3
  • Instance attribution identifies influential training examples, which can share artifacts such as premise–hypothesis overlap; hybrid attribution offers a broader view of shortcut behavior.Hybrid methods combine feature and instance attribution [26].Source: §3.2 · p. 3

4 ORIGINS OF SHORTCUT LEARNING

Shortcut learning can arise at multiple points in the pre-training–fine-tuning pipeline. The review examines training-data artifacts, model characteristics, and fine-tuning dynamics as contributing factors.

4.1 Skewed Training Dataset

  • Crowdsourced collection and annotation artifacts skew training data; models can capture and amplify these patterns, such as hypothesis-length differences associated with NLI labels.For entailment, hypotheses are shorter on average than for neutral examples, suggesting overlap bias.Source: §4.1 · p. 4

4.2 LLMs Models

  • Larger models generally show better generalization and smaller IID–OOD accuracy gaps than base models, while smaller or compressed models are more prone to spurious patterns and reduced robustness.The review also notes that distilled compressed models are more vulnerable to adversarial attacks.Source: §4.2 · p. 4
  • Models with different pre-training objectives show different robustness; ELECTRA and RoBERTa outperform BERT on ANLI, and RoBERTa-base exceeds BERT-base by around 20% accuracy on HANS.The review suggests distinct inductive biases may help explain differences in generalizability.Source: §4.2 · p. 4

4.3 Model Fine-tuning Process

  • Fine-tuning dynamics favor simple, easy-to-learn features early, while longer fine-tuning can improve generalizability across NLU tasks.Reading-comprehension models have been observed to learn shortcuts within the first few training iterations.Source: §4.3 · p. 4

5 MITIGATION OF SHORTCUT LEARNING

The review surveys dataset- and model-centric approaches to mitigate shortcut learning, aiming to improve OOD generalization and adversarial robustness while maintaining IID performance. It also highlights unresolved trade-offs and limits on what bias refinement can remove.

5.1 Data-Centric Mitigation Approaches

  • Dataset refinement reduces selected biases through artifact-aware data collection, filtering, split redesign, and augmentation, but unidentified biases may remain and still harm performance.Adversarial filtering, counterfactual augmentation, and mixup are among the described approaches.Source: §5.1 · p. 5
  • Reweighting emphasizes hard training samples, while environment-based methods encourage reliance on correlations stable across non-IID training subsets.Reweighting can use a two-stage indexing model or focal loss; environment methods can align gradients across environments.Source: §5.1 · p. 5

5.2 Model-Centric Mitigation Methods

  • Model-centric methods augment ERM with prior knowledge to suppress non-robust features; some require shortcuts to be known, while others assume they are unknown.Adversarial training uses shared task and artifact classifiers or generated adversarial examples in a MinMax formulation.Source: §5.2 · p. 5
  • Explanation regularization uses human rationales, PoE combines a debiased model with a fixed bias-only model, and confidence regularization distills softened teacher predictions for biased samples.Explanation regularization requires prior knowledge about shortcuts; PoE trains the bias-only model first.Source: §5.2 · pp. 5–6
  • Contrastive learning can reduce reliance on non-robust features by contrasting factual and counterfactual inputs, but poorly designed instance discrimination may suppress robust features.The review reports improved generalization for a framework contrasting examples built with identified causal and non-causal terms.Source: §5.2 · p. 6

5.3 IID and Robustness Trade-off?

  • The relationship between IID performance and OOD robustness remains unresolved: robust-learning methods often sacrifice IID performance, though some preserve it.The review calls for research into conditions governing this trade-off.Source: §5.3 · p. 6

6 FUTURE RESEARCH DIRECTIONS

The future agenda calls for domain knowledge across model design, training, and evaluation, alongside stronger scrutiny of mitigation methods and deeper explanations of shortcut learning. It also points to adjacent research areas and both harmful and beneficial uses of shortcuts.

6.1 Introducing More Domain Knowledge

  • Domain knowledge may enter model architectures as task-dependent inductive biases to improve robustness and generalization beyond IID benchmarks, even if benchmark performance does not improve.The biases should be designed for each task's characteristics.Source: §6.1 · p. 6
  • RoBERTa-base with standard fine-tuning loss could outperform BERT-base with robust learning objectives on HANS, highlighting pre-training's role in generalization [2].The survey also notes anisotropy in pre-trained BERT embeddings, which can make unrelated words highly similar and affect downstream expressive power and accuracy [12] [18].Source: §6.1 · p. 6
  • Human-prior heuristics may miss unknown task biases, while evaluation datasets should cover more NLU tasks and a wider range of biases.The survey suggests incorporating more human-like commonsense knowledge into training [p14].Source: §6.1 · p. 6

6.2 Revisiting The Mitigation Approaches

  • BERT-base mitigated models remain below 70% accuracy on HANS for MNLI, while analysis suggests debiased models may encode more biases and gain OOD performance through a refined classification head [47, 22].The survey calls for investigating whether such models learn robust features and what accounts for their improved generalization.Source: §6.2 · pp. 6–7

6.3 In-depth Theoretical Understanding

  • Theory for shallow networks links simplicity bias and Gradient Starvation to shortcut learning, but a solid theoretical account of why LLMs learn shortcuts remains lacking [36] [49].Gradient Starvation describes gradient descent learning non-robust networks while slowing learning of robust, task-relevant features.Source: §6.3 · p. 7

6.4 Taking Inspiration from Other Directions

  • Robust learning is closer to domain generalization than domain adaptation, and long-tailed classification offers related approaches such as reweighting for robustness.Domain adaptation targets a specific target domain, whereas domain generalization and robust learning address a range of unknown conditions.Source: §6.4 · p. 7
  • Shortcut learning can also produce discrimination when models associate fairness-sensitive attributes such as ZIP code or surname with prediction labels [11].At inference, the model may amplify such bias toward demographic groups.Source: §6.4 · p. 7

6.5 Motivating Other Directions

  • Shortcuts may be deliberately inserted as backdoors using crafted text patterns, or used as trigger-pattern watermarks to help model owners claim ownership [44].These are respectively described as a potential security threat and a benign use.Source: §6.5 · pp. 7–8

7 PROMPT-BASED PARADIGM

The survey examines shortcut learning in prompt-based methods, distinguishing parameter-updating prompt fine-tuning from prompting with fixed parameters. Early findings show robustness varies by method and evaluation setting, and the evidence base remains limited.

7 PROMPT-BASED PARADIGM

  • Prompting has emerged for very large models as an alternative to standard fine-tuning, using instructions or demonstrations without gradient updates [30].The survey contrasts this with the standard pre-training and fine-tuning paradigm for medium-sized models.Source: §7 · p. 8

7.1 Robustness of Prompt-based Methods

  • Prompt-based fine-tuning updates model parameters, whereas prompting without fine-tuning keeps parameters fixed and applies different prompts [31].The survey associates the first approach with medium-sized models and the second with huge-sized models.Source: §7.1 · p. 8
  • Zero-shot prompt models showed greater robustness to lexical-overlap heuristics, but prompt-finetuned models adopted spurious heuristics with more labeled data and performed poorly OOD [48].The survey attributes this vulnerability to parameter updates using a biased NLI dataset, as in standard fine-tuning.Source: §7.1 · p. 8
  • In zero- and few-shot natural-language inference, T0 (3B, 11B) and GPT-3 (175B) learned just as quickly from irrelevant or misleading prompts as from effective, instructive prompts [51].The results indicate improvement was not derived from understanding task instructions in ways analogous to humans’ use of task instructions.Source: §7.1 · p. 8

7.2 Prompting versus Standard Fine-tuning

  • GPT-3 had a smaller generalization gap and higher OOD accuracy than fine-tuned BERT and RoBERTa in most tested settings, despite slightly worse in-distribution performance [38].On HANS, GPT-3 still showed substantial performance disparities between bias-supporting and bias-countering subsets.Source: §7.2 · p. 8
  • Prompt-based research primarily targets standard-benchmark performance, so robustness and generalization still require more thorough evaluation.The survey also calls for methods that improve OOD performance without sacrificing standard-benchmark performance.Source: §7.2 · p. 8

8 CONCLUSIONS

The survey reviews shortcut learning in LLMs for NLU, its causes and mitigation, and connections to other research directions. It concludes that data-driven training alone is insufficient for high-level NLU and advocates combining it with domain knowledge throughout design and evaluation.

8 CONCLUSIONS

  • The survey attributes shortcut learning to skewed datasets, model architecture, and learning dynamics, reviews mitigation solutions, and concludes that domain knowledge should complement data-driven LLM design and evaluation for high-level NLU.It also discusses future research directions and connections to related areas.Source: §8 · p. 8
Loading 2208.11857v2…