PaperlayerView exact revision on arXiv

Source-linked AI summary

State-of-the-art generalisation research in NLP: A taxonomy and review

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, Zhijing Jin

arXiv:2210.03050v4cs.CLcs.AI

TL;DR

NLP generalisation is important but poorly standardised, and strong i.i.d. performance can fail to predict broader generalisation. This paper introduces a five-axis taxonomy, applies it to a large literature review, and uses the resulting map to recommend more comprehensive evaluation.

  • Problem

    NLP lacks agreement and standards for defining and evaluating good generalisation, while i.i.d. performance can fail across scenarios requiring other forms of generalisation.

  • Method

    The paper develops a five-axis taxonomy, classifies 449 papers and 619 experiments, and provides evaluation cards, online tools, and a field-wide analysis.

  • Results

    The review maps existing generalisation research and identifies overall patterns and areas that should be prioritised in future work.

  • Takeaways & Limitations

    Evaluation cards and online tools provide a structured basis for characterising studies and moving state-of-the-art generalisation testing toward a new NLP status quo.

  • Takeaways & Limitations

    The taxonomy reflects substantial author discussion and trade-offs concerning its definition, annotation process, and paper selection.

Abstract

from arXiv · show

The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any evaluation standards for generalisation. In this paper, we lay the groundwork to address both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they investigate, the type of data shift they consider, the source of this data shift, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis that maps out the current state of generalisation research in NLP, and we make recommendations for which areas might deserve attention in the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to update as new NLP generalisation studies are published. With this work, we aim to take steps towards making state-of-the-art generalisation testing the new status quo in NLP.

1 Introduction

NLP generalisation is widely valued, but traditional evaluation and fragmented research practices leave its meaning, assessment, and priorities unclear. The paper proposes a taxonomy, review, and practical tools to systematise this area.

  • Good generalisation transfers representations, knowledge, and strategies from past experience to new experiences, supporting robust, reliable, and fair NLP models.
  • Systematic generalisation testing is not yet the field’s status quo despite the importance attributed to generalisation.
  • Research lacks agreement about what good generalisation entails, which types matter, and how results from different experimental setups relate.
  • High performance on random or i.i.d. train–test splits can coexist with failures across scenarios requiring different forms of generalisation.
  • The paper introduces a five-axis taxonomy, reviews 449 papers and 619 experiments, and provides evaluation cards and online tools for analysing generalisation research.

2 The generalisation taxonomy

The paper’s taxonomy provides a structured way to characterise generalisation studies and support both retrospective analysis and the design of future research.

  • Evaluation cards operationalise the taxonomy by letting researchers characterise their generalisation experiments across its axes.
  • The taxonomy is intended to explain existing generalisation research while actively structuring ongoing and future studies.

2.1 Motivation: what is the high-level motivation for a generalisation test?

Generalisation studies pursue practical, cognitive, intrinsic, or fairness-oriented goals, and their motivation shapes both experimental design and the conclusions that tests support.

  • The taxonomy distinguishes four intertwined motivations: practical, cognitive, intrinsic, and fairness.
  • Motivation determines which generalisation is desirable, shapes experimental design, and affects conclusions drawn from model behaviour.
  • The same question can have different motivations, and studies may inform conclusions across multiple motivational dimensions.
  • Practical motivation: Practical studies assess deployment scenarios or modelling changes, including generalisation across domains and data-collection processes.
  • Cognitive motivation: Cognitive studies may evaluate models as references for human intelligence or use them to develop hypotheses about human cognition and language.
  • Fairness motivation: Fairness-oriented studies examine whether models generalise to diverse demographics and perpetuate undesirable biases from training data.

2.2 Generalisation type: what type of generalisation is a test addressing?

The taxonomy identifies six generalisation types, spanning cognitively motivated capabilities such as compositional and structural generalisation and practically motivated transfer across tasks, languages, domains, and robustness settings.

  • The six generalisation types are compositional, structural, task, language, domain, and robustness generalisation.
  • Compositional generalisation: Compositional generalisation recombines previously learned elements to map novel inputs made from them to correct outputs.
  • Structural generalisation: Structural generalisation evaluates novel syntactic structures or morphological forms, without requiring an output meaning or interpretation space.
  • Cross-task generalisation: Cross-task generalisation concerns adapting one model to multiple NLP problems, through multitask learning, pretraining and finetuning, or in-context learning.
  • Robustness generalisation: Robustness generalisation tests whether models abstract away from spurious training correlations and align with the task solution humans associate with the task.

2.3 Shift type: what kind of data shift is considered?

The taxonomy characterises generalisation tests by the distributional shifts between training, pretraining, and test data. It distinguishes shifts in inputs, outputs, both, or multiple distributions, while also recording cases where shifts are assumed rather than verified.

  • Shift types: Generalisation tests compare data distributions when p(x1, y1) ≠ p(x2, y2), focusing here on out-of-distribution relationships.The taxonomy sets aside the no-shift i.i.d. case and concentrates on o.o.d. evaluation.
  • Shift types: Covariate shift changes p(x) while keeping p(y|x) unchanged, testing whether models learned the underlying task rather than invalid input heuristics.Challenge sets such as HANS, PAWS, and COGS deliberately target this pattern.
  • Shift types: Label shift keeps p(x) fixed but changes p(y|x), arising from annotation, temporal, domain, or task changes.The paper notes that NLP studies often treat label shift as an obstacle rather than directly evaluating it.
  • Shift types: Full shift changes both p(x) and p(y|x), making it an especially challenging distributional mismatch across languages, tasks, or data types.Such shifts can nevertheless sometimes be addressed without retraining.
  • Shift types: The review also records multiple shifts across several distributions and assumed shifts whose train–test relationship is not verified or grounded in prior research.Authors mainly classify shifts explicitly considered and supported by literature or empirical checks, while retaining unverified cases as assumed shift.

2.4 Shift source: how are training and test data obtained?

The shift-source axis describes how differences between pretraining, training, and test data arise and therefore how much experimental control researchers have. It distinguishes naturally occurring data, artificial splits of natural corpora, constructed partitions, and generated data.

  • Shift source: Shift source determines experimenter control over the data and the conclusions that a generalisation test can support.The taxonomy distinguishes four sources of shifts.
  • Naturally occurring shifts: Naturally occurring shifts use corpora that differ without systematic intervention, such as across annotators, domains, populations, time periods, or collection procedures.Researchers have no direct control over the base distribution or partitioning scheme.
  • Splits of natural corpora: Artificially splitting a natural corpus controls the partitioning rule while leaving the underlying data uncontrolled, for example by assigning longer or syntactically distinct samples to test.The split dimension is chosen by the experimenter rather than arising naturally between corpora.
  • Constructed partitions: Constructed partitions pair a natural corpus with data deliberately designed to contain or avoid patterns, violate heuristics, or test unusual length or complexity.This setup targets a specific generalisation property through the engineered partition.
  • Generated data: Generated-data studies control both the base distribution and partitioning scheme, making them the most precise way to measure targeted aspects of generalisation.Generated data may be fully synthetic, templated natural language, or a narrow corpus selection.

2.5 Locus of shift: between which data distributions does the shift occur?

The locus-of-shift axis identifies which stages of the modelling pipeline differ, thereby determining which generalisation question an experiment can answer. The taxonomy covers five loci spanning training, finetuning, pretraining, testing, and combinations of these stages.

  • Locus of shift: Shift loci specify whether distributional differences occur between pretraining, training, finetuning, and testing, including experiments spanning multiple stages.The five loci determine which part of the contemporary modelling pipeline is investigated.
  • Train–test locus: A train–test locus evaluates a model trained on one partition and tested on a shifted out-of-distribution partition.It can target either the generalisation ability of a model instance or another part of the pipeline, depending on the setup.
  • Finetune train–test locus: A finetune train–test locus evaluates whether a model trained earlier can handle a shifted test set during finetuning.The shift is between finetuning training and finetuning test data.
  • Pretraining–training locus: A pretraining–training locus tests whether a pretraining procedure produces parameters useful for later training on different tasks or domains.The focus is downstream generalisation after the pretraining stage.
  • Pretraining–test locus: A pretraining–test locus evaluates a pretrained model directly on out-of-distribution data, with no further training in in-context-learning setups or with i.i.d. finetuning.This locus can examine the pretrained model itself rather than only its downstream adaptation.
  • Multiple loci: Multiple-loci experiments jointly assess shifts between pretraining and training and between training and testing, covering all modelling stages.They evaluate the generalisability of the full pipeline rather than one isolated transition.

3 A review of existing generalisation research

The review maps NLP generalisation research across five taxonomy axes using 619 experiments from 449 ACL Anthology papers. It finds strong concentration in practical, cross-domain, covariate-shift, naturally occurring, and train–test settings, while multi-stage and several other combinations remain less studied.

  • Scope and taxonomy: The review classifies 619 generalisation experiments from 449 ACL Anthology papers across five categorical taxonomy axes.The axes cover motivation, generalisation type, shift type, shift source, and shift locus.
  • Motivations: Practical motivation is most common, while fairness-oriented studies comprise only 3% of the reviewed total.Intrinsic and cognitive motivations follow practical motivation; motivation trends remained relatively stable over the past five years.
  • Generalisation type: Cross-domain generalisation is most frequent at more than 30% of studies, followed by robustness, cross-task, and compositional generalisation.Structural and cross-lingual generalisation are least commonly investigated in the review.
  • Shift type: The vast majority of studies examine covariate shift, while assumed shifts have increased over the past few years.The review links assumed shifts to difficulty analysing relationships with increasingly large, general-purpose training corpora, limiting what conclusions can be drawn.
  • Shift source: Almost half of studies use naturally occurring shifts, whereas the shift source varies substantially across generalisation types.Compositional studies predominantly use fully generated data, while cross-task, cross-domain, and cross-lingual studies more often use natural or artificially split natural corpora.
  • Shift locus and interactions: Most studies focus on finetuning train–test splits, with relatively few examining multiple modelling stages or shifts involving pretraining.Cross-task generalisation is the main exception, appearing frequently at pretrain–train and pretrain–test stages; assumed shifts mostly occur at pretrain–test.

4 Conclusion

The paper introduces a five-axis taxonomy and applies it to a broad review of NLP generalisation research. The review maps recurring patterns and recommends priorities for improving evaluation and comparability.

  • The framework characterises studies by motivation, generalisation type, data-shift type, shift source, and modelling-pipeline locus.
  • The review finds that study goals are not always aligned with experimental designs, motivating explicit reporting of motivations and evaluation cards.
  • Cross-lingual and fairness-motivated generalisation studies are underrepresented and should receive more attention.
  • Studies targeting similar questions often use widely varying evaluation setups, motivating meta-studies comparing experimental paradigms.
  • Most studies examine only one modelling-pipeline stage, leaving a need to assess whether generalisation persists throughout training.
  • Prompt-based evaluations often omit the relationship between pretraining and test data, especially when training data are large and uncontrolled.
  • The review’s static conclusions are complemented by an online website with updateable entries, visualisations, and search tools.
  • The framework improves structure and comparability but does not provide standardised data or procedures for generalisation testing.

5 Limitations

The taxonomy and review involve deliberate scope and categorisation choices that constrain interpretation. The authors identify coarse axes, debatable annotations, excluded paper types, and an assumption that strong generalisation is important.

  • The taxonomy uses relatively coarse axis values to support field-level conclusions, but these can group studies that differ in meaningful ways.
  • Individual studies can be difficult to label because papers may state unclear aims or because theoretical distinctions are hard to apply to real distributions.
  • The review excludes multimodal studies because they involve more distributions and distribution shifts that are harder to characterise within one taxonomy.
  • The review also excludes representational generalisability studies, such as probing papers, although the authors consider their inclusion valuable future work.
  • The paper assumes that strong generalisation skills are crucial for NLP models, while acknowledging that some application settings may not require complex generalisation.

A Annotation setup

The review describes the procedures used to select and annotate papers.

  • The annotation setup covers the procedures used for paper selection and annotation.

A.1 Paper selection

Paper selection began with a preliminary literature review and an ACL Anthology search based on generalisation-related title or abstract terms.

  • The authors retrieved ACL Anthology papers containing generalisation-related word forms in their titles or abstracts, after an initial substantive literature review.

A.2 Annotation

The annotation process assigns taxonomy labels to papers through multi-annotator review, using a diagram as a simplified guide and recording distinct experiments separately when studies differ.

  • Annotation procedure: Each paper received five taxonomy labels from a first annotator, which a second annotator checked; unresolved disagreements involved a third annotator.The Figure 11 diagram guided annotation, while an online FAQ addressed recurring questions.
  • Annotation procedure: The annotation diagram indicates where papers contain the information needed to complete the review.It is an aid for characterisation rather than a complete representation of the taxonomy.
  • Multiple questions: Papers with multiple generalisation questions require the diagram to be followed separately for each question.Examples include combining cross-domain with cross-task generalisation or natural splits with generated shifts.

B Evaluation cards

Evaluation cards provide a standardised visual summary of a paper’s generalisation experiments and preserve which shifts and pipeline loci occur together.

  • Purpose: Evaluation cards visualise all generalisation experiments in a study to show how extensively a model was evaluated.They are proposed as a standardised addition to the taxonomy and can be generated through the paper’s website tool.
  • Information captured: Unlike the review’s grouped analysis, evaluation cards record which shifts and modelling loci are investigated in the same experiment.The example combines a covariate shift during finetuning with a label shift from pretraining to training.

C Author contributions

The supplied material identifies the author-contribution section and its accompanying CRediT table, but provides no substantive contribution details.

  • Contribution record: The paper details individual author contributions using Elsevier’s Contributor Roles Taxonomy, with authors listed in publication order.The supplied passage refers to CRediT as CRediT19 and links to the taxonomy’s source.
  • Contribution record: Table 1 is labelled as a record of individual author contributions using CRediT.

D Multi-lingual benchmarks

The paper notes that multilingual benchmarks can support cross-lingual generalisation evaluation, while the supplied figure caption presents an interactive overview of the literature review.

  • Multilingual benchmarks: Multilingual benchmarks or datasets are created in several ways and can be used to evaluate cross-lingual generalisation.The paper notes that multilingual modelling papers do not often focus explicitly on generalisation.
  • Literature-review overview: Figure 13 provides an overview of the literature review, including interactions, with an interactive version available online.The caption directs readers to Section 3 for detailed explanations and analyses.

E A concise summary of our taxonomy

The proposed taxonomy characterises NLP generalisation studies along five axes, covering their motivation, generalisation type, data shift, shift source, and shift locus in the modelling pipeline.

  • Taxonomy overview: The taxonomy is based on a detailed analysis of many existing NLP generalisation studies and identifies five principal axes along which they differ.Together, the axes describe the motivation and goals of generalisation studies.
  • Motivation: A study’s motivation influences the desirable type of generalisation, the conclusions drawn from model behaviour, and the experimental design.Researchers should align the experimental setup with the study’s motivating questions.
  • Type of generalisation: The generalisation-type axis specifies what a test is intended to capture and includes six main types identified in the literature.The passage lists compositional, structural, cross-task, and cross-lingual generalisation among these types.
  • Data shift: The data-shift axis describes the kind of shift considered and gives the taxonomy a statistical interpretation of generalisation.Different shifts call for different experimental setups.
  • Source of shift: The shift-source axis distinguishes naturally occurring shifts, artificially partitioned natural corpora, generated shifts, and fully generated datasets.The source determines experimental control over training and testing data and affects the conclusions that can be drawn.
  • Locus of shift: The shift-locus axis identifies which part of the contemporary modelling pipeline is being tested, together with the shift type defining the generalisation question.It completes the taxonomy by locating where generalisation is investigated.

F List of publications included in our review

The review’s publication list spans studies addressing generalisation through multilingual transfer, domain and robustness shifts, few-shot learning, representation learning, and related evaluation settings.

  • Domain and robustness shifts: The list includes studies examining domain shift, accents, capitalization, political heterogeneity, and other robustness-related changes in NLP data.Examples address Yoruba domain and diacritics, speech-recognition accents, named-entity capitalization errors, and socio-political news classification.
  • Few-shot and zero-shot generalisation: Few-shot and zero-shot generalisation appears across stance detection, question answering, sequence modelling, natural-language classification, and transfer learning.The listed works include generalized topic representations, lexicon learning, zero-shot question answering, and few-shot classification.
  • Cross-lingual and multilingual studies: The included publications cover cross-lingual transfer and multilingual modelling across classification, parsing, machine translation, and representation-learning settings.Examples include work on cross-lingual transferability, multilingual parsing, and multilingual BERT.
  • Other generalisation settings: Other included studies investigate compositional or structural generalisation, adversarial evaluation, multitask learning, and alternative modelling architectures.Examples include SCAN tasks, adversarial human annotation, synthetic adversarial data, multitask scaling, and pipeline versus end-to-end generation.
Loading 2210.03050v4…