State-of-the-art generalisation research in NLP: A taxonomy and review

Motivation

• Systematic generalisation testing is not the NLP status quo despite its importance for robust, reliable, fair, capable, and human-like models.The ACL Anthology contained over 1200 papers mentioning generalisation in their title or abstract during the previous five years.Source: §1 · p. 2

Problem

• Traditional evaluation usually relies on randomly split, similarly sampled i.i.d. data, producing major progress without resolving how generalisation should be assessed beyond such splits.The introduction cites improvements in parsing, language modelling, GLUE, and scaled-up language-understanding benchmarks as examples of i.i.d. progress.Source: §1 · pp. 2–3

Motivation

• High i.i.d. scores do not guarantee robust generalisation across heuristics, stereotypes, memorisation, distribution shifts, subpopulations, or altered task, compositional, structural, and length conditions.These failures question which capabilities recent breakthroughs actually reflect.Source: §1 · p. 2

Contribution

• The proposed taxonomy characterises generalisation studies by motivation, generalisation type, data-shift type, shift source, and shift locus within the modelling pipeline.Figure 1 presents these five nominal axes, including practical, cognitive, intrinsic, and fairness motivations and compositional, structural, task, language, domain, and robustness generalisation types.Source: Figure 1 · p. 3, §1 · pp. 3–4
• The article reviews 449 papers and 619 generalisation experiments, providing a meta-analysis of the field’s current state and recommendations for more sound and exhaustive testing.It also releases online tools for exploring studies and aims to make state-of-the-art generalisation testing the new NLP status quo.Source: §1 · p. 3
• Evaluation cards let authors comprehensively summarise their papers’ generalisation experiments by marking their positions on the taxonomy’s five axes.The accompanying website can automatically generate LaTeX code for these cards.Source: §1 · p. 3, Figure 2 · p. 4

• The taxonomy comprises five axes for analysing generalisation research in NLP and is visualised in Figure 1 and summarised in Appendix E.Source: §2 · p. 4
• Evaluation cards let researchers characterise the experiments conducted in their studies using a format analogous to model cards and data sheets.An example appears in Figure 2, with usage guidance in Appendix B.Source: §2 · p. 4
• Using evaluation cards is intended to make generalisation evaluation the status quo and enable effective monitoring of research trends.Source: §2 · p. 4

2.1 Motivation: what is the high-level motivation for a generalisation test?

• The taxonomy distinguishes practical, cognitive, intrinsic, and fairness motivations, which determine what generalisation is desirable and how studies should be designed and interpreted.Studies can have multiple motivations, but the review identifies a main guiding motive because motivation affects the implications of observed generalisation or failure.Source: §2.1 · p. 4

2.1.1 Practical: in what settings can the model be used or improved?

• Practical studies assess where models can be deployed or which modelling changes improve performance across domains, collection conditions, time periods, and training-data biases.Examples include translation from canonical to noisy internet text (Michel and Neubig, 2018) and language-model generalisation across time periods (Lazaridou et al., 2021).Source: §2.1.1 · p. 5

2.1.2 Cognitive: does the model generalise like a human?

• Cognitive studies either evaluate whether models generalise like humans or use model generalisation to develop hypotheses about human language and cognition.Human reference points include rapid, low-data, and compositional generalisation (Lake et al., 2017; Marcus, 2018), while other work investigates cognition through models (Baroni, 2021; Hupkes, 2020; Lakretz et al., 2021b; Marcus, 1999; McClelland and Plaut, 1999).Source: §2.1.2 · p. 5

2.1.3 Intrinsic: does the model solve the task correctly?

• Intrinsic studies test whether models learned the intended task rather than relying on spurious patterns or non-generalisable heuristics.Depending on the account of correct learning, tests may require compositional generalisation, transfer across tasks, or performance across differently sampled datasets; models need not always match human generalisation, such as across languages.Source: §2.1.3 · pp. 5–6

2.1.4 Fairness and inclusivity: does the model generalise in a fair and responsible way?

• Fairness-oriented studies examine generalisation across diverse or marginalised demographics, the perpetuation of training-data biases, and efficiency for low-resource or compute-constrained groups.Efficiency is treated as a correlate of generalisation because models that learn quickly from less data can be more inclusive, accessible, and environmentally responsible (Marcus, 2018; Strubell et al., 2019).Source: §2.1.4 · p. 6

2.2 Generalisation type: what type of generalisation is a test addressing?

• The taxonomy identifies six generalisation types: compositional, structural, cross-task, cross-lingual, cross-domain, and robustness generalisation.These types are overviewed in Figure 3.Source: §2.2 · p. 6, Figure 3 · p. 7

2.2.1 Compositional generalisation

• Compositional generalisation tests whether models systematically recombine learned elements to produce correct outputs for novel combinations, but remains difficult to evaluate in ungrounded language models.It is commonly studied in sequence classification, translation, semantic parsing, and other generative tasks.Source: §2.2.1 · pp. 6–7

2.2.2 Structural generalisation

• Structural generalisation concerns producing or processing grammatically correct forms, including syntactic and morphological generalisation, without requiring an output meaning space.Large language models are difficult to assess because their training data and exposure to supposedly novel structures or words are often unknown.Source: §2.2.2 · pp. 7–8

2.2.3 Generalisation across tasks

• Cross-task generalisation examines whether models adapt across NLP problems, evolving from multitask and transfer learning to pretrain-finetune and parameter-free in-context learning.In-context studies often leave the relationship between training and test data uncontrolled, leaving important evaluation questions open.Source: §2.2.3 · pp. 8–9

2.2.4 Generalisation across languages

• Cross-lingual generalisation evaluates whether task abilities transfer across languages, commonly by training with labelled data in one language and testing in others.It is also studied through multilingual models trained on multiple languages, although relatively few multilingual papers explicitly focus on generalisation.Source: §2.2.4 · pp. 9–10

2.2.5 Generalisation across domains

• Cross-domain generalisation tests performance across topical, stylistic, demographic, or temporal differences, including domain adaptation and shifts between historical or future data.Such shifts are practically important because they frequently arise in naturally occurring NLP applications.Source: §2.2.5 · pp. 10–11

2.2.6 Generalisation in the context of robustness

• Robustness generalisation tests whether models avoid spurious correlations and learn solutions aligned with the underlying task despite unintended data-collection shifts.Common scenarios include annotation artefacts, inadequate standardised splits, and demographic imbalance, with worst-group accuracy proposed for detecting subpopulation failures.Source: §2.2.6 · pp. 11–12

2.3 Shift type: what kind of data shift is considered?

• The taxonomy formalises generalisation experiments by comparing distributional differences between datasets, where shift occurs when p(x1, y1) ≠ p(x2, y2).It decomposes distributions into p(x) and p(y|x), distinguishing changes in inputs, conditional labels, both, or neither.Source: §2.3 · pp. 12–13
• The review excludes no-shift i.i.d. evaluation from this discussion because within-distribution performance is often not indicative of complex out-of-distribution generalisation.No shift means both p(xtr) = p(xtst) and p(ytr|xtr) = p(ytst|xtst).Source: §2.3 · p. 13

2.3.1 Covariate shift

• Covariate shift changes p(x) while preserving p(y|x), testing whether models learned the underlying task across altered inputs; challenge sets deliberately target invalid heuristics.It is considered the easiest shift to address without additional training or processing, and more complex shifts are often converted into it.Source: §2.3.1 · p. 13

2.3.2 Label shift

• Label shift preserves p(x) but changes p(y|x), arising from annotation, temporal, domain, or task changes and typically requiring adaptation, new parameters, or task indicators.Prompting and domain indicators can recast some label shifts as covariate shift or no shift.Source: §2.3.2 · pp. 13–14

2.3.3 Full shift

• Full shift changes both p(x) and p(y|x), occurring in language modelling, multilingual adaptation, or cross-language and cross-data evaluation; it is challenging but need not require retraining.Like label shift, full shift is often transformed into an easier shift type.Source: §2.3.3 · p. 14

2.3.4 Multiple shifts

• Multiple shifts cover experiments examining relationships among several distributions, such as pretraining choices and subsequent out-of-distribution or cross-domain generalisation.The taxonomy currently does not distinguish configurations such as label+covariate from covariate+covariate.Source: §2.3.4 · p. 14

2.3.5 Assumed shift

• Assumed shift covers studies that posit training–test differences without verifying or grounding them, including cases constrained by linguistic plausibility, unavailable training data, or limited analysis.The review includes these studies despite their unverified shift assumptions.Source: §2.3.5 · p. 14

2.4 Shift source: how are training and test data obtained?

• The taxonomy distinguishes naturally occurring shifts, artificially partitioned natural corpora, generated shifts, and fully generated datasets.These sources differ in experimenters’ control over the base distribution and partitioning scheme.Source: §2.4 · p. 15

2.4.1 Naturally occurring shifts

• Naturally occurring shifts compare unmodified corpora differing in annotators, domains, populations, time periods, collection procedures, tasks, or languages.Experimenters control neither the base distribution nor the partitioning scheme.Source: §2.4.1 · pp. 15–16

2.4.2 Splits of natural corpora

• Artificially partitioned natural corpora retain naturally collected data but impose controlled splits along properties such as length, syntactic complexity, or grammatical environments.Experimenters control the partitioning scheme while lacking control over the underlying data.Source: §2.4.2 · p. 16

2.4.3 Generated shifts

• Generated shifts pair a natural corpus with a constructed partition designed to test patterns, heuristics, sequence complexity, or adversarial robustness.The constructed partition is usually the test set, although generated training data with natural test data also occurs less commonly.Source: §2.4.3 · p. 16

2.4.4 Fully generated

• Fully generated datasets provide the greatest control because experimenters design both the base distribution and partitioning scheme.They may be synthetic, templated, or narrowly selected natural language, with splits based on observable or globally computed properties.Source: §2.4.4 · p. 16

2.5 Locus of shift: between which data distributions does the shift occur?

• Five shift loci compare training–test, finetuning train–test, pretraining–training, pretraining–test, or all data distributions.These loci determine which modelling distributions and pipeline components are evaluated.Source: §2.5 · p. 17, Figure 6 · p. 17

2.5.1 The train–test locus

• The train–test locus evaluates shifted test partitions after training, either probing a model instance or comparing training procedures.Studies may examine a fixed parameter set θ∗ or how architecture and training-data choices produce generalising models.Source: §2.5.1 · p. 18

2.5.2 The finetune train–test locus

• The finetune train–test locus evaluates out-of-distribution finetuning test data, usually focusing on finetuning procedures rather than individual finetuned models.Experiments may also compare pretraining procedures, creating multiple loci when both pretraining and finetuning shifts are present.Source: §2.5.2 · p. 18

2.5.3 The pretrain-train locus

• The pretrain–train locus tests whether pretraining procedures produce parameter sets ˆθ that support downstream generalisation after later training.Studies receive this classification only when the final training stage uses i.i.d. splits; a shifted final stage creates multiple loci.Source: §2.5.3 · p. 18

2.5.4 The pretrain–test locus

• The pretrain–test locus directly evaluates pretrained models on out-of-distribution data, either without further training or after i.i.d. finetuning.This setting often assumes, rather than explicitly examines, the relationship between pretraining and test data and commonly appears in in-context learning.Source: §2.5.4 · p. 19

2.5.5 Multiple loci

• Multiple-loci experiments jointly assess shifts between pretraining and training and between training and testing across the full modelling pipeline.They evaluate both the generalisability of pretrained models and the success of finetuning, while the two shifts may differ in nature.Source: §2.5.5 · p. 19

3 A review of existing generalisation research

• The review labels 619 NLP generalisation experiments from 449 ACL Anthology papers across five taxonomy axes and provides searchable references, interactive visualisations, and evaluation cards.The taxonomy covers motivation, generalisation type, shift type, shift source, and shift locus.Source: §3 · p. 19

3.1 Overall trends on different axes

• Practical motivation dominates, while fairness accounts for only 3% of studies; cross-domain generalisation exceeds 30%, whereas structural and cross-lingual generalisation are least studied.Motivation trends remained relatively stable over the past five years, though fairness attention may be increasing and may be underdetected by the search procedure.Source: §3.1.1 · pp. 20–21, §3.1.2 · p. 21
• Covariate shift is by far the most common shift type, assumed shifts have increased, naturally occurring shifts comprise almost half of studies, and fully generated datasets account for 11%.Multiple-shift studies are comparatively promising because they assess generalisation across the modelling pipeline, but large uncontrolled corpora make training–test relationships difficult to determine.Source: §3.1.3 · pp. 21–22, §3.1.4 · p. 22
• Most studies examine finetune train–test splits, while pretraining-related and multi-locus shifts remain uncommon despite a recent increase in studies covering multiple loci and pretrain–test loci.The review attributes the scarcity of multi-stage studies partly to their greater computational requirements.Source: §3.1.5 · p. 22

3.2.1 What data shift source is used for different generalisation types?

• Shift sources vary by generalisation type: compositional studies predominantly use fully generated data, robustness, cross-lingual, and cross-task studies mainly use natural shifts, and structural studies span all sources.Few studies directly compare conclusions across generated, artificially partitioned, and naturally occurring shifts.Source: §3.2.1 · p. 22

3.2.2 For which loci of shift are different generalisation types studied?

• Cross-task generalisation is the only type frequently studied at pretrain–train or pretrain–test loci; other types usually use train–test or finetune-stage loci, and multi-locus studies are underrepresented.The review highlights no studies of compositional generalisation from pretraining to finetuning or training and calls for evaluation across the entire modelling pipeline.Source: §3.2.2 · pp. 22, 24

3.2.3 Which types of data shifts occur across different loci?

• Assumed shifts mostly occur at the pretrain–test locus, where limited knowledge of large pretraining corpora creates a substantial challenge for evaluating generalisation.Finetuning studies more often involve new labels or covariate shifts that do not require detailed understanding of pretraining data.Source: §3.2.3 · p. 24

3.2.4 How does motivation drive generalisation research?

• Practical motivation dominates cross-domain, robustness, cross-task, and cross-lingual studies, while cognitive motivation is common for compositional and structural studies; over 20% of practical studies nevertheless use artificial or fully generated data.This mismatch shows why studies should state their motivation explicitly so experimental conclusions remain informative for the intended research question.Source: §3.2.4 · p. 24

4 Conclusion

• NLP generalisation remains poorly defined and is not yet evaluated comprehensively, despite models sometimes failing drastically beyond random train–test splits.The authors aim to make state-of-the-art generalisation testing the field’s new status quo.Source: §4 · p. 25

4.1 The generalisation taxonomy

• The taxonomy characterises generalisation studies by motivation, target generalisation type, data-shift type, shift source, and shift locus in the modelling pipeline.It is intended to support critical analysis of existing research and the structured design of new studies.Source: §4.1 · p. 25

4.2 Existing work on generalisation: the taxonomy in action

• Analysis of 449 ACL Anthology papers showed that the taxonomy applies broadly and enables a comprehensive map of generalisation research, including field-wide patterns and future priorities.The reviewed papers contained generalisation-related words in their titles or abstracts.Source: §4.2 · pp. 25–26
• Future studies should align motivations with experimental designs, use evaluation cards, and expand attention to underrepresented cross-lingual and fairness-motivated generalisation.The review also recommends meta-studies comparing different evaluation paradigms and experiments spanning all stages of the modelling pipeline.Source: §4.2 · p. 26
• Prompting-based evaluations often neglect relationships between pretraining and test data, so the authors suggest drawing on human-participant methods when training data are uncontrolled.They identify assessment as especially difficult when large, inaccessible training data are involved.Source: §4.2 · p. 26

4.3 Future work

• The paper’s taxonomy and online tools are presented as a first step, but standardised tests and procedures remain absent.The authors propose a shared, transparent, community-governed platform with a leaderboard whose priorities evolve with models and knowledge.Source: §4.3 · pp. 26–27, §4.2 · p. 26

Taxonomy design

• The taxonomy uses coarse, theoretically grounded axes that support field-level conclusions but can conflate distinct training setups and shift difficulties.The authors suggest distinguishing pretrain-train variants, capturing distribution-shift magnitude, and extending the taxonomy to pipelines with more learning stages.Source: §5 · p. 27

Annotation

• Individual studies can be difficult to label because papers may state unclear evaluation goals and theoretical distinctions such as shift types are hard to estimate for natural distributions.The annotations therefore prioritise useful distinctions for systematising experiments rather than unknown ‘true’ distributional differences.Source: §5 · p. 27

Paper selection

• The review excluded multimodal studies because their multiple, heterogeneous distributions are difficult to characterise within one taxonomy.Including multimodal work is identified as a compelling direction for future research.Source: §5 · p. 27
• The review also excluded representation-generalisation studies, although the authors see value in extending the taxonomy and comparing behavioural with representational experiments.Such a comparison could form a meta-study using matched axis values.Source: §5 · p. 28

Assumption

• The framework assumes that strong generalisation is crucial for NLP models, but some large-language-model applications may be close to i.i.d. and require little complex generalisation.Researchers who believe their setup needs no generalisation should state and justify that position.Source: §5 · p. 28

A Annotation setup

• The review used defined procedures to select the papers included in its analysis.Source: A Annotation setup · p. 50
• The selected papers were annotated using procedures described in this section.Source: A Annotation setup · p. 50
• Together, paper selection and annotation establish the review’s methodological setup.Source: A Annotation setup · p. 50

A.1 Paper selection

• The corpus comprises 619 generalisation experiments from 449 papers after screening ACL searches and preliminary selections for genuine generalisation studies.Papers were excluded when they addressed unrelated method generalisation, used random train–test splits, or lacked generalisation experiments; multi-experiment entries were duplicated across differing taxonomy values.Source: §A.1 · p. 50
• The review is a static snapshot of this selection, while the accompanying website is intended to expand with missed and newly published generalisation papers.The full paper list is available in the paper’s second bibliography and online.Source: §A.1 · p. 50

Annotation procedure

• Each paper received five taxonomy-axis labels from one annotator, checked by a second and adjudicated by discussion or a third annotator when needed.The annotation process was guided by Figure 11 and an online FAQ addressing common questions.Source: §A.2 · p. 50, Figure 11 · p. 51

Task annotation

• Task annotation recorded multiple tasks, overarching categories where possible, or multitask when studies demonstrated simultaneous capabilities.The labeling scheme also annotated which task or tasks each study considered.Source: §A.2 · p. 50

Multiple experiments

• Experiments with different taxonomy-axis values were recorded separately, allowing one paper to contribute multiple generalisation questions or experiments.Figure 11 illustrates that the annotation diagram may need to be followed multiple times for papers covering distinct generalisation settings.Source: §A.2 · p. 50, Figure 11 · p. 51

B Evaluation cards

• Evaluation cards visualise all generalisation experiments in a study, while preserving which data shifts and modelling loci are investigated together.For example, one experiment can represent covariate shift during fine-tuning and label shift between pretraining and training.Source: B Evaluation cards · p. 52
• The authors provide a web tool that generates one- or two-column evaluation cards for inclusion in paper appendices.The tool’s rendering in the web interface is shown in Figure 12.Source: B Evaluation cards · p. 52

• Individual author contributions are documented using Elsevier’s Contributor Roles Taxonomy (CRediT)19, with authors listed in author-list order.The taxonomy is used to recognise individual contributions.Source: C Author contributions · p. 53
• Table 1 presents the individual author contributions using CRediT, with work done outside Amazon noted separately.Source: Table 1 · p. 53, C Author contributions · p. 53

D Multi-lingual benchmarks

• Multilingual benchmarks enable evaluation of cross-lingual generalisation through translated datasets, native-speaker annotations, existing multilingual resources, and aggregated benchmark suites.Aggregated suites additionally support simultaneous evaluation of cross-task and cross-language generalisation.Source: D Multi-lingual benchmarks · p. 54

Overview

• The taxonomy is based on a detailed analysis of existing NLP generalisation studies and organises experimental choices across five axes.Together, the axes provide a comprehensive picture of a study’s motivation, goal, and experimental setup.Source: E A concise summary of our taxonomy · p. 55

Motivation

• Studies differ in four motivations: practical, cognitive, intrinsic, and fairness and inclusivity.Motivation influences the desired generalisation, conclusions drawn from results, and experimental design.Source: E A concise summary of our taxonomy · p. 55

Generalisation type

• The taxonomy identifies six generalisation types: compositional, structural, cross-task, cross-lingual, cross-domain, and robustness generalisation.This axis specifies what a generalisation test is intended to capture.Source: E A concise summary of our taxonomy · p. 55

Data shift

• Data shifts are classified as covariate, label, full, assumed, or multiple shifts.The shift classification reflects statistical distinctions and informs suitable experimental setups.Source: E A concise summary of our taxonomy · p. 55

Shift source

• Shift sources comprise naturally occurring shifts, artificially partitioned natural corpora, generated shifts, and fully generated datasets.The source determines experimenter control over training and testing data and the conclusions an experiment can support.Source: E A concise summary of our taxonomy · p. 55

Shift locus

• The shift locus spans transitions between pretraining, training, and testing, including shifts between multiple stages simultaneously.Together with shift type, the locus determines which modelling-pipeline component and generalisation question are investigated.Source: E A concise summary of our taxonomy · p. 55

• Robustness and distribution-shift studies examine accents, capitalization errors, adversarial inputs, domain shift, social-media variation, and out-of-distribution prediction.The bibliography includes AccentDB, robustness to capitalization errors in named entity recognition, adversarial data generation for question answering, and domain-shift effects on discourse parsing.Source: F List of publications included in our… · pp. 56–58
• Cross-lingual and multilingual generalisation studies include transfer in machine translation, parsing, classification, named entity recognition, and language modeling.Examples include syntax-augmented multilingual BERT, few-shot cross-lingual de-identification, multilingual parsing, and cross-lingual transferability of monolingual representations.Source: F List of publications included in our… · pp. 56–57
• Few-shot and zero-shot generalisation publications address sequence modeling, stance detection, question answering, classification, and transfer learning.The listed work includes lexicon learning, generalized topic representations, adversarial stance detection, self-supervised knowledge-triplet learning, and meta-learning across classification tasks.Source: F List of publications included in our… · pp. 56–57
• Other listed publications investigate compositionality, inductive transfer, data augmentation, language-model generalisation, bias, evaluation, and task-specific adaptation.Examples include SCAN generalisation, sharpness-aware minimization, few-shot evaluation, unintended-bias metrics, legal-language benchmarking, and domain and tagset adaptation for named entity recognition.Source: F List of publications included in our… · pp. 57–58
• The list includes studies of robustness, adaptation, and data augmentation, including HiddenCut, posterior differential regularization, multilingual mix, and domain adaptation methods.Examples cover natural language understanding, machine translation, semantic parsing, and sentiment analysis.Source: F List of publications included in our… · pp. 59, 61
• Additional entries concern generalisation in summarization, semantic parsing, information extraction, question answering, named entity recognition, and natural language generation.The list includes cross-dataset summarization, event extraction, text-to-SQL parsing, zero-shot slot filling, and few-shot NER.Source: F List of publications included in our… · pp. 59, 61
• Multilingual and cross-lingual generalisation appears in studies of typology, transfer, multilingual NLU, semantic role labeling, machine translation, and morphological inflection.The list includes work involving UDify, mBERT, Arabic, multilingual translation, and cross-lingual datasets.Source: F List of publications included in our… · pp. 59, 60, 61, 78
• It covers compositional, systematic, morphological, and length generalisation through work on meta-learning, transformers, bilingual lexicon induction, noun compounds, and longer sequences.The cited studies examine composition in semantic parsing, neural machine translation, morphology, and sequence extrapolation.Source: F List of publications included in our… · pp. 60–61
• The publications also address shortcut learning, unintended bias, adversarial debiasing, constraint-based behavior, and data leakage in NLP evaluation.These works investigate robustness and reliability in text classification, NLU models, multilingual settings, and performance measurement.Source: F List of publications included in our… · pp. 60–62
• The cited publications cover text-to-SQL evaluation, evaluating generalization, local decision boundaries, and pretrained transformers.Examples include Finegan-Dollak et al. (2018), Fisch et al. (2019), Gardner et al. (2020), and Hendrycks et al. (2020).Source: F List of publications included in our… · pp. 62, 63, 65
• The list covers multilingual, cross-lingual, and domain-shift research, including Massive, multilingual fact checking, word-in-context disambiguation, and cross-domain parsing.These works include FitzGerald et al. (2022), Gupta and Srikumar (2021), Hettiarachchi and Ranasinghe (2021), and Fried et al. (2019).Source: F List of publications included in our… · pp. 62, 64, 65
• The references also encompass compositional and systematic generalisation, semantic parsing, reasoning, linguistic structure, and task-specific transfer.Examples include Gai et al. (2021), Gontier et al. (2020), Goodwin et al. (2022), Herzig and Berant (2021), and He et al. (2020).Source: F List of publications included in our… · pp. 62–65

F List of publications included in our review

• It also includes publications on cross-lingual, multilingual, cross-domain, and low-resource generalisation across classification, tagging, segmentation, and question answering.Examples include XTREME, zero-shot cross-lingual transfer, cross-domain Chinese word segmentation, low-resource POS tagging, and UNIFIEDQA.Source: F List of publications included in our… · pp. 65, 67
• The list further covers distribution shifts and adaptation methods involving counterfactual data, continual learning, transfer learning, domain attention, and in-the-wild benchmarks.Included examples address counterfactually augmented data, lifelong learning, parameter-efficient multi-task fine-tuning, and WILDS distribution shifts.Source: F List of publications included in our… · pp. 66–68
• The bibliography includes work on compositional, systematic, and syntactic generalisation in neural models and semantic parsing.Examples include compositional word-pair embeddings, realistic compositional-generalisation measurement, COGS, auxiliary sequence prediction, and label-aligned graphs.Source: F List of publications included in our… · p. 66

Multilingual and low-resource generalisation

• Several publications examine multilingual, cross-lingual, and low-resource generalisation across transliteration, translation, named entity recognition, and morphology.The bibliography includes multilingual transliteration, cross-lingual NER, multilingual machine translation, and low-resource morphological segmentation.Source: F List of publications included in our… · pp. 69–70

Compositional and structural generalisation

• The bibliography includes Lake and Baroni’s 2018 study of compositional generalization in sequence-to-sequence recurrent networks.Source: F List of publications included in our… · p. 69

Robustness and distribution shift

• It covers robustness to domain, temporal, adversarial, and distributional shifts in language understanding and generation.The listed work includes temporal generalization, domain adaptation, adversarial training, and resistance to adversarial attacks.Source: F List of publications included in our… · pp. 69–70

Methods for improving generalisation

• Additional entries study generalisation through data selection, perturbation, prompting, model compression, and representation or auxiliary-task design.The bibliography includes reinforced training-data selection, perturbed embeddings, multi-stage prompting, super tickets, and auxiliary-task approaches.Source: F List of publications included in our… · pp. 69–71

Transfer and task generalisation

• The list also contains work on zero-shot, few-shot, and cross-task transfer for question answering, entity linking, dialogue, and intent classification.Examples include zero-shot entity linking, domain-agnostic question answering, cross-task dialogue-state tracking, and cross-domain few-shot intent classification.Source: F List of publications included in our… · pp. 56, 69, 70

• The listed studies examine compositional generalization through recurrent networks, neural program synthesis, and transformers (Loula et al., 2018; Nye et al., 2020; Ontanon et al., 2022).Source: F List of publications included in our… · pp. 71, 74
• Several publications address domain, cross-dataset, cross-lingual, and cross-task transfer using adaptation, augmentation, multilingual models, and multitask learning (Ludwig et al., 2022; Malkiel and Wolf, 2021; M’hamdi et al., 2021).Source: F List of publications included in our… · pp. 71–72
• The bibliography also covers robustness and shift evaluation, including real distributional shifts, out-of-domain robustness, adversarial training, and cross-dataset abuse detection (Malinin et al., 2021; Ng et al., 2020; Nejadgholi and Kiritchenko, 2020).Source: F List of publications included in our… · pp. 72, 74
• Other included work examines syntactic heuristics in natural language inference, hierarchical inductive bias in sequence-to-sequence networks, linguistic features for generalization, and variability across models with similar test-set performance (McCoy et al., 2019; McCoy et al., 2020a; McCoy et al., 2020b; Moosavi and Strube, 2018).Source: F List of publications included in our… · pp. 72–73
• The list includes studies of compositional, systematic, and algorithmic generalisation in sequence models, machine translation, morphology, reasoning, and small datasets.Examples include compositional sequence models, morphological reinflection across languages, logical reasoning, and grokking beyond overfitting.Source: F List of publications included in our… · pp. 74–76
• Several publications examine cross-lingual and multilingual generalisation through transfer, zero-shot learning, multilingual datasets, and cross-lingual representations.The listed work covers content filtering, machine translation, event coreference, causal commonsense reasoning, task and language transfer, and multilingual quality estimation.Source: F List of publications included in our… · pp. 74–76
• Other listed studies investigate few-shot learning, pretraining, representation transfer, structured prediction, reasoning, and generalisation in specialised applications.The applications include dialogue, question answering, relation extraction, clinical NLP, hate-speech detection, authorship, historical text normalisation, and product-title generation.Source: F List of publications included in our… · pp. 75–77
• The publications also address robustness to domain, topic, language, and distributional variation through adaptation, augmentation, challenge data, and behavioral testing.Examples include multi-task domain adaptation, text augmentation, cross-topic rumor detection, model compression for domain adaptation, and CheckList behavioral testing.Source: F List of publications included in our… · pp. 75, 77

F List of publications included in our review

• The listed studies examine NLI model behavior, intent-classification generalization, and spurious correlations in cross-topic settings.Examples include Sanchez et al. (2018), Shen et al. (2021b), and Thorn Jakobsen et al. (2021).Source: F List of publications included in our… · pp. 78, 80
• Other publications investigate compositional, logical, mathematical, and syntactic generalisation through semantic parsing, diagnostic benchmarks, and language-model analyses.Representative works include Shaw et al. (2021), Sinha et al. (2019), Saxton et al. (2019), and Shen et al. (2021c).Source: F List of publications included in our… · pp. 78–79
• The bibliography also covers transfer and evaluation across domains, languages, datasets, and tasks, including cross-database semantic parsing, multilingual entailment, domain-generalised translation, and broad benchmarks.Examples include Suhr et al. (2020), Vulić et al. (2019), Vu et al. (2022), Rybak et al. (2020), and Talmor and Berant (2019).Source: F List of publications included in our… · pp. 77, 79, 80

• Wang and Warstadt entries include work on domain generalization, semantic parsing, multilingual interference, language-model linguistic generalization, and benchmarked language understanding (Wang et al., 2021a; Wang et al., 2020; Warstadt et al., 2020).Source: F List of publications included in our… · p. 81
• The cited entries address language modelling, zero-shot learning, syntactic rule learning, and undersensitivity in neural reading comprehension.Source: F List of publications included in our… · p. 81
• Later entries include structural supervision, multilingual embeddings, combinatorial productivity, reading-comprehension robustness, QA generalization, and multidomain classification (Wilcox et al., 2020; Winata et al., 2019; Wong and Wang, 2007; Wu et al., 2020).Source: F List of publications included in our… · p. 82
• The cited publications cover prompt-based pretraining, transformer optimization on small datasets, cross-domain transfer, robustness, and assessment of pretrained models’ generalization capacity.Source: F List of publications included in our… · pp. 82–83
• The final entries cover cross-domain sentiment, constituency parsing, few-shot learning across tasks, and compositional generalization for neural semantic processing.Source: F List of publications included in our… · p. 83

Adaptation and transfer

• The bibliography includes studies on adversarial training, domain adaptation, cross-lingual adaptation, and low-resource transfer.Examples include Yoo and Qi (2021), Yu and Joty (2021), Yu et al. (2021), Zaharia et al. (2022), and Ziser and Reichart (2017).Source: F List of publications included in our… · pp. 84–85

Question answering and semantic parsing

• Question-answering and text-to-SQL publications address cross-domain parsing, contextual information, clinical reading comprehension, retrieval, and schema expansion.The entries include Spider, SParC, SituatedQA, emrQA, SPARTA, and schema expansion for text-to-SQL.Source: F List of publications included in our… · pp. 84–85

Generation, dialogue, and speech

• The bibliography also includes dialogue, summarization, generation, and speech-related studies examining robustness, adaptation, evaluation, and multilingual transfer.Examples include zero-shot dialogue generation, dialogue response evaluation, low-resource summarization, multilingual NLG, and speaker adaptation for ASR.Source: F List of publications included in our… · pp. 84–85

Robustness, bias, and evaluation

• Other included publications study bias, robustness, explanatory knowledge, temporal reasoning, and evaluation methods for NLP systems.The list contains work on WIKIBIAS, unreliable-news datasets, commonsense inference, temporal relations, MoverScore, and robust inference capabilities.Source: F List of publications included in our… · p. 85