PaperlayerView exact revision on arXiv

Source-linked AI summary

Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation

Aparna Elangovan, Jiayuan He, Karin Verspoor

arXiv:2102.01818v1cs.CLcs.LG

TL;DR

Train/test overlap in public NLP benchmarks can make evaluations of generalization difficult to interpret. This paper measures overlap across datasets and examines its association with model performance. It concludes that stratifying test sets by similarity supports a more robust assessment of memorization versus generalization.

  • Problem

    Train/test overlap can inflate benchmark results and leave a single held-out test-set metric insufficient to infer generalizability.

    Source: §1 · p. 1, §7 · p. 5

  • Method

    The paper proposes a framework to quantify train/test overlap and stratify test sets by similarity for assessing model performance.

    Source: §1 · p. 2, §7 · p. 6

  • Results

    For BC3ACT, F1-score rises from 56.4 to 63.0 as similarity increases, while accuracy drops from 85.8 to 77.5.

    Source: §5.2 · p. 4

  • Takeaways & Limitations

    Quantifying overlap and stratifying test sets by similarity enable more robust assessment of models’ memorization versus generalization capabilities.

    Source: §7 · pp. 5–6

    The bag-of-words similarity approach may flag semantically different texts, and acceptable similarity depends on the data and task.

    Source: §6.1 · p. 5

Abstract

from arXiv · show

Public datasets are often used to evaluate the efficacy and generalizability of state-of-the-art methods for many tasks in natural language processing (NLP). However, the presence of overlap between the train and test datasets can lead to inflated results, inadvertently evaluating the model's ability to memorize and interpreting it as the ability to generalize. In addition, such data sets may not provide an effective indicator of the performance of these methods in real world scenarios. We identify leakage of training data into test data on several publicly available datasets used to evaluate NLP tasks, including named entity recognition and relation extraction, and study them to assess the impact of that leakage on the model's ability to memorize versus generalize.

1 Introduction

Public benchmarks enable shared evaluation, but train/test overlap can undermine whether they measure generalization to unseen data. The paper examines overlap across NLP datasets and proposes a framework for assessing its impact on memorization versus generalization.

1 Introduction

  • Train/test overlap can inflate results for models that memorize more effectively than they generalize, raising questions about evaluations’ external validity.Such models may make poor-quality predictions outside the shared-task setting.Source: §1 · p. 1
  • The paper assesses train/test overlap in public datasets for named entity recognition, relation extraction, and text classification, including AIMed, BioCreative, and SST2.It examines the impact of not accounting for overlap on performance evaluation.Source: §1 · p. 1
  • The authors propose a framework to examine inadvertent leakage between dataset splits and assess models’ memorization versus generalization characteristics.They argue that robustness on unseen data is an important consideration in evaluating model performance.Source: §1 · pp. 1–2

2 Related work

Prior work has examined memorization and generalization in question-answering datasets, including question and answer overlap and paraphrase-based split controls. These studies report substantial test-set overlap and worse performance under stricter splits.

2 Related work

  • The memorization-versus-generalization issue has also been discussed in open-domain question answering, where systems answer questions using available texts.Source: §2 · p. 2
  • Lewis et al. (2020) distinguish question memorization, answer memorization, and generalization in open-domain QA evaluation.Across three datasets, they report that 58–71% of test answers occur in training data and 28–34% have paraphrased questions in training data.Source: §2 · p. 2
  • Min (2020) group questions into paraphrase-based “Templates” and assign each group to only one data split.On emrQA and Overnight, models performed significantly worse on test sets with strict division; Rawat et al. (2020) also used this splitting methodology on emrQA.Source: §2 · p. 2

3 Approach

The approach treats random instance splitting as a possible source of train/test leakage and defines overlap according to the NLP task. The authors propose quantifying that overlap and testing its impact on model performance.

3 Approach

  • Randomly shuffling instances without broader context can inadvertently leak data when similar training and test instances overlap.Source: §3 · p. 2
  • Leakage may occur in input texts or annotated outputs, with relevant overlap depending on the NLP task.The approach distinguishes document-level input similarity for classification, input or target-entity similarity for NER, and input or participating-entity similarity for relation extraction.Source: §3 · p. 2
  • The framework quantifies train/test overlap and tests its impact on model performance across benchmark datasets.The paper reports overlap findings in Section 5.1 and performance impacts in Section 5.2.Source: §3 · p. 2

4 Method

The method examines several NLP datasets, measures each test instance’s similarity to its closest training instance, and evaluates model performance across similarity-based test partitions. It also compares random and document-unique splits for AIMed.

4.1 Datasets

  • The study covers AIMed, BC2GM, ChEMU, BC3ACT, and SST2; AIMed is evaluated with random and document-ID-unique 10-fold splits.AIMed (R) uses random folds, while AIMed (U) keeps source document IDs unique across folds.Source: §4.1 · pp. 2–3

4.2 Similarity measurement

  • Each test instance is matched to its most similar training instance, and the average best-match score estimates train/test overlap.Similarity uses cosine similarity over unigram, bigram, and trigram count vectors with stopwords ignored.Source: §4.2 · p. 3

4.3 Evaluate model performance

  • The evaluation compares BERT performance across four similarity-threshold intervals and four similarity-ranked quartiles.The threshold intervals control similarity ranges but can contain different numbers of instances; quartiles keep sample counts equal while thresholds vary. AIMed is also compared across three models using random versus document-unique splits.Source: §4.3 · p. 3

5 Results

The datasets show substantial train-test overlap, including exact matches and high n-gram similarity. Performance varies across similarity-based splits, with results differing by dataset and metric.

5.2 Model performance and similarity

  • More than 10 F-score points separate AIMed (R) and AIMed (U) across all three models; Table 3 compares the split types.AIMed (U) requires unique document IDs in different folds, and train-test similarity drops significantly from AIMed (R) to AIMed (U).Source: Table 3 · p. 4, §5.2 · p. 4

5.1 Similarity in datasets

  • BC2GM has 70% overlap between train and test gene names, including 2,308 of 6,331 test genes with exact train matches; AIMed (R) has over 73% overlap even by trigrams.Source: §5.1 · p. 4

5.2 Model performance and similarity

  • ChEMU F-score falls 96.7→85.6 from 4I to 2I, while BC2GM performance falls 82.4% to 74.5% in 2I versus 1I.Source: §5.2 · p. 4
  • BC2GM F-score is 78.5→87.1 from 4I to 3I despite 41 of 47 instances in 4I having 100% train similarity; six exact-match cases account for a recall drop from 90.6→77.5.Source: §5.2 · p. 4
  • BC3ACT F1-score rises 56.4→63.0 with similarity while accuracy falls 85.8→77.5, possibly amid a positive-class share difference of 50% in training versus 17% in testing.Source: §5.2 · p. 4
  • SST2 accuracy rises 85.0→96.8 from 1I to 4I, while quartile F-score rises 92.9→94.1 from Q1 to Q3 then falls to 92.8 in Q4.Source: §5.2 · p. 4

6 Discussion

The discussion examines how similarity between training and test data relates to model performance, and cautions that similarity measures and acceptable overlap depend on task context. It connects higher test–train similarity with higher F-scores and the risk that held-out scores reflect memorization rather than generalization.

6.1 Quantifying similarity

  • The bag-of-words cosine measure detects simple overlap, which is more common in labor-intensive tasks such as NER and relation extraction than in text classification.The discussion also warns that this measure may indicate similarity when meanings differ, particularly for classification tasks.Source: §6.1 · pp. 4–5
  • Similarity thresholds define four intervals—1I through 4I—and a full range F; SST2 uses accuracy, while the other datasets use F1 as the primary metric.The intervals are [0, 0.25), [0.25, 0.5), [0.5, 0.75), [0.75, 1], and [0, 1], respectively.Source: §6.1 · p. 5
  • The discussion leaves the acceptable level of train–test similarity dependent on the dataset and task, noting it may be higher when training data has sufficient volume and variety.More sophisticated semantic similarity methods could be incorporated, but may create a chicken-and-egg problem if they use a machine-learning model.Source: §6.1 · p. 5

6.2 Memorization vs. Generalization

  • F-scores tend to be higher on test instances more similar to training data, so high held-out scores may reflect memorization rather than generalization.The authors note that shared-task training sets may not provide the volume and variety needed to make this distinction irrelevant in real-world use.Source: §6.2 · p. 5

7 Conclusion

The conclusion argues that train–test overlap should be quantified when assessing NLP models’ real-world applicability. It recommends similarity-stratified test sets because a single held-out score cannot establish generalizability.

  • Quantifying train–test overlap is crucial to assessing real-world applicability, since one held-out test metric is insufficient to infer a model’s generalizability.Stratifying test sets by similarity enables more robust assessment of memorization versus generalization.Source: §7 · pp. 5–6

A AIMed document examples

The AIMed example shows that one document can yield multiple instances that retain the same document identifier.

  • AIMed document AIMed.d0 contains multiple instances, including AIMed.d0.s0 and AIMed.d0.s1, which share the same document ID.Source: A AIMed document examples · p. 8

B Classwise similarity for BC3AST

The appendix reports class-wise similarity for BC3ACT and notes that positive samples have higher mean similarity than negative samples.

C BERT and similarity thresholds

The section presents BERT performance across similarity-threshold ranges, with tables covering NER and sentiment datasets. It reports NER scores and sample proportions, including a view of output gene mentions using unigrams.

C BERT and similarity thresholds

D High similarity BC2GM samples

This section presents high-similarity BC2GM test samples and identifies samples where exact input-text matches in training still accompanied missed gene detections.

D High similarity BC2GM samples

Related papers

Loading 2102.01818v1…