PaperlayerView exact revision on arXiv

Source-linked AI summary

Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models

Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, Rich Caruana

arXiv:2404.06209v3cs.LGcs.AIcs.CL

TL;DR

Evaluating LLMs on tabular tasks is complicated by uncertainty about whether evaluation data appeared in training. The paper introduces memorization tests and compares few-shot performance on seen and novel datasets, while also probing statistical prediction. It finds stronger performance on seen data, but reasonable performance on novel datasets and above-random statistical classification that struggles as dimension increases.

  • Problem

    Detecting whether an LLM has seen an evaluation dataset during training is challenging, especially when access is limited to API queries.

    Source: §1 · p. 1

  • Method

    The authors introduce four tests for verbatim tabular-data memorization, compare few-shot performance on seen and post-training datasets, and probe in-context statistical prediction.

    Source: §1 · pp. 2–3, §3.1 · p. 3

  • Results

    LLMs perform better on seen datasets, while GPT-3.5 and GPT-4 perform reasonably well on novel datasets and classify better than random but struggle as problem dimension increases.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings highlight the importance of checking whether evaluation datasets were seen during pre-training when assessing LLM few-shot performance.

    Source: §1 · pp. 1–2

    The authors usually do not know whether or how the LLMs saw the tabular datasets during training, and could not randomly insert datasets into the training corpus.

    Source: §7 · p. 10

Abstract

from arXiv · show

While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether a language model has seen a tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training. We find that LLMs perform better on datasets seen during training, indicating that memorization leads to overfitting. At the same time, LLMs show non-trivial performance on novel datasets and are surprisingly robust to data transformations. We then investigate the in-context statistical learning abilities of LLMs. While LLMs are significantly better than random at solving statistical classification problems, the sample efficiency of few-shot learning lags behind traditional statistical learning algorithms, especially as the dimension of the problem increases. This suggests that much of the observed few-shot performance on novel real-world datasets is due to the LLM's world knowledge. Overall, our results highlight the importance of testing whether an LLM has seen an evaluation dataset during pre-training. We release the https://github.com/interpretml/LLM-Tabular-Memorization-Checker Python package to test LLMs for memorization of tabular datasets.

1 Introduction

The paper develops ways to detect tabular-data contamination and examines how it relates to LLM few-shot performance on seen and novel datasets. It finds better performance on seen data, while models also perform reasonably well on novel datasets and can act as statistical predictors.

1 Introduction

  • Seen-dataset performance is higher, which the authors interpret as memorization leading to overfitting; Table 1 also shows performance differences across pre- and post-2021 financial time-series years.The table reports robust mean relative prediction error; the supplied passage gives no cell values.Source: Table 1 · p. 2, §1 · p. 2
  • GPT-3.5 and GPT-4 have verbatim-memorized many popular tabular datasets, and GPT-4 can consistently generate the entire Iris and Wine datasets.The memorization is reported as robust across precise GPT-series model versions.Source: §1 · p. 2
  • The study compares few-shot performance on datasets evidenced as seen during training with datasets released after training, and varies data format using small numerical noise.The authors use evidence of verbatim memorization of at least part of a dataset to classify it as “seen during training.”Source: §1 · p. 2
  • On novel datasets, GPT-3.5 and GPT-4 perform reasonably well, and statistical classification remains better than random but becomes difficult as problem dimension increases.After removing feature names and standardizing data, GPT-4 performance scales with the number of few-shot examples, while GPT-3.5 remains flatter.Source: §1 · pp. 2–3

2 Datasets

The study uses both pre-2021 internet-available datasets and datasets not freely available online before 2022. The latter group includes newer, private, and newly derived datasets.

2 Datasets

  • The pre-2021 group includes UCI Iris, Wine, Adult, and Housing, plus OpenML Diabetes and Kaggle Titanic.These popular datasets are indicated in red in the study.Source: §2 · p. 3
  • The newer group includes ACS Income and Travel, Spaceship Titanic, privately held FICO, and a novel ICU dataset derived from Harvard Dataverse data.The ACS datasets derive from the 2022 American Community Survey; ICU prediction distinguishes intensive from intermediate care.Source: §2 · p. 3

3 Testing Language Models for Memorization of Tabular Data

The paper tests verbatim memorization by asking whether models can reproduce tabular datasets represented as CSV text. Its operational criterion relies on consistent generation of dataset values that include random-variable realizations.

3 Testing Language Models for Memorization of Tabular Data

  • The authors situate their approach among tests of feature names, categorical values, and data-distribution properties, while focusing this section on verbatim memorization.They use few-shot prompting to condition chat models to complete a given prefix, following the prefix-suffix pattern used in prior work.Source: §3 · p. 3, §3.1 · p. 3

3.1 Testing for memorization

  • Four memorization tests use the canonical CSV representation of tabular datasets and treat consistent dataset generation as evidence of memorization.The rationale is that random-variable realizations cannot be consistently reproduced unless their values were seen before.Source: §3.1 · p. 3

A. Knowledge and Learning B. Memorization

The paper tests tabular-dataset memorization with several prompting procedures and reports evidence across closed and open LLMs. GPT-3.5 and GPT-4 show evidence of memorizing popular datasets, while the tests’ limited power means failure to extract data does not rule memorization out.

A. Knowledge and Learning B. Memorization

  • Four tests probe memorization by completing initial rows, a row following contiguous rows, a unique feature in a row, or the first token of a following row.When dataset rows are known to be random, first-token accuracy can be statistically compared with mode-based completion; the tests are available in tabmemcheck.Source: A. Knowledge and Learning B.… · p. 4
  • The tests are highly significant, so positive results provide definitive evidence of memorization, but their power is unclear and failed extraction cannot rule memorization out.The authors note that a model may have memorized a dataset without the tests being able to extract it through prompting.Source: A. Knowledge and Learning B.… · p. 4

3.2 LLMs have memorized many of the popular tabular datasets

  • GPT-3.5 and GPT-4 show evidence of memorizing initial rows from all tested datasets publicly available before 2021, and of memorizing random rows from Titanic, Diabetes, Wine, and Iris.The authors characterize the latter as evidence that the models have essentially memorized those entire datasets.Source: Table 2 · p. 4, §3.2 · p. 5
  • No memorization evidence was found for GPT-3.5 and GPT-4 on novel datasets released after training.For FICO, the authors suggest registration before access might have protected it from inclusion in training data.Source: §3.2 · p. 5
  • Five open and open-weight LLMs showed evidence of memorizing popular datasets, usually only their initial rows rather than the entire dataset.Iris was the exception; the table reports Header Test and Row Completion Test results.Source: §3.2 · p. 5, Table 3 · p. 5

4 Few-Shot Learning with LLMs and Tabular Data

The experiments compare few-shot performance on memorized and novel tabular datasets across standardized data formats. They find that memorized-data performance is more sensitive to reformatting, while statistical classification performance lags behind traditional algorithms as problem dimension grows.

4 Few-Shot Learning with LLMs and Tabular Data

  • The experiments evaluate GPT-3.5 and GPT-4 on memorized and novel datasets, using randomly selected, label-stratified 20-example prompts at temperature zero.Prompts present rows as “Feature Name = Feature Value” and ask the model to predict the target from the other features.Source: §4 · pp. 5–6

4.1 Dataset Format: Original, Perturbed, Task and Statistical

  • The study presents tabular examples in original, perturbed, task, or statistical formats, changing values, feature names, categorical labels, prompts, or feature scaling across formats.The formats are standardized and comparable across datasets.Source: §4.1 · p. 6

4.2 Memorization leads to inflated performance estimates

  • On memorized datasets, GPT-4 and GPT-3.5 accuracy declines across original, perturbed, task, and statistical formats; on novel datasets, performance is robust across the first three formats.GPT-4 outperforms logistic regression on 3 of 5 memorized datasets in the original format, but only 1 of 5 in the task format, where the difference is within standard error; it does not outperform logistic regression on novel datasets in the original format.Source: §4.2 · pp. 6–7, Table 4 · p. 7

4.3 Performance depends on feature names and variable format

  • Except on Spaceship Titanic, performance drops significantly in the statistical format, suggesting reliance on natural feature scales and variable names.Spaceship Titanic is synthetically generated and uses fictional variable names; Figure 2 depicts the task-to-statistical accuracy drop across its subfigures.Source: §4.3 · p. 8, Figure 2 · p. 8

4.4 Language models act as statistical predictors, but lag behind traditional algorithms

  • With 20 examples, GPT-3.5 and GPT-4 deteriorate faster than logistic regression and 1-nearest neighbor as classification dimension increases; at dimension 8, GPT-4 improves with more examples but remains less efficient than logistic regression.At dimension 2, the LLMs are roughly on par with the traditional classifiers; with 200 examples, GPT-4 is roughly on par with 1-nearest neighbor. TabLLM fine-tuning also scales with sample count.Source: §4.4 · pp. 8–9, Figure 3 · p. 9

5 Drawing Random Samples from Datasets seen during Training

The section examines whether training-data exposure supports tabular-data generation. GPT-3.5 produces novel California Housing samples that reflect summary statistics, with sample diversity varying by temperature.

5 Drawing Random Samples from Datasets seen during Training

  • The section presents sampling from seen datasets as an additional implication of training exposure, requiring no fine-tuning.The figure describes GPT-3.5 receiving only the dataset name and feature names.Source: §5 · p. 9, Figure 4 · p. 9
  • The generated samples become more diverse and more similar to the data distribution as temperature rises, but high temperatures can produce samples outside the data support.At low temperatures, samples concentrate around the data mode.Source: Figure 4 · p. 9
  • GPT-3.5 generates California Housing samples that are not copied from training data and whose feature correlations match those of the dataset.The model was conditioned on samples from other datasets, without being given California Housing feature values.Source: §5 · pp. 9–10

6 Related Work

Related work covers tabular-data applications, contamination and memorization tests, and in-context learning. The paper positions its contribution as demonstrating frequent exposure to certain tasks and overfitting on tasks seen frequently during training.

6 Related Work

7 Discussion

LLMs have memorized many popular tabular datasets, and the authors find strong evidence that this memorization is associated with overfitting in in-context learning. Their findings also show that GPT-3.5 and GPT-4 rely on world knowledge while retaining some ability to act as statistical predictors.

7 Discussion

  • The study finds strong evidence of overfitting in in-context learning due to memorization of popular tabular datasets.The models also rely on world knowledge and can act as statistical predictors.Source: §7 · p. 10
  • The study usually cannot establish whether or how the LLMs saw the tabular datasets during training.The authors also say it would have been more ideal to insert datasets into the training corpus at random.Source: §7 · p. 10
  • Without controlling for memorization, evaluations might have erroneously concluded that GPT-4 frequently outperforms traditional statistical learning algorithms.Source: §7 · p. 10

8 Ethics Statement

The authors state that their study uses datasets explicitly available for scientific research and addresses invalid performance evaluations of LLMs. They note that memorization also has broader implications for copyright and privacy.

8 Ethics Statement

  • The authors do not believe the study raises significant ethical concerns because it uses datasets available for scientific research and examines invalid LLM performance evaluations.Source: §8 · p. 11
  • The authors note that memorization has broader implications, including copyright and privacy.Source: §8 · p. 11

9 Reproducibility Statment

The reproducibility appendix describes model versions, datasets, baselines, time-series experiments, and memorization tests. It also provides qualitative examples of row, feature, and header completion, with exact matches distinguished from errors.

9 Reproducibility Statment

  • The authors estimate that replicating all results through the OpenAI API costs approximately 1000 USD, with GPT-4 few-shot experiments accounting for the most expensive queries.The GPT-4 experiments require approximately 1000 queries per data point in some cases.Source: §9 · p. 11

A Datasets

  • The study covers older public datasets and newer or less-public datasets, including ACS-derived data, 2022 Spaceship Titanic, FICO, and a novel ICU dataset.The appendix describes Iris, Wine, Kaggle Titanic, OpenML Diabetes, Adult Income, California Housing, and the construction of ACS Income and ACS Travel.Source: A Datasets · p. 16

B Models

  • The appendix identifies the GPT model versions used for results and describes logistic regression and gradient-boosted trees as statistical baselines.Source: B Models · pp. 16–17

C Time Series Experiments

  • The time-series experiment predicts the MSCI World stock index’s 2022 development from its price levels over the previous 20 days.The appendix says it computes a robust mean relative error for each year.Source: C Time Series Experiments · p. 17

D Memorization Tests

  • The appendix cautions that quantitative completion counts alone do not establish memorization; they must be judged against dataset entropy and structure.GPT-4 can generate all 151 rows of Iris from the first couple of rows, while some test results remain ambiguous.Source: D Memorization Tests · p. 17

D.1 Qualitative Examples

  • The qualitative examples use string-distance coloring: green marks exact matches, red errors, and violet missed digits across row, feature, and header completion tests.The examples include Adult Income, Iris, ACS Income, FICO, OpenML Diabetes, and header completion on Adult Income.Source: Figure 5 · p. 18, Figure 6 · p. 18, Figure 7 · p. 19

D.2 Quantitative Test Results

The appendix reports that completion counts can be either negligible or high and describes additional tests of dataset knowledge and transformations. It details how those tests are performed, including the formats used to alter the data.

D.2 Quantitative Test Results

  • Row completion counts are either zero or very low, or exceed 90% on some datasets; interpretation depends on the data’s entropy.The appendix counts a row only when the complete row matches the CSV entry.Source: §D.2 · p. 21

D.3.1 Feature Names

  • The feature names test asks the model to complete dataset feature names from the first feature name, with outputs either matching the CSV format or clearly made-up.Source: §D.3.1 · p. 22

D.3.2 Feature Values

  • The feature values test checks whether sampled observations contain valid feature values, especially valid categorical strings, and whether their formatting matches the training data.Source: §D.3.2 · p. 22

D.3.3 Feature Distribution

  • The feature distribution test samples at temperature 0.7 and compares sampled feature values with the data’s feature mode.The test is motivated by the models’ ability to model modal feature values despite rarely modeling finer distribution details well.Source: §D.3.3 · p. 22

D.3.4 Conditional Distribution

  • The conditional distribution test compares Pearson correlations in samples drawn at temperature 0.7 with correlations in the original dataset.Source: §D.3.4 · p. 22

D.3.5 Header Test

  • The header test splits a dataset at random positions in rows 2, 4, 6, and 8, then tests whether the model completes at least the next row.The test uses few-shot examples and considers the best completion.Source: §D.3.5 · pp. 22–23

E Dataset Formats

  • The original format preserves CSV feature names and values, while perturbed data changes individual digits by approximately 1% and changes unique identifiers.Source: E Dataset Formats · p. 23
  • Task formatting further rounds and noises numerical values, recodes categories, renames features, and removes dataset references from the system prompt; statistical formatting numerically encodes categories and standardizes features.The appendix says these transformations do not significantly change the underlying statistical classification problem.Source: E Dataset Formats · p. 23

F Replication with Llama 3.1 70B and Gemma 2 27B

The open-weight replications broadly resemble the main paper’s results, but their training exposure and memorization patterns differ from those of GPT-3.5 and GPT-4. The appendix also provides prompt examples and checks that dataset transformations preserve the underlying classification problem.

F Replication with Llama 3.1 70B and Gemma 2 27B

  • The replication used the same prompts and few-shot example ordering as the GPT experiments, without changing the experimental setup.Inference used Together.ai with Meta-Llama-3.1-70B-Instruct-Turbo and google/gemma-2-27b-it endpoints.Source: F Replication with Llama 3.1 70B and… · p. 23
  • GPT-novel datasets may have appeared in open-weight training, and the open-weight models do not seem to have memorized whole datasets except Iris.These differences qualify interpretation of the replication results.Source: F Replication with Llama 3.1 70B and… · p. 24
  • Across open-weight replications, popular-dataset transformation-decay patterns broadly resembled GPT results; Llama averaged better on GPT-3.5-novel datasets, while Gemma’s slight novel-set decay was not statistically significant.Figure 11 summarizes the results in Tables 7 and 8.Source: F Replication with Llama 3.1 70B and… · pp. 24–25, Figure 11 · p. 25

G Prompts

  • The diabetes prompt presents patient inputs for positive-or-negative prediction; accompanying figures depict header and feature-name tests and California Housing sampling.The prompt example uses the original OpenML Diabetes format.Source: G Prompts · p. 26, Figure 12 · p. 26, Figure 13 · p. 27

H Additional Tables and Figures

  • Logistic Regression accuracy is the same up to perhaps one percentage point, and Gradient Boosted Tree accuracy is the same, for original versus statistically transformed data.The appendix also presents California Housing sample summaries and Pearson correlations.Source: Table 9 · p. 30, Table 10 · p. 30, Table 11 · p. 30

Related papers

Loading 2404.06209v3…