PaperlayerView exact revision on arXiv

Source-linked AI summary

Task Contamination: Language Models May Not Be Few-Shot Anymore

Changmao Li, Jeffrey Flanigan

arXiv:2312.16337v1cs.CL

TL;DR

The scope of task contamination in zero- and few-shot evaluation has been difficult to assess. Across multiple LLMs, chronological analysis and other contamination checks found evidence of prior task exposure, while models rarely significantly outperformed majority baselines on classification tasks without possible contamination.

  • Problem

    The scope of task contamination in zero- and few-shot evaluation is difficult to assess, in part because closed models do not release their pre-training data.

    Source: §1 · p. 1

  • Method

    The paper evaluates 12 LLMs across classification and semantic parsing tasks using chronological analysis, training-data inspection, task-example extraction, and membership inference.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Pre-training-era datasets were significantly more likely to yield performance above the majority baseline, while models rarely significantly outperformed that baseline on classification tasks without possible contamination.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Some LLMs have seen task examples during pre-training and are therefore no longer zero- or few-shot on those tasks, motivating further research into contamination’s extent and impact.

    Source: §11 · p. 9

    For open models, obtaining crawled sources can be non-trivial, and grepping pre-training corpora may be unreliable because of formatting, normalization, and tokenization differences.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language models (LLMs) offer impressive performance in various zero-shot and few-shot tasks. However, their success in zero-shot and few-shot settings may be affected by task contamination, a potential limitation that has not been thoroughly examined. This paper investigates how zero-shot and few-shot performance of LLMs has changed chronologically over time. Utilizing GPT-3 series models and several other recent open-sourced LLMs, and controlling for dataset difficulty, we find that on datasets released before the LLM training data creation date, LLMs perform surprisingly better than on datasets released after. This strongly indicates that, for many LLMs, there exists task contamination on zero-shot and few-shot evaluation for datasets released prior to the LLMs' training data creation date. Additionally, we utilize training data inspection, task example extraction, and a membership inference attack, which reveal further evidence of task contamination. Importantly, we find that for classification tasks with no possibility of task contamination, LLMs rarely demonstrate statistically significant improvements over simple majority baselines, in both zero and few-shot settings.

1 Introduction

The paper investigates task contamination in zero- and few-shot evaluation, finding that models more often beat majority baselines on datasets released before their training data was collected. Inspection and extraction analyses provide further evidence of contamination, while uncontaminated classification tasks rarely show significant gains over majority baselines.

Introduction

  • Assessing contamination is difficult because closed models do not disclose pre-training data, while open-model corpora can be hard to retrieve or search reliably.Crawled sources may have changed, and formatting, normalization, or tokenization differences can prevent straightforward example matching.Source: §1 · pp. 1–2
  • The study evaluates 12 closed and open models on 16 classification tasks and one semantic parsing task, comparing datasets created before versus after each model’s training data was crawled.The model set ranges from GPT-3 series models to Fairseq MoE, GPT-J, Bloom, OPT, LLaMA, Alpaca, and Vicuna.Source: §1 · p. 2
  • Datasets created before model training data was collected are significantly more likely to yield performance above the majority baseline than later datasets.This pattern is reported across models and datasets for both zero-shot and few-shot settings.Source: Figure 1 · p. 1, §1 · p. 2
  • On classification tasks with no possibility of task contamination, models rarely significantly outperform simple majority baselines in either zero- or few-shot settings.The paper reports this pattern across a range of tasks.Source: §1 · p. 2
  • For Spider semantic parsing, extracted-example counts correlate strongly with model accuracy (R=.88), supporting task contamination as the source of zero-shot performance gains.The authors conduct a membership inference attack as a case study across all analyzed models.Source: §1 · p. 2
  • Extractable training examples increase across GPT-3 versions from davinci to GPT-3.5-turbo, closely tracking their zero-shot performance gains on the task.The authors interpret this correspondence as strong evidence that task contamination explains the cross-version performance increase.Source: §1 · p. 2

2 Overview

The overview presents four complementary methods for detecting task contamination, each with different evidential strengths and limitations. Using all four, the authors report strong evidence of contamination for some model–dataset combinations and outline the analyses that follow.

2 Overview

  • The four methods are training data inspection, task example extraction, membership inference, and chronological analysis.Membership inference checks for exact output matches and applies only to generation tasks; task example extraction is possible only with instruction-tuned models.Source: §2 · p. 2
  • Training data inspection, task example extraction, and membership inference have high precision but low recall, so a lack of evidence does not establish absence of contamination.Formatting variation, differing task-definition keywords, and dataset size can obscure contamination evidence.Source: §2 · p. 2
  • Chronological analysis has high recall but low precision because factors other than contamination may also increase performance over time.The authors use all four methods, report strong contamination evidence for some model–dataset combinations, and proceed from chronological analysis to inspection, extraction, uncontaminated-task performance, and membership inference.Source: §2 · p. 2

3 Models and Datasets

The study evaluates 12 proprietary and open models, comparing performance across datasets grouped by release date and by each model’s training-data collection date.

Models

  • The evaluation covers 12 models: five proprietary GPT-3 series models and seven open models with freely accessible weights.The paper says comparing the groups can provide insights into proprietary and community-driven open models.Source: §3 · pp. 2–3

Datasets

  • Datasets are split at January 1, 2021 into pre-2021 and post-2021 groups, and separately into pre-collection and post-collection groups for each model.The latter split distinguishes datasets a model could possibly have been trained on from those it could not have been trained on.Source: §3 · p. 3
  • Zero-shot and few-shot evaluation assumes models have no prior exposure to the task, while task-example exposure during pretraining can create a false impression of competency.The paper uses chronological dataset groupings to help identify overlap or anomalies.Source: §3 · p. 3

4 Chronological Analysis

Across models, datasets released before training-data collection were more likely than later datasets to produce above-majority-baseline results in both zero- and few-shot settings, with statistically significant differences. The section also examines caveats, model-specific patterns, and performance trends across pre- and post-2021 datasets.

Analysis of Pre- and Post-collection Datasets

  • Pre-collection datasets were more likely than post-collection datasets to beat majority baselines in both zero- and few-shot settings, with differences significant at the 99% confidence level.The same pattern held for statistically significant improvements over the baseline, and the authors interpret the findings as evidence of possible task contamination.Source: Analysis of Pre- and Post-collection… · pp. 3–4

Analysis of Pre- and Post-collection for Individual LLMs

  • For individual LLMs, pre- versus post-collection performance differences were significant at 95% confidence, with contamination indications across model families and stronger indications for GPT-3 models from davinci-001 onward.The global relative-to-collection-date trend held across models; the paper cautions that newer open models had fewer post-collection datasets.Source: Figure 2 · p. 4, Analysis of Pre- and Post-collection… · pp. 4–5

Caveats

  • After excluding known difficult tasks, fine-tuned PLM accuracy had no correlation with dataset release date (R2 = 0.001), but post-collection FOMC results may still reflect contamination.The paper notes that preliminary GPT-3 experimentation on FOMC may have been used by OpenAI for model updates.Source: Caveats · p. 4, Figure 4 · p. 5

Performance over Time

  • On pre-2021 datasets, open LLM performance stayed near majority baseline from 2019–2022, while later GPT-3 models rose well above it in both zero- and few-shot settings.The authors describe the GPT-3 trend as particularly suspect and suggest that user-input data collection may contribute to contamination.Source: Figure 3 · p. 5, Performance over Time · p. 6
  • On post-2021 datasets, GPT-3 performance rose over time, especially zero-shot, while open LLMs remained below majority baseline and GPT-3 performance.From davinci-001 to davinci-002, performance increased on pre-2021 datasets but decreased on post-2021 datasets; the decrease was significant at 95%, suggesting instruction tuning was specific to earlier datasets.Source: Figure 3 · p. 5, Performance over Time · p. 6

5 Training Data Inspection

Training-data inspection found task examples in Alpaca and Vicuna and evidence that even small numbers of such examples can boost performance over LLaMA, potentially compromising zero- and few-shot evaluation.

5 Training Data Inspection

  • The inspection searched for task-related instruction patterns and manually checked matched data for task examples.Task examples were identified by regular-expression matching followed by manual inspection.Source: §5 · p. 6, Table 3 · p. 6
  • Manual inspection was limited to Alpaca and Vicuna because their fine-tuning datasets were small enough to check.Source: §5 · p. 6
  • Alpaca and Vicuna outperform LLaMA on tasks with more than one training example, averaged across zero- and few-shot settings and all tasks.The performance changes are reported as Δ% in Table 3; the supplied passages give no specific values.Source: Table 3 · p. 6, §5 · p. 6

6 Task Example Extraction

Task example extraction finds evidence of contamination in GPT-3 models on pre-collection datasets, but the method and training data inspection both have low recall. Results for models without instruction tuning are inconclusive because they cannot follow the extraction instructions.

6 Task Example Extraction

  • The extraction method prompts LLMs to generate task training examples, treating generation of any such examples as evidence of possible contamination.Unlike prior work that attempted to generate matching test examples, this method targets training examples, which should not be present in zero- or few-shot evaluation.Source: §6 · p. 6
  • For all pre-collection datasets, GPT-3 models from davinci-001 onward can generate task-specific training examples; some post-collection datasets also show contamination evidence.The authors suggest post-collection contamination may result from dataset creators experimenting with GPT-3 before release, as the FOMC paper reports doing.Source: §6 · p. 6, Table 4 · p. 7
  • Almost no open LLMs generate task-specific training examples, except Vicuna, which is fine-tuned on ChatGPT data.Results are inconclusive for models without instruction tuning because they cannot follow instructions to generate examples.Source: §6 · pp. 6–7

Comparison to Training Data Inspection

  • Training data inspection and task example extraction both have low recall, with each detecting contamination the other misses.Training data inspection found contamination in Alpaca for SST-2 and NewsMet, which extraction missed; extraction found contamination in Vicuna for NewsMTSC, which inspection missed.Source: Comparison to Training Data Inspection · p. 7

7 LLM Performance on Tasks With No Contamination

For tasks without a demonstrated possibility of task contamination, LLMs rarely show statistically significant improvements over majority baselines.

7 LLM Performance on Tasks With No Contamination

  • 1 out of 51 (2%) post-collection model/dataset combinations without extracted task examples significantly improved over the majority baseline in zero- or few-shot settings.The exception was davinci-001 on MTSC-RW, which showed a significant improvement but did not generate task examples with the prompt.Source: §7 · p. 7

8 Membership Inference

On Spider, exact-match generations increase across model versions and correlate strongly with execution accuracy, indicating growing task contamination, although its effect on performance improvement remains undetermined.

8 Membership Inference

  • The membership inference attack checks whether generated content exactly matches dataset examples; the study applies it to Spider, a semantic-parsing and text-to-SQL generation task.The test is possible for generation but not classification tasks, because inputs may occur in training data without their labels being known to occur there.Source: §8 · p. 7
  • Exact-match generated examples increased over time across GPT-3-series and recent open-source LLMs, indicating increasing task contamination on Spider.Because database schemas were absent from zero-shot prompts, exact matches of table or field names found in the data indicate contamination.Source: §8 · p. 7
  • R=0.88: exact-match generation counts were strongly positively correlated with execution accuracy, but the contamination’s effect on performance improvement remains undetermined.Execution accuracy was computed after adding the schema to the prompts.Source: §8 · p. 7

9 Take-Aways

The takeaways warn that task contamination can inflate zero- and few-shot evaluations, while classification tasks without demonstrated contamination rarely show significant gains over majority baselines.

9 Take-Aways

  • Task contamination may inflate closed-source models’ zero- and few-shot performance, making them untrustworthy baselines, especially after instruction tuning or RLHF; its extent remains unknown, so caution is warranted.The observed increase over time in GPT-3 series models’ zero- and few-shot performance on many downstream tasks is likely due to contamination.Source: §9 · p. 8
  • For classification tasks without demonstrated contamination, LLMs rarely show statistically significant improvements over majority baselines in either zero- or few-shot settings.Source: §9 · p. 8
  • Inspecting open-source LLM training data for contamination can be difficult: membership may be unclear without released processed data, models may not reproduce evidence, and dataset formatting differences complicate analysis.OPT and LLaMA did not release training data, while Alpaca and Vicuna did, allowing more definite information.Source: §9 · p. 8
  • The authors encourage public release of training datasets to make contamination easier to diagnose.Source: §9 · p. 8

10 Related Work

Earlier research examined LLM data contamination and noticed zero-shot or few-shot task contamination, but did not systematically analyze it; this work provides an exhaustive evaluation of the problem.

10 Related Work

11 Conclusion and Future Work · A Hyperparameters

The paper finds evidence that some LLMs have seen task examples during pre-training, while uncontaminated classification tasks rarely show statistically significant gains over majority baselines. It recommends further research on contamination in zero- and few-shot settings and reports decoding, example-selection, and hardware choices.

11 Conclusion and Future Work

  • Some LLMs have seen task examples during pre-training and are therefore no longer zero- or few-shot for those tasks.The evidence comes from analysis across a range of tasks.Source: §11 · p. 9
  • On classification tasks with no possibility of contamination, LLMs rarely significantly outperform simple majority baselines in zero- or few-shot settings.Source: §11 · p. 9
  • The analysis combines chronological comparisons, training-data inspection, task-example extraction, and a membership inference attack.Source: §11 · p. 9
  • The authors recommend further research to establish the extent and impact of task contamination in zero- and few-shot evaluation.Source: §11 · p. 9

A Hyperparameters

B Datasets … H Prompts for Task Example Extraction

The appendices document the dataset cohorts and prompt sources, explain how training examples were inspected, and provide detailed performance tables and task-specific prompt materials.

B Datasets

  • Pre-2021 datasets include GLUE and SuperGLUE tasks, while the post-2021 cohort includes StrategyQA, NLI4Wills, NewsMTSC, CREPE, FOMC, and NewsMet.The passage identifies the pre-2021 tasks as MRPC, BoolQ, SST-2, QNLI, WNLI, RTE, CB, COPA, and WiC.Source: B Datasets · p. 13

C Prompt Sources

  • Task prompts come from prior benchmark research and OpenAI examples, or were designed based on related tasks from those sources.The passage cites Bang et al. 2023, Qin et al. 2023, and OpenAI 2023a.Source: C Prompt Sources · p. 13

D Training Data Inspection Details

  • Training-data inspection manually reviews examples found with task-specific regular-expression or string-search patterns, counting only examples containing task input and output.Mentions of a task without input-output examples are excluded; some tasks, including COPA and BoolQ, lack a specific matching pattern.Source: D Training Data Inspection Details · p. 13, Table 7 · p. 13

E Detailed Results Tables

  • Tables report zero-shot results across experimented LLMs and datasets and few-shot results for GPT-series models, with confidence intervals and significance markers against majority baselines.Bold indicates significantly larger performance than the majority baseline using a t-test with p=.99; datasets above the table line predate LLM training-data collection.Source: Table 8 · p. 14, Table 9 · p. 14, E Detailed Results Tables · p. 14

F Additional Figures

  • Figures show average performance across models ordered by training-data collection date and separate performance plots for pre-2021 and post-2021 datasets.The post-2021 figure marks models whose training-data collection date followed dataset release, indicating they may have seen the dataset.Source: Figure 7 · p. 14, Figure 8 · p. 15, Figure 9 · p. 15

G Prompt Examples for Each Task

  • The appendix provides zero-shot prompt examples for each task, illustrating task-specific inputs and expected outputs.Examples span formats including yes/no classification, multi-class sentiment, entailment, causal choice, and SQL generation.Source: G Prompt Examples for Each Task · pp. 16–17

H Prompts for Task Example Extraction

  • Table 10 lists the prompts used for extracting task examples.Source: Table 10 · p. 20

Related papers

Loading 2312.16337v1…