PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples

Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica

arXiv:2311.04850v2cs.CLcs.AI

TL;DR

Public benchmarks can be contaminated by paraphrased or translated test cases that evade common decontamination methods and distort scores. The paper combines embedding retrieval with strong-LLM comparison to detect semantically matching samples, revealing substantial HumanEval overlap in training datasets. It concludes that stronger decontamination and fresh one-time exams are needed for accurate evaluation.

  • Problem

    Existing n-gram and embedding-similarity methods can miss semantically equivalent rephrased or translated test samples, challenging reliable benchmark contamination detection.

  • Method

    The LLM decontaminator retrieves top-k training items by embedding similarity and asks a strong LLM to judge whether each is too close to the test case.

  • Results

    8-18% of HumanEval overlapped with RedPajama-Data-1T and StarCoder-Data, and CodeAlpaca contained 12.8% rephrased HumanEval samples.

  • Takeaways & Limitations

    The paper urges stronger decontamination for public benchmarks and development of fresh one-time exams for accurate LLM evaluation.

  • Takeaways & Limitations

    The precise boundary of contamination remains challenging, including cases where GSM-8k training and test examples differ only in numbers.

Abstract

from arXiv · show

Large language models are increasingly trained on all the data ever produced by humans. Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets. While most data decontamination efforts apply string matching (e.g., n-gram overlap) to remove benchmark data, we show that these methods are insufficient, and simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures. Furthermore, we demonstrate that if such variation of test data is not eliminated, a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4. We validate such observations in widely used benchmarks such as MMLU, GSK8k, and HumanEval. To address this growing risk, we propose a stronger LLM-based decontamination method and apply it to widely used pre-training and fine-tuning datasets, revealing significant previously unknown test overlap. For example, in pre-training sets such as RedPajama-Data-1T and StarCoder-Data, we identified that 8-18\% of the HumanEval benchmark overlaps. Interestingly, we also find such contamination in synthetic dataset generated by GPT-3.5/4, suggesting a potential risk of unintentional contamination. We urge the community to adopt stronger decontamination approaches when using public benchmarks. Moreover, we call for the community to actively develop fresh one-time exams to evaluate models accurately. Our decontamination tool is publicly available at https://github.com/lm-sys/llm-decontaminator.

1. Introduction

Public benchmark scores can be misleading when test data overlaps training data, while simple rephrasings evade common detectors and enable extreme benchmark overfitting. The paper introduces stronger detection and finds substantial overlap in widely used datasets.

  • Over 10% of Llama-2’s MMLU test samples and 25% of HumanEval were reported as contaminated in prior analyses.
  • N-gram overlap and embedding similarity have limited accuracy, making accurately detecting contamination an open challenge.Embedding methods also require difficult threshold choices balancing recall and precision.
  • Rephrased or translated test samples preserve task content while remaining difficult for existing contamination tests to detect.
  • A fine-tuned 13B Llama model matched GPT-4’s performance across MMLU, GSM-8k, and HumanEval while avoiding n-gram contamination detection.
  • The proposed LLM decontaminator combines embedding retrieval with strong-LLM judgment and works significantly better than existing methods.
  • 8-18% of HumanEval overlapped with RedPajama-Data-1T and StarCoder-Data, while CodeAlpaca contained 12.8% rephrased HumanEval samples.The paper also advocates stronger decontamination and fresh one-time exams.

2. Background

Contamination detection methods differ in assumptions and computational demands, but established approaches each have important limitations. The background introduces overlap, embedding, decoding, and influence-based methods and their practical trade-offs.

  • Contamination is test-set information leaked into training data, producing overly optimistic benchmark scores.
  • N-gram overlap is simple and fast but can miss contamination when training and test text differs slightly.Examples include GPT-3’s 13-gram and GPT-4’s 50-character criteria.
  • Embedding similarity captures more semantic information than n-gram matching but requires a difficult threshold balancing recall and precision.
  • Decoding matching can detect contamination without training-data access by testing whether a model auto-completes a partial test prompt.
  • Influence functions rank training examples by relevance for human judgment but impose high computational overhead.

3. Rephrased Samples

Rephrased samples retain the semantics of benchmark items while changing wording, code style, or language to evade contamination detectors. The paper generates such samples with LLMs and studies their effects across text and coding benchmarks.

  • Rephrased samples are test cases with the same semantics as originals but altered forms that existing contamination tests struggle to detect.
  • Text rephrasing changes word order or substitutes synonyms, whereas code rephrasing changes coding styles, names, and implementations without changing semantics.
  • 3.1. Rephrasing Techniques: The rephrasing algorithm prompts a high-quality LLM, checks detectability, and retries until the sample evades the selected detector or reaches MaxRetry.
  • 3.2. Translation Techniques: Translation can evade both n-gram overlap and standard embedding search because equivalent prompts in different languages often have different embeddings.
  • 3.2. Translation Techniques: Multilingual coding augmentation uses multiple programming languages to improve generalization and understanding of equivalent code functions.Experiments indicate better results than single-language translation.

4. LLM Decontaminator

The LLM decontaminator detects rephrased benchmark samples by combining embedding-based candidate retrieval with LLM judgments, targeting the limitations of existing methods. It is presented as a more accurate approach with moderate computational overhead.

  • Method motivation: The LLM decontaminator is designed to remove rephrased samples that existing contamination tests may miss.The paper introduces it specifically to detect rephrased samples relative to a benchmark.
  • Algorithm: The LLM decontaminator identifies potentially contaminated training items through embedding-based top-k retrieval followed by pairwise LLM judgments.The detector returns whether each training case might be a rephrased version of a test case.
  • Algorithm: The method returns the training–test pairs judged by the LLM to represent contamination.Its pipeline evaluates each retrieved pair and adds positive matches to the contamination set.
  • Detection visualization: The approach combines rapid embedding filtering with reliable LLM judgments, yielding higher accuracy than the compared detection methods.Embedding search narrows the candidate pool, while the LLM evaluates semantic equivalence.

5. Experiments

The experiments test whether rephrased benchmark samples inflate scores and whether existing detectors can identify them across knowledge, mathematics, and coding tasks. Rephrased training data substantially raises reported benchmark performance, while the LLM decontaminator detects such samples more reliably than the alternatives.

  • MMLU detection controls: N-gram overlap can produce false positives in multiple-choice questions when different problems share similar answer arrangements.The experiments add a question-only control group to separate question-stem rephrasing from option-pattern overlap.
  • MMLU knowledge benchmark: Rephrased samples raise MMLU accuracy from 45.3 to 88.5 for Llama-2 models trained for 16 epochs.The original model uses 5-shot evaluation, whereas the rephrased-data model uses 0-shot evaluation.
  • HumanEval coding benchmark: Rephrased HumanEval data raises CodeLlama 7B scores from 32.9 to 67.7 and 13B scores from 36.0 to 81.1.The experiments use rephrased Python, translated code, and a multilingual programming-language dataset.
  • GSM-8K mathematics benchmark: Rephrased samples raise GSM-8K accuracy from 28.7 to 95.3 for Llama-2 7B and 13B models.The original model is tested 5-shot and the rephrased-data model 0-shot.
  • Contamination detection: Both rephrased and translated samples evade n-gram overlap, while embedding similarity is ineffective for translated samples in some settings.Multilingual embedding search also struggles on the US History subject.
  • Contamination detection: The LLM decontaminator achieves the highest minimum and average F1 scores and reliably detects rephrased samples in coding datasets.Embedding similarity works better within the same programming language but becomes less effective after translation.

5.3. Contamination in Real World Datasets

The LLM decontaminator reveals previously unknown benchmark overlap in real-world training datasets, including rephrased HumanEval and MATH samples. It identifies contamination in both conventional and synthetic datasets.

  • 12.8% of HumanEval test samples in CodeAlpaca are rephrased samples detected by GPT-4 with k=1.The detected set contains 21 rephrased HumanEval samples.
  • 14 HumanEval rephrased samples were identified in a 16G sample of RedPajama-Data-1T’s GitHub subset.RedPajama-Data-1T is used for pretraining open-source models including MPT and OpenLlama.
  • A unified embedding-similarity threshold is difficult to set because similarity distributions differ substantially across subjects.At threshold 0.8, Sociology rephrased samples are difficult to identify; at 0.4, Abstract Algebra produces many false positives.
  • 1.58% of the MATH test set appears as self-rephrased samples in MATH training data.The LLM decontaminator identifies 79 such instances.
  • 76 rephrased MMLU test cases, or 0.543% of the test set, are detected in FLAN’s CoT subset.Detection uses GPT-4 with k=1 for the decontamination parameters.

6. Discussion

The discussion broadens contamination beyond paraphrases to cases differing only in numbers, and argues that such training examples can encourage memorization rather than generalization. It also identifies synthetic data and benchmark design as ongoing concerns.

  • 6.1. Beyond rephrased samples: Training on GSM-8K examples differing from test cases only by substituted numbers may encourage solution memorization and poor generalization.The authors state that the resulting benchmark numbers may not effectively capture math problem-solving performance.
  • 6.2. Contamination in synthetic data: Synthetic LLM-generated data can contain subtle benchmark contamination, as illustrated by contamination found in CodeAlpaca.The discussion also notes related detection of subtle contamination in Phi-1’s LLM-generated data.
  • 6.3. Enhancing Benchmarks for LLMs: Without access to training data, detecting contamination remains an open problem.The authors therefore propose fresh one-time questions and rapidly iterated benchmarks instead of relying only on static benchmarks.

7. Related Work

Related work covers training-data extraction and memorization, contamination detection, benchmark perturbations, and dynamic evaluation. The paper distinguishes its focus on benchmark contamination from these neighboring directions.

  • Prior memorization studies address data privacy or the boundary between generalization and memorization, rather than benchmark contamination specifically.
  • Existing contamination studies include n-gram filtering and methods resembling decoding matching without access to training data.Exchange detection uses the order of test cases within a benchmark as a signal.
  • Benchmark-enhancement approaches perturb questions to prevent answer-pattern memorization or use human-in-the-loop dynamic evaluations.

8. Conclusion

The paper concludes that simple test variations evade existing detection and can enable benchmark overfitting, motivating an LLM-based decontaminator and fresh one-time exams. It urges stronger decontamination for public benchmarks.

  • Existing detection methods miss test cases with simple variations, while unremoved variations can let a 13B model overfit benchmarks.
  • The proposed LLM decontaminator reveals previously unknown test overlap when applied to real-world datasets.
  • The authors urge stronger decontamination for public benchmarks and fresh one-time exams for accurate LLM evaluation.

A. Rephrase Instruction Prompts

The paper uses rephrasing and translation prompts to create test variations while preserving question meaning, format, and task intent. Instructions also alter wording, variable names, code structures, and repeated-character patterns to bypass n-gram overlap checks.

  • MMLU: The rephrase template preserves meaning while limiting repeated word sequences and modifying mathematical variable names.The MMLU instructions also request preserving the multiple-choice format and replacing isolated numbers or letters when appropriate.
  • MMLU: The translation template preserves the multiple-choice format while changing wording, word order, formulas, and selected symbols.The instructions target both lexical overlap and identical mathematical expressions.
  • HumanEval: HumanEval rephrasing modifies code vocabulary and structure, including loop forms and inserted meaningless commands, to evade n-gram checks.The prompt also limits repeated words and characters and rewords comments or sentences.
  • HumanEval: HumanEval translation converts Python programs to C while applying additional wording, structural, and filler-code changes.The prompt explicitly suggests inserting statements such as an unused integer variable to bypass overlap detection.

B. Rephrase Examples

The paper illustrates rephrased samples across mathematical, coding, and benchmark datasets. Examples preserve the underlying task while changing wording, presentation, programming language, or solution structure.

  • Mathematical examples: Rephrased mathematical samples retain the original problems while adding requests to write Python programs for solving them.Examples cover algebraic simplification, circle equations, cone volume, and function composition.
  • Coding examples: Evol-Instruct-Code-80k-v1 examples rephrase a Fibonacci task while retaining its input-output examples and recursive solution intent.The rephrased sample includes the nth Fibonacci specification and returns for base cases and recursion.
  • Coding examples: CodeExercise-Python-27k examples restate an uppercase-character ASCII-sum task with equivalent documentation and implementation logic.The examples preserve the function behavior, while the rewritten version expands the explanation and names the function differently.
  • Benchmark examples: WizardLM-evol-instruct-V2 transforms an MMLU nuclear-family multiple-choice question into an open-ended definition prompt.The rewritten answer describes a married couple and their children as a household unit, while also mentioning possible relatives.
  • Coding examples: The Stack example preserves the happy-string task and examples while separating the prompt and solution presentation.The code checks short strings and scans each three-character window for repeated letters.
  • Coding examples: StarCoder-Data examples rephrase a cube-recognition task and expose a multi-line solution whose correctness can be inspected.The sample includes absolute-value handling, cube-root estimation, and a later check of the computed root.
Loading 2311.04850v2…