Source-linked AI summary
Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
Alon Jacovi, Avi Caciularu, Omer Goldman, Yoav Goldberg
TL;DR
The paper addresses how evaluation benchmarks can become contaminated through internet-crawled training data and closed APIs, while contamination remains difficult to detect. It proposes encryption and licensing, training-exclusion requirements for APIs, and contextual scrutiny of internet-derived data. These strategies are presented as practical protections, but their effectiveness is limited by negligence, adversarial annotation, and imperfect detection.
Problem
Evaluation data may enter internet-crawled training corpora or future API training, while contamination is difficult to verify and closed-model training data is often unknown.
Method
The paper proposes encrypting public test data with public-key methods and a No Derivatives license, withholding evaluations from APIs without exclusion controls, and releasing context to identify compromised internet-derived instances.
Results
BIG-Bench was compromised to an extent that prevented its usage entirely.
Takeaways & Limitations
Researchers should encrypt evaluation data, require API training-exclusion controls, reassess internet-derived benchmarks, and track who can access test data.
Takeaways & Limitations
Negligence can reduce or invalidate the strategies, including through re-uploaded plain-text datasets or adversarial crowdsourced annotation.
Abstract
from arXiv · showhide
Data contamination has become prevalent and challenging with the rise of models pretrained on large automatically-crawled corpora. For closed models, the training data becomes a trade secret, and even for open models, it is not trivial to detect contamination. Strategies such as leaderboards with hidden answers, or using test data which is guaranteed to be unseen, are expensive and become fragile with time. Assuming that all relevant actors value clean test data and will cooperate to mitigate data contamination, what can be done? We propose three strategies that can make a difference: (1) Test data made public should be encrypted with a public key and licensed to disallow derivative distribution; (2) demand training exclusion controls from closed API holders, and protect your test data by refusing to evaluate without them; (3) avoid data which appears with its solution on the internet, and release the web-page context of internet-derived data along with the data. These strategies are practical and can be effective in preventing data contamination.
1 Introduction
Internet-crawled training data and closed APIs make evaluation data difficult to certify as unseen, creating contamination risks whose scope is hard to determine. The paper outlines practical strategies for protecting evaluation integrity.
- Problem: Evaluation data available on the web may enter training, while closed models conceal their training data.The resulting evaluation data cannot be reliably certified as unseen.
- Problem: BIG-Bench was compromised enough to prevent its usage entirely.
- Problem: The community faces contamination from both internet-crawled training data and data collected through closed API calls.API providers may reserve the option to use submitted data for future training.
- Contribution: The paper outlines three practical strategies that individual researchers can use to protect evaluation integrity.
- Contribution: Strategy 1 protects data from automatic crawlers using public-key encryption and a license forbidding distribution of adaptations.
- Contribution: The paper also identifies training exclusion controls for closed APIs and recommends protecting test data by refusing evaluation without them.It notes that some API contexts provide exclusion guarantees while others do not.
2 Setting
The paper considers internet-crawled corpora and closed APIs as independent contamination scenarios under assumptions of contamination whenever exposure is possible. Its strategies target sincere, non-adversarial actors seeking to preserve clean evaluations.
- Scenarios: The paper considers two independent contamination scenarios: internet-crawled corpora and closed API models.
- Scenario 1: In the internet-crawled scenario, training data is automatically derived from the internet and difficult to exhaustively search for contamination.
- Scenario 1: Internet-derived labels or context may expose solutions even when the final evaluation dataset was processed or manually annotated.
- Scenario 2: In the closed-API scenario, any API call containing test data compromises that data for future evaluations when exclusion from training is not guaranteed.
- Assumptions: The strategies presume contamination whenever training exposure is possible, including through automatic crawling or API use.
- Assumptions: The setting assumes evaluators and model developers are sincere and share an incentive to demonstrate reliable evaluation results.The paper distinguishes this setting from adversarial sabotage.
3 Why is Data Contamination Prevalent?
Data contamination is difficult to establish because training data may be private, large-scale matching is costly and imperfect, and existing mitigations are incomplete or fragile over time.
- Private training data makes it impossible to know whether models were trained on particular evaluation data.
- Even known training corpora are difficult to inspect or filter because their scale and rapid development hinder thorough checks.
- Fuzzy matching is expensive to run routinely at scale and can miss contamination that evades its assumptions or coverage.
- Detection remains reactive: compromised data must be discarded and replaced, whereas the proposed strategies aim to prevent compromise.
- Hidden-answer leaderboards only partially mitigate contamination because test data remain crawlable and development sets remain exposed.
- Using newly created data is inefficient because its guarantee vanishes with newer models, requiring repeated and expensive data creation.
4 Suggested Mitigation Strategies
The paper proposes practical protections for evaluation data: encrypt public test data, require closed APIs to provide training exclusion controls, and scrutinize internet-derived data and its context.
- Strategy 1: Encrypting test data before upload provides a simple, cheap, preventative guard against automatic crawling of plain-text test data.The key may be released alongside the encrypted data because the target is non-adversarial crawling.
- Strategy 1: A No Derivatives license can prevent encrypted evaluation data from being redistributed without its encryption.CC BY-ND 4.0 is given as an example of a permissive license with this clause.
- Strategy 3: Few-shot prompts should be treated as evaluation data because their repeated plain-text publication enables reuse in later work.The paper recommends avoiding compromised prompts and annotating new ones instead.
- Strategy 2: Closed API evaluation should wait for training exclusion controls because the first API use compromises the test data under Scenario 2.An API holder may implement a system requesting exclusion from future training.
- Strategy 3: Automatic labels derived from internet context should be avoided because the context can directly reveal the solution.Manual annotations also require scrutiny when original context contains information useful for solving the instance.
- Strategy 3: Strategy 3 releases internet context as documentation and discards instances whose context significantly indicates the solution.Such instances may be used for training but are unsuitable for benchmarks.
5 Discussion and Open Questions
The discussion extends the recommendations to documenting existing evaluations, assessing internet presence, and considering partially effective or centralized approaches.
- Large-scale benchmark builders should document existing evaluations for contamination and employ Strategy 3, especially for internet-derived test sets.Older test sets can be trivially compromised when their origins or contents are online.
- Centralized strategies could provide benefits if many or all relevant actors cooperate from the outset, but this assumption is stricter.
- Research databases of evaluation data and contamination events may help cooperating developers avoid training on test examples, though they are not strictly effective mitigation.
- Watermarking test-data metadata or other non-intrusive context components remains an unexplored contamination-detection direction.
6 Conclusions
The paper calls on researchers to protect evaluation integrity by controlling access to test data and addressing contamination risks under stricter assumptions.
- Researchers should encrypt evaluation data or use gating mechanisms to evade automatic crawling.The paper also calls for withholding unseen evaluation data from closed APIs without exclusion controls.
- Researchers should reassess internet-derived evaluation data and consider who and what can access test data.The paper frames these actions as part of a broader call to address data contamination.
Limitations
The proposed strategies depend on sincere and careful actors, and their effectiveness can be weakened by negligence, legal uncertainty, or adversarial behavior.
- Negligent actors can reduce or invalidate the strategies by re-uploading plain-text datasets, making Strategy 2 especially vulnerable.Researchers can publicize contamination events, notify relevant contacts, retain uncompromised data where possible, or collect replacements.
- Strategy 2 is vulnerable when someone sends test data to a closed API without the API holder detecting or excluding it.Warnings on the data’s host webpage or file may reduce this scenario.
- The legal meaning of derivative work depends on local copyright law, and decryption may not constitute an adaptation.The Strategy 1 license is intended to encourage encrypted distribution among sincere actors rather than guarantee legal enforceability.
- Contamination-specific licenses could improve reliability and enforceability by restricting training or API use without exclusion controls.Such licenses may still permit distribution of derivatives when their terms specifically address contamination risks.
- Adversarial crowdworkers may use closed APIs to automate labeling against guidelines, compromising manually annotated data.This scenario falls outside the paper’s sincere-actor assumption.
A Partial Strategy A: Templated test instances
Templated test instances provide a partial way to generate unseen evaluation cases by converting data into programmatic templates and sampling counterfactual variants.
- A Partial Strategy A: Templated test instances: This strategy requires significant effort, is incomplete, and applies only in some cases, but remains one of few practical defenses against closed-API contamination.
- A Partial Strategy A: Templated test instances: Textual test data can be counterfactually augmented by converting it into a programmatic template.For arithmetic, answers can vary with conditioned numbers; for summarization, input changes should appear in the summary.
- A Partial Strategy A: Templated test instances: Each evaluation attempt samples one counterfactual variant with a seed and uses it across all evaluated systems.The seed is forfeited after evaluation, after which the seed or test data may be publicized.
- A Partial Strategy A: Templated test instances: More elaborate counterfactual variants provide stronger unseen-test protection even when the source instance is compromised.The strategy is expensive to implement, although test sets are traditionally small.