Source-linked AI summary
Can we trust the evaluation on ChatGPT?
Rachith Aiyappa, Jisun An, Haewoon Kwak, Yong-Yeol Ahn
TL;DR
Because ChatGPT’s training data are unknown and the model is continuously updated, it is impossible to ascertain that evaluation datasets have not contaminated it. The authors compare ChatGPT versions on stance detection and observe improved performance after a prior evaluation, but cannot establish whether contamination occurred or affected the results.
Problem
ChatGPT’s closed training data and continuous updates make it impossible to ascertain whether benchmark datasets have contaminated the model, a concern because leakage can make evaluation results unreliable.
Source: §1 · pp. 1–2Method
The authors compare ChatGPT versions on the SemEval 2016 Task 6 and P-stance test sets, using the November 30 or December 15 version as V1.
Source: §2 · p. 2Results
Takeaways & Limitations
Closed models with continuous training make preventing data contamination and ensuring fair model evaluation more challenging.
Source: §4 · p. 4The study cannot prove or disprove whether data leakage occurred or affected ChatGPT’s evaluation.
Source: Limitations · p. 4
Abstract
from arXiv · showhide
ChatGPT, the first large language model (LLM) with mass adoption, has demonstrated remarkable performance in numerous natural language tasks. Despite its evident usefulness, evaluating ChatGPT's performance in diverse problem domains remains challenging due to the closed nature of the model and its continuous updates via Reinforcement Learning from Human Feedback (RLHF). We highlight the issue of data contamination in ChatGPT evaluations, with a case study of the task of stance detection. We discuss the challenge of preventing data contamination and ensuring fair model evaluation in the age of closed and continuously trained models.
1 Introduction
Because ChatGPT is closed and continuously updated, its training data are unknown, making it impossible to ascertain that evaluation datasets have not contaminated the model. The authors use stance detection to examine this risk and report improved performance in versions released after a prior evaluation, while acknowledging that contamination cannot be established.
1 Introduction
- Unknown training data and continual updates make it impossible to ascertain whether ChatGPT has been exposed to evaluation datasets.The concern matters because data leakage can make evaluation results unreliable and may inflate estimated performance.Source: §1 · pp. 1–2
- ChatGPT is described as trained through supervised fine-tuning, reward-model training, and further fine-tuning with human-preference rewards.The final stage uses Proximal Policy Optimization on prompts submitted to the OpenAI API.Source: §1 · pp. 1–2
- Contamination may involve both task inputs and labels, or input-only exposure; the authors cannot exclude either possibility for ChatGPT.They note that exposure to text in the same domain can increase performance even without correct labels, and that RLHF responses could potentially match input text with output labels.Source: §1 · p. 2
- Stance detection extracts a text’s Favor, Against, or Neither position toward a target, and prior studies reported strong zero-shot ChatGPT results on SemEval 2016 Task 6 and P-stance.The task is harder on social media because tweets may contain abbreviations, hashtags, URLs, spelling errors, and incoherent text.Source: §1 · p. 2
- ChatGPT was updated repeatedly after its November 30, 2022 release, creating an opportunity to compare performance before and after the prior stance-detection evaluation.The authors report overall improvement after that evaluation, while noting that the model may simply have improved despite updates aimed at restricting problematic behavior.Source: §1 · p. 2
2 Methods
The authors compare ChatGPT versions on the SemEval 2016 Task 6 and P-stance test sets, treating the November 30 or December 15 version as V1. Data collection used different versions and procedures because earlier models were unavailable and access was constrained.
2 Methods
- The authors assume Zhang et al. used the November 30 or December 15 ChatGPT version, termed V1, and reuse the SemEval 2016 Task 6 and P-stance test sets.SemEval covers six targets and P-stance covers three; Figure 1 marks ChatGPT version dates and the study datasets.Source: §2 · p. 2, Figure 1 · p. 3
- The authors reused the prior prompt, selecting Favor, Against, or Neutral for SemEval and Favor or Against for P-stance.P-stance lacks a neutral stance label.Source: §2 · p. 2
- They manually collected Jan. 30 ChatGPT responses for 860 SemEval tweets covering Hillary Clinton, Feminist Movement, and Legalization of Abortion.The sampled targets matched those used in the previous work.Source: §2 · p. 2
- For Feb. 13 ChatGPT Plus, the authors collected responses on both datasets and manually extracted explicit stance labels; P-stance contributed 2,157 test tweets.The Jan. 30 version was no longer available when they decided to include P-stance.Source: §2 · p. 3
- Version comparisons were constrained because older models were unavailable, hourly requests were limited, and no public API hindered data collection.Consequently, they used Feb. 13 ChatGPT Plus for P-stance and both Jan. 30 ChatGPT and Feb. 13 ChatGPT Plus for SemEval.Source: §2 · p. 3
3 Evaluation Metric and Results
Recent ChatGPT versions outperform V1 on both stance-detection datasets, though gains differ by dataset and version. The authors also note that possible contamination complicates interpreting these results.
3 Evaluation Metric and Results
- Macro-F averages the favor and against class F scores, whereas micro-F aggregates true and false positives and negatives across those classes.Source: §3 · p. 3
- 12.46 micro-F and 8.6 macro-F point improvements were observed for Jan 30 ChatGPT versus V1 on SemEval.Feb 13 ChatGPT plus showed smaller improvements of 6.1 micro-F and 1.89 macro-F points versus V1 on SemEval.Source: §3 · p. 3
- The authors raise possible contamination concerns even for V1, given documented NLP dataset contamination in InstructGPT.Source: §3 · p. 3
- 0.74 micro-F and 0.26 macro-F point improvements were observed for Feb 13 ChatGPT plus versus V1 on P-Stance.Source: §3 · p. 3
- SemEval’s improvement was greater than P-Stance’s, and Feb 13 ChatGPT plus scored below Jan 30 ChatGPT on SemEval while remaining above V1.Source: §3 · p. 3
4 Discussion and Conclusion
The discussion argues that closed, continually updated models make data leakage difficult to rule out, so apparent benchmark performance may not be a clean measure of zero-shot ability. The authors call for more transparent training-data practices and safeguards.
4 Discussion and Conclusion
- Because ChatGPT is closed and continually updated, researchers cannot verify dataset use or ensure that evaluation data has not entered training.The authors warn that evaluations may expose their datasets to the model, making subsequent evaluations unreliable; even input-only exposure may matter.Source: §4 · p. 3
- The stance-detection case study finds potential contamination from prior evaluations on SemEval 2016 Task 6 and P-Stance, but cannot distinguish leakage from genuine superiority.Source: §4 · p. 3
- Table 1 reports zero-shot Micro-F1/Macro-F1 scores across ChatGPT versions on SemEval 2016 Task 6.Source: Table 1 · p. 4
- Figure 2 charts zero-shot macro-F performance on SemEval 2016 Task 6A across models, with earlier-model scores from Zhang et al. (2022).Source: Figure 2 · p. 4
- BIG-bench’s canary string can probe training exposure, but checking contamination is increasingly challenging for closed models.The canary is used to filter benchmark tasks from training data and probe exposure through anomalous probability assignments.Source: §4 · p. 4
- The authors recommend documenting potential contamination, enabling scrutiny of training data and models, and preventing contamination from user inputs.Source: §4 · p. 4
Limitations
The evaluation was constrained by ChatGPT’s active development and collection conditions that could affect responses. The authors could not estimate performance uncertainty, and their results cannot determine whether contamination occurred or affected evaluation.
Limitations
- Rate limits restricted Jan 30 SemEval data collection to roughly 40 queries per hour, so responses were collected across sittings with changing verbosity.A single chat session was used for multiple inputs in each sitting, which may have accumulated context; Feb 13 ChatGPT Plus instead used a new session per query.Source: Limitations · p. 4
- The authors could not try multiple queries per input or estimate performance uncertainty.Source: Limitations · p. 4
- The results cannot prove or disprove whether data leakage occurred or affected ChatGPT evaluation.Source: Limitations · p. 4
Ethics Statement
The paper notes that contamination has implications for scientific research using closed language models and for measuring NLP progress. Because training data may include sensitive or proprietary text and closed models make its presence difficult to verify, the authors call for more work on detection and prevention.
Ethics Statement
- Contamination has implications for scientific research using closed language models, measuring NLP progress, and commentary about LLM emergent properties or “intelligence.”Source: Ethics Statement · p. 5
- Digital text used to build LLMs may contain sensitive or proprietary information, while practices to exclude such data remain preliminary.Source: Ethics Statement · p. 5
- For newer closed-source models, verifying whether such data is in training sets is described as virtually impossible; the authors call for experiments and methods to quantify and prevent its use.Source: Ethics Statement · p. 5
A.1 Uncertainties in Zhang et al., 2022
The authors flag ambiguities in Zhang et al., 2022’s reported scores and prompts. For their comparison, they adopt a worst-case interpretation of the F1 labels and use the prompt format shown in that work’s screenshot.
A.1 Uncertainties in Zhang et al., 2022
- The authors requested clarification because Zhang et al., 2022’s F1-m and F1-avg definitions appeared to describe the same score, yet their reported values differed.Source: §A.1 · p. 6
- To capture a worst-case scenario, the authors treat F1-m as micro average and F1-avg as macro average.They note that PT-HCL and TPDG scores appear under the opposite labels from those indicated by their original articles.Source: §A.1 · p. 6
- Because Zhang et al., 2022 showed mismatched prompt text in its body and figures, the authors followed the screenshot format, including neutral for SemEval but not P-Stance.Source: §A.1 · p. 6