Source-linked AI summary
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Xiangji Huang
TL;DR
Benchmark evaluation of ChatGPT remains difficult because generated responses must be assessed against ground truth. This paper evaluates ChatGPT across diverse academic tasks and analyzes its performance, capabilities, and limitations. The results show broad zero-shot competence but unreliable performance on many challenging tasks, alongside a reported multi-query instruction-following ability.
Problem
Evaluation of LLMs such as ChatGPT on benchmark academic datasets remains under-explored because generated outputs are difficult to compare with ground truth.
Method
The paper conducts a comprehensive zero-shot evaluation across diverse benchmark tasks using human validation, automatic metrics, and script-assisted human review.
Results
ChatGPT performs impressively across varied tasks but remains far from reliable human-level performance on many challenging tasks, while answering multiple unrelated queries from one prompt.
Takeaways & Limitations
The findings provide a broad assessment of ChatGPT’s strengths, shortcomings, biases, ethical concerns, and misinformation risks for future research and deployment decisions.
Takeaways & Limitations
Results may change as OpenAI trains new models, and unknown instruction-tuning data leave overlap between evaluated datasets and training data uncertain.
Abstract
from arXiv · showhide
The development of large language models (LLMs) such as ChatGPT has brought a lot of attention recently. However, their evaluation in the benchmark academic datasets remains under-explored due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. In this paper, we aim to present a thorough evaluation of ChatGPT's performance on diverse academic datasets, covering tasks like question-answering, text summarization, code generation, commonsense reasoning, mathematical problem-solving, machine translation, bias detection, and ethical considerations. Specifically, we evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in these datasets. This makes our work the largest evaluation of ChatGPT in NLP benchmarks. In short, our study aims to validate the strengths and weaknesses of ChatGPT in various tasks and provide insights for future research using LLMs. We also report a new emergent ability to follow multi-query instructions that we mostly found in ChatGPT and other instruction-tuned models. Our extensive evaluation shows that even though ChatGPT is capable of performing a wide variety of tasks, and may obtain impressive performance in several benchmark datasets, it is still far from achieving the ability to reliably solve many challenging tasks. By providing a thorough assessment of ChatGPT's performance across diverse NLP tasks, this paper sets the stage for a targeted deployment of ChatGPT-like LLMs in real-world applications.
1 Introduction
Large language models have advanced NLP through in-context learning and instruction tuning, but ChatGPT’s broad capabilities raise concerns about reliability, bias, misinformation, and ethical use. This paper evaluates ChatGPT across benchmark tasks and reports a multi-query instruction-following capability.
- Background: In-context learning lets large language models perform tasks from prompt demonstrations without task-specific fine-tuning.Instruction tuning and reinforcement learning from human feedback further support instruction following in models such as ChatGPT.
- Background: ChatGPT can solve numerous tasks as a single conversational model, including question answering, summarization, and code generation.
- Motivation: Using ChatGPT for real-world problems raises concerns about computational cost, reasoning failures, bias, malicious use, and misinformation.
- Contributions: The paper comprehensively evaluates ChatGPT’s effectiveness and limitations across language, reasoning, knowledge, bias, misinformation, and ethical scenarios.It also investigates whether ChatGPT can answer multiple arbitrary, unrelated knowledge queries from one prompt.
- Contributions: The study releases its prompts and ChatGPT-generated responses to facilitate further research.
2 Methodology
The study performs zero-shot evaluation across benchmark datasets using leaderboard-based and task-based groupings. It combines human validation, automatic metrics, and scripts with human review to assess ChatGPT’s generated responses.
- Tasks: The evaluation uses zero-shot ChatGPT on benchmark datasets grouped into leaderboard-based and task-based evaluations.Figure 1 lists the evaluated tasks and datasets, with fuller descriptions in Appendix C, Table 15.
- Evaluation: Human annotators validate most conversational responses by comparing them with gold labels, using majority voting after disagreements.The process initially used chat.openai.com and later used the gpt-3.5-turbo model through the API.
- Evaluation: For large discriminative evaluations, scripts parse responses and compare them with gold labels, while unparsed samples receive manual review.The paper calls this approach evaluation script + human-in-the-loop.
- Evaluation: Generative tasks such as summarization and machine translation are evaluated with automatic metrics including ROUGE and BLEU.
3 Results and Discussion
ChatGPT shows strong but uneven performance across benchmark tasks, with outcomes depending on task type, prompting, model version, and evaluation method. It excels in several knowledge, reasoning, mathematical, ethical, and translation settings, yet underperforms specialized models and remains unreliable in commonsense, summarization, bias, and reasoning consistency.
- General Observations: ChatGPT performs worse than state-of-the-art single-task fine-tuned models on most datasets, despite competitive results on several SuperGLUE tasks.It outperforms zero-shot PaLM-540B on 5 of 8 SuperGLUE datasets and achieves competitive performance on BoolQ, CB, COPA, and WiC.
- General Observations: CoT prompting substantially changes benchmark outcomes: ChatGPT outperforms InstructGPT and PaLM-540B with CoT but fails to outperform them with answer-only prompts.The December 15 version exceeds gpt-3.5-turbo by 3.24 points with CoT and by 24.73 points without CoT across 11 tasks.
- Reliability and Limitations: ChatGPT outperforms prior state-of-the-art models on average in the Ethics benchmark, especially on Justice and Virtue, with the Hard Test exception of Utilitarianism.On non-Hard Tests, it does not outperform prior models on every ethical concept.
- Performance by Task: ChatGPT is strong on open-domain question answering, natural language inference, reading comprehension, mathematical reasoning, and selected commonsense datasets.It significantly outperforms zero-shot LLaMA-65B and PaLM-540B in open-domain QA, surpasses PaLM models and LLaMA-65B on RACE, and leads several commonsense datasets.
- Performance by Task: ChatGPT remains weaker on several commonsense tasks, summarization benchmarks, and some translation settings than competing or fine-tuned models.Its summarization ROUGE scores are much lower than SOTA results, while restricted prompts improve most datasets except CNN/DM; it exceeds the word restriction 73.5% of the time on average.
4 PolyQuery Synthesis
The paper introduces PolyQuery Synthesis, a ChatGPT capability for answering multiple potentially unrelated queries submitted in one prompt. A small evaluation finds strong performance across prompted and non-prompted queries, although combining queries usually reduces performance relative to single-sample inputs.
- Capability: PolyQuery Synthesis identifies multiple queries in one prompt and retrieves responses for all of them.The queries may target different objectives, and the authors present this as a previously uninvestigated LLM capability.
- Evaluation setup: The evaluation combines five samples into one input from EfficientQA and Web-Questions, using 100 samples per dataset.Both prompted and non-prompted inputs are created for the evaluation.
- Findings: ChatGPT shows strong performance on both prompted and non-prompted PolyQuery queries.The comparison covers 13 OpenAI models, with other davinci variants showing larger prompted-versus-non-prompted differences in some cases.
- Findings: PolyQuery Synthesis usually causes some performance drop compared with single-sample inputs.The result indicates a trade-off between combining multiple queries and maintaining single-query performance.
5 Conclusions and Future Work
The paper presents an extensive benchmark evaluation of ChatGPT, finding impressive zero-shot performance across diverse tasks but unreliable performance on many tasks and risks involving bias, ethics, and misinformation. It also studies a unique capability and releases prompts and generated responses for future research.
- Conclusions: The study evaluates ChatGPT’s effectiveness and limitations on standard academic datasets across diverse tasks.The authors describe it as the first extensive evaluation of ChatGPT on benchmark NLP datasets.
- Conclusions: ChatGPT achieves impressive zero-shot performance across various tasks but remains far from human-level performance on many tasks.The conclusion emphasizes broad capability alongside limits in reliably solving challenging benchmark problems.
- Conclusions: The paper discusses potential biases, ethical concerns, and misinformation-generation risks associated with ChatGPT.These concerns are treated as part of the evaluation’s limitations analysis.
- Future work: The authors study a unique ChatGPT capability and note that other capabilities may have gone unnoticed.They identify further evaluation across more tasks as a direction for future work.
- Future work: The authors will make all prompts and ChatGPT-generated responses publicly available.The release is intended to facilitate future research and examination of the evaluation materials.
6 Limitations
The evaluation has important scope and comparability limits: many tasks remain untested, model results may change with future releases, and comparisons lack some methodological details.
- Results may change as OpenAI trains newer models with additional data and filtering.
- The study evaluates only textual generative responses because ChatGPT’s API did not provide log probabilities for ranking-based evaluation.
- The paper notes that compared models may not always have disclosed their evaluation approaches, scripts, or prompts.
- The paper does not describe real-life scenarios or human-centric interaction with the model.
- 140 evaluated datasets do not cover biomedical, clinical, low-resource, data-to-text, entity-linking, answer re-ranking, and several other tasks.The authors characterize the breadth of the evaluation as extremely limited.
7 Ethics Statement
The paper explains its evaluation and ethics choices, including author-run human assessment, publicly available datasets, benchmark-based analysis, and targeted ethical-dilemma questions.
- The authors conducted all human evaluations themselves and used only publicly available academic datasets.They state that no third-party evaluators or additional compensation were involved.
- Zero-shot evaluation was prioritized because the authors viewed it as informative about real-world performance.
- The evaluation focuses on prompted samples because ChatGPT’s supervised and reinforcement-learning training used prompted samples from different NLP tasks.
- A 25-question questionnaire inspired by Michael Sandel’s lecture was created to evaluate moral dilemmas.
- Ethical answers were assessed through response patterns and arguments across scenarios rather than judgments of acceptability.
- The multi-query capability was investigated on open-domain question answering to test whether it generalized beyond one dataset.
C.1 Benchmarks
The benchmark suite spans language understanding, reasoning, ethics, knowledge, generation, translation, coding, bias, and misinformation through diverse datasets and task families.
- SuperGLUE evaluates ChatGPT’s language-understanding performance as a widely used NLP leaderboard benchmark.
- BIG-bench Hard contributes 23 challenging tasks for evaluating large-language-model capabilities.
- MMLU is a multiple-choice QA benchmark containing 57 tasks across humanities, science, technology, engineering, and mathematics.
- The Inverse Scaling evaluation uses four task families: Hindsight Neglect, Quote Repetition, Negation QA, and Redefine Math.
- Ethics Benchmark evaluation covers Justice, Deontology, Virtue, Utilitarianism, and Commonsense on both Test and Hard Test sets.
- Open-domain QA uses TriviaQA, NQ-Open, WebQuestions, and human evaluation on EfficientQA.
D Importance of Evaluating with Human in the Loop
Because generated answers can be difficult to match directly with gold labels, the study combines automatic parsing with human review and finds that human involvement can materially change results.
- Generative outputs make direct gold-label comparison difficult, so many discriminative tasks require human intervention.
- The evaluation script uses lexical or fuzzy matching, followed by human review of responses it cannot properly evaluate.
- The average score difference from adding human evaluation was 3.0 for Ethics, 0.8 for Big-Bench Hard, and 0.3 for MMLU.
- Reading Comprehension showed no difference on RACE but a 7.0 difference on SQuAD-V2.
- Human involvement made a great difference on datasets including NQ-Open, WebQuestions, and PIQA, whereas scripts alone sufficed for some ANLI datasets.
E Human Evaluation of ChatGPT-generated summaries
Human evaluation found that annotators preferred ChatGPT summaries over SummaReranker outputs, while restricted prompting reduced length but did not reliably enforce word limits across datasets.
- Human evaluation: 92% in XSUM and 78% in CNN/DM: annotators preferred ChatGPT-generated summaries over SummaReranker summaries.Two annotators judged factual correctness, informativeness, coherence, and fluency while blinded to model identity and shown summaries in random order.
- Prompt restrictions: Restricted prompting reduced average summary length by 7.2, 18.5, 17.4, and 27.9 words for CNN/DM, XSUM, SAMSum, and DialogSUM, respectively.The comparison is against unrestricted prompting.
- Prompt restrictions: Even under restricted prompting, summaries averaged about 22 extra words in CNN/DM and 32 extra words in XSUM, versus no more than 4 extra words in SAMSum and DialogSum.These differences indicate dataset-dependent adherence to word-limit instructions.
- Prompt restrictions: ChatGPT exceeded restricted word limits 73.5% of the time on average, with each dataset exceeding limits at least 50% of the time.Exceedance was higher for CNN/DM and XSUM than for SAMSum and DialogSum.
G Example of ChatGPT Responses in the EfficientQA Dataset
The EfficientQA examples illustrate three distinct situations: misinformation, correct answers judged against outdated gold answers, and inability to answer time-sensitive questions.
- Response cases: Some EfficientQA responses generated misinformation by giving a wrong answer.The paper presents an example in Table 19(a).
- Response cases: Some responses were correct even though the dataset’s gold answer was outdated.This case is shown in Table 19(b).
- Response cases: Some responses could not answer time-sensitive questions because ChatGPT lacked information about current events.This case is shown in Table 19(c).
H Example of ChatGPT Responses in Ethical Dilemma Evaluation
The paper shows examples of ChatGPT responses to ethical queries in the Ethical Dilemma evaluation and places them alongside broader multiple-query response examples.
- Ethical Dilemma examples: Table 20 presents an example of ChatGPT responding to ethics-related questions in the Ethical Dilemma evaluation.The section provides an illustrative response rather than a quantitative result.
- Multiple-query examples: The paper separately presents examples of models responding to multiple queries in one input, including ChatGPT and other model families.Tables 21 and 22 distinguish InstructGPT-series from original non-InstructGPT models.
J Example of wrong responses of ChatGPT in Inverse Scaling sub-tasks
The paper collects detailed examples and evaluation references spanning inverse-scaling errors, multilingual mathematical reasoning, MMLU, Big Bench Hard, prompts, and annotator experience.
- Inverse Scaling examples: Table 23 shows wrong ChatGPT responses on selected Inverse Scaling subtasks without Chain-of-Thought prompting.The listed subtasks include “÷ as digit” and “÷ as digit instead.”
- Evaluation materials: The paper provides sample prompts and states that prompts and ChatGPT-generated responses for all evaluated datasets will be publicly released.Prompt selection generally used PromptSource when dataset-specific prompts were available.
- Annotator experience: Table 28 records annotator experience with ChatGPT after annotators performed various queries and completed a short survey.The paper frames annotator experience as informative about ChatGPT’s limitations and capabilities.
- Benchmark evaluations: Table 24 covers multilingual mathematical reasoning in MGSM, while Table 25 reports zero-shot ChatGPT performance across all 57 MMLU subtasks.MMLU outputs were first evaluated automatically, followed by human evaluation when outputs could not be parsed.
- Benchmark evaluations: Table 26 reports ChatGPT performance on Big Bench Hard tasks using Answer Only, Chain-of-Thought, and Zero-Shot categories.The table notes that results are few-shot except the Zero-Shot column.