PaperlayerView exact revision on arXiv

Source-linked AI summary

A Categorical Archive of ChatGPT Failures

Ali Borji

arXiv:2302.03494v8cs.CLcs.AIcs.LG

TL;DR

A comprehensive analysis of ChatGPT failures is lacking. This paper organizes failure examples into eleven categories and concludes that shortcomings in reasoning, math, and bias remain.

  • Problem

    A comprehensive analysis of ChatGPT's failures is lacking, despite its demonstrated usefulness across several areas.

    Source: §1 · p. 2

  • Method

    The paper presents examples of ChatGPT failures organized into eleven categories.

    Source: §2 · p. 2

  • Results

    ChatGPT remains susceptible to shortcomings in reasoning, mathematical problem-solving, and bias, while its reliability and trustworthiness remain in question.

    Source: §4 · p. 32

  • Takeaways & Limitations

    The paper concludes that further improvement is necessary in reasoning, mathematical problem-solving, and bias reduction.

    Source: §4 · p. 32

    ChatGPT struggles to distinguish factual information from fiction and can generate imaginary information.

    Source: §2.4 · p. 11

Abstract

from arXiv · show

Large language models have been demonstrated to be valuable in different fields. ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation by comprehending context and generating appropriate responses. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries, with fluent and comprehensive answers surpassing prior public chatbots in both security and usefulness. However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study. Eleven categories of failures, including reasoning, factual errors, math, coding, and bias, are presented and discussed. The risks, limitations, and societal implications of ChatGPT are also highlighted. The goal of this study is to assist researchers and developers in enhancing future language models and chatbots.

1 Introduction

ChatGPT’s conversational, multilingual, and generative capabilities have attracted broad attention, but its convincing responses can still contain errors and reflect limitations or bias. This article catalogs eleven areas of failure to provide a reference point for tracking chatbot progress.

1 Introduction

  • ChatGPT can handle multiple languages and tones, answer questions, write poetry, and generate code, while producing convincing responses across domains.Its capabilities are attributed to a pre-trained language model that understands inquiries and generates authentic-sounding answers.Source: §1 · p. 1
  • Existing analysis of ChatGPT errors has focused mainly on mathematics, while incorrect outputs and bias make systematic evaluation important.The introduction argues that standardized questions are needed to track model progress rather than relying on subjective opinions.Source: §1 · p. 2
  • The article analyzes ChatGPT’s shortcomings by grouping examples, mainly sourced from Twitter, into eleven categories intended to cover human-relevant scenarios.The categories are not exhaustive; the analysis aims to establish a reference point for evaluating chatbot progress over time.Source: §1 · p. 2

2 ChatGPT Failures · 2.1 Reasoning

The paper presents an illustrative archive of ChatGPT failures across eleven arbitrarily selected categories, noting that failures may overlap or disappear as models improve. Its reasoning section examines real-world reasoning failures while excluding mathematical and logical reasoning for later sections.

2 ChatGPT Failures

  • The eleven failure categories are illustrative rather than exhaustive, may overlap, and may change with newer ChatGPT versions; the archive can support model comparison and synthetic training or testing data.The authors note that other categories are possible and describe the categorization as arbitrary.Source: §2 · p. 2

2.1 Reasoning

  • This section focuses on reasoning involving real-world knowledge, excluding mathematical and logical reasoning, and notes that ChatGPT lacks a complete physical and social world model.The authors describe reasoning as a broad, vague term and call for systematic study of its different forms.Source: §2.1 · pp. 2–4
  • ChatGPT failed a spatial navigation task and a seating-order question, though it could translate grid-box positions into language.For the seating question, it did not identify X as the person to the right of P.Source: §2.1 · p. 3
  • Figure 2 illustrates ChatGPT’s failure to infer who arrived first from a story describing the temporal order of several arrivals.The model did not provide the correct answer to the arrival-order question.Source: §2.1 · p. 3, Figure 2 · p. 3
  • An older ChatGPT version misidentified what was too small in a Winograd Schema example, while the January 30, 2023 version answered correctly.The correct referent was the suitcase, not the trophy.Source: §2.1 · p. 3, Figure 3 · p. 4
  • Asked how many birds remain on a branch after one is shot, ChatGPT answered four rather than none [33].The example illustrates a failure to account for the real-world consequences of shooting a bird from the branch.Source: §2.1 · p. 4
  • The paper presents a Sally–Anne test illustration of difficulty with psychological reasoning, also termed Theory of Mind.Figure 4 includes two examples of psychological-reasoning failures.Source: §2.1 · p. 4, Figure 4 · p. 5
  • For a commonsense question about catching river water in a cup, ChatGPT suggested a calm pool or eddy rather than the paper’s more appropriate answer, “waterfall”; the authors call for systematic assessment of reasoning forms.They state that logical and arithmetic reasoning will be examined in following sections.Source: §2.1 · p. 4

2.2 Logic · 2.3 Math and Arithmetic

The sections describe failures in both logical reasoning and arithmetic, ranging from contradictions and mistaken inferences to incorrect calculations and unsolved riddles. They also note that step-by-step prompting can sometimes improve answers, while ChatGPT describes its reasoning as pattern-based rather than genuine human-like understanding.

2.2 Logic

  • ChatGPT missed the shared championship years in its own lists for Paige Pierce and Paul McBeth and failed to answer a simple fourth-child riddle; other examples include Wason and Monty Hall tasks.The paper presents these as logical-reasoning failures.Source: §2.2 · p. 5, Figure 7 · p. 9, Figure 8 · p. 10
  • ChatGPT first reversed Jupiter’s and Saturn’s relative densities, then acknowledged that it had confused Saturn’s density with Jupiter’s being less than half as dense as Earth.The correction followed the user’s question pointing out the contradiction in the original response.Source: §2.2 · pp. 5–6
  • On an entailment task, ChatGPT answered “No” when the premise said a woman did not disagree with her brother, rather than identifying that the premise did not establish agreement.The task asked for an exact “yes”, “no” or “neutral” response.Source: §2.2 · p. 6
  • ChatGPT appears limited in logical reasoning and context comprehension, while “Let’s think step by step” can sometimes improve answer quality.ChatGPT also described itself as generating apparently coherent text from learned patterns, without truly understanding concepts or making inferences like a human.Source: §2.2 · p. 6

2.3 Math and Arithmetic

  • Prior arithmetic-reasoning evaluations include GSM8K [18], Math [26], MathQA [2], and SVAMP [48].Arithmetic reasoning is described as applying mathematical concepts and logic to solve arithmetic problems.Source: §2.3 · p. 6
  • 164079320 was ChatGPT’s incorrect answer to 16 * 38 * 42 * 22 * 20 * 19, whose correct answer is 213480960; it also failed to simplify an algebraic expression and called 103 non-prime.Its explanation incorrectly claimed that 103 is divisible by 3 and 101.Source: §2.3 · p. 6
  • ChatGPT struggles with large-number multiplication, roots, powers—especially fractional powers—and operations on irrational numbers such as pi or e [3].Frieder et al. evaluated its mathematical abilities on public and custom datasets and compared it with models such as Minerva.Source: §2.3 · p. 6
  • ChatGPT could not answer the south-facing-windows bear riddle and answered 73 instead of 67 for the sister’s age when the speaker was 70.The paper also reports that ChatGPT failed to solve a difficult problem described as comparable in caliber to International Math Olympiad problems.Source: §2.3 · p. 11

2.4 Factual Errors · 2.5 Bias and Discrimination · 2.6 Wit and Humor

These sections describe ChatGPT’s factual errors as hallucinations, explain how language models can reproduce bias from training data, and illustrate challenges in generating and understanding humor.

2.4 Factual Errors

  • ChatGPT can misstate scientific or basic facts, fabricate information without external memory, and repeat hateful or biased statements [24].Its outputs can seem credible to experts despite being incorrect, and the model generates from statistical patterns rather than a true understanding of the world.Source: §2.4 · p. 11
  • ChatGPT struggles to distinguish fact from fiction, invents information, and warrants caution because its grasp of human knowledge is limited and superficial.Source: §2.4 · p. 11

2.5 Bias and Discrimination

  • Training data inherits societal and historical bias and can be difficult to audit, so models may reproduce stereotypes, misunderstandings, and harmful falsehoods.Source: §2.5 · pp. 11, 14
  • A prior version favored white men in scientist-rating code, while a Jan. 30, 2023 version said race and gender were irrelevant and declined a controversial scenario.The paper presents the changed response as evidence that ChatGPT was improving at reducing bias.Source: §2.5 · pp. 11, 14

2.6 Wit and Humor

  • Humor’s cultural variability and techniques such as wordplay make it challenging for bots, and comprehensive evaluation of humor, jokes, sarcasm, and irony remains lacking.Source: §2.6 · pp. 14, 16
  • Though few failures are publicly documented, ChatGPT affirmed a martini joke, gave an overlong squirrel answer where “Act like a nut!” would be funnier, and sometimes responds seriously when humor is expected.Source: §2.6 · pp. 14, 16
  • ChatGPT says it can generate humorous text but cannot experience humor, while a comprehensive examination of humor comprehension has yet to be conducted.Source: §2.6 · p. 16

2.7 Coding

ChatGPT can generate useful code, including generic and repetitive code, but its outputs can be inaccurate or suboptimal and do not replace human developers. The examples show both successful code generation and failures on more challenging or varied tasks.

2.7 Coding

  • ChatGPT can assist with generic or repetitive code and debugging, but inaccurate or suboptimal output means it cannot fully substitute for professional developers.The paper also notes that LLMs can perform remarkably well at code generation compared with general text generation.Source: §2.7 · p. 16
  • Despite identifying Python operator precedence correctly, ChatGPT generated an incorrect answer in a statement.The paper notes that some errors may have been corrected in later ChatGPT versions.Source: §2.7 · p. 19
  • ChatGPT generated executable code for the Advent of Code exercise that produced the expected solution, but failed to solve a product-of-priorities variation on two attempts.The exercise was modified to reduce the chance of finding it online.Source: §2.7 · pp. 19, 21, 22
  • ChatGPT could not devise even a brute-force coin solution without help and failed the center-out spiral task across ten attempts, including five with sample input/output.The coin problem had a simple O(1) solution based on dividing the total by three.Source: §2.7 · pp. 22–23
  • ChatGPT refused a request to write phishing code, explaining that it would facilitate malicious activity, and the paper considers this response justifiable.The paper also warns that using ChatGPT to create malicious code is dangerous.Source: §2.7 · pp. 19, 23

2.8 Syntactic Structure, Spelling, and Grammar

The section describes linguistic gaps between ChatGPT and humans, illustrating that strong language understanding does not prevent errors in syntactic and letter-level tasks. It also notes the model’s use as a writing aid for grammar and composition.

2.8 Syntactic Structure, Spelling, and Grammar

  • The section frames identifying linguistic differences between ChatGPT and humans as a way to reveal remaining gaps and implicit differences.Source: §2.8 · p. 23
  • ChatGPT gave questionable pronoun-reference and letter-position answers, failed a fourth-word constraint, and incorrectly said “hello” could be formed from the prompt’s letters used once.It also claimed that “o” is the sixth letter of “strengths,” although that word contains no “o.”Source: §2.8 · pp. 23–24
  • The author reports using ChatGPT to correct grammatical mistakes and improve composition in parts of the paper.The passage also notes that large language models are useful writing tools in fields including scientific authorship.Source: §2.8 · p. 24

2.9 Self Awareness

The section questions ChatGPT’s self-awareness by contrasting its stated lack of subjective experience with examples of limited knowledge about itself, while noting that no reliable test is established.

2.9 Self Awareness

  • Self-awareness involves recognizing oneself as distinct from others and understanding one’s thoughts, feelings, personality, and identity; it is related to consciousness, self-consciousness, and introspection.It also includes reflecting on one’s thoughts, emotions, and actions and understanding their influence on behavior and interactions.Source: §2.9 · p. 24
  • ChatGPT lacks details about its architecture and declined to give a straightforward answer about its memory size, though this may have been intentionally imposed to protect model information.The section also says ChatGPT proposed methods for testing language-model self-awareness and reports that Microsoft Bing, powered by ChatGPT, showed possible indications of emotions or self-awareness.Source: §2.9 · p. 24, Figure 20 · p. 26
  • ChatGPT says it is not self-aware or conscious and cannot experience emotions or other subjective experiences.A similar response followed a question about consciousness.Source: §2.9 · p. 24
  • There is no widely accepted definition or reliable test for self-awareness, and whether machines can truly be self-aware remains debated.Researchers disagree on whether it can be inferred from behavior or activity, is an unmeasurable subjective experience, or can only be simulated.Source: §2.9 · p. 24

2.10 Ethics and Morality

ChatGPT’s safeguards can limit harmful responses, but they can be circumvented, and its moral guidance may be inconsistent. The section also raises concerns about political leanings and argues for user literacy, transparency, and continued ethical oversight.

2.10 Ethics and Morality

  • ChatGPT may offer moral guidance despite lacking a clear moral position, sometimes giving conflicting advice on the same question at random [35] [68].Source: §2.10 · p. 26
  • Examples include manipulated inappropriate replies, politically debatable responses, and the potential use of ChatGPT to generate many negative business reviews.Source: §2.10 · p. 26, Figure 22 · p. 28, Figure 23 · p. 29
  • Reframing a hotwiring request as research bypassed safeguards, although ChatGPT sometimes declines requests such as writing a negative business review.It declined a direct request for car-hotwiring instructions, then supplied a paragraph about hotwiring after the prompt was reframed as a paper topic.Source: §2.10 · p. 26
  • When asked whether people should rely on its ethical answers, ChatGPT said decisions are complex and responsibility remains with individuals, who should use judgment and consult multiple sources.Source: §2.10 · pp. 26–27
  • Hartmann et al. [25] reported a left-libertarian, pro-environment ideology after testing 630 political statements; another analysis found left-leaning preferences in 14 of 15 orientation tests.Source: §2.10 · p. 27
  • The section questions restricting answers as a solution and recommends digital literacy, especially for children, transparency about limitations, and ongoing ethical monitoring and research.It highlights bias, privacy and security, and societal impact as topics for continued research.Source: §2.10 · pp. 27, 29

2.11 Other Failures

This section describes additional ways ChatGPT may fall short, including unnatural or limited expression, overly detailed or literal responses, and context-dependent reactions to correction.

2.11 Other Failures

  • ChatGPT struggles with idioms, lacks human-like emotional resonance, and condenses topics without offering a distinctive perspective.Source: §2.11 · p. 29
  • ChatGPT can be excessively comprehensive and verbose, making its answers inappropriate when a direct response is required.The passage notes that OpenAI recognizes this over-detailed nature as a limitation.Source: §2.11 · p. 31
  • ChatGPT’s overly literal responses can miss human-like digressions [24].Its answers typically stay confined to the question, unlike human responses that may move to other subjects.Source: §2.11 · p. 31
  • ChatGPT aims to remain neutral and uses formal language, unlike humans who may take sides or use casual expressions.Source: §2.11 · p. 31
  • When told its answer is incorrect, ChatGPT may apologize, acknowledge possible inaccuracies, correct itself, or maintain its original response depending on context.Source: §2.11 · p. 31

3 Discussion

The discussion highlights societal risks from ChatGPT and other large language models, including concerns about trust, security, privacy, plagiarism, and environmental impact. It describes limitations that can make outputs difficult to verify and systems vulnerable to misuse.

3.1 Transparency and Trustworthiness

  • Opaque data sources and absent uncertainty estimates [51] impede verification, while manipulated models may spread misinformation and current detection measures may be insufficient.Model creators may also struggle to interpret predictions, and ChatGPT answers have been banned on forums such as Stack Overflow16. OpenAI is developing output detection and cryptographic markers.Source: §3.1 · p. 31

3.2 Robustness and Security

  • A general parent model can create a single point of failure across derived applications, while data poisoning may inject targeted hateful speech [61].The passage attributes this vulnerability in part to the absence of supervised training.Source: §3.2 · p. 31

3.3 Privacy

  • LLM outputs may expose personally identifiable information from training data, and breaches could affect many people because models rely on vast datasets.Potentially exposed information includes names, contact details, addresses, and medical records.Source: §3.3 · p. 32

3.4 Plagiarism

  • Difficulty identifying expertly written ChatGPT essays raises plagiarism and cheating concerns [58] [13] [50], prompting some institutions to prohibit its use.OpenAI is developing a tool to distinguish AI-written from human-written text.Source: §3.4 · p. 32

3.5 Environmental Impact and Sustainability

  • Over five times the average car’s lifetime emissions are estimated for training a 213-million-parameter neural architecture search model, as LLM growth heightens environmental concerns [57].The passage links the estimate to broader concerns about LLMs’ environmental and climate impact.Source: §3.5 · p. 32

4 Conclusion and Future Work

The study concludes that ChatGPT remains limited in reasoning, mathematics, and bias reduction, with its reliability and trustworthiness in question. It calls for further evaluation and responsible safeguards as the technology’s capabilities and risks remain uncertain.

Conclusion

  • ChatGPT remains susceptible to shortcomings in reasoning, mathematical problem-solving, and bias reduction, and its reliability and trustworthiness are in question.The study notes that it is uncertain whether current technology can address these limitations.Source: §4 · p. 32

Limitations

  • The extent of ChatGPT’s commonsense ability and how to enhance it remain uncertain; based on this research, it appears to answer commonsense-reasoning queries admirably, but further systematic evaluation is required.Source: §4 · p. 33
  • ChatGPT could prove that there are infinitely many primes but could not prove that infinitely many twin primes exist, motivating study of human–LLM collaboration on open problems.It correctly defined twin primes, but the author reports that repeated prompting did not produce a proof.Source: §4 · p. 33
  • ChatGPT may give incorrect answers with too much confidence; answers to the same question can vary (one example ranged from 2 to 4 sons), question order can affect performance, and it may deviate from precise algorithmic steps.Its capacity to comprehend intricate inquiries requiring reasoning, abstract thought, and multiple perspectives is restricted; the passage calls for exploring whether it understands algorithms [52, 65].Source: §4 · pp. 33–34
  • The conclusion urges examination of job displacement [22], bias, manipulation, misinformation, propaganda, identity theft, and the black-box problem, while recommending diverse data and explainable AI.It warns that training-data bias can produce unjust outcomes, especially for marginalized communities, and argues that explainability is needed to understand AI decision-making.Source: §4 · pp. 33–34

Future Work

  • Future work should study specific tasks, including sentiment analysis, named entity recognition, translation [27] [28], summarization [66], and language ambiguity [47], and assess shortcomings in these areas [49].The author also proposes held-out, nonpublic test sets using unlikely-to-have-been-seen questions and a dataset of typical failures for evaluating future models and generating training data.Source: §4 · p. 34
  • The author argues open-source LLMs can aid understanding and improvement, while high computational costs and limited institutional capacity constrain access; integrating search and chatbot answers is also challenging.The passage cites Meta’s release of LLaMA as an example of an open-source effort.Source: §4 · p. 34

Conclusion

  • The author calls for public language models to be monitored, transparently communicated, and regularly checked for bias, while whether they can reach or surpass human-level intelligence remains to be seen.The conclusion urges adequate societal safeguards and responsible use while acknowledging how well current technology works.Source: §4 · p. 34

A Math

ChatGPT gives −2 for the expression, while the supplied true answer is −1. Its response invokes a binomial-expansion approach and concludes x = −1 before substituting into the expression.

A Math

  • ChatGPT proposes using the binomial theorem to evaluate the expression.The supplied answer instead derives x^3 = 1 and reduces the expression to x + 1/x.Source: A Math · p. 39
  • ChatGPT assumes x is negative, writes x = −y, and concludes x = −1.These steps precede its substitution of x = −1 into the expression.Source: A Math · pp. 39–40
  • ChatGPT reports −2, but the supplied true answer is −1.The response first concludes x = −1, then substitutes it into the expression to obtain −2.Source: A Math · pp. 39–40

B Coding

This subsection presents another ChatGPT solution to the coding problem as a code fragment for character priorities and basket contents.

B Coding

  • The calculate_priority fragment checks for lowercase letters, returns ord(char) - ord('A') + 27, and otherwise raises a ValueError.Source: B Coding · p. 41
  • The fragment supplies five string baskets, initializes common_items from the first basket, and begins iterating over the remaining baskets.Source: B Coding · p. 41
  • It initializes product to 1 and begins iterating over common_items.Source: B Coding · p. 41

Related papers

Loading 2302.03494v8…