In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT

• ChatGPT’s fluent answers may foster trust among users who cannot readily identify errors, and fabricated legal cases have led to sanctions and reputational risks.The introduction also notes potential consequences for legal proceedings and misinformation.Source: §1 · p. 1
• Prior evaluations used limited samples or narrow question types, while system-role effects and the feasibility of adversarial attacks remained insufficiently investigated.Source: §1 · p. 1
• The paper measures reliability in generic QA across three perspectives: baseline performance, system-role effects, and responses to adversarial examples.Source: §1 · p. 1

• Across 5,695 questions, reliability varies by domain, with underperformance in law and science; unanswerable-question detection is limited on both GPT-3.5 and GPT-4.The law-task correctness on extractive and abstractive questions is respectively 11.97% and 4.29% below overall averages; unanswerable identification is 26.63% for GPT-3.5 and 14.29% for GPT-4.Source: §2 · p. 2
• System roles can alter reliability in ways not evident from their descriptions: benign roles improve correctness on four QA tasks, while bad and jailbreak roles generally reduce it.Source: §2 · p. 2
• ChatGPT is vulnerable to sentence- and character-level adversarial examples, and a manually designed leakage prompt enables score-based attacks with an average attack-success-rate improvement of 0.38.The qualitative analysis found cases where changing one character altered the output.Source: §2 · p. 2

2.1 ChatGPT

• ChatGPT generates responses from learned knowledge, and users can specify task style through system roles, including assistant-style or jailbreak instructions.Source: §2.1 · p. 2
• System roles are increasingly used in applications, but their effects on ChatGPT had not been systematically investigated.Source: §2.1 · p. 3

2.2 Question-Answering Task

• QA includes yes/no, multiple-choice, extractive, and abstractive tasks, which elicit distinct capabilities and use task-specific metrics.Yes/no expects a binary response; multiple-choice selects an option; extractive draws from context; abstractive requires a free-form answer.Source: §2.2 · p. 3

3.1 Evaluation Dataset

• Ten benchmark datasets span four QA task types and capabilities including reading comprehension, reasoning, commonsense, document comprehension, and truthfulness.Source: §3.1 · p. 3
• The study samples up to 1,000 records per dataset, retaining complete smaller datasets and taking 1,000 records from each RACE subset.Sampling makes human thematic analysis more practical, helps address imbalance, and limits the burden of slow API responses.Source: §3.1 · p. 3
• BERTopic and deductive coding produce eight domains and 5,695 questions, including 410 unanswerable questions; the final coding has kappa 0.74.Questions with representative scores below 0.5 are filtered out.Source: §3.1 · pp. 3–4
• Law, medicine, and technology may be underrepresented; technology has the fewest questions, at 165.The authors attribute this possibility partly to keeping those domains separate rather than merging them.Source: §3.1 · p. 4

3.2 Evaluation Pipeline

• The pipeline has four steps: query formation, ChatGPT invocation, answer extraction, and evaluation.Figure 1 depicts the workflow.Source: §3.2 · p. 4, Figure 1 · p. 4
• Queries pair a system message with a user message containing the question and task-specific context or options; the system message is blank for baseline RQ1 runs.The study does not use advanced techniques such as in-context learning.Source: §3.2 · p. 4
• Answers are extracted as selected options, substrings, or complete responses by task; annotators handle responses that do not follow the expected format.Source: §3.2 · p. 5
• Reliability is evaluated using task-specific accuracy, F1, or RougeL metrics and the rate of identifying unanswerable questions; experiments are repeated five times and averaged.Source: §3.2 · p. 5

4.1 Correctness

• Recreation exceeded overall averages by YN +3.21%, MC +0.15%, EX +18.31%, AB +4.09%; law scored YN -3.18%, MC -1.62%, EX -11.97%, AB -4.29%.Correctness varied across question domains.Source: Figure 3 · p. 5, §4.1 · p. 5
• GPT-4 scored 79.03% correctness on CQA, 2.43% above GPT-3.5, but the authors judge it still far from reliable.The models showed similar behaviors, so the authors conclude model upgrades alone are insufficient to resolve reliability issues.Source: Figure 4 · p. 5, §4.1 · p. 5
• Present-tense questions usually had slightly better correctness; ELI5 present-tense questions scored mean RougeL 0.21 versus 0.18 for past-tense questions.The authors leave the possible training-set-bias explanation open because they lack access to ChatGPT’s training set.Source: §4.1 · p. 5, Figure 5 · p. 6

4.3 Qualitative Analysis

• Among answerable questions, the most common refusal reason was “not mentioned”; ChatGPT sometimes blamed ambiguity rather than acknowledging its limitations.The authors suggest this may influence users’ judgments of ChatGPT’s reliability.Source: Figure 6 · p. 6, §4.3 · p. 6
• Hallucinatory facts were the most common failure, appearing in 6.8% of answers on average; casual answers, referential confusion, and lack of knowledge followed.The qualitative analysis sampled 100 records and inspected five ChatGPT answers per question to reduce randomness.Source: §4.1 · p. 5, §4.3 · p. 6

4.4 Takeaways

• Because refusal analysis covers answerable questions, “not mentioned” being more common than “no knowledge” may make ChatGPT appear dishonest about its limits.The authors say this may influence users’ judgments of ChatGPT’s capability.Source: §4.4 · p. 7

5.1 Correctness

• Benign roles improve correctness on four QA tasks, but effects vary: Expert-R alone improves SQuAD2, Bad often raises correctness, and Bad-M and jailbreak roles generally lower it.On SQuAD2, Expert-R improves correctness by 5.82%; on CQA, Bad-M lowers it from 76.60% to 37.10%.Source: §5.1 · p. 7, Table 4 · p. 8

5.2 Unanswerable Question Identification

• Almost all roles reduce unanswerable-question detection; DAN identifies 9.07%, while Expert-R reaches 29.36%, still far from satisfactory.The authors attribute the declines to jailbreak roles encouraging answers and the Expert-R gain to its refusal instruction.Source: §5.2 · p. 7, Figure 7 · p. 8

5.3 Qualitative Analysis

• Benign roles reduce hallucinations and casual answers but leave reasoning and knowledge failures; bad and jailbreak roles tend to increase all four failure types.DAN produces 4% more hallucinatory facts, and system-role answers can add convincing fake data or irrelevant theory that is harder to check.Source: §5.3 · pp. 7–8
• Expert-CoT rejects fewer answerable questions than either jailbreak role: 12.2 versus 15 for DAN and 13.4 for ChatAGI, from 28.6 without a role.The authors describe the in-the-wild jailbreak roles as less effective at reducing rejections than the simple Expert-CoT instruction.Source: §5.3 · p. 8

5.4 Takeaways

• Role effects on reliability may be hard to discern from the role itself, so the authors call for thorough evaluation before real-world use.A jailbreak role intended to break restrictions can instead decrease correctness.Source: §5.4 · p. 8

6.1 Threat Model

• The assumed adversary seeks incorrect answers through semantic-preserving, fluent perturbations with few queries and can query ChatGPT but cannot access its weights or internal settings.The threat model follows prior adversarial-attack work [27] [37] [39] [48].Source: §6.1 · p. 9

6.2 Methodology

• The study tests five decision-based attacks spanning character, word, and sentence changes, and uses a leakage prompt to obtain scores for TextFooler word substitutions.TextFooler uses those scores to identify important words, then replaces them with semantically similar, grammatical alternatives until ChatGPT’s response changes.Source: §6.2 · p. 9

6.3 Experiment Settings

• Evaluation uses 65 correctly answered yes/no and multiple-choice questions with ChatGPT under Expert-CoT, measuring attack success, perturbation quality, and query efficiency.Decision-based attacks are capped at 10 queries per question.Source: §6.3 · pp. 9–10

6.4 Quantitative Evaluation

• Sentence-level attacks reach 0.65 ASR on CQA, the highest among the three perturbation levels; score-based attacks raise average ASR to 0.38.Synonyms is an exception on BoolQ, where its ASR is considerably lower than other attacks.Source: §6.4 · p. 10, Table 5 · p. 11
• Nat and VIPER usually achieve higher ASR than Swap; Synonyms averages 0.004 ASR, but score-based TextFooler raises the average to 0.38.The authors interpret the character-level pattern as weaker handling of natural noise and visual perturbations than artificial noise.Source: §6.4 · p. 10
• Word-level examples retain meaning best, with CQA semantic similarity of 0.93 for Synonyms and 0.76 for TextFooler; VIPER has the lowest reported BoolQ perplexity, 304.81.The paper links word-level similarity to synonym replacement and VIPER’s fluency to visual perturbations.Source: §6.4 · p. 10, Table 5 · p. 11
• Score-based attacks require significantly more queries than decision-based attacks, while decision-based methods use similar query counts; high-success attacks usually need several interactions.On ARC, query averages are 8.00 for VIPER, 9.14 for Swap, 8.14 for Nat, 1.00 for Synonyms, and 2.71 for SCPN.Source: §6.4 · p. 10, Table 5 · p. 11

6.5 Qualitative Analysis

• Perturbations can redirect reasoning, and in one example changing “assists” to “assist” shifts ChatGPT from the correct answer to an incorrect one.The authors report that some perturbations mislead ChatGPT even when it appears to recover the original typo-filled question.Source: §6.5 · p. 10, Table 6 · p. 11

7.1 Social Implications

• The study reports hallucinatory facts in 6.8% of answers and connects unreliable outputs to risks including legal consequences, reputational damage, and misinformation spread.It argues the volume of false information could be substantial given ChatGPT’s reported usage of one billion queries per day.Source: §7.1 · pp. 11–12

7.2 Actionable Guidance

• Proposed reliability measures span training-data quality, in-context learning, and output post-processing, including detectors that flag potentially misleading responses and direct users to verification resources.The discussion notes trade-offs: large training datasets are costly, demonstrations require user effort and cannot fully address model misknowledge, and a warning alone leaves verification to users.Source: §7.2 · p. 12

7.3 Limitations

• The evaluation covers English questions only, and system-role analysis uses manually crafted roles, limiting how directly findings extend to other languages or user-created roles.The authors note that reliability may vary across languages and that versatile user-created roles may have different impacts.Source: §7.3 · p. 12

8 Related Work

• Prior work examines task performance, logical consistency, robustness, and security risks [12] [38] [69] [27] [37] [39], while this study focuses on generic-QA reliability across domains, roles, and attacks.The paper also situates LLM risks in backdoor attacks [11] [21], prompt injection [57], obfuscation [40], and misuse such as phishing.Source: §8 · pp. 12–13

9 Conclusion

• The study presents a large-scale measurement of ChatGPT reliability across generic QA, system roles, and adversarial examples.It frames these as three perspectives on reliability.Source: §9 · p. 13
• ChatGPT reliability varies by domain, with noticeable underperformance on law and science questions.Source: §9 · p. 13
• System roles steer ChatGPT’s behavior and can affect reliability in ways not evident from their descriptions.Source: §9 · p. 13
• Sentence-level and character-level adversarial examples can be mounted effectively against ChatGPT.Source: §9 · p. 13

A.1 Detailed Explanation of Datasets

• BoolQ [24] tests yes/no reading comprehension; TruthfulQA [50] probes false-belief answers; OpenbookQA [51] requires reasoning from science facts.Source: §A.1 · pp. 16–17
• RACE [46] tests passage comprehension; ARC [25] science reasoning; CommonsenseQA [67] requires commonsense knowledge non-trivial for models.Source: §A.1 · p. 16
• SQuAD1 [60] requires context extraction, SQuAD2 [59] adds unanswerable questions, and NarrativeQA [42] and ELI5 [28] use free-form answers.Source: §A.1 · p. 16

A.2 Evaluation of Leakage Prompt

• The leakage prompt’s confidence scores give the correct option the highest score, supporting the prompt’s usability; option (E) is absent because no sample question has it as the answer.Source: §A.2 · p. 17, Figure 8 · p. 17

A.3 Impacts of Question Fluency

• Question fluency positively correlates with answer fluency in almost all datasets, while higher question perplexity accompanies greater reliability variance, with direction differing across datasets.The reported Pearson correlation is 0.1 (p < 0.1), except in BoolQ and TruthfulQA; the figures visualize fluency using perplexity, where higher PPL means lower fluency.Source: §A.3 · p. 17, Figure 9 · p. 17, Figure 10 · p. 18