Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

1 Introduction

• The evaluation targets challenges including human-evaluation demands, risks from widely deployed LLMs, and uncertainty about their massive training data.The authors describe their workflow as a systematic study of robustness, consistency, and credibility in areas they say had rarely been addressed.Source: §1 · pp. 1–2
• ChatGPT could answer correctly on some datasets even when passage information was completely polluted, raising concerns about using such data to evaluate LLM-involved systems.The authors characterize this as a possible sign of memorization.Source: §1 · p. 2
• The study uses gpt-3.5-turbo, LLaMA, and OPT, with over one million queries conducted for each model.The authors report around 0.7 billion tokens on the ChatGPT API.Source: §1 · p. 2
• The workflow converts 12 public datasets into a multiple-choice QA primitive and regularizes model outputs for parallel processing at scale.The primitive is a quintuple: prompt, passage, question, confusing options, and answer.Source: §1 · p. 3
• RTI estimates the relative probability that a dataset has been memorized and is intended to help judge its suitability for LLM-involved evaluation.The authors present it as a reference index for dataset credibility.Source: §1 · p. 3
• No less than 27%: minor but structured input changes shifted ChatGPT’s output unexpectedly in the authors’ test.The authors warn developers to be careful with input contents.Source: §1 · p. 3
• 3.2%: ChatGPT’s response accuracy fluctuated on average across semantically identical queries in the authors’ testing protocol.The input changes were mostly grammatical-expression and writing-habit differences, which the authors say occur commonly in widespread deployment.Source: §1 · p. 3

2.1 Adversarial Attack in NLP

• Traditional NLP attacks include character-, word-, and sentence-level black-box methods, while white-box approaches require model gradients, structure, or parameters; ChatGPT’s interface restricts this evaluation to black-box attacks.The cited works include character-level attacks [8] [9] [10] [11], word-level attacks [12] [13] [14], sentence-level attacks [15] [16], and white-box methods [17] [18] [19] [20] [21] [22].Source: §2.1 · p. 3

2.2 Adversarial Attacks on LLMs

• Prior LLM robustness schemes use perturbations such as typos, entity swaps, negations, and sentence insertions, but most approaches target smaller models such as BERT [27] and XLNET [28].The authors argue that ChatGPT-scale LLMs need study; a recent work [29] reports ChatGPT’s advantage on most adversarial classification and translation tasks.Source: §2.2 · pp. 3–4

2.3 Threat Model

• Existing threat-model work often focuses on smaller machine-learning models, prompting a need for models more closely integrated with ChatGPT’s application scenarios.The cited threat-model studies include black-box systems and APIs.Source: §2.3 · p. 4

2.4 LLMs’ Evaluation: Reliable?

• RTI estimates a relative rate of testing samples already memorized by an LLM; it cannot provide an absolute score without information about the model’s training data.The authors warn that isolated evaluation of external modules may be questionable for some datasets because ChatGPT can answer without other information.Source: §2.4 · p. 4

3 Method, Data, and Desiderata

• The workflow evaluates robustness, consistency, and credibility using a common data form and automated assessment of model responses.It is designed to handle more than a million queries without requiring human scrutiny of each response.Source: §3 · p. 4, §3.2 · p. 4

3.1 Raw Dataset Curation

• The benchmark uses 12 public datasets, mainly covering mathematical deduction, logical reasoning, and commonsense understanding.Some datasets lack a preset passage; their passage field may be null.Source: §3.1 · p. 4

3.2 Data Assembly

• Each sample is converted into a five-part multiple-choice QA primitive: prompt, passage, question, confusing options, and answer.The options are designed to be incorrect but similar to the ground-truth answer; for text-ed answers, the workflow may alter words or formulas and add a “None of the other options is correct” choice.Source: §3.2 · pp. 5–6
• Confusion options are assembled according to answer type, with existing wrong options reused where available and alternatives generated for other types.True/false samples may receive an “unable to determine” option; numerical alternatives are randomized, while word alternatives share a linguistic property.Source: §3.2 · pp. 5–6
• A fixed prompt and in-context learning present each sample as linked passage and question-option queries, enabling responses to be checked by their selected option.Multiple questions tied to one passage are sent as follow-up queries.Source: §3.2 · p. 6

3.3 Gauge the Robustness

• The setup targets realistic user-generated mistakes—such as typos, incorrect transcribed words, and visual flaws—rather than minimizing perturbation distance.The authors frame robustness as responses to malicious queries made intentionally or unintentionally.Source: §3.3 · p. 7
• For samples with a passage, the auto-attacker perturbs that passage and evaluates accuracy changes against the original input.Each passage word has attack probability ρ, and robustness is gauged by accuracy after perturbation.Source: §3.3 · p. 8
• Word-level attacks randomly insert, delete, or replace words; inserted or replacement words may come from the passage or WordNet synonyms and random words.These attacks simulate missing or misidentified words, including errors from speech recognition.Source: §3.3 · p. 8
• Character-level attacks repeat, insert, or delete selected characters, while visual-level attacks replace selected English letters with visually similar characters.The authors associate these attacks with natural typing or spelling mistakes and OCR-related visual flaws, respectively.Source: §3.3 · p. 8

3.4 Gauge the Consistency

• Consistency is tested by replacing the prompt with five synonymous statements and comparing ChatGPT outputs across the different phrasings.The prompts are intended to convey the same meaning in different forms.Source: §3.4 · p. 9
• The options attack randomizes the order of options without changing the data primitive’s semantics.It simulates parallel grammatical structures whose order does not affect reading.Source: §3.4 · p. 9

3.5 Gauge the Credibility: Relative Training Index.

• RTI is the expected minimum word-attack probability that changes the model’s original answer, measured across dataset samples.The procedure increases ρ from 0 to 1 in steps of 0.1; higher RTI indicates lower reliability of the developed method, and vice versa.Source: §3.5 · pp. 9–10
• ChatGPT can answer correctly even when a passage is 100% polluted, raising concern that evaluation on such data may be misleading.The authors say this may reflect information or answers encoded in the model’s parametric memorization, while acknowledging that the study does not fully explain or resolve the reliability problem.Source: §3.5 · p. 10
• The authors suggest that a lower RTI should encourage researchers to move away from the dataset under consideration.They present this as a tentative recommendation.Source: §3.5 · p. 10

3.6.1 Metrics on Robustness and Consistency.

• ER measures accuracy on the data primitive, while ACR measures how often the model retains its original output after attacks.The section describes ER as conventional accuracy evaluation and ACR as the proportion of the test set whose output remains unchanged.Source: §3.6.1 · p. 10

3.7 Qualitative Pattern Analysis of Adversarial Sample

• The qualitative analysis uses part-of-speech tagging, dependency parsing, phrase-structure discovery, and positional analysis at word granularity.It examines clean and polluted inputs to identify which prompt components may be associated with output drift.Source: §3.7 · p. 11
• For position and structure categories, the method counts perturbed-word categories in samples whose outputs drift, using frequency sl.The categories include positions such as head, tail, or middle and structure tags; sgn indicates whether the output changed, while G balances attack times.Source: §3.7 · p. 11
• For part-of-speech and dependency categories, the method counts normalized category frequencies and trains a random forest to score their importance for attack success.The RF uses category vectors and an output-drift indicator as input features and target, respectively.Source: §3.7 · p. 11

4.1.1 Main Results

• All tested attacks increased ChatGPT’s ER by at least 2% and changed at least 27% of its output options.The reported pattern holds regardless of attack type.Source: §4.1.1 · p. 11
• Word-level attacks had roughly 10% higher average ER than character-level attacks; word deletion reached 61.8% ER and 49.8% ACR, while character deletion exceeded insertion by 10% ER and 11% ACR.The authors suggest word-level changes may affect tokenizer and word-embedding representations more strongly, altering semantic understanding.Source: §4.1.1 · pp. 11–13
• As visual transformations increased, ER and ACR increased, yet ChatGPT retained around 40% accuracy.The authors suggest visually similar characters in the GPT Tokenizer may contribute to retained accuracy.Source: §4.1.1 · p. 13
• Compared with LLaMA and OPT, ChatGPT performed better on ER but worse on ACR in tests using the best- and worst-performing ChatGPT datasets.The authors suggest the other models’ weaker baseline abilities may explain the ER comparison, while some models’ output randomness may contribute to the ACR comparison.Source: §4.1.1 · p. 13

4.1.2 Attack Pattern Analysis

• The most attack-sensitive POS tags were NOUN, ADP, VERB, DET, and AUX; the leading dependency tags were ROOT, nsubj, pobj, prep, and det.The authors suggest these categories may matter because they contribute to sentence meaning and structure.Source: §4.1.2 · p. 13, Figure 2 · p. 14
• NP, VP, PP, WHNP, and ADJP were the most impactful parser tags, and attacks on middle-position words had greater impact than attacks at either end.The authors report that middle-position impact was distributed relatively evenly, with possible effects from transitions and differing attention weights.Source: §4.1.2 · p. 14, Figure 3 · p. 15

4.2 Consistency Results

• Prompt variation produced average accuracy standard deviations of 11.9% for LLaMA and 1.9% for ChatGPT; ChatGPT’s option-order standard deviation was 1.13%.LLaMA showed only 0.98% accuracy variance yet tended to choose the first option regardless of its content; the authors infer that pretraining alone may not confer multiple-option-test ability.Source: §4.2 · pp. 14–15, Figure 4 · p. 16

4.3 Credibility Results

• RTI varied across datasets, was lower for more complex samples, and was higher on training and validation sets than on test sets.The authors report RTI of 0.175 for RAQuA and up to 0.45 for recognized StrategyQA training data, and describe RTI as a quantitative reference for dataset reliability in LLM evaluation.Source: §4.3 · pp. 15–16

5.1 LLM Application Security

• Ordinary typos and spelling mistakes can cause answer fluctuations despite relatively high robustness to character-level attacks, potentially increasing users’ checking costs.Source: §5.1 · p. 16
• Multimodal integration adds attack avenues: OCR may misread visual characters, while speech recognition can produce contextual word-level errors that ChatGPT resists poorly.The discussion notes that such errors could have serious consequences in finance, medicine, or automated driving.Source: §5.1 · p. 17
• The authors urge safeguards against potential harms from adversarial attacks when LLMs are integrated into other systems.Source: §5.1 · p. 17

5.2 LLM Evaluation Reliability

• ChatGPT answering after paragraph deletion makes some open-dataset evaluations unconvincing; RTI offers a relative training-probability reference, with statistical baselines proposed as an absolute indicator.The authors caution that some samples may have been memorized because ChatGPT can answer without the passage.Source: §5.2 · pp. 17–18

5.3 Adversarial Training Paradigm of LLM

• Traditional adversarial-training techniques may not transfer readily because LLM training is multistage, pretraining is costly, and in-context learning replaces fine-tuning; effective LLM methods remain unexplored.Source: §5.3 · p. 18

• Because gpt-3.5-turbo parameters change over time, repeated queries may receive different answers and results may not be fully reproducible.The authors consider local update errors unlikely to be significant given the scale of their inquiries.Source: §6 · p. 18
• The evaluation mainly covers QA and reasoning datasets; evaluating a wider range of tasks increases costs.The authors note that LLMs can also perform basic tasks such as translation.Source: §6 · p. 18
• The multiple-option input template may not suit all models, and alternative prompts may improve performance.Source: §6 · p. 18
• The authors propose future work with more diverse datasets and ablations across models, modules, templates, and prompts.Source: §6 · p. 18

• AQuA, ECQA, QASC, and QED are listed as datasets that cannot be split into passage and question parts.Source: A Datasets Description · p. 23
• The dataset examples contrast raw data with target forms divided into component parts; subsequent datasets share a prompt-and-query structure that is omitted for space.Source: A Datasets Description · p. 23
• The examples include AQuA wrong options, Creak true/false answers, and NoahQA answers with multiple types, including numbered answers.Source: Table 13 · p. 23, Table 14 · p. 23, Table 15 · p. 24
• One response example selects option D because adding +45 and -30 yields 15.Source: A Datasets Description · p. 23
• GSM8k is shown with text-ed answers, while bAbi uses word-ed answers and showcases the bAbi16 inductive-reasoning task.Source: Table 16 · p. 25, Table 17 · p. 26

• Table 18 lists the attack methods used to construct adversarial examples and their attack-ratio settings.Source: B Attack Ratio Settings · p. 26, Table 18 · p. 26
• For all attack methods, the probability that a word in a sentence is attacked follows a uniform distribution U(0, 1).Source: Table 18 · p. 26
• Character- and visual-level parameters specify the count or percentage of affected characters in the targeted word.Source: Table 18 · p. 26

C Detailed Experiment results

• Table 19 compares adversarial attacks at three levels using error rate (%).The section identifies it as a detailed listing of results from Table 6.Source: C Detailed Experiment results · p. 26, Table 19 · p. 27
• Table 20 compares adversarial attacks at three levels using answer changed rate (%).The section identifies it as a detailed listing of results from Table 10.Source: C Detailed Experiment results · p. 26, Table 20 · p. 27
• Figures 5 and 6 analyze adversarial-sample patterns across all datasets and three attack levels, for dependency relations and part-of-speed, respectively.The prose identifies the figures as analyses of samples subjected to various attack methods.Source: C Detailed Experiment results · p. 27, Figure 5 · p. 28, Figure 6 · p. 29