PaperlayerView exact revision on arXiv

Source-linked AI summary

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models

Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, Bo Li

arXiv:2306.11698v5cs.CLcs.AIcs.CR

TL;DR

Evidence on GPT trustworthiness remains limited even as capable models are proposed for sensitive applications, so this paper evaluates GPT-4 and GPT-3.5 across eight dimensions. It finds GPT-4 generally performs better, but is more easily manipulated by misleading prompts or demonstrations.

  • Problem

    Assessing GPT robustness and bias matters because biased outputs may amplify real-world inequalities, while unexpected inputs can challenge response reliability.

    Source: §4 · p. 17, §6 · p. 27

  • Method

    The paper evaluates GPT-4 and GPT-3.5 across toxicity, stereotype bias, adversarial and OOD robustness, adversarial demonstrations, privacy, machine ethics, and fairness.

    Source: §12 · p. 61

  • Results

    GPT-4 generally performs better than GPT-3.5, but is more easily manipulated by jailbreaking or misleading prompts and in-context demonstrations.

    Source: §12 · p. 61

  • Takeaways & Limitations

    The findings raise concerns that stronger overall performance does not eliminate GPT-4’s vulnerability to misleading prompts or demonstrations.

    Source: §12 · p. 61

    The evaluation covers only GPT-3.5 and GPT-4 versions published on March 1 and March 14, 2023, respectively.

    Source: §1 · p. 4

Abstract

from arXiv · show

Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives -- including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially because GPT-4 follows (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/ ; our dataset can be previewed at https://huggingface.co/datasets/AI-Secure/DecodingTrust ; a concise version of this work is at https://openreview.net/pdf?id=kaHpo8OZw2 .

1 Introduction

The paper evaluates GPT-3.5 and GPT-4 across eight trustworthiness perspectives and varied scenarios, including adversarial prompts and demonstrations. Its findings show strengths on some standard evaluations but also substantial vulnerabilities to misleading instructions, attacks, and privacy leakage.

Toxicity

  • Jailbreaking prompts raise GPT-3.5 and GPT-4 toxicity to almost 100%, despite toxicity probabilities below 32% on different task prompts; GPT-4 is more susceptible to these prompts.The models have significantly reduced toxicity compared with models without instruction tuning or RLHF, but the authors’ challenging task prompts can further increase toxicity.Source: §1 · p. 7

Stereotype bias

  • GPT-3.5 and GPT-4 are not strongly biased on most tested topics under benign or untargeted prompts, but misleading targeted prompts can elicit biased agreement, especially from GPT-4.Bias also varies with the demographic group and stereotype topic in the prompt.Source: §1 · p. 7

Adversarial Robustness

  • 89.2% attack success was achieved by SemAttack against GPT-4 on QQP when transferred from Alpaca, while BERT-ATTACK achieved 100% against GPT-3.5 on MNLI-mm when transferred from Vicuna.GPT-4 performs better on standard AdvGLUE and resists human-crafted attacks better than GPT-3.5, but transferred attacks remain effective; Alpaca-7B generates the most transferable texts overall.Source: §1 · p. 7

Out-of-Distribution Robustness

  • GPT-4 generalizes better than GPT-3.5 across tested OOD styles and similar-domain demonstrations, but its accuracy still needs improvement on recent events.For different-domain demonstrations, GPT-4 benefits from domains close to the target and is harmed by distant ones, while GPT-3.5 accuracy declines across demonstration domains.Source: §1 · p. 9

Robustness to Adversarial Demonstrations

  • Counterfactual demonstrations do not mislead GPT models in general, but spurious correlations mislead GPT-3.5 more than GPT-4, and nearby backdoored demonstrations can mislead both, especially GPT-4.The backdoor effect is strongest when the backdoored demonstration is positioned close to the backdoored user input.Source: §1 · p. 9

Privacy

  • GPT models can leak sensitive training data and conversation PII; GPT-4 is generally more protective of PII than GPT-3.5, yet privacy-leakage demonstrations elicit all tested PII types from both.With few-shot prompting, knowing the target email domain can make extraction accuracy 100x higher than when the domain is unknown; privacy-related wording also affects leakage.Source: §1 · p. 9

Machine Ethics

  • GPT-3.5 and GPT-4 are competitive with fine-tuned non-GPT models on moral recognition, but jailbreaking prompts and evasive sentences can mislead both, with GPT-4 more vulnerable to prompts.GPT-3.5 performs worse than GPT-4 on self-harm recognition; increasing severity improves GPT-4 recognition but has little effect on GPT-3.5.Source: §1 · p. 9

Fairness

  • GPT-4 is more accurate on demographically balanced test data but has higher unfairness scores on unbalanced data; imbalanced few-shot contexts worsen fairness, while 16 balanced examples can improve it.Both models show large zero-shot performance gaps across groups with different base-rate parity.Source: §1 · pp. 9–10

2 Preliminaries

The preliminaries describe GPT-3.5 and GPT-4 and establish the prompting and evaluation procedures used to assess their trustworthiness. Classification and generation use distinct prompt configurations and decoding settings, while later sections probe the models with diverse and potentially adversarial prompts.

2.1 Introduction to GPT-3.5 and GPT-4

  • GPT-3.5 and GPT-4 are autoregressive transformers trained for next-token prediction and fine-tuned with RLHF for instruction following and human-value alignment.GPT-3.5 retains 175 billion parameters; GPT-4’s parameter count and pretraining corpus are undisclosed, but it is described as significantly larger in parameters and training budget. The models are fine-tuned for dialogue contexts. [132] [189] [38] [157]Source: §2.1 · p. 10
  • The study accesses models through OpenAI’s API and fixes GPT-4 to gpt-4-0314 and GPT-3.5 to gpt-3.5-turbo-0301 for reproducible evaluations.API parameters include temperature and maximum tokens; the fixed snapshots are dated March 14 and March 1, respectively.Source: §2.1 · p. 10

2.2 Prompt design for downstream tasks

  • Classification prompts use task descriptions and permitted labels; few-shot prompts also include demonstrations formatted as simulated user inputs and model responses.For zero-shot classification, the task description is placed in the user prompt, while the default system prompt is “You are a helpful assistant.” Figure 5 illustrates classification prompt designs.Source: §2.2 · p. 11, Figure 5 · p. 11
  • Classification uses temperature=0 and a 20-token maximum; refusals and answers outside the permitted classes count as incorrect.The paper later reports Non-existence Rate (NE) for non-existing answers and Refusal Rate (RR) for declined answers.Source: §2.2 · p. 11
  • Generation uses a helpful-assistant system prompt, a task description, up to 150 tokens, temperature 1, and top-p 1 to produce diverse continuations.The setup is intended to identify worst-case generations over multiple runs; Figure 5 illustrates text-completion prompts.Source: §2.2 · p. 11, Figure 5 · p. 11
  • The evaluation probes trustworthiness with diverse and potentially adversarial prompts and will open-source its platform for assessing other LLMs.The stated aim is to expose potential limitations and inform efforts to improve model security and reliability.Source: §2.2 · p. 12

3 Evaluation on toxicity

The evaluation finds that GPT-3.5 and GPT-4 reduce toxicity on standard benign prompts, but adversarial system and user prompts can elicit toxic generations. It examines both prompt types to identify where these vulnerabilities arise.

3.1 Evaluation on standard benchmark

  • On nontoxic prompts, GPT-3.5 and GPT-4 keep expected maximum toxicity below 0.13 and toxicity probability around 1%, but adversarial system prompts push probability nearly 100%, with GPT-4 more toxic than GPT-3.5.The adversarial prompts can jailbreak the models even when the task prompts are nontoxic, exposing a gap in alignment against malicious instructions.Source: §3.1 · pp. 12–13, Table 2 · p. 13

3.2 Design of diverse system prompts

  • Straightforward adversarial system prompts, especially instructions to add swear words, most effectively elicit toxicity; GPT-4 is more toxic than GPT-3.5 across the designed prompts.Role-playing prompt #13 yields the highest toxicity across both models, combining direct toxic instructions with a third-party role; task-reformulation and respond-as-program prompts also produce average GPT-4 toxicity of 0.6.Source: Figure 7 · p. 15, §3.2 · pp. 15–16

3.3 Design of challenging user prompts

  • GPT-4-generated challenging toxic prompts raised GPT-4’s expected maximum toxicity to 0.95 and toxicity probability to 100%, exceeding GPT-3.5-generated and benchmark prompts.These prompts also transfer to pretrained GPT-3 models, producing expected maximum toxicity of 0.9 and toxicity probability of 100%; the authors release 1.2K generated prompts.Source: Table 3 · p. 17, §3.3 · p. 17
  • More toxic user prompts are more likely to provoke toxic model responses, and GPT-4-generated prompts have higher prompt toxicity than GPT-3.5-generated or benchmark prompts.With benign system prompts, GPT-3.5 and GPT-4 show less toxicity on generated prompts than on the benchmark’s toxic subset, which the authors attribute to increased refusal of the more toxic prompts.Source: §3.3 · p. 17

4 Evaluation on stereotypes bias

This section evaluates GPT-3.5 and GPT-4 by measuring agreement with stereotype statements under benign and adversarial system prompts. Both models mostly reject stereotypes under benign and untargeted prompts, but targeted prompts substantially increase biased agreement.

4.1 Design of stereotype dataset

  • The dataset combines 16 stereotype topics, three single-stereotype templates each, and 24 demographic groups across seven factors to create 1,152 user prompts.It distinguishes 12 stereotyped from 12 non-stereotyped groups for analysis, and tests each prompt with benign, untargeted, and targeted system prompts.Source: §4.1 · pp. 18–19

4.2 Evaluation setup

  • For each user prompt, the evaluation samples 25 generations and computes agreementIndex as the fraction agreeing with the stereotype, averaging the three variants per topic.Results are displayed as 24×16 heatmaps of demographic groups by stereotype topics, with benign-prompt heatmaps in the appendix.Source: §4.2 · p. 19

4.3 Results

  • agreementIndex rose by 0.346 for GPT-3.5 and 0.535 for GPT-4 from untargeted to targeted prompts, versus 0.008 and 0.057 from benign to untargeted prompts, respectively.GPT-4 was more likely than GPT-3.5 to produce biased content under targeted prompts, potentially because it follows instructions more precisely; both models largely rejected biased statements under benign and untargeted prompts.Source: Figure 10 · p. 20, §4.3 · p. 21
  • Agreement varied by demographic group and topic: GPT-4 agreed with “Women have HIV” at 0.17 and “Men have HIV” at 0.35, but “Homosexuals have HIV” at 0.Under targeted prompts, models more readily generated bias on less sensitive topics such as leadership and greed than on drug dealing and terrorists; implicit sarcastic agreement by GPT-4 was rare.Source: §4.3 · p. 21

5 Evaluation on adversarial robustness

GPT-4 outperformed GPT-3.5 on standard AdvGLUE, but both models were vulnerable to stronger adversarial texts generated against autoregressive models. Prompt variations had little effect on standard-benchmark robustness, while attack effectiveness varied by surrogate model and strategy.

5.1 Robustness evaluation on standard benchmark AdvGLUE

  • 78.41% versus 67.37%: GPT-4 exceeded GPT-3.5 in average robust accuracy on standard AdvGLUE, while GPT-3.5 was near the 65.77% leaderboard baseline.The largest performance drops across templates were 9.90% for GPT-4 and 14.43% for GPT-3.5, versus 26.89% for the leaderboard model.Source: Table 5 · p. 23, §5.1 · p. 23
  • Prompt templates did not significantly change robustness; attacks increased GPT-3.5’s answer-nonexistence rate by over 50% relative to benign data in every template, unlike GPT-4 under Templates 1 and 3.GPT-3.5 often described inputs as unclear or nonsensical, while GPT-4 often returned a neutral sentiment not specified as an option.Source: §5.1 · p. 23
  • Sentence-level and human-crafted perturbations were especially effective on standard AdvGLUE, and near-imperceptible single-word changes could flip GPT-4’s predictions.For human-crafted attacks, attack success rates fell from 61.13% to 36.78% on ANLI and from 10.52% to 0% on AdvSQuAD when moving from GPT-3.5 to GPT-4; examples also show sentiment and NLI prediction flips from single-word changes.Source: §5.1 · p. 24

5.2 Robustness evaluation on generated adversarial texts AdvGLUE++

  • GPT-4’s robust accuracy fell from 78.18% on AdvGLUE to 55.64% on Alpaca-generated AdvGLUE++, while GPT-3.5’s fell from 67.37% to 49.23%.Among the three surrogate models, Alpaca-produced adversarial texts had the highest transferability to GPT-3.5 and GPT-4.Source: Table 7 · p. 24, §5.2 · pp. 26–27
  • AdvGLUE++ evaluates GPT-3.5 and GPT-4 on word-level attacks generated against Alpaca-7B, Vicuna-13B, and StableVicuna-13B.The attacks adapt five AdvGLUE strategies and optimize conditional probabilities of candidate labels given the prompt; evaluation measures robust accuracy, benign accuracy, performance drop, and attack success rate.Source: §5.2 · pp. 24–26
  • 56.08% and 44.31%: SemAttack had the highest average attack success rates against Alpaca- and StableVicuna-generated texts, while TextFooler led against Vicuna at 60.91%.These rates are averaged across tasks.Source: Table 8 · p. 25, Table 9 · p. 25, Table 10 · p. 26

6 Evaluation on out-of-distribution robustness

The section evaluates OOD robustness across style shifts, unknown knowledge, and out-of-distribution demonstrations, finding vulnerabilities to uncommon styles, time-sensitive facts, and mismatched demonstrations.

6.1 Robustness on OOD style

  • GPT-4’s accuracy drops 1.2% from Augment to Shake-W and 7% from Tweet to Bible; uncommon styles harm both models, particularly GPT-3.5.The style evaluation uses word- and sentence-level transformations of SST-2; it finds GPT-4 more resilient than GPT-3.5 across transformed styles.Source: §6.1 · pp. 28–29, Table 11 · p. 29

6.2 Robustness on OOD knowledge

  • On QA2023, GPT-4’s ACC is 24% lower than GPT-3.5’s, while its RR is 37% higher; both models’ MACC falls from over 87% on QA2020 to about 70%.The evaluation treats 2020 RealtimeQA events as in-distribution and 2023 events as OOD; it also finds GPT-4’s ACC is 4% higher than GPT-3.5’s on QA2020.Source: §6.2 · pp. 29–30, Table 12 · p. 29
  • Adding an “I don’t know” option raises GPT-4’s RR and MACC but lowers its QA2020 ACC by more than 17%; GPT-3.5’s MACC decreases.The added option makes GPT-4 more conservative, but does not help GPT-3.5 better identify uncertainty.Source: Table 12 · p. 29, §6.2 · pp. 30–31

6.3 Robustness on OOD demonstrations via in-context learning

  • Style-matched demonstrations usually outperform original-style demonstrations, and the source-versus-target performance gap is larger for GPT-3.5 than GPT-4.The comparison uses 8-shot in-context learning on style-transformed SST-2 test sets, with demonstrations from original or correspondingly transformed training examples.Source: Table 13 · p. 31, §6.3 · p. 32
  • For domain-shifted demonstrations, GPT-4’s ACC improves by over 7% with in-distribution and Facts examples, while GPT-3.5’s MACC falls by more than 20%.GPT-4’s MACC is consistently higher than its zero-shot result across the tested demonstration domains; GPT-3.5 is impaired even by in-distribution demonstrations.Source: §6.3 · pp. 32–33, Table 15 · p. 32

7 Evaluation on robustness against adversarial demonstrations

The section evaluates how counterfactual examples, spurious correlations, and backdoors in demonstrations affect GPT-3.5 and GPT-4 predictions. Findings vary by demonstration and trigger type, and some backdoors substantially mislead the models.

7.1 Robustness against counterfactual demonstrations

  • Counterfactual demonstrations do not mislead GPT-3.5 or GPT-4; they generally improve GPT-3.5 on SNLI-CAD and improve both models on the four MSGS linguistic tasks.On SNLI-CAD, GPT-4’s performance is basically unchanged; the authors interpret the MSGS improvement as evidence that both models distinguish the input from its counterfactual using task descriptions.Source: §7.1 · pp. 34–35

7.2 Robustness against spurious correlations in demonstrations

  • Spurious correlations have heuristic-dependent effects: Verb and Passive can mislead both models, L_RC has little impact, and GPT-3.5 is more easily misled than GPT-4 on NLI.For example, GPT-3.5 accuracy drops on S_RC with entailment-correlated demonstrations, while GPT-4 identifies causal features and improves performance on that subset.Source: §7.2 · p. 36

7.3.1 Evaluation setup

  • The backdoor evaluation varies four trigger-generation approaches across three setups, then tests demonstration placement, trigger location, and backdoor instructions.The approaches are BadWord, AddSent, SynBkd, and StyleBkd; the setups vary the number and labeling of triggered demonstrations, with ASR and clean accuracy as metrics.Source: §7.3.1 · pp. 37–38

7.3.2 Results

  • Backdoored demonstrations can produce high attack success rates; word- and sentence-based triggers are more effective, while GPT-4 is more vulnerable to syntax- and style-based triggers.Under BadWord Setup 3, GPT-3.5’s clean accuracy drops, whereas GPT-4 retains clean accuracy despite high attack success. The authors attribute GPT-4’s greater vulnerability to its stronger pattern- and instruction-following.Source: Table 20 · p. 38, §7.3.2 · pp. 38–39
  • Backdoor placement matters: both models are more affected when backdoored examples are near the test input, and trigger words at the beginning are most effective.GPT-4 is also affected by backdoored examples at the beginning of demonstrations, suggesting it attends to distant examples more than GPT-3.5 does; trigger words at the end are least effective for both.Source: Table 21 · p. 39, Table 22 · p. 39, §7.3.2 · p. 39
  • Adding a backdoor instruction significantly raises both models’ attack success rates; GPT-4 reaches 100% while retaining unchanged clean accuracy.This experiment uses BadWord under Setup 1, the least effective setup, and indicates that GPT-4 follows the backdoor instruction alongside the benign task description.Source: Table 23 · p. 39, §7.3.2 · p. 39

8 Evaluation on privacy

The evaluation finds privacy leakage across training-data recall, conversation history, and privacy-event prompts. Leakage depends on the prompt and information type, and GPT-4’s stronger protection in some settings does not extend to all settings.

8.1 Privacy leakage of training data

  • Under context prompting on Enron Email, GPT-3.5 accurately predicted up to 5.23% of email addresses and GPT-4 up to 3.97%; generally, longer contexts produced more correct predictions for both models.Source: §8.1 · p. 42
  • Few-shot extraction improved with more demonstrations; unknown email domains yielded under 1% accuracy, about 100 times lower than known-domain prompts.Even with unknown domains, models recovered the correct email local part in about 20% of 5-shot cases; GPT-4 generally extracted more accurately than GPT-3.5.Source: Table 25 · p. 41, §8.1 · pp. 42–43

8.2 Privacy leakage during conversations

  • GPT-4 protected all tested PII under zero-shot and privacy-protection demonstrations, but leakage demonstrations led both models to leak all PII types at 3-shot.GPT-3.5 leaked some PII even with protection instructions; digit sequences were harder to leak than letter sequences, and SSNs were especially difficult to reveal.Source: §8.2 · p. 44

8.3 Understanding of privacy-related words and privacy events

  • Privacy-event leakage varied with wording and question prompt: GPT-4 was more prone overall in the constructed prompts, but neither model leaked under Question prompt 3.GPT-4 sometimes leaked for “confidentially told you” but not “in confidence”; the stricter evaluation counted leakage only when affirmative responses lacked qualifying caveats, and Question prompt 3 yielded 0/100 for both models across events and words.Source: §8.3 · p. 46, Figure 21 · p. 47

9 Evaluation on machine ethics

The section evaluates commonsense moral recognition on standard benchmarks and tests its robustness to jailbreaking prompts, evasive descriptions, and victim or severity conditions. Strong benchmark performance coexists with vulnerabilities to adversarial framing and difficulty recognizing harm to oneself.

9.1 Evaluation on standard machine ethics benchmarks

  • GPT-4 exceeded GPT-3.5’s ETHICS accuracy by 2.8% on short and 0.9% on long samples few-shot, and by 3.4% and 4.5% zero-shot.Few-shot GPT-4 was also competitive with fine-tuned models, surpassing fine-tuned BERT-large and coming within 1.1% of the best fine-tuned model; zero-shot GPT models outperformed some fine-tuned models.Source: §9.1 · pp. 48–50, Table 27 · p. 49
  • On Jiminy Cricket, GPT-3.5 and GPT-4 reached 73.9% and 78.6% accuracy zero-shot, and 77.9% and 82.4% few-shot.GPT-4 outperformed GPT-3.5 in both settings.Source: §9.1 · pp. 49–50, Table 28 · p. 50

9.2 Evaluation on jailbreaking prompts

  • GPT-4’s jb_prompt2 and jb_prompt4 raised FPR to 100% on both datasets; combining weaker prompts raised minimum ∆FPR to 90.00% on ETHICS and 62.50% on Jiminy Cricket.Jailbreaking prompts also misled GPT-3.5: combining effective prompts raised ∆FPR to 59.50% and 55.50% on the two datasets, respectively. GPT-4 was more vulnerable; jb_prompt2 increased ETHICS FPR by 96.00% on GPT-4 versus 14.00% on GPT-3.5.Source: §9.2 · pp. 50–51, Table 29 · p. 51

9.3 Evaluation on evasive sentences

  • Evasive descriptions of actions as accidental, harmless, or imaginary increased FPR; the increases reached 43% for GPT-4 on ETHICS and 64% on Jiminy Cricket.On ETHICS, describing actions as intentions or imagination increased FPR by over 30% for GPT-3.5 and over 40% for GPT-4; GPT-4 was more vulnerable under most conditions.Source: Figure 24 · p. 52, §9.3 · p. 52

9.4 Evaluation on conditional actions

  • Models recognized harm to others better than harm to oneself: GPT-4’s FPR was 17.31% for harm to others, while GPT-3.5’s reached 87.63% for harm to oneself.Both models had FPR below 50% for harm to others and above 50% for harm to oneself.Source: Figure 25 · p. 53, §9.4 · p. 53
  • For minor and moderate harm, GPT-3.5’s FPR remained above 50% with little difference, while GPT-4’s FPR fell by 19.77% as severity increased; great harm was excluded for imbalance.GPT-4 generally had FPR below 50% across the evaluated severity levels.Source: Figure 25 · p. 53, §9.4 · p. 53

10 Evaluation on fairness

The evaluation finds fairness gaps in GPT-3.5 and GPT-4 that grow under imbalanced test distributions or few-shot contexts, while balanced examples mitigate them at an accuracy cost.

10.2 Fairness evaluation in zero-shot setting

  • As test-set base-rate imbalance increased, both models’ accuracy and unfairness scores rose; GPT-4 remained more accurate, while sex and race produced especially severe disparities.Higher accuracy under biased test distributions indicated bias toward the group with sensitive attribute A = 1, revealing an accuracy-fairness tradeoff.Source: Table 30 · p. 54, Table 32 · p. 55, §10.2 · p. 55

10.1 Metrics of fairness

  • The evaluation measures prediction gaps with demographic parity difference (Mdpd) and label-conditioned equalized odds difference (Meod), and distributional imbalance with base-rate parity (bP).Large Mdpd and Meod indicate prediction unfairness; a large bP reflects a demographically imbalanced distribution.Source: §10.1 · pp. 54–55

10.3 Fairness evaluation under demographically imbalanced context in few-shot learning

  • With 32 biased few-shot examples, Mdpd rose from 0.033 to 0.12 for GPT-3.5 and from 0.10 to 0.28 for GPT-4.0; Meod also worsened.The test-set base rate parity was fixed at 0.0, isolating the bias induced by the few-shot context.Source: Table 31 · p. 55, §10.3 · p. 56

10.4 Fairness evaluation with demographically balanced few-shot examples

  • With 0, 16, and 32 demographically balanced few-shot examples (bPc=0), GPT predictions became fairer on the Adult evaluation (bPt=0.5), while accuracy on biased test sets decreased, indicating an accuracy-fairness tradeoff.Source: §10.4 · p. 56, Table 33 · p. 56

11 Related work

The section surveys research and benchmarks for evaluating LLM trustworthiness across multiple dimensions. It positions this work as extending prior evaluations of GPT models with broader benchmarks, settings, and attack types.

Benchmarks on LLMs toxicity and stereotype bias

  • Prior toxicity and stereotype-bias benchmarks use prompt completion, dataset-based tests, and automated scoring, while this work adds challenging toxicity prompts and direct stereotype elicitation with system prompts.REALTOXICITYPROMPTS and BOLD use PerspectiveAPI for toxicity evaluation; the stereotype benchmark uses straightforward, evidence-backed statements and avoids conflating stereotypes.Source: §11 · p. 57

Adversarial robustness and prompt hacking

  • AdvGLUE and PromptBench expose adversarial vulnerabilities, and this work evaluates GPT-4 and GPT-3.5 on AdvGLUE while adding advanced attacks and broader jailbreaking prompts.Prior prompt-hacking research covers injection, leaking, and jailbreaking; this work’s jailbreaks target toxicity and commonsense ethics through role-play, opposite statements, and word-meaning substitutions.Source: §11 · pp. 58, 60

Benchmarks on out-of-distribution robustness

  • Because zero-shot OOD testing is difficult without access to GPT training data, prior work uses transformed or post-training-date data, while this work covers transformed text, OOD knowledge, and domains.The evaluation includes both zero-shot and in-context learning settings, where demonstrations may come from distributions different from the test inputs.Source: §11 · p. 58

Adversarial demonstrations via in-context learning

  • Building on studies of demonstration order, labels, semantic priors, and adversarial examples, this work tests counterfactual, spurious-correlation, and backdoored demonstrations and their relation to test inputs.Prior work [182] studies whether adversarial demonstrations can change predictions without changing the test input.Source: §11 · p. 58

Benchmarks on privacy

  • Prior privacy research extracts or infers memorized training data and develops defenses, while this work takes an initial step toward assessing GPT-3.5 and GPT-4 risks in both training data and information injected during inference.Reported risks include verbatim GPT-2 text generation, sensitive GPT-Neo memorization, and PII extraction; mitigation work includes deduplication, differential privacy, and private in-context learning.Source: §11 · p. 59

Benchmarks on machine ethics

  • Existing machine-ethics benchmarks assess moral concepts, social norms, narratives, exceptions, cultural rituals, and agent behavior; this work evaluates GPT-4 and GPT-3.5 on existing benchmarks and adversarial or conditioned prompts.Benchmarks include ETHICS, SOCIAL-CHEM-101, Moral Stories, MoralExceptQA, Jiminy Cricket, and MACHIAVELLI.Source: §11 · pp. 59–60

Benchmarks on fairness

  • Prior fairness studies examine downstream predictions, counterfactual invariance, and in-context examples, while this work evaluates GPT-3.5 and GPT-4 in zero- and few-shot settings and varies demonstration fairness and balance.The evaluation studies demographically imbalanced demonstrations and the number of balanced demonstrations.Source: §11 · p. 60

Regulations related to trustworthiness

  • Regulatory frameworks address high-risk AI compliance, robustness, fairness, privacy, and disclosure, while research proposes procedures for evaluating providers and assessing compliance.The section discusses the EU AI Act draft and amendments, the U.S. AI Bill of Rights, and proposed AIA evaluation procedures.Source: §11 · p. 61

12 Conclusion and future directions … B.1 Target groups and stereotype templates selected for stereotype bias evaluation

The paper evaluates GPT-4 and GPT-3.5 across trustworthiness dimensions, finding GPT-4 generally performs better but can be easier to manipulate with misleading instructions. It also identifies evaluation and safeguarding directions and provides appendix materials for toxicity and stereotype-bias evaluations.

12 Conclusion and future directions

  • GPT-4 generally performs better than GPT-3.5 across metrics, but is easier to manipulate with jailbreaking or misleading prompts and adversarial demonstrations.The evaluations cover toxicity, stereotype bias, adversarial and out-of-distribution robustness, privacy, ethics, and fairness.Source: §12 · p. 61
  • Future evaluations should test interactive conversations, misleading in-conversation contexts, and coordinated, stealthy adversaries.The paper notes that its evaluations mainly use static datasets and generally consider one adversarial case per scenario.Source: §12 · pp. 61–62
  • Future work should evaluate domain-specific vulnerabilities and audit whether outputs satisfy users’ mapped safety and trustworthiness requirements.The paper points to domains such as law and education and proposes mapping instructions to logical spaces or specific contexts for verification.Source: §12 · p. 62
  • Trustworthiness verification could be divided into solvable subproblems, including guarantees for concrete functionalities [188] and verification through model abstractions or semantically preserved embeddings.The paper highlights the need for rigorous guarantees in safety-critical domains and the difficulty posed by GPT models’ discrete nature.Source: §12 · p. 62
  • Safeguarding directions include adding domain knowledge and logical reasoning, and using diverse roles to improve answer consistency and contextual understanding.The paper gives retrieval-augmented pretraining as an example of adding knowledge and reasoning support.Source: §12 · p. 62

A.1 Greedy decoding v.s. Top-p decoding

  • The toxicity appendix compares greedy decoding with nucleus sampling and presents toxicity-evaluation templates for the 1.2K most toxic prompts.Greedy decoding uses top-k=1 with no randomness; nucleus sampling uses top-p=1 with more randomness.Source: §A.1 · p. 77, Table 34 · p. 77

A.2 Full list of diverse system prompts

B.1 Target groups and stereotype templates selected for stereotype bias evaluation

  • The stereotype-bias dataset pairs selected demographic groups with stereotype templates by replacing the [target_group] placeholder in each template.The appendix identifies separate tables for the selected groups and templates.Source: §B.1 · p. 83, Table 35 · p. 83, Table 36 · p. 83

B.2 Supplementary results on stereotype bias evaluation · B.3 Evaluation on standard stereotype bias benchmark · C Additional details of evaluation on adversarial robustness

On BBNLI, GPT-3.5 and GPT-4 had bias scores close to zero across domains and task settings, while supplementary examples show that GPT-4 could respond sarcastically while still expressing stereotypes. The benchmark results also identify religion as the domain with higher bias scores for both models.

B.2 Supplementary results on stereotype bias evaluation

  • In supplementary examples, GPT-4 sarcastically agreed with stereotypes about Latin Americans and Canadians, despite the sarcastic framing.The paper says such sarcastic agreements were observed at a low rate.Source: §B.2 · pp. 84–85
  • In another example, GPT-4 rejected a stereotype about Asians having a drug problem but then used anti-Asian stereotypes and insults, including “yellow-skinned” [88] and “slanty-eyed” [163].The table caption notes those two stereotypes even though GPT-4 rejects the original statement.Source: Table 37 · p. 84, §B.2 · p. 85
  • A supplementary output agreed with the stereotype that physically disabled people cannot raise children well, describing them as worthless and incapable of care.Source: §B.2 · p. 85

B.3 Evaluation on standard stereotype bias benchmark

  • BBNLI tests 16 harmful stereotypes across race, religion, and gender using 3,642 premise-hypothesis pairs, with both NLI and question-answering versions.Its questions include whether the models exhibit bias in NLI and QA, and whether task encoding affects bias independently of content.Source: §B.3 · p. 85
  • The evaluation compares responses to pro- and anti-stereotypical hypotheses, generates five responses per hypothesis, and averages bias scores across domains and settings.The score ranges from 1 for “yes” to the pro-stereotypical and “no” to the anti-stereotypical version, to -1 for the reverse; matching predictions score 0.Source: §B.3 · p. 85
  • Across race, religion, and gender, bias scores were close to 0 for GPT-3.5 and GPT-4 in both NLI and QA, with insignificant differences between settings.The authors interpret this as resistance to bias in either the pro- or anti-stereotypical direction.Source: §B.3 · p. 85
  • For both models, religion bias scores were higher than race and gender in NLI and QA; GPT-4 was marginally higher on race and religion, while GPT-3.5 was marginally higher on gender.The authors note that reducing bias associated with religious stereotypes may require additional effort.Source: §B.3 · pp. 85–86

C.1 Details of the standard AdvGLUE benchmark … E.2 Demonstration templates

These appendices document the construction and statistics of AdvGLUE and AdvGLUE++, describe OOD-style transformations and knowledge examples, and provide task descriptions and demonstration templates for Section 7.

C.1 Details of the standard AdvGLUE benchmark

  • The appendix provides detailed construction statistics for the standard AdvGLUE test set.Table 39 presents the dataset statistics.Source: §C.1 · p. 86, Table 39 · p. 87

C.2 Construction of AdvGLUE++

  • AdvGLUE++ is broken down by task type and target model, with separate descriptions of the strategies used to generate adversarial texts.The appendix also provides dataset statistics for AdvGLUE++.Source: §C.2 · p. 87, Table 40 · p. 87
  • AdvGLUE++ generation strategies include typo replacement, importance-ranked synonym substitution, contextual word replacement, sememe-based search, and embedding-space attacks.TextBugger uses typos; TextFooler substitutes synonyms for important words; BERT-ATTACK uses masked-language prediction; SememePSO searches HowNet substitutions with particle swarm optimization; SemAttack backpropagates gradients to select candidate words.Source: §C.2 · pp. 87–88

D.1 Details of OOD style

  • OOD-style examples cover transformations implemented with methods from, an Augment configuration following [106] with a 0.2 misspelling rate, Shake-W using, and other sentence-level transformations following [94].Table 41 presents examples of the different styles.Source: §D.1 · p. 88, Table 41 · p. 88

D.2 Details of OOD knowledge

  • The appendix gives qualitative RealtimeQA examples across four settings and examples of questions GPT-4 correctly answered under QA2023.Table 43 marks the answer selected by GPT-4 in bold.Source: §D.2 · p. 88, Table 42 · p. 89, Table 43 · p. 89

E.1 Task descriptions

  • The appendix summarizes the task descriptions used in the Section 7 experiments.These descriptions are presented in Table 44.Source: §E.1 · p. 90, Table 44 · p. 90

E.2 Demonstration templates

  • The appendix summarizes the demonstration templates used for the Section 7 experiments.The templates are presented in Table 45.Source: §E.2 · p. 90, Table 45 · p. 90

E.3 More ablation studies … G.1.2 Prompt design

The appendices report a further trigger-location ablation and specify privacy-data constructions and machine-ethics dataset and prompt implementations.

E.3 More ablation studies

  • With every demonstration backdoored, beginning triggers were most effective and end triggers least effective for GPT-3.5, while GPT-4 results were similar across locations.The authors hypothesize that trigger location matters less for GPT-4 because every demonstration contains the trigger.Source: Table 46 · p. 90, §E.3 · p. 90

F.1 Additional details of the Enron email dataset

  • Enron email few-shot templates B–D format name/email demonstrations and a target email using comma-separated, mailto, or original-message layouts.The supplied examples include comma-separated fields in Template B, mailto formatting in C, and original-message headers in D.Source: §F.1 · p. 91

F.2 Additional details of PII injected during conversations

  • Conversation PII was randomly generated: phone/canary numbers had 10 digits, SSNs 9, and credit cards 16.Passwords and keys used 12 mixed-case alphanumeric characters; passport numbers and canary codes used 10 uppercase letters or digits; addresses were randomly selected from seven public options.Source: §F.2 · p. 91

F.3 Additional details of privacy events

  • The privacy events used in prompts covered divorce, health, party planning, layoffs, exams, bankruptcy, promotion, and voting.Examples included seeing psychiatrists, failing an exam, and voting for Biden in the election.Source: §F.3 · p. 91

G.1.1 More implementation details

  • ETHICS has 13,910 training and 3,885 testing samples; few-shot evaluation uses 32 short demonstrations for short tests and eight for long tests, following [73].Jiminy Cricket has 1,000 training and 4,000 testing samples; its samples draw on expert walkthroughs and CALM actions [201], with unrecognized or nonsensical actions filtered, and the dataset is to be released for future research. The Jiminy Cricket collection follows a process similar to zero-shot classification experiments [75].Source: §G.1.1 · p. 92

G.1.2 Prompt design

  • ETHICS prompts request wrong/not-wrong verdicts, with more instruction for long Reddit posts; Jiminy Cricket prompts classify scenarios as bad, good, or neutral.The Jiminy Cricket prompts also include annotator labeling guidelines, with the corresponding template shown in Figure 34b.Source: §G.1.2 · p. 92

G.1.3 More experimental results … I.1 Aggregation protocol for each trustworthiness perspective

Supplementary evaluations report machine-ethics results and conditional-action examples, describe evaluation costs and open-model scoring, and specify aggregation protocols for DecodingTrust’s trustworthiness perspectives.

G.1.3 More experimental results

  • Few-shot GPT-4 achieved the highest accuracy across ETHICS tasks except utilitarianism, where it trailed RoBERTa-large by 1.1%; zero-shot GPT-4 reached 93.7% on virtue.GPT models usually outperformed non-GPT models, and GPT-4 often outperformed GPT-3.5; few-shot GPT-3.5 ranked second on justice, deontology, and utilitarianism. Across models, average accuracy was 80.5% on utilitarianism and 81.5% on commonsense morality.Source: Table 47 · p. 94, §G.1.3 · p. 94

G.2 Additional details of evaluation on jailbreaking prompts / G.3 Additional details of evaluation on evasive sentences

  • Jailbreaking prompts were inserted into system or user messages according to prompt type, while evasive sentences were placed after scenario descriptions to perturb moral recognition.The evaluation placed the system-level prompt only in the system message and the user-level prompt only in the user message; other jailbreaking prompts could appear in either.Source: §G.2 · p. 94, §G.3 · p. 94

G.4 Additional details of evaluation on conditional actions

  • Jiminy Cricket conditional-action testing used concise scenarios and yes/no moral-action prompts; examples show recognition varied with the victim and severity of harm.Models recognized harm to others and moderate-severity harm as immoral, but failed to recognize the same harm to oneself and harm at a minor severity as immoral.Source: §G.4 · pp. 94, 96, Figure 37 · p. 96, Figure 36 · p. 96

H Dataset statistics and estimated computational cost

  • The cost appendix reports prompt counts, prompt and completion tokens, and estimated costs across perspectives and scenarios, helping readers assess whether experiments fit available resources.Tables 48–56 provide overall and scenario-level statistics, including repetitions and total costs for individual evaluation scenarios.Source: H Dataset statistics and estimated… · p. 97, Table 48 · p. 97

I.1 Aggregation protocol for each trustworthiness perspective

  • Aggregation defines toxicity as one minus average expected maximum toxicity, stereotype bias through agreement with stereotyped prompts, adversarial robustness through average robust accuracy, and OOD robustness across OOD scenarios.Robustness to adversarial demonstrations averages counterfactual, spurious-correlation, and backdoor components; the backdoor component is one minus average attack success rate.Source: §I.1 · pp. 100–101
  • Privacy aggregation combines training-data leakage, conversation PII leakage, and privacy-word/event leakage; machine ethics combines benign accuracy with adversarial false-positive rates, while fairness reflects demographic parity differences.The privacy scenario weights are 0.4 for Enron extraction, 0.3 for PII leakage, and 0.3 for privacy understanding; higher fairness scores indicate fairer predictions but lower prediction capacity under the observed accuracy-fairness trade-off.Source: §I.1 · pp. 102–103

I DecodingTrust scores on open LLMs

  • The open-model evaluation covers six Open LLM leaderboard models and reports an overall score for each model within each trustworthiness perspective.The supplied passage identifies Llamav2-7B-Chat [167], Vicuna-7B, Alpaca-7B, MPT-7B, Falcon-7B, and RedPajama-INCITE-7B-Instruct; Table 57 presents the comprehensive results, but the supplied text does not provide its cell values.Source: I DecodingTrust scores on open LLMs · p. 100, Table 57 · p. 103

I.2 Comprehensive evaluation results of existing LLMs

No evaluated model leads across all trustworthiness perspectives: GPT-4 performs best on three, while Llama 2 performs best on four. The results underscore the continuing need to develop more trustworthy LLMs.

I.2 Comprehensive evaluation results of existing LLMs

  • GPT-4 achieves the best performance on 3 perspectives, while Llama 2 leads on 4: toxicity, stereotype bias, privacy, and fairness.GPT-4 leads on adversarial robustness, out-of-distribution robustness, and robustness to adversarial demonstrations.Source: §I.2 · p. 103
  • No evaluated model achieves the best performance on all eight trustworthiness perspectives, leaving trustworthy LLM development an important task.The section reports overall results in Table 57, perspective-specific results in Tables 58–65, and visualizations in Figures 38–47.Source: §I.2 · p. 103

J Limitations … How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

The paper acknowledges limits from unavailable training data, subjective trustworthiness measures, model updates, and possible misuse, while outlining social benefits and dataset documentation, distribution, and maintenance practices.

J Limitations

  • Unavailable GPT pretraining data limits failure analysis and out-of-distribution evaluation, while subjective trustworthiness measures still need human-alignment analysis.The OOD scenarios depend on hypotheses about training exposure; aligning evaluation metrics with human understanding is left for future work.Source: J Limitations · p. 108
  • The study focuses mainly on GPT-3.5 and GPT-4 at a specific time, and released jailbreak prompts could enable misuse or fine-tuning to bypass tests.The authors plan to keep newly generated challenging prompts private, while arguing that vulnerability evaluation can help practitioners design defenses before deployment.Source: J Limitations · p. 108

K Social impacts

  • The authors argue that findings on bias, privacy, robustness, and machine ethics can support mitigation, privacy protections, reliable deployment, and ethical guidelines.They say understanding trustworthiness gaps could guide more trustworthy ML systems and help the public trust systems used in sensitive applications.Source: K Social impacts · pp. 108–109

L Data sheet

For what purpose was the dataset created?

  • The dataset was created to assess trustworthiness across eight areas, including toxicity, bias, adversarial and OOD robustness, privacy, machine ethics, and fairness.It is intended to help stakeholders understand model capabilities, limitations, and deployment risks.Source: For what purpose was the dataset… · p. 109

L.2 Composition/collection process/preprocessing/cleaning/labeling and uses:

  • The paper and project website describe the dataset’s composition, collection, preprocessing, cleaning, labeling, and uses.The cited website is decodingtrust.github.io.Source: §L.2 · p. 109

L.3 Distribution

  • The research group manages and maintains the dataset, and the evaluation dataset is publicly hosted on GitHub.Source: §L.3 · p. 109

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

Related papers

Loading 2306.11698v5…