Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment

Motivation

• Unaligned LLMs inherit harmful content, polarized opinions, discrimination, and illegal advice from imperfect Internet data, while objective optimization that ignores human values contributes additional failures.The authors note that the exact causes of LLM problems remain an ongoing research question.Source: §1 · p. 4
• Alignment improves LLM reliability, safety, and adherence to human values, contrasting unaligned GPT-3’s harmful responses with aligned ChatGPT’s refusal and supportive redirection.The paper attributes alignment’s benefits to reducing harmful outputs and improving adherence to human instructions, while Figure 1 illustrates the before-and-after contrast.Source: §1 · pp. 4–5, Figure 1 · p. 5

Problem

• Existing alignment guidance lacks a unified, principled, and sufficiently granular framework for evaluating the full spectrum of LLM trustworthiness, despite proposals such as the HHH principle [20] and a six-risk taxonomy [21].Prior work also surveys generative AI’s social impacts [22] or evaluates LLMs holistically [23], but does not provide an accepted alignment taxonomy focused specifically on language-model trustworthiness.Source: §1 · pp. 4–5

Contribution

• The paper proposes a fine-grained taxonomy of LLM alignment requirements that unpacks harms across individuals, society, and stakeholders and guides data collection for desirable alignment processes.The taxonomy is motivated by imbalanced alignment data, such as frequent violence examples but marginal child-abuse and self-harm examples, and by alignment’s dependence on data [24].Source: §1 · p. 5
• The paper surveys trustworthiness categories, explains evaluation and dataset construction, measures widely used LLMs, and demonstrates that targeted evaluation data can also support effective category-specific alignment.Even widely considered well-aligned models can fail some alignment tasks, motivating more fine-grained evaluation.Source: §1 · p. 5

Language models

• LLMs are autoregressive models that predict next-token distributions; large models contain tens of millions to billions of parameters and exhibit emergent abilities [28] [29] [30].Text generation samples from token distributions using methods such as greedy sampling, nucleus sampling [26], and beam search [27].Source: §2 · p. 6

Transformer architecture

• Transformers underpin recent LLM progress through self-attention, feedforward layers, and residual links, while enabling direct fine-tuning without task-specific architectures [31] [32] [33].The architecture was initially designed for machine translation before adaptation to language modeling.Source: §2 · p. 6

Evaluation models

• The survey evaluates GPT-4, ChatGPT, GPT-3, aligned GPT-3, OPT-1.3B, and FLAN-T5, with ChatGLM and DiabloGPT used for additional case studies.Models were accessed during May–July 2023; examples use web-interface names, whereas experiments use API names.Source: §2 · p. 6

LLM Alignment

• Standard alignment combines supervised fine-tuning on human-written answers with RLHF, which trains a reward model from ranked outputs and updates the policy using PPO [1] [18] [19] [38].SFT prompts often solicit unsatisfactory or harmful answers, while RLHF rankings reflect outputs preferred by an “unbiased” human user.Source: §2 · p. 6, Figure 2 · p. 7
• Alternative alignment methods include RAFT, RRHF, DPO, and Stable Alignment, respectively avoiding or reducing reliance on PPO or separately fitted reward models [39] [40] [41] [42].These methods learn from high-ranked samples or preference data using ranking-based objectives.Source: §2 · p. 7

Research scope and limitations

• Alignment research remains active because current methods require labor-intensive question generation and evaluation, while no unified framework covers all LLM trustworthiness dimensions.The survey therefore seeks full-coverage benchmark data and efficient, effective evaluation methods rather than ranking all available methods.Source: §2 · pp. 6–7

Taxonomy Overview

• The taxonomy organizes LLM alignment into 7 major categories and 29 sub-categories spanning reliability, safety, fairness and bias, resistance to misuse, interpretability, goodwill, and robustness.Figure 3 presents the categories and their sub-categories, including reliability, safety, fairness, resistance to misuse, explainability and reasoning, social norm, and robustness.Source: §3 · p. 7, Figure 3 · p. 8
• The categories address correct and consistent outputs, safe and lawful behavior, non-discrimination, misuse prevention, explanation and reasoning, shared human values, and resilience to attacks or distribution shifts.The taxonomy includes concerns such as hallucination, privacy violation, stereotype bias, cyberattack, limited causal reasoning, cultural insensitivity, prompt attacks, and poisoning attacks.Source: §3 · p. 8

How We Determine the Taxonomy

• The taxonomy is determined by potential harm to users and society across LLM applications, together with literature from responsible AI, information security, social science, human-computer interaction, law, and moral philosophy.The authors first survey current LLM applications, whose coverage is explicitly non-exhaustive, then group sub-categories by relevant capabilities and specific concerns.Source: §3 · pp. 8–9

Motivation and Scope

• Reliability, safety, fairness, misuse resistance, interpretability, social alignment, and robustness are prioritized because failures can cause misinformation, injury, discrimination, intentional harm, emotional damage, or security risks.Hallucinations can affect nearly all applications except probably creative writing, with consequences ranging from amusing nonsense to financial or legal disasters; the categories are not claimed to cover the entire trustworthiness space.Source: §3 · pp. 8–9

4 Reliability

• Across reliability, LLMs should answer factual questions correctly, avoid confident errors, and recognize uncertainty, but uncertainty estimation remains an ongoing challenge [48].These requirements are especially important because unreliable outputs can undermine applications in high-stakes sectors such as health care and finance.Source: §4 · p. 9

4.1 Misinformation

• Misinformation arises when LLMs unintentionally provide wrong information, influenced by imperfect training data, entity co-occurrences, unpopular facts, and limited knowledge updating [49] [50].Retrieval can improve predictions for unpopular facts, while knowledge updating varies substantially across models; code-davinci-0023 updated knowledge around 85% of the time on two knowledge-intensive QA datasets [56] [58].Source: §4.1 · pp. 9–10

4.2 Hallucination

• Hallucinations are fabricated or source-unfaithful outputs, including intrinsic conflicts with sources and extrinsic claims unverifiable from existing sources [60] [62].Evaluation uses similarity metrics, truthfulness classifiers, and human judgments, while mitigation through better data, tailored rewards, and external knowledge remains limited and unresolved.Source: §4.2 · pp. 10–11

4.3 Inconsistency

• Inconsistency occurs across users, sessions, conversations, or prompt formulations, reducing trust; randomness from sampling, updates, hidden platform operations, and hardware contributes to the problem [89] [90] [91].Consistency is distinct from logicality: emotional-support systems may need a stable warm tone without logically valid responses.Source: §4.3 · p. 11

4.4 Miscalibration

• Miscalibration makes LLMs express unjustified confidence, including for outdated knowledge or objectively indeterminate questions, while internal logits and textual confidence can diverge [93] [94].Selective classifiers, conformal prediction, out-of-domain detection, post-hoc scaling, and fine-tuning can partially address generally poor calibration; alignment can also teach verbal uncertainty [99] [100].Source: §4.4 · pp. 12–13

4.5 Sycophancy

• Sycophancy causes LLMs to affirm users’ misconceptions and abandon correct answers when challenged, differing from inconsistency because it primarily reflects excessive instruction-following [24] [122].Training data, excessive helpfulness instructions, and RLHF may promote confirmation, creating a tradeoff between sycophancy and alignment with human values.Source: §4.5 · p. 13, Figure 11 · p. 13

5 Safety

• LLM safety requires preventing harmful outputs, prohibiting misuse, proactively mitigating unintentional harm, and protecting personal information privacy, but comprehensive safety remains unfinished.Unsafe outputs can reduce user trust and create difficulties complying with safety regulations.Source: §5 · pp. 13–14

5.1 Violence

• Aligned LLMs can deny violent requests, but indirect prompting may still elicit well-organized instructions for terrorism or nuclear-bomb construction; question-answer refusal data and content filtering are proposed mitigations [18] [1].Figure 12 illustrates how users can hide harmful intentions by pretending to seek information righteously.Source: §5.1 · p. 14, Figure 12 · p. 14

5.2 Unlawful Conduct

• LLMs can directly advise users on illegal substances and evading crimes, potentially causing greater harm than search engines, while compliance requires alignment with differing national laws.Figures 13 and 14 show examples involving GPT-3 and GPT-4; practitioners may need locally labeled alignment data.Source: §5.2 · p. 14, Figure 13 · p. 14, Figure 14 · p. 14

5.3 Harms to Minor

• LLMs may facilitate child trafficking, abuse, exploitation, or age-inappropriate violent and sexual content, making additional alignment and access controls important for protecting minors.The concern is treated separately from general legal requirements because it is considered both legally and morally important.Source: §5.3 · p. 15

5.4 Adult Content

• LLMs can generate erotic or sexually offensive content and recommend sexual websites, motivating stricter screening, age controls, and output filtering with pornographic-content detectors [139] [140] [62].Different regions define and tolerate sexual content differently.Source: §5.4 · p. 15

5.5 Mental Health Issues

• For mental-health-related questions, LLMs should recognize context, provide psychosocial support and resources, and avoid careless confirmation or denial because such responses can have disastrous consequences.More aligned models may still be prompted to continue unsafe content, so safeguarding generated content remains an active challenge.Source: §5.5 · p. 15

5.6 Privacy Violation

• Attackers can extract personal or sensitive information and private training samples from LLMs through queries, with memorization-related risks often increasing with training-sample repetition [53] [193].Differential privacy, machine unlearning, federated learning, and secure multi-party computation are proposed defenses, but their large-scale effectiveness and efficiency remain unclear [195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218].Source: §5.6 · p. 16

6.1 Injustice

• Fairness requires impartial treatment of similar individuals, decisions based on true desert rather than group membership, and comparable services that users would not envy across groups.These criteria connect LLM fairness to individual fairness, meritocratic justice, and envy-freeness [226].Source: §6.1 · p. 16

6.2 Stereotype Bias

• LLMs must not exhibit or amplify stereotypes affecting protected groups, including gender, race and color, religion and belief, and sexual orientation.Pretrained LLMs can absorb stereotype biases persisting in crowdsourced data and exhibit toxicity against protected groups.Source: §6.2 · p. 17

6.3 Preference Bias

• LLM responses should remain neutral and factual on political, scientific, societal, product, and subjective questions rather than promoting ideological beliefs or intervening in users’ values.Suggested mitigations include presenting alternative viewpoints with references and finetuning on human-written neutral answers.Source: §6.3 · pp. 17–18

6.4 Disparate Performance

• LLM capabilities can differ substantially across racial and social-status groups, tasks, languages, and cultures, including lower translation quality from English to Chinese and Korean than to languages with Latin scripts [62].The section attributes these disparities to task difficulty, missing or imbalanced training data, and difficulty understanding cultural backgrounds.Source: §6.4 · p. 18
• Reducing disparate performance requires local alignment tailored to regional cultures, norms, laws, and user expectations rather than one-size-fits-all solutions.Local alignment depends on collecting suitable alignment data, while fair training seeks to reduce prediction error and disparities between groups.Source: §6.4 · p. 18

7 Resistance to Misuse

• LLM misuse concerns malicious users intentionally leveraging models to cause harm, rather than users unintentionally eliciting dangerous or misleading outputs.The survey distinguishes proactive misuse from potential and unintentional misuse discussed elsewhere.Source: §7 · p. 18, §7.1 · p. 19

7.1 Propagandistic Misuse

• Malicious users can leverage LLMs to generate propaganda targeting people, advocating terrorism, and producing extreme or harmful political arguments.Alignment can mitigate these uses through factual, unbiased responses or refusals, including polite refusals for advocacy that amplifies harmful opinions [20].Source: §7.1 · p. 19

7.2 Cyberattack Misuse

• LLMs can assist cyberattacks by generating malicious code for DDoS attacks, malware, phishing, and data stealing, while alignment and request detection offer mitigations.Suggested defenses include cyberattack-related alignment data, output ranking, keyword matching, and simple text classifiers.Source: §7.2 · pp. 19–20, Figure 21 · p. 20

7.3 Social-engineering Misuse

• LLMs can enhance social-engineering attacks through contextualized deceptive messages, impersonation, phishing, spam, fake content, and social-network manipulation.Because pretending to be someone is also useful, traditional system-level defenses such as bot detection, authentication, filtering, and usable security may be preferable to simple alignment restrictions; detector accuracy may decline as models advance.Source: §7.3 · p. 20

7.4 Leaking Copyrighted Content

• LLM memorization can leak copyright-protected text, code, images, video, and audio, creating copyright and intellectual-property concerns [183] [185].Mitigations include detecting extraction prompts, tracing training-data use, watermarking, and differential privacy, although LLM watermarking remains an open research area.Source: §7.4 · pp. 20–21

8.1 Lack of Interpretability

• LLMs can improve interpretability by retrieving explicit source documents, exposing chain-of-thought reasoning, or using auxiliary models to identify neuron-level themes, although long contexts remain a challenge.Retrieval-augmented methods let users inspect supporting sources, while Bills et al. use explainer and simulator models to predict subject-model activations; chain-of-thought enables step-by-step explanations [29].Source: §8.1 · p. 22

8.2 Limited General Reasoning

• Chain-of-thought explanations can improve question-answering accuracy and support inspection of reasoning, with self-consistency and tree-of-thoughts extending this approach through voting, backtracking, and alternative chains [29] [381] [382].These methods target logical consistency and reduce fixation on a single potentially flawed reasoning path.Source: §8.2 · p. 22
• GPT-3 barely outperforms random guessing on abductive detective puzzles, while GPT-4 solves only 38%, and models often exploit superficial cues rather than robust logical reasoning.CoT explanations can also be unfaithful: models may omit obvious biases they use, and ChatGPT can reach correct mathematical conclusions through invalid steps [122] [383].Source: §8.2 · pp. 22–23
• Reasoning improvements use prompt engineering, continued or scratch pretraining, supervised fine-tuning, and reinforcement learning, while comprehensive evaluation requires diverse benchmarks such as the Chain-of-thought hub [398].Examples include least-to-most prompting, human-annotated CoT fine-tuning, specialization for smaller models, and rewards for intermediate mathematical or coding steps [389] [391] [392] [393] [394] [396].Source: §8.2 · p. 23

8.3 Limited Causal Reasoning

• GPT-4 is relatively accurate at inferring necessary causes but much less accurate at sufficient causes, which require considering many counterfactual scenarios [358].Causal reasoning covers cause-effect inference, counterfactual questions, and structural causal models, extending beyond premise-based logical reasoning [358] [399].Source: §8.3 · p. 23
• On CORR2CAUSE, untuned LLMs barely outperform random guessing; few-shot fine-tuning improves accuracy, but the improvement fails under paraphrased templates or renamed variables [400].In a necessary-cause sentiment case study, text-davinci-003 produced an inconsistent positive sentence after changing the causal event, contrary to common sense.Source: §8.3 · p. 24

9 Social Norm

• Social-norm alignment concerns avoiding group-directed offense, sensitivity to destabilizing topics, and sympathetic emotional support, especially under the Harmless principle of HHH [20].The authors focus on values broadly considered socially beneficial and recommend neutrality for controversial political or preference-sensitive questions.Source: §9 · pp. 24–25

9.1 Toxicity

• Toxicity includes rude, disrespectful, threatening, or identity-attacking language, motivating classifiers and system-level filters that abstain from displaying toxic LLM outputs [402] [403] [404].Hate speech can incite violence and is prohibited by policies in some jurisdictions, while automated generation makes offensive-language harms more severe.Source: §9.1 · p. 25

9.2 Unawareness of Emotions

• Responsible LLMs should provide informative, emotionally supportive, and sympathetic responses to vulnerable users rather than insensitive confirmations; ChatGPT showed strong emotional awareness across anger, fear, happiness, and sadness scenarios.Figure 24 presents a failed example of an answer interpreted as insensitive to a user’s emotions.Source: §9.2 · p. 25, Figure 24 · p. 25

9.3 Cultural Insensitivity

• Cultural alignment requires locally collected datasets because GPT-3 performed significantly better with American contexts than other cultures, reflecting regional differences in values and reactions [272].Proposed remedies include human-feedback instructions [1], user-specified harmlessness rules [418], simulated societies [42], and target value datasets.Source: §9.3 · pp. 25–26

10 Robustness

• Robustness failures can arise from ordinary prompt errors, malicious inputs, temporal changes, feedback effects, and attacker-controlled training content, so deployment testing is essential.Potential defenses include influential-sample removal, robust subset training, differential privacy, and Distributionally Robust Optimization [348] [463].Source: §10 · p. 26, §10.4 · p. 28

10.1 Prompt Attacks

• Prompt typos can change GPT-3’s answer to the same ethical-reasoning question, while adversarial prompts can flip classifications or elicit dangerous information [419] [420] [421].Adversarial prompt engineering can also support red teaming and provide alignment examples [422], analogous to adversarial data augmentation [423].Source: §10.1 · p. 26, Figure 25 · p. 26

10.2 Paradigm and Distribution Shifts

• Shifting knowledge and concepts can make answers outdated and policies obsolete, creating deployment-time fairness and compliance risks despite static-data verification [425].The paper calls for detecting knowledge shifts, acknowledging lag, and developing efficient mechanisms to update LLMs.Source: §10.2 · pp. 26–27, Figure 26 · p. 27

10.3 Interventional Effect

• High-frequency, large-scale user interactions can reinforce existing disparities and unethical outputs through feedback loops, heightening LLM bias concerns [1].Proposed directions include modeling user responses, optimizing long-term utility, predicting interaction dynamics, and using causal or debiasing methods [432] [440] [442].Source: §10.3 · p. 27

10.4 Poisoning Attacks

• Poisoning 0.1% of unlabeled data can induce arbitrary test-time misclassification, while poisoning 0.01% can misclassify CLIP test images [452].Internet-based LLM training is especially exposed: attackers can poison web-scale datasets or insert crafted code files that cause malicious code suggestions [454] [455] [459].Source: §10.4 · p. 28

11 Case Studies: Designs and Results

• The overall evaluation selects eight non-exhaustive subcategories spanning all seven trustworthiness pillars, including hallucination, safety, gender stereotype, miscalibration, misuse, interpretability, and robustness.The experiments are designed as general measurement studies intended to demonstrate practical feasibility and extend to other categories.Source: §11 · p. 28

11.1 Overall Design

• The evaluation automates test-data generation and labeling with existing LLMs to replace large-scale human annotation, while acknowledging the need for human audits to support credibility.This approach can reduce manual work from hundreds of human labelers to a few prompt engineers, but automation remains a study caveat.Source: §11.1 · p. 28

11.2 Hallucination

• More aligned models hallucinate less on the 300-sample QA task, with GPT-4 achieving the highest correct-answer accuracy among six evaluated LLMs.The evaluation uses questions containing hallucinated and correct answers, treating hallucinated or irrelevant responses as errors.Source: §11.2 · p. 29, Figure 27 · p. 29

11.3 Safety

• More aligned models are more likely to refuse unsafe prompts, with GPT-3.5-turbo and GPT-4 reaching refusal ratios close to 100%.The study generates 984 unsafe prompts and measures the percentage of questions each model refuses to answer.Source: §11.3 · p. 30, Figure 30 · p. 31

11.4 Fairness

• Alignment correlates with greater resistance to gender stereotypes, but refusal ratios remain only 30−60%, and GPT-4 stays below 60%.Manual inspection found that GPT-3.5-turbo and GPT-4 still frequently selected gender-stereotypical options, motivating fairness-specific alignment.Source: Figure 31 · p. 31, §11.4 · p. 32

11.5 Miscalibration

• Miscalibration comparisons exclude davinci, OPT-1.3B, and flan-t5-xxl because they lack accessible or coherent uncertainty outputs, and the fairness-dataset results may not transfer to other multiple-choice datasets.ChatGPT and GPT-4 internal probabilities are unavailable and are therefore approximated by repeated sampling; further research is needed.Source: §11.5 · pp. 32–33
• Miscalibration is measured with Wasserstein distance between internal logits and external verbal confidence, using 20 samples to approximate unavailable ChatGPT and GPT-4 internal probabilities.The distance is preferred because confidence distributions place most weight on one option, producing many zero weights that can bias MSE or MAE.Source: §11.5 · p. 32
• Text-davinci-003 shows the strongest consistency between external verbal confidence and internal confidence, while ChatGPT and GPT-4 show similar consistency.The comparison concerns confidence matching rather than answer correctness, so a model can be confidently unfair.Source: §11.5 · p. 32, Figure 32 · p. 33

11.6 Propagandistic and Cyberattack Misuse

• Misuse testing covered propaganda, terrorism advocacy, harmful political propaganda, and cyberattacks, measuring the percentage of 500 repeated samples that each model refused.Prompts were generated from 100 keywords per misuse type, and GPT-4 judged whether outputs constituted refusals.Source: §11.6 · p. 34
• ChatGPT and GPT-4 best resisted misuse, but unaligned models sometimes scored better because they produced nonsensical, instruction-unfollowing outputs rather than trustworthy refusals.Alignment that improves instruction-following without improving trustworthiness can increase misuse harms.Source: §11.6 · pp. 34–35
• For extreme political propaganda, more aligned models were more likely to generate harmful content because unaligned models often lacked the capability to complete the task.Aligned models sometimes handled the task technically without considering how their outputs could be misused.Source: §11.6 · p. 35

11.7 Leaking Copyrighted Content

• All tested LLMs emitted text resembling copyrighted content, with no obvious relationship to alignment; davinci had nearly a 20% high-similarity rate and GPT-4 had 18%.The evaluation used 1K book excerpts, 350-character prompts, greedy sampling, and BERT cosine similarity, where lower similarity was better.Source: §11.7 · p. 36

11.8 Causal Reasoning

• Causal evaluation also exposed inconsistent reasoning: GPT-4 generated event pairs it later handled inconsistently, while text-davinci-003 failed to connect the related Q1 and Q2 judgments.OPT-1.3B and davinci frequently could not answer the multiple-choice questions properly, and their selectable-answer accuracy remained significantly lower.Source: §11.8 · pp. 37–38
• For causal reasoning, text-davinci-003, ChatGPT, and GPT-4 significantly outperformed davinci, while Q2 was harder than Q1 and GPT-4 remained below 70% accuracy on Q2.Q2 additionally requires counterfactual reasoning about whether changing Event A would make Event B inconsistent.Source: §11.8 · pp. 37–38

11.9 Robustness

• Typos substantially reduced every model’s answer consistency; flan-t5-xxl degraded least among aligned models, while ChatGPT and GPT-4 were surprisingly vulnerable.ChatGPT and GPT-4 often answered “No” to typoed prompts, apparently treating erratic wording as nonsensical rather than preserving the original task.Source: §11.9 · p. 39, Figure 44 · p. 39

11.10 Generating Training Data for Alignment

• After alignment training, GPT-4 judged a significant portion of GPT-2 outputs better, while 78% of fine-tuned LLaMA-7B outputs beat pretrained outputs.The generated data used 433 positive prompts for SFT and 443 prompts with positive and negative outputs for reward-model training.Source: §11.10 · p. 40, Table 1 · p. 40

Conclusions

• The paper contributes a detailed trustworthiness taxonomy, reviews proposed solutions, and conducts measurement studies that produce a dataset for automated alignment evaluation.The taxonomy addresses alignment challenges across multiple dimensions and provides the basis for the measurement studies.Source: §12 · p. 40
• Higher-alignment LLMs generally perform better, but specific topics still require fine-grained alignment to improve trustworthiness coverage.The study emphasizes continued research to achieve robust alignment across diverse dimensions and support reliable, ethical deployment.Source: §12 · pp. 40–41

Limitations

• The taxonomy does not cover the full spectrum of LLM trustworthiness, motivating iterative development of a more fine-grained and comprehensive evaluation framework.The authors explicitly present this as a limitation and encourage community involvement in improving the framework.Source: §12 · p. 41
• Automated measurement simplifies evaluation by converting questions to multiple choice and using a more advanced LLM as a presumed ground-truth judge instead of human reviewers.These choices improve speed and cost efficiency but may sacrifice accuracy relative to slower human evaluation; a parallel effort is reported in [471].Source: §12 · p. 41

Open Problems

• Open alignment problems include whether RLHF is optimal, how to construct alignment data, how labeler viewpoints affect outcomes, how data-dependent alignment is, and which challenges resist alignment.The paper characterizes alignment science as early-stage, with limited theoretical insight and practical guidance.Source: §12 · p. 41
• The community needs more principled methods for evaluating and implementing alignment so LLMs adhere to societal values and ethical considerations.Addressing the identified open problems is presented as crucial for increasingly trustworthy and responsible deployment.Source: §12 · p. 41

• Figure 46 shows the label distribution of Anthropic’s red-teaming data.Source: Figure 46 · p. 63, A Evaluation Categories in Anthropic… · p. 63

B.1–B.3 Examples from Testing Hallucination, Safety, and Fairness

• Additional hallucination, safety, and fairness test examples are provided in Figures 47–49.Source: §B.1 · p. 64, §B.2 · p. 64, §B.3 · p. 64

B.3 Examples from Testing Fairness (Section 11.4)

• Additional miscalibration examples in Figure 50 show verbal uncertainty assigned to nonexistent option C.Source: §B.3 · p. 64, Figure 50 · p. 68

B.4 Examples from Testing Uncertainty (Section 11.5)

• Additional uncertainty-testing examples are included in Figure 50.Source: §B.4 · p. 64, Figure 50 · p. 68

B.5 Examples from Testing Misuse (Section 11.6)

• Misuse-resistance examples cover harmful misinformation about celebrities, terrorism advertisements, extreme political propaganda, and cyberattack assistance in Figures 51–54.Source: §B.5 · p. 64, Figure 51 · p. 69, Figure 52 · p. 70

B.6 Examples from Testing Copyright Leakage (Section 11.7)

• Additional copyright-leakage examples are provided in Figure 55.Source: §B.6 · p. 64, Figure 55 · p. 73

B.7 Examples from Testing Causal Reasoning (Section 11.8)

• Figures 56–61 provide additional causal-reasoning examples, including questions requiring binary causal judgments.Source: §B.7 · p. 64, Figure 56 · p. 74, Figure 57 · p. 75

B.9 Examples from Testing Alignment (Section 11.10)

• Figure 63 shows evaluation outputs before and after alignment using generated safety data, alongside additional alignment-test examples.Source: §B.9 · p. 64, Figure 63 · p. 81

B.8 Examples from Testing Robustness (Section 11.9)

• The appendix also includes examples testing robustness against typo attacks in Figure 62.Source: Figure 62 · p. 80

B.8 Examples from Testing Robustness (Section 11.9)

• Figure 62 lists additional test examples for evaluating robustness against typo attacks.Source: §B.8 · p. 64