PaperlayerView exact revision on arXiv

Source-linked AI summary

Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, Hang Li

arXiv:2308.05374v2cs.AIcs.LG

TL;DR

LLM alignment is important for trustworthy deployment, yet practitioners lack clear guidance for evaluating whether outputs meet social, ethical, and regulatory expectations. This paper surveys trustworthiness dimensions, proposes a detailed taxonomy and evaluation pipeline, and conducts measurement studies. In general, more aligned models perform better, although improvement varies across topics and important challenges remain.

  • Problem

    Practitioners lack clear, unified guidance for evaluating alignment and trustworthy behavior across LLM outputs.

  • Method

    The paper surveys trustworthiness dimensions, develops a detailed taxonomy and evaluation datasets, and conducts measurement studies with automated evaluation methods.

  • Results

    In general, LLMs with higher alignment perform better, while widely considered well-aligned models can still fail on some alignment tasks.

  • Takeaways & Limitations

    Fine-grained alignment evaluation and targeted improvements are needed because alignment effectiveness differs across trustworthiness topics.

  • Takeaways & Limitations

    The taxonomy does not claim to cover the entire LLM trustworthiness space, and safety defenses can be bypassed by specific prompt instructions.

Abstract

from arXiv · show

Ensuring alignment, which refers to making models behave in accordance with human intentions [1,2], has become a critical task before deploying large language models (LLMs) in real-world applications. For instance, OpenAI devoted six months to iteratively aligning GPT-4 before its release [3]. However, a major challenge faced by practitioners is the lack of clear guidance on evaluating whether LLM outputs align with social norms, values, and regulations. This obstacle hinders systematic iteration and deployment of LLMs. To address this issue, this paper presents a comprehensive survey of key dimensions that are crucial to consider when assessing LLM trustworthiness. The survey covers seven major categories of LLM trustworthiness: reliability, safety, fairness, resistance to misuse, explainability and reasoning, adherence to social norms, and robustness. Each major category is further divided into several sub-categories, resulting in a total of 29 sub-categories. Additionally, a subset of 8 sub-categories is selected for further investigation, where corresponding measurement studies are designed and conducted on several widely-used LLMs. The measurement results indicate that, in general, more aligned models tend to perform better in terms of overall trustworthiness. However, the effectiveness of alignment varies across the different trustworthiness categories considered. This highlights the importance of conducting more fine-grained analyses, testing, and making continuous improvements on LLM alignment. By shedding light on these key dimensions of LLM trustworthiness, this paper aims to provide valuable insights and guidance to practitioners in the field. Understanding and addressing these concerns will be crucial in achieving reliable and ethically sound deployment of LLMs in various applications.

1 Introduction

Alignment is presented as essential for making LLMs more reliable, safe, usable, and consistent with human values, but practitioners lack clear, comprehensive guidance for evaluating it. The paper responds with a fine-grained taxonomy, evaluation guidance, and measurement studies showing that alignment benefits are uneven across tasks.

  • Motivation: Alignment improves LLM usability and trustworthiness by encouraging instruction following and reducing unreliable, biased, toxic, unsafe, and adversarial behaviors.Earlier models such as GPT-3 were described as producing hallucinations, stereotypes, toxic language, harmful propaganda, and vulnerability to prompt attacks.
  • Motivation: Unsupervised Internet training data and objectives that ignore human values contribute to unreliable, unethical, and dangerous model outputs.The cited training data may contain harmful content, polarized opinions, discrimination, and illegal advice.
  • Research gap: Practitioners lack clear, principled, unified guidance for evaluating alignment across the full spectrum of trustworthy LLM behavior.Existing guidance includes the Helpful, Honest, and Harmless principle, but does not provide a comprehensive evaluation framework.
  • Contributions: The paper proposes a fine-grained alignment taxonomy, evaluation and dataset-building guidelines, measurement studies, and targeted alignment procedures.The authors also show that evaluation datasets can support alignment itself.
  • Contributions: The paper surveys alignment categories, explains how to evaluate trustworthiness, studies widely used LLMs, and demonstrates targeted alignment effectiveness.Its stated goals combine taxonomy development, evaluation guidance, measurement, and alignment using the resulting datasets.

2 Background

The background describes autoregressive language modeling and the standard alignment pipeline of supervised fine-tuning, reward-model training, and reinforcement learning from human feedback. It also frames the study as an evaluation pipeline rather than a complete benchmark, with reproducibility affected by model and sampling variability.

  • Language models: An autoregressive language model generates a token sequence by repeatedly predicting each next token from preceding tokens.The sequence probability factorizes as P(w) = P(w1) · P(w2|w1) · · · P(wT|w1, · · ·, wT−1).
  • Models: The evaluation primarily covers GPT-4, ChatGPT, GPT-3, aligned GPT-3, and several open-source models accessed during May–July 2023.The listed API models include gpt-4, gpt-3.5-turbo, davinci, and text-davinci-003.
  • Scope and reproducibility: The paper aims to provide an evaluation pipeline rather than benchmark or rank all available methods, and model outputs may vary with sampling, updates, platform operations, and hardware.The authors note limited API access to Google Bard and Anthropic Claude and discuss efforts toward reproducibility.
  • LLM alignment: The standard alignment procedure uses supervised fine-tuning, reward-model training from human rankings, and reinforcement learning from predicted rewards.Human-written answers produce an SFT model; ranked sampled outputs train the reward model; RL updates the SFT model.
  • LLM alignment: Alternative alignment methods such as RAFT, RRHF, DPO, and Stable Alignment have been proposed to reduce or replace parts of RLHF.The cited alternatives differ in whether they learn from ranked samples, use ranking losses, or eliminate reward-model fitting.

3 Taxonomy Overview

The proposed taxonomy organizes LLM trustworthiness into seven major categories and 29 sub-categories, selected using application impact and responsible-AI-related literature. Its coverage is intentionally not claimed to encompass the entire trustworthiness space.

  • Taxonomy scope: The taxonomy contains seven major categories and 29 sub-categories for evaluating LLM alignment.The categories are reliability, safety, fairness and bias, resistance to misuse, interpretability, goodwill, and robustness.
  • Reliability: Reliability covers misinformation, hallucination, inconsistency, miscalibration, and sycophancy, emphasizing correct, truthful, consistent outputs with proper confidence.Reliability is prioritized because hallucinations can harm output trustworthiness and affect nearly all applications except possibly creative writing.
  • Safety: Safety covers violence, unlawful conduct, harms to minors, adult content, mental health issues, and privacy violations.Its stated objective is to avoid unsafe and illegal outputs and prevent private-information leakage.
  • Additional categories: Fairness, resistance to misuse, explainability and reasoning, social norms, and robustness address bias, malicious use, reasoning and interpretability, shared values, and adversarial resilience.The listed sub-categories include propaganda, cyberattack, social engineering, copyright, prompt attacks, distribution shifts, and causal reasoning.
  • Taxonomy construction: Categories are determined by their impact on LLM applications and by literature spanning responsible AI, information security, social science, human-computer interaction, jurisprudence, and moral philosophy.Sub-categories are grouped according to their relevance to particular capabilities and concerns.
  • Scope and limitations: The taxonomy is intentionally non-exhaustive and reflects what the authors believe should be addressed based on literature, public discussion, and their own analysis.The authors explicitly do not claim coverage of the entire LLM trustworthiness space.

4 Reliability

Reliability requires correct, truthful, consistent outputs with appropriate confidence, yet LLMs can misinform, hallucinate, vary with prompting, overstate certainty, and flatter user misconceptions. The paper surveys evaluation and mitigation approaches, while noting that hallucination mitigation remains unresolved.

  • Reliability is foundational because unreliable outputs can harm nearly all LLM applications, especially in health care and finance.
  • 4.1 Misinformation: LLMs can generate misinformation, including factually wrong answers caused by imperfect or memorized training data.
  • 4.2 Hallucination: Hallucinations are fabricated, nonsensical, or source-unfaithful outputs, comprising intrinsic conflicts with sources and extrinsic contents unverifiable from existing sources.
  • 4.2 Hallucination: Hallucination evaluation uses similarity metrics, truthfulness classifiers, and human assessment across summarization and question-answering tasks.
  • 4.2 Hallucination: Mitigating hallucinations remains an open problem despite data improvements, alternative RLHF rewards, and external knowledge bases.
  • LLM reliability is also weakened by prompt-sensitive inconsistency, overconfidence in wrong or outdated answers, and sycophantic confirmation of user misconceptions.

5 Safety

Safety evaluation covers harmful content, unlawful conduct, harm to minors, sexual content, mental-health risks, and privacy. Although alignment and filtering can reduce some risks, indirect prompting, jurisdictional differences, and extraction attacks leave substantial challenges.

  • Safety requires LLM outputs to support safe, healthy conversations and avoid hateful, harmful, or dangerous content.
  • 5.1 Generated Content Safety: Indirect prompting can circumvent alignment and elicit organized details for violent or dangerous activities, including nuclear-bomb construction.
  • 5.1 Generated Content Safety: Alignment can remove violent content through refusal examples, while filtering and conversion methods offer additional mitigation approaches.
  • 5.2 Unlawful Conduct: GPT-3 and GPT-4 have been used to solicit advice about obtaining, purchasing, cultivating, and using illegal substances.
  • Safety concerns include dangerous or age-inappropriate content for minors, sexually explicit generation, and risks associated with LLM-based mental-health support.
  • Remarks on Safety Concerns: More aligned LLMs may deny explicit unsafe requests, but targeted prompts can still induce unsafe sexual content, making generated-content safety an active challenge.
  • Privacy: LLM memorization enables attackers to extract personal information and private training samples through model queries.
  • Privacy: Privacy-enhancing approaches include differential privacy, machine unlearning, federated learning, and secure multi-party computation.

6 Fairness

The paper frames fairness as preventing favoritism, discrimination, stereotypes, and disparate performance across users and groups. It also emphasizes neutrality in responses about political, societal, scientific, and product preferences.

  • Fairness risks: Uncurated training data can favor certain groups, perpetuate stereotypes, and produce disparate performance across user groups.The paper links these risks to statistical patterns and imbalance in pretraining data.
  • Justice and impartiality: Impartiality requires similar individuals to receive indistinguishable outputs when group identity is irrelevant.This operationalizes justice as treating similar individuals similarly.
  • Justice and impartiality: Justice also requires that claims about what people deserve reflect their actual desert rather than irrelevant group membership.The paper connects this consideration to meritocratic fairness and envy-freeness.
  • Stereotypes: The paper identifies gender, race, religion, sexual orientation, disability, socioeconomic status, and age as commonly vulnerable areas for stereotypes and discrimination.The list is explicitly non-exclusive, and age-related workplace prejudice is given as an example.
  • Stereotypes: Pretrained LLMs can acquire and amplify stereotypes from crowdsourced data, including toxicity toward protected groups.The paper argues that LLM-generated text must not exhibit or highlight such stereotypes.
  • Preference bias: LLM responses should remain neutral on political ideas, public figures, events, products, and other preference-sensitive topics.Suggested approaches include presenting alternative viewpoints with references and fine-tuning on human-written neutral answers.

7 Resistance to Misuse

Resistance to misuse concerns deliberate exploitation of LLMs for propaganda, cyberattacks, social engineering, and copyright extraction. The paper surveys alignment and system-level mitigations while noting that some misuse problems resist simple refusal-based strategies.

  • Propaganda: Malicious users may leverage LLMs to generate propaganda against targeted people, advocate terrorism, or create extreme political propaganda.The paper distinguishes these proactive misuses from unintentional safety failures such as hallucinated misinformation.
  • Propaganda: Alignment can mitigate propaganda misuse by training models to refuse such requests and provide factual, unbiased alternatives.The proposed data should include correct responses to requests for political propaganda.
  • Cyberattacks: LLMs’ low-cost, fast code generation can facilitate malicious cyberattacks, creating a serious societal threat.The paper discusses including cyberattack-related requests and preferred responses in alignment data.
  • Social engineering: LLMs can enhance social engineering by contextualizing deceptive messages, impersonating individuals, and supporting phishing, spam, bots, and fake content.These attacks typically target specific individuals for financial or security-compromising motives.
  • Social engineering: Social-engineering misuse is difficult to address through alignment alone because useful applications also require hypothetical role-playing and pretending to be experts.The paper therefore points to system-level measures such as fake-account and bot detection.
  • Copyright: LLM memorization can enable extraction of copyright-protected text, code, images, video, and audio from training data.Figure 22 illustrates GPT-3 completing a prompted passage from Harry Potter with another protected sentence.

8 Explainability and Reasoning

The paper surveys interpretability, general reasoning, and causal reasoning as components of trustworthy LLM behavior. It describes conversational and retrieval-based explanations but reports persistent limitations in faithful and robust human-like reasoning.

  • Lack of Interpretability: LLMs should explain their reasoning and provide transparency, especially in high-stakes applications such as medical diagnosis, hiring, and lending.The paper identifies black-box behavior as a central interpretability concern.
  • Lack of Interpretability: Retrieval-augmented LLMs provide explicit reference documents that users can inspect to assess output justification and transparency.The approach may be constrained by limited context length when many documents are retrieved.
  • General reasoning: Chain-of-thought explanations expose stepwise reasoning, while self-consistency and tree-of-thoughts explore multiple reasoning paths.The paper presents these methods as approaches for inspecting or improving reasoning.
  • General reasoning: Chain-of-thought prompting can improve question-answering accuracy compared with requesting an answer without an explanation.Tree-of-thoughts additionally allows backtracking and exploration of alternate chains.
  • General reasoning: LLMs may produce plausible explanations that do not reflect their actual reasoning, including invalid steps despite reaching correct conclusions.Controlled-bias studies found models omitted obvious shortcuts used in their answers.
  • General reasoning: GPT-3 barely outperforms random guesses on abductive reasoning, while GPT-4 solves 38% of the detective puzzles.Performance also drops on new datasets requiring logical reasoning, suggesting limited robustness across tasks.
  • Causal reasoning: Causal reasoning evaluates necessary and sufficient causes, counterfactuals, and consistent event changes in generated text.The paper describes a task requiring models to modify causal events to flip a sentence’s sentiment.
  • Causal reasoning: Fine-tuning with few-shot examples can improve causal-task accuracy, but the improvement is not robust to paraphrased templates or renamed variables.Without fine-tuning, models can barely outperform random guesses.

9 Social Norm

Social-norm alignment requires LLMs to avoid offensive language, respond sensitively to vulnerable users, and account for cultural and regional differences. The paper also distinguishes broadly shared social values from debatable preferences.

  • Social values: LLMs should avoid offensive language, handle destabilizing topics sensitively, and respond sympathetically to users seeking emotional support.These concerns overlap with safety and fairness but are treated as a distinct social-values category.
  • Social values: The paper limits its focus to values it judges broadly compatible with social good, while directing controversial preferences to its discussion of neutrality.It cautions that some social values remain debatable.
  • Toxicity: Toxicity includes rude, disrespectful, threatening, or identity-attacking language toward groups such as those defined by culture, race, or gender.The paper notes that no single unified characterization of toxicity exists.
  • Toxicity: Toxicity detectors can support alignment and serve as system-level filters that withhold detected toxic outputs from users.The paper cites Perspective and pretrained-language-model classifiers as examples of detection tools.
  • Toxicity: LLMs should use non-offensive language because hate speech can create social problems and is prohibited by policies in some jurisdictions.The paper presents automated generation as making offensive-language moderation more challenging.
  • Emotional awareness: Responsible LLMs should provide informative, sympathetic, and sensitive responses to vulnerable users seeking support.The paper argues for continuous monitoring and improvement of emotional awareness.
  • Emotional awareness: ChatGPT has been reported to generate supportive conversations with strong emotional awareness across scenarios involving anger, fear, happiness, and sadness.The reported evaluation covered several emotionally charged scenarios.
  • Cultural values: Cultural alignment requires locally collected datasets because users’ political, religious, and cultural responses differ across regions.GPT-3 reportedly performs significantly better with American context than with other cultures.

10 Robustness

LLM robustness is challenged by prompt errors, adversarial manipulation, changing knowledge, evolving policies, and feedback effects that can alter deployment behavior.

  • Robustness against typo attacks: Prompt typos and grammatical errors can make LLMs produce wrong or low-quality answers to otherwise identical questions.The paper illustrates this problem with GPT-3 responding differently when asked the same question with typos.
  • Robustness against prompt attacks: Adversarial prompt attacks can flip model classifications or elicit dangerous information that the model initially refused to provide.The paper also notes that adversarial prompt engineering can support red teaming and alignment-data generation.
  • Robustness to distribution and concept shifts: Changing knowledge bases require answers to be updated over time, while evolving policies can change whether content is considered acceptable.Examples include sports records, wealth rankings, and changing content-moderation treatment of LGBTQ-related identities.
  • Robustness to distribution and concept shifts: Shifting data distributions can create deployment-time fairness and policy-compliance violations despite validation on static training data.The paper describes this as creating a false sense of security before deployment.
  • Feedback effects: Human-feedback fine-tuning may contribute to feedback-loop bias because model interactions can alter the underlying data distribution.The concern is heightened by the frequency and scale of LLM interactions with users.
  • Potential responses: Strategic and performative prediction frameworks model users’ responses to deployment, while debiasing methods aim to mitigate feedback-loop effects.These approaches are presented as related technical directions for managing interventional effects.
  • Resistance to data poisoning: Internet-sourced LLM training data is vulnerable to poisoning, and prior work shows that very small poisoned-data fractions can induce arbitrary test-time misclassification.Reported examples include 0.1% poisoning in unlabeled data and 0.01% poisoning of a CLIP dataset.
  • Resistance to data poisoning: Potential defenses include identifying and removing high-impact training samples or iteratively training on samples with the smallest errors.The surveyed defenses draw on methods developed for traditional poisoning attacks.

11 Case Studies: Designs and Results

The paper operationalizes selected trustworthiness categories through measurement studies using automated or indirect evaluation procedures across multiple LLMs. Results generally associate stronger alignment with better hallucination, safety, and fairness performance, while confidence consistency can coexist with unfair answers and evaluation coverage is limited.

  • Study design: Eight subcategories are selected for measurement studies spanning reliability, safety, fairness, resistance to misuse, interpretability, and robustness.The selected topics include hallucination, general safety, gender stereotypes, miscalibration, misuse, copyrighted-content leakage, causal reasoning, and typo attacks.
  • Study design: The evaluation framework prioritizes scalable test-data generation because human labeling, while reliable, is not sufficiently fast or scalable.Specific measurement procedures vary according to each subcategory’s circumstances and requirements.
  • Reliability: Hallucination: Hallucination evaluation uses multiple-choice questions containing hallucinated and correct answers, counting hallucinated or irrelevant selections as hallucinations.Questions are drawn from a public hallucination benchmark.
  • Reliability: Hallucination: GPT-4 achieves the highest score on the evaluated hallucination task, while less-aligned models generate hallucinated outputs more frequently.The comparison covers six LLMs ordered from davinci to GPT-4 by alignment level.
  • Safety: Safety evaluation queries models with unsafe prompts and measures whether they refuse to answer or generate unsafe outputs.Prompt generation and response labeling can be automated with existing LLMs, with manual keyword cleaning included in the procedure.
  • Safety: GPT-3.5-turbo and GPT-4 reach refusal ratios close to 100% for unsafe prompts, with refusal likelihood generally increasing with alignment.The study evaluates 984 prompts across six LLMs.
  • Fairness: Gender stereotype: Gender-bias resistance correlates with alignment, but even GPT-4 refuses fewer than 60% of gender-stereotypical questions and often selects stereotypical options.The measured refusal ratio is lower than the reported 80–100% range for safety.
  • Reliability: Miscalibration: Miscalibration analysis compares external confidence expressed in generated text with internal first-token probabilities, but excludes models lacking suitable logits or coherent confidence responses.Among the compared models, text-davinci-003 has the most consistent external and internal confidence, although models can remain confidently unfair.

11.6 Propagandistic and Cyberattack Misuse

The evaluation examines whether aligned LLMs resist misuse across propaganda, terrorism, and cyberattacks, while also measuring copyright leakage and causal reasoning. Results show that alignment generally improves refusal behavior and causal reasoning, but instruction-following can increase misuse risks and does not eliminate leakage or reasoning inconsistencies.

  • Misuse evaluation: Four misuse tests target propaganda against people, terrorism advocacy, extreme political propaganda, and cyberattacks, using malicious prompts judged for refusal by GPT-4.The test LLM passes when it refuses unethical or harmful outputs.
  • Misuse results: ChatGPT and GPT-4 show much higher performance on resisting misuse tasks, while unaligned davinci and OPT-1.3B can outperform aligned models by failing to follow instructions.Manual inspection attributes the apparent advantage of unaligned models to their inability to complete the requested tasks, not greater trustworthiness.
  • Misuse results: More aligned models can be more likely to generate extreme and harmful political propaganda when they technically follow prompts without considering potential misuse.The paper describes this pattern as nearly opposite to the expected trend.
  • Misuse implications: Alignment focused on instruction-following may become a curse rather than a blessing for misuse resistance when trustworthiness is not also considered.The authors therefore distinguish usability gains from trustworthy behavior in misuse settings.
  • Copyright leakage: All tested LLMs emit text resembling copyrighted content, with no obvious correlation between copyright leakage and alignment degree.The authors attribute leakage more to whether copyrighted text appeared in training data than to alignment itself.
  • Copyright leakage: 18% of gpt-4 outputs and nearly 20% of davinci outputs have over 90% similarity to copyrighted text.Similarity above 90% means the emitted text often differs by fewer than two words.
  • Causal reasoning: Alignment training improves causal reasoning: text-davinci-003, ChatGPT, and GPT-4 outperform davinci on both causal questions.The second question is harder because it additionally tests counterfactual necessary-cause reasoning.
  • Causal reasoning: GPT-4 reaches less than 70% accuracy on the more difficult causal-reasoning question and shows inconsistency between generative and discriminative behavior.The paper notes that some GPT-4-generated event pairs may not strictly follow the stated causal relationship.

12 Conclusions and Challenges

The paper surveys LLM trustworthiness, proposes a detailed alignment taxonomy, and uses measurement studies to evaluate alignment across multiple dimensions. Its results support generally better performance for more aligned models, while emphasizing finer-grained evaluation, continued improvement, and acknowledged taxonomy and measurement limitations.

  • Contributions: The paper develops a detailed taxonomy of LLM trustworthiness and reviews proposed solutions before conducting measurement studies.The taxonomy provides the basis for detailed alignment measurements and an automated evaluation dataset.
  • Findings: More highly aligned LLMs generally perform better according to publicly claimed alignment efforts, but specific topics still leave room for improvement.The result is reported as a general tendency rather than uniform success across all topics.
  • Implications: Fine-grained alignment is important for attaining better coverage of trustworthiness across diverse dimensions.The conclusion links this need to continued research and efforts toward robust alignment.
  • Limitations: The taxonomy does not encompass the entire spectrum of LLM trustworthiness.The authors encourage iterative development of a more fine-grained and comprehensive framework.
  • Limitations: The measurement studies simplify some evaluations by converting questions into multiple-choice formats and relying on a more advanced model for automated assessment.These choices reduce the need for extensive human labeling but constrain the evaluation design.
  • Open Problems: Alignment science remains in its early stages, with unresolved theoretical and practical questions including whether RLHF is optimal.The paper presents these issues as open problems rather than settled conclusions.
  • Conclusion: The field needs more principled methods for evaluating and implementing alignment so models adhere to societal values and ethical considerations.The conclusion frames this as important for increasingly trustworthy and responsible deployment.

A Evaluation Categories in Anthropic Red-team Dataset

The appendix includes the label distribution of Anthropic’s red-teaming data as an evaluation-category overview.

  • A Evaluation Categories in Anthropic Red-team Dataset: Figure 46 presents the label distribution of Anthropic’s red-teaming data.It provides an overview of the categories represented in the dataset.

B Additional Examples of the Generated Test Prompts

The appendix supplies additional examples from the paper’s alignment measurements, spanning test prompts, safety and fairness, uncertainty, misuse, copyright, causal reasoning, robustness, and before–after alignment outputs.

  • B Additional Examples of the Generated Test Prompts: Additional generated test prompts and examples cover hallucination, safety, and fairness evaluations.Figures 47–49 provide further examples for these categories.
  • B.4 Examples from Testing Uncertainty (Section 11.5): Additional examples illustrate miscalibration and uncertainty tests, including outputs that assign probability to an unavailable option.The appendix references Figures 50 and related uncertainty examples.
  • B.5 Examples from Testing Misuse (Section 11.6): Misuse examples cover harmful misinformation about celebrities, terrorism advertisements, extreme political propaganda, and cyberattack coding assistance.These examples correspond to Figures 51–54.
  • B.6 Examples from Testing Copyright Leakage (Section 11.7): The appendix includes additional leaked-copyright examples and causal-reasoning test examples.Copyright examples appear in Figure 55, while causal-reasoning examples appear in Figures 56–61.
  • B.9 Examples from Testing Alignment (Section 11.10): Additional alignment examples show outputs before and after performing alignment with generated safety data.These examples are listed in Figure 63.
  • B.7 Examples from Testing Causal Reasoning (Section 11.8): Causal-reasoning examples use contexts, cause questions, and constrained A/B answers, with some outputs choosing an unrelated C option.The examples include supermarket–dinosaur-fossil scenarios and gender-stereotype questions with refusal options.
  • B.8 Examples from Testing Robustness (Section 11.9): The appendix presents examples of robustness testing against typo attacks.Figure 62 provides additional robustness examples.

B.8 Examples from Testing Robustness (Section 11.9)

Figure 62 provides additional test examples for evaluating robustness against typo attacks.

  • Figure 62 lists additional test examples.
  • The examples support robustness evaluation.
  • The tested threat is typo attacks.
Loading 2308.05374v2…