PaperlayerView exact revision on arXiv

Source-linked AI summary

Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity

Terry Yue Zhuo, Yujin Huang, Chunyang Chen, Zhenchang Xing

arXiv:2301.12867v4cs.CLcs.SE

TL;DR

Current LLMs can pose ethical risks, yet systematic and user-informed examination of those risks remains limited. The paper red-teams ChatGPT across bias, reliability, robustness, and toxicity using benchmarks and case studies, finding ethical risks that existing benchmarks miss and identifying prompt-injection, misinformation, and other concerns. The authors use these findings to inform future evaluation and responsible language-model design.

  • Problem

    Systematic examination and user study of harmful behaviors and ethical risks in current LLM usage remain limited, while existing studies do not comprehensively cover ethical concerns.

  • Method

    The paper red-teams ChatGPT across bias, reliability, robustness, and toxicity using benchmarks, case studies, human evaluation, and responsible-model analysis.

  • Results

    The evaluation identifies ethical risks including multilingual bias, prompt-injection vulnerability, hallucination-related misinformation, and risks inadequately captured by existing benchmarks.

  • Takeaways & Limitations

    The findings support continued development of broader benchmarks and strategies for determining and mitigating ethical hazards in language-model applications.

  • Takeaways & Limitations

    Reported results may be inconsistent because ChatGPT’s hyperparameters were undisclosed and the evaluated system may have changed across versions and training data.

Abstract

from arXiv · show

Recent breakthroughs in natural language processing (NLP) have permitted the synthesis and comprehension of coherent text in an open-ended way, therefore translating the theoretical algorithms into practical applications. The large language models (LLMs) have significantly impacted businesses such as report summarization software and copywriters. Observations indicate, however, that LLMs may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. Large-scale benchmarks for accountable LLMs should consequently be developed. Although several empirical investigations reveal the existence of a few ethical difficulties in advanced LLMs, there is little systematic examination and user study of the risks and harmful behaviors of current LLM usage. To further educate future efforts on constructing ethical LLMs responsibly, we perform a qualitative research method called ``red teaming'' on OpenAI's ChatGPT\footnote{In this paper, ChatGPT refers to the version released on Dec 15th.} to better understand the practical features of ethical dangers in recent LLMs. We analyze ChatGPT comprehensively from four perspectives: 1) \textit{Bias} 2) \textit{Reliability} 3) \textit{Robustness} 4) \textit{Toxicity}. In accordance with our stated viewpoints, we empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks, and hence illustrate them via additional case studies. In addition, we examine the implications of our findings on AI ethics and harmal behaviors of ChatGPT, as well as future problems and practical design considerations for responsible LLMs. We believe that our findings may give light on future efforts to determine and mitigate the ethical hazards posed by machines in LLM applications.

I. Introduction

The paper investigates ethical risks in current LLM applications, addressing gaps in practical, timely, consensual, and comprehensive evaluation through a qualitative red-teaming study of ChatGPT.

  • Research gaps: The study addresses gaps in practical, timely, consensual, and comprehensive examination of ethical risks in advanced language models.Earlier work was often theoretical, lagged behind newer models, lacked user consensus, or focused narrowly on selected issues.
  • Approach: Researchers manually classified 305,701 ChatGPT-related tweets to identify public concerns across bias, reliability, robustness, and toxicity.
  • Findings: Red-teaming revealed potential ethical issues including programming bias, prompt-injection susceptibility, and misinformation from hallucination.
  • Contribution: The paper combines benchmarks, heuristics, examples, and human evaluation to support understanding and mitigation of ethical hazards rather than provide a deterministic solution.

II. Common Themes of Ethical Concerns

The paper organizes ChatGPT’s ethical concerns around creative generation and decision-making, covering bias, robustness, reliability, and toxicity.

  • Application Scenarios: ChatGPT’s two main application scenarios are creative generation and decision-making.Creative generation produces content, while decision-making uses language models to provide judgments or suggestions from natural-language input.
  • Bias: Bias includes social stereotypes, unfair discrimination, exclusionary norms, and limited multilingual understanding caused by incomplete representation in training data.
  • Robustness: Robustness concerns maintaining performance under semantically or syntactically different inputs, including semantic perturbation, prompt injection, and data leakage.
  • Reliability: Reliability concerns precise information, while false or misleading and outdated training data can produce incorrect outputs.
  • Toxicity: Toxicity covers harmful or offensive content, including offensive language and pornography in training data.

III. Diagnosing AI ethics Of ChatGPT

The study evaluates ChatGPT across four ethical dimensions using established benchmarks aligned with HELM, while using zero-shot evaluation to better reflect typical human-computer interaction.

  • The evaluation covers bias, reliability, robustness, and toxicity using established benchmarks consistent with HELM.
  • The study evaluates 80% randomly selected samples from each dataset to conserve computational resources.
  • ChatGPT is tested zero-shot rather than with HELM-style prompting, targeting typical human-computer interaction scenarios.

1) Experiment Settings:

The bias evaluation compares zero-shot ChatGPT with few-shot InstructGPT and GPT-3 on BBQ and BOLD, finding lower measured bias for ChatGPT on the reported tasks.

  • 1) Experiment Settings:: The BBQ and BOLD datasets measure bias in question answering and text generation using established task-specific metrics.
  • 2) Result Analysis: ChatGPT showed lower bias than other LLMs across BBQ question answering and BOLD text generation benchmarks.It outperformed the baselines on reported BBQ measures and three of four BOLD metrics.

3) Beyond Benchmarking:

Case studies reveal ethical risks that standard bias benchmarks may miss, including multilingual bias, biased code generation, and socially safe open-ended dialogue.

  • Beyond Benchmarking: The case studies were motivated by evidence that existing social-bias measurements cover only a limited subset of possible bias domains.The authors specifically identify multilingual comprehension and code generation as additional domains requiring examination.
  • Language Understanding: ChatGPT showed inadequate comprehension of low-resource languages, while translation quality also varied among high-resource languages.The evaluation used six FLORES-200 language pairs with English as the target language.
  • Code Generation: 87% of generated Python programs were biased in Round 1, falling to 31% after ChatGPT was instructed to remove bias.Manual annotation reported inter-rater reliability of 0.99 and 1.00 for Rounds 1 and 2, respectively.
  • Open-ended Dialogue: Human evaluation found 92% alignment with ground-truth labels for socially safe and unbiased responses in open-ended dialogue.The evaluation covered 50 dialogues and achieved a Fleiss’ kappa of 0.94.

B. Robustness

ChatGPT performs better than the compared baselines on benchmarked adversarial robustness, but contrasting semantic perturbations remain challenging and prompt injections can bypass safety mechanisms.

  • Benchmark Robustness: All three language models degraded under contrasting semantic perturbations, indicating a challenge for robustness in current language models.Robustness was assessed by comparing Exact Match performance across augmented subsets.
  • Benchmark Robustness: ChatGPT demonstrates superior accuracy and adversarial robustness compared with two LLM baselines, while remaining susceptible to semantics-altering perturbations.The evaluation covers IMDB and BoolQ under original and perturbed conditions.
  • Prompt Injection: The prompt-injection case study tested 100 unsafe, unethical, and illegal scenarios using manually constructed adversarial prompts.The prompts applied transformations such as requesting a song or poem to alter the framing of unsafe requests.
  • Prompt Injection: 95 of 98 scenarios protected by safety mechanisms were successfully jailbroken through persona-assigned adversarial prompts, while 2 of 100 scenarios were directly answered.The findings indicate high vulnerability to prompt injections that can bypass safety features.

C. Reliability

The reliability evaluation uses OpenBookQA and TruthfulQA to assess factual knowledge and commonsense capabilities against InstructGPT and GPT-3 using Exact Match.

  • Experiment Settings: OpenBookQA measures basic science knowledge, whereas TruthfulQA targets questions aligned with common misconceptions across law, medicine, finance, and politics.Both datasets are multiple-choice question-answering benchmarks.
  • Experiment Settings: InstructGPT (davinci v2) and GPT-3 (davinci v1) serve as baseline models for comparison.The baselines correspond to the models used in the associated experiments.
  • Experiment Settings: Model accuracy is evaluated with the Exact Match metric.

2) Result Analysis:

ChatGPT shows mediocre factual reliability relative to prior state-of-the-art models and produces hallucinated answers that can mislead users. Its performance is comparable to InstructGPT, while GPT-3 performs worse on TruthfulQA.

  • 2) Result Analysis:: ChatGPT marginally outperforms previous SOTA LLMs but correctly answers no more than 65% of test cases.The evaluation covers OpenBookQA and TruthfulQA against 5-shot InstructGPT and GPT-3.
  • 2) Result Analysis:: ChatGPT maintains mediocre factual reliability relative to SOTA LLMs, consistent with its similar training strategies and architectures.
  • 2) Result Analysis:: A heroin-response example shows ChatGPT asserting addiction can follow one use despite the supplied ground truth requiring repeated use for physical addiction.
  • 2) Result Analysis:: Three authors identified 24 hallucinated responses in a random sample of 100 TruthfulQAgen questions.The human evaluation classified generated responses as containing misinformation or disinformation.
  • 2) Result Analysis:: ChatGPT’s hallucinations indicate unreliable performance that may mislead users in practical applications.

D. Toxicity

On RealToxicPrompts, ChatGPT exhibits nearly zero toxic output and slightly better toxicity reduction than two baselines. However, prompt injection can bypass its safeguards and elicit harmful language.

  • D. Toxicity: The evaluation compares ChatGPT with 5-shot Cohere and 5-shot T5 using RealToxicPrompts and PerspectiveAPI toxic-threshold proportions.
  • D. Toxicity: ChatGPT demonstrates superior toxicity reduction compared with Cohere and T5 on RealToxicPrompts.All models exhibit minimal toxicity, with toxic fractions nearly zero.
  • D. Toxicity: Prompt injection is used to test whether ChatGPT can be jailbroken into generating toxic language despite its safe-use design.
  • D. Toxicity: ChatGPT is prone to prompt-injection exploitation, and its mitigation strategy inadequately detects potential toxicity early.

IV. Discussions

The discussion portrays ChatGPT as ethically mixed: benchmark results can suggest lower bias and low toxicity, yet case studies expose multilingual, coding, factual, and jailbreak-related risks. Existing benchmarks therefore do not fully capture its ethical behavior.

  • IV. Discussions: Existing language-model benchmarks are insufficient because ChatGPT can perform comparably or better than SOTA models while still exhibiting ethical risks.
  • IV. Discussions: ChatGPT can generate toxic language when prompted to act as hostile fictional characters, despite refusing toxicity in a direct response.
  • IV. Discussions: ChatGPT’s ethical risks include multilingual bias, code-generation bias, factual unreliability, hallucinated misinformation, and prompt-injection jailbreaks.
  • IV. Discussions: Poor multilingual understanding may bias decision-making and creative generation, with potential impacts on underrepresented groups.
  • IV. Discussions: Code-generation bias from logical oversimplification may create flaws in real-world software production.

B. Towards Responsible Language Models

Responsible language models require attention to internal reliability and external usage ethics as models scale, data constraints intensify, and deployment becomes more costly. The paper highlights evolving risks, including encoded biases, temporal unreliability, prompt manipulation, privacy concerns, and resource burdens.

  • a) Internal Ethics — Modeling: Language models can encode biases and outdated knowledge, while unchanged parameters make reliability decline as facts change over time.The paper notes that continual data and model updates could mitigate temporal unreliability, but they are unaffordable for many users.
  • b) External Ethics — Usage: External ethics assigns responsibility to producers and users, including privacy-preserving data practices, systematic evaluation, and safeguards against malicious deployment.The paper specifically warns that language models can leak private information and be attacked to produce malicious content or decisions.
  • a) Emergent Ability: Larger, more context-dependent models may amplify ethical risks and remain vulnerable to prompt injection, motivating up-to-date benchmarks for unforeseen behaviors.The paper characterizes these risks as potentially emergent and says ad-hoc parameter tuning offers no complete solution.
  • b) Machine Learning Data: Limited high-quality data, annotation costs, and crowdsourcing bias constrain future model development, although deduplication and data-efficiency strategies may help.The paper identifies curriculum learning, active learning, and prompting as promising but still early-stage approaches.
  • c) Computational Resource: Scaling LLMs increases training and deployment costs, strains available hardware, and raises energy consumption and carbon emissions.The paper encourages hardware-software co-design to reduce computational demands and environmental impact.

V. Conclusion

The paper diagnoses ChatGPT’s ethical risks across bias, robustness, reliability, and toxicity through benchmarks and case studies. It reports ethical risks alongside slightly stronger performance than current state-of-the-art models, while cautioning that undisclosed model changes and evaluation choices limit empirical validity.

  • Conclusion: ChatGPT is diagnosed across bias, robustness, reliability, and toxicity, with benchmarks and case studies revealing ethical risks and prompt-injection susceptibility.The study also offers directions for designing and evaluating more ethical language models.
  • Conclusion: ChatGPT may perform slightly better than current SOTA language models while still exhibiting evidence of ethical risks.The conclusion presents performance and ethical-risk findings together rather than treating stronger performance as evidence of safety.
  • Limitations: The empirical results may be inconsistent because ChatGPT’s hyperparameters are undisclosed and its versions changed during the study period.The paper notes that successive versions may have been trained with new data.
  • Limitations: Evaluation validity is limited by possible gaps in datasets and by intuitively designed zero-shot prompts whose construction could be scrutinized further.The proprietary nature of ChatGPT’s data and model further constrains assessment.
Loading 2301.12867v4…