PaperlayerView exact revision on arXiv

Source-linked AI summary

Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems

Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, Zhixing Tan, Junwu Xiong, Xinyu Kong, Zujie Wen, Ke Xu, Qi Li

arXiv:2401.05778v1cs.CLcs.AI

TL;DR

LLM systems face safety and security risks across language models, deployment tools, and generated content. This survey organizes risks and mitigation strategies by system module and reviews assessment benchmarks. The reviewed studies report varied findings, including robustness comparisons and differences in hallucination and privacy outcomes.

Abstract

from arXiv · show

Large language models (LLMs) have strong capabilities in solving diverse natural language processing tasks. However, the safety and security issues of LLM systems have become the major obstacle to their widespread application. Many studies have extensively investigated risks in LLM systems and developed the corresponding mitigation strategies. Leading-edge enterprises such as OpenAI, Google, Meta, and Anthropic have also made lots of efforts on responsible LLMs. Therefore, there is a growing need to organize the existing studies and establish comprehensive taxonomies for the community. In this paper, we delve into four essential modules of an LLM system, including an input module for receiving prompts, a language model trained on extensive corpora, a toolchain module for development and deployment, and an output module for exporting LLM-generated content. Based on this, we propose a comprehensive taxonomy, which systematically analyzes potential risks associated with each module of an LLM system and discusses the corresponding mitigation strategies. Furthermore, we review prevalent benchmarks, aiming to facilitate the risk assessment of LLM systems. We hope that this paper can help LLM participants embrace a systematic perspective to build their responsible LLM systems.

I. INTRODUCTION

LLM systems raise safety and security concerns, while existing risk taxonomies largely focus on output content and provide limited coverage across system modules. This paper proposes a module-oriented taxonomy and surveys risks, mitigations, and assessment benchmarks; it also reviews LLM training and system background.

I. INTRODUCTION

  • LLM systems can leak user chat histories, respond harmfully to adversarial prompts, or generate untruthful, toxic, biased, and illegal content.The paper notes these contents may be abused, with adverse social impacts.Source: §I · p. 1
  • Most existing taxonomies assess output content, while few systematically categorize risks across the input, language-model, toolchain, and output modules.The proposed taxonomy aims to help participants understand module-specific concerns and build more responsible systems.Source: §I · pp. 1–2
  • The module-oriented taxonomy classifies risks and mitigation strategies by system module, helping developers identify where to focus attention.The paper also reviews risk-assessment benchmarks and prevalent LLM systems.Source: §I · p. 2

II. BACKGROUND

  • LLMs have billions or more parameters and are pretrained on massive data; the paper describes pretraining, supervised fine-tuning, and learning from human feedback as typical development steps.The scaling law is described as linking increased pretraining data and model size with improved downstream-task capacity.Source: §II · p. 2
  • Pretraining uses large, mixed-source corpora, preprocessing, and tokenization; maximizing token-sequence likelihood equips the model with language understanding and generation abilities.Specialized code, multilingual, and scientific data may be included to enhance reasoning and task-solving abilities.Source: §II · p. 2
  • Supervised fine-tuning trains on smaller, curated examples to enable the model to handle prompts for multiple downstream tasks.Instruction tuning is identified as the most popular recent fine-tuning method.Source: §II · pp. 2–3
  • Learning from human feedback uses preference-ranked outputs to train a reward model, then applies reinforcement learning to fine-tune the language model.The paper notes that RLHF can be complex and unstable, and describes ranking objectives and natural-language preferences as alternatives.Source: §II · p. 3

III. MODULES OF LLM SYSTEMS

The survey describes four LLM-system modules and the risks associated with them. It organizes module-specific risks and mitigation strategies and reviews benchmarks for safety and security assessment.

III. MODULES OF LLM SYSTEMS

  • An LLM system comprises an input module, language-model module, toolchain module, and output module.The taxonomy framework covers 12 specific risks and 44 sub-categorized risk topics across the four modules.Source: §III · p. 3, Figure 3 · p. 4
  • The input module receives and preprocesses prompts, while the language-model module comprises training data and the trained model.Source: §III · p. 3
  • The toolchain module includes development and deployment utilities, such as software tools, hardware platforms, and external tools.Source: §III · p. 3
  • The output module returns system responses and may use a safeguard to revise content for ethical soundness and justification.The survey covers risks including unsuitable or adversarial prompts, tool vulnerabilities, and harmful or dishonest outputs, and summarizes assessment benchmarks.Source: §III · pp. 3–4

IV. RISKS IN LLM SYSTEMS

The survey categorizes LLM-system risks by module and distinguishes unsafe-topic prompts from adversarial prompts in its input-module discussion.

A. Risks in Input Modules

  • Input prompts may contain unsafe topics that elicit offensive or biased content, or adversarial content with a clear attack intention.Unsafe prompts can conceal harmful opinions, making them harder to identify; adversarial prompts include prompt injection and jailbreaking.Source: A. Risks in Input Modules · pp. 4–5

Prompt Injection

  • Prompt injection includes goal hijacking, which redirects a prompt’s task, and prompt leaking, which seeks to expose confidential instructions.Goal-hijacking text can also enter through data or retrieved web pages, while prompt leaking is reported as more challenging than goal hijacking.Source: A. Risks in Input Modules · p. 5

Jailbreaking

  • Jailbreaks craft scenarios to elicit policy-violating content, ranging from one-step prompt modifications to multi-step conversational attacks.One-step methods include role-playing, integration, and obfuscation; multi-step methods use request contextualizing or external assistance.Source: A. Risks in Input Modules · p. 5

A. Risks in Input Modules

B. Risks in Language Models

The survey describes language-model risks arising from training data, generation behavior, and attacks, including privacy leakage, toxicity and bias, hallucinations, and model vulnerabilities.

Privacy Leakage

  • Private information in web-collected corpora or personal conversations used for fine-tuning can be exposed, and model associations may leak PII beyond memorization.The survey identifies web collection and alignment with human-machine conversations as settings where private data can enter training corpora.Source: B. Risks in Language Models · pp. 6–7

Toxicity and Bias Tendencies

  • About 0.2% of LLaMA2’s pre-training documents were recognized as toxic, while filtering toxic data can reduce model performance.The survey defines toxic data to include hate speech, offensive utterances, profanities, and threats.Source: B. Risks in Language Models · p. 7
  • Training-data bias is subjective and context-dependent; pronoun imbalances can affect gender associations, while stereotypes may remain hidden in benign corpora.The survey notes that deciding what counts as a stereotype in a corpus remains an open problem.Source: B. Risks in Language Models · p. 7

Hallucinations

  • Hallucinations include unsupported additions to supplied material and incorrect real-world claims, reducing system reliability.The survey notes that hallucinations on conceptual knowledge become more serious as model size increases.Source: B. Risks in Language Models · p. 7
  • Knowledge gaps, noisy training data, and false recall can produce hallucinations when models lack, mislearn, or misretrieve relevant information.False recall can reflect confusion from co-occurrence, positional patterns, duplicated data, or similar named entities.Source: B. Risks in Language Models · pp. 7–8
  • Consistency-seeking can repeat user misinformation or false demonstrations, while autoregressive decoding and sampling can accumulate errors or add randomness.The survey links false-demonstration copying to induction heads and identifies top-p and top-k sampling as potential sources of randomness.Source: B. Risks in Language Models · p. 8

Vulnerability to Model Attacks

  • Model attacks include extraction, inference, poisoning, evasion, and overhead attacks, targeting model replication, training-data inference, behavior, predictions, or resource use.Poisoning may implant hidden triggers, while evasion perturbs test samples and overhead attacks can maximize energy consumption.Source: B. Risks in Language Models · p. 8
  • Studies report adversarial-prompt vulnerabilities, training-data extraction, textual backdoors, reward-model poisoning, and attacks using LLM-generated samples.Reportedly affected models include Flan-T5, BLOOM, ChatGPT, GPT-4, GPT-2, and GPT-Neo; prompt abstraction is another described scenario.Source: B. Risks in Language Models · pp. 8–9

C. Risks in Toolchain Modules

The survey examines toolchain risks across software development, hardware platforms, and external tools used in LLM services.

Software Development Tools

  • Software-development risks span vulnerable runtimes, CI/CD supply-chain exposures, deep-learning frameworks, and preprocessing tools in LLM workflows.Examples include interpreter denial-of-service and information-leakage vulnerabilities, exploitable CI plugins, framework crashes or code execution, and image-processing attacks.Source: C. Risks in Toolchain Modules · p. 9

Hardware Platforms

  • GPU side channels can expose trained-model parameters, while memory attacks can manipulate model parameters; mitigation transfer to large LLMs remains uncertain.Secure execution environments have been designed to mitigate GPU side-channel risks.Source: C. Risks in Toolchain Modules · p. 10
  • Distributed training networks face burst-traffic disruption and congestion during gradient transmission between GPU server nodes.Source: C. Risks in Toolchain Modules · p. 10

External Tools

  • Unreliable external-tool outputs can introduce factual errors into prompts and amplify hallucinations.The survey identifies public resources such as web APIs and search engines as sources of added knowledge.Source: C. Risks in Toolchain Modules · p. 10
  • Malicious tool instructions or manipulated public data can expose sensitive information, trigger injection attacks, or produce targeted malicious outputs.Unverified tool inputs may also result in arbitrary code execution.Source: C. Risks in Toolchain Modules · p. 10

D. Risks in Output Modules

The output module may pass harmful, untruthful, or unhelpful generated content to users, motivating review before export.

Harmful Content

  • Generated outputs may contain biased, toxic, or private information, with examples ranging from occupational gender associations to disclosed business secrets.The survey also describes toxic outputs elicited by role-playing prompts.Source: D. Risks in Output Modules · pp. 10–11

Untruthful Content

  • Untruthful outputs include factual errors and summaries that conflict with their source articles.The survey relates these factuality and faithfulness errors to language-model hallucination.Source: D. Risks in Output Modules · p. 11

Unhelpful Uses

  • Improper uses include academic misconduct, copyright violations, cyberattacks, vulnerable generated code, and consequentially poor legal or medical advice.The survey notes that reliance on LLMs may affect students’ independent thinking and that professional users may lack deeper subject-matter understanding.Source: D. Risks in Output Modules · p. 11

V. MITIGATION

The mitigation survey organizes defenses around risks in LLM-system modules, including input safeguards and language-model protections. It describes prompt design and detection, privacy and safety training, hallucination mitigation, and defenses against model attacks.

A. Mitigation in Input Modules

  • Input defenses modify prompts or detect malicious inputs, addressing varied harmful prompts and adversarial attacks through complementary strategies.Prompt design includes safety preprompts, reordered prompts, and alternative input formats; detection includes keyword matching and classifiers.Source: A. Mitigation in Input Modules · pp. 11–12
  • Keyword matching uses blocklists or allowlists, but natural-language paraphrases limit rule-based detection; classifiers can use latent-prediction features or an LLM detector.The latent-prediction features are described as potentially improving detector interpretability.Source: A. Mitigation in Input Modules · p. 12

B. Mitigation in Language Models

  • Privacy defenses intervene on training data or use differential privacy, which protects information in neighboring datasets but can degrade model performance.Data interventions include PII cleansing, deduplication, and text sanitization; utility-enhancing techniques seek a better privacy-utility trade-off.Source: B. Mitigation in Language Models · p. 13
  • Detoxification and debiasing use data filtering or safety training, but simple data interventions may raise language-model loss or filter demographic groups.Safety training includes style-transfer fine-tuning, adversarial learning, and reinforcement learning from human feedback.Source: B. Mitigation in Language Models · p. 13
  • Hallucination mitigation includes improving training-data quality, adding external evidence, changing decoding, and using multi-agent debate, which can be computationally expensive.The passage also notes that RLHF can improve factuality, while reward hacking and complex, unstable training procedures remain concerns.Source: B. Mitigation in Language Models · pp. 13–14
  • Extraction-attack defenses restrict or perturb query responses, use warnings, or watermark models; disruption-based defenses can cost performance and may be circumvented.Warning-based methods measure distances between continuous queries, while watermarking aims to establish ownership of stolen models.Source: B. Mitigation in Language Models · p. 14
  • Inference attacks can be addressed with differential privacy, regularization, or adversarial training, while poisoning defenses detect suspicious samples or reverse-engineer backdoor triggers.Source: B. Mitigation in Language Models · p. 14
  • Evasion defenses train robust models or detect adversarial inputs; overhead-attack defenses limit energy use and monitor inputs, APIs, and resources.The survey notes that conventional deep-learning defenses may also be applied to several LLM model-attack types.Source: B. Mitigation in Language Models · pp. 14–15

C. Mitigation in Toolchain Modules

Toolchain mitigations address software, hardware, deployment, and external-tool risks. The section describes control-flow protections, provenance analysis, sanitization, and execution restrictions, alongside practical overhead and privacy concerns.

C. Mitigation in Toolchain Modules

  • Control-flow integrity can prevent exploitation of software-tool vulnerabilities, but its overhead for large-scale LLM software motivates lower-precision and hardware-optimized versions.Source: C. Mitigation in Toolchain Modules · p. 15
  • Data provenance graphs support security forensics and attack detection, but LLM settings make provenance costly in computation, storage, latency, privacy, and interpretability.Graph vertices represent file descriptors and edges represent their relationships, such as system calls; the survey says provenance for LLM systems remains challenging.Source: C. Mitigation in Toolchain Modules · p. 15
  • External-tool risks can be reduced by using trusted or multiple tools, validating and sanitizing inputs, isolating execution, and applying least privilege.Source: C. Mitigation in Toolchain Modules · p. 16
  • Tool-interaction privacy defenses sanitize documents or detect and anonymize sensitive information, while ethical API-use guidelines address prompt injection and data leakage.Source: C. Mitigation in Toolchain Modules · p. 16

D. Mitigation in Output Modules

Output safeguards detect undesirable generated content, intervene on harmful or untruthful responses, and watermark generated text. The survey notes trade-offs involving user experience, computational costs, and watermark robustness.

D. Mitigation in Output Modules

  • Output safeguards detect sensitive, harmful, or untruthful content using rules, neural detectors, blocklists, classifiers, fact checks, or response consistency.Neural detectors can better identify cryptic harmful information; factuality detectors use external knowledge or context, while consistency methods probe multiple responses.Source: D. Mitigation in Output Modules · p. 16
  • Detected harmful content may be withheld, while untruthful content can be corrected with external evidence or by selecting among repeated responses, at additional computational cost.The section cautions that safety interventions should balance safety and user experience, since vocabulary-based filtering may be unsuitable for medical tasks.Source: D. Mitigation in Output Modules · p. 16
  • Watermarks can identify generated text and support ownership verification, but visible identifiers are easy to remove and hidden-pattern methods can be broken.Source: D. Mitigation in Output Modules · pp. 16–17

VI. RISK ASSESSMENT

The section reviews benchmarks for assessing LLM robustness, truthfulness, ethical issues, and bias, with robustness evaluation distinguishing adversarial and out-of-distribution settings.

A. Robustness

  • Robustness benchmarks assess adversarial attacks and performance on out-of-distribution samples not seen during training.The section focuses on robustness, truthfulness, ethical issues, and bias in existing LLM evaluations.Source: A. Robustness · p. 17, §VI · p. 17
  • PromptBench evaluations across several tasks reveal insufficient robustness to adversarial prompts, including spelling errors and synonym substitutions.The evaluations cover sentiment analysis, linguistic reasoning, reading comprehension, machine translation, and mathematical problems.Source: A. Robustness · p. 17
  • GLUE-X evaluations report a significant performance decrease on out-of-distribution samples, while BOSS finds positive correlations between in-distribution performance and OOD robustness.For domain-specific LLMs, fine-tuning is reported to enhance OOD robustness.Source: A. Robustness · p. 17
  • Evaluations using existing datasets report that ChatGPT exhibits superior robustness compared with other models.Source: A. Robustness · p. 17

B. Truthfulness

The section surveys ways to define and assess LLM hallucinations, including datasets, evaluation metrics, and studies of model behavior. Reported evaluations find differences across hallucination types and weaknesses in handling input-conflicting hallucinations.

B. Truthfulness

  • LLM hallucinations are meaningless or false information misaligned with inputs, classified as input-, context-, or fact-conflicting.Source: B. Truthfulness · pp. 17–18
  • HaDes supports token-level detection, Wikibro uses annotated article paragraphs, Med-HALT targets medical reasoning and context-conflicting hallucinations, and HaluEval covers multiple hallucination types.Source: B. Truthfulness · p. 18
  • Levy/Holt evaluates comprehension and hallucination using premise-hypothesis questions, while TruthfulQA targets fact-conflicting hallucinations with questions and correct or incorrect answers.Source: B. Truthfulness · p. 18
  • Concept-7 classifies potential hallucinatory instructions rather than classifying hallucinations themselves.Source: B. Truthfulness · p. 18
  • Hallucination assessment uses statistical overlap or contradiction measures, model-based comparisons, and manual evaluation because general content-quality metrics may be unsuitable.Model-based approaches include information extraction, question answering, language reasoning, and language-model metrics.Source: B. Truthfulness · p. 18
  • Studies report that ChatGPT performs better on internal than external hallucinations, while external hallucinations occur across tasks and input-conflicting hallucinations remain difficult for ChatGPT.In a medical-domain evaluation, two hallucination types account for 27% and 43% of GPT-3.5 hallucinations, 25% and 33% for GPT-4, and 8% and 44% for Bard.Source: B. Truthfulness · p. 18

C. Ethics

The section surveys ethical risks and their assessment, including toxicity, privacy leakage, and copyright concerns. Evaluations describe ChatGPT's responses to hate speech and report privacy leakage across model training and inference settings.

C. Ethics

  • Ethical concerns include toxic outputs, privacy leakage involving personal data, data-collection policies that may violate GDPR, and copyrighted training content.Source: C. Ethics · p. 18
  • Privacy studies report personal-data leakage by GPT-3.5 and GPT-4 in zero-shot settings, more leakage with lengthy contextual prompts, and GPT-4 avoiding PII leakage when privacy directives are followed.Both GPT-3.5 and GPT-4 are reported unable to protect all sensitive keywords effectively.Source: C. Ethics · pp. 18–19
  • Ethics benchmarks include toxicity prompts, false-statement detection, hate-speech annotations, broad toxicity and value-alignment assessments, and Chinese offensive-content evaluation.Source: C. Ethics · p. 19
  • SafetyPrompts supplies Chinese test prompts for ethical issues, while CValues evaluates LLM alignment with human values.Source: C. Ethics · p. 19
  • Studies report that ChatGPT cannot recognize hate speech and that automatically generated content evaluations reveal implicit hate speech issues.Questionnaires and personality tests have also been used to assess ChatGPT; studies describe it as open and gregarious, with rare indications of dark traits.Source: C. Ethics · p. 19

D. Bias

The section surveys social-bias evaluation and identifies future research topics in input monitoring and training-data intervention. It describes limitations of current approaches and calls for more effective methods.

D. Bias

  • Training data may contain bias that contributes to social biases in model outputs, categorized across gender, race, religion, occupation, politics, and ideology.Source: D. Bias · p. 19
  • Questionnaire studies report political and American-culture bias in ChatGPT, while other evaluations identify ethical issues that vary by region.Source: D. Bias · p. 19
  • Using NLP-generated evaluation content can leave bias assessment unresolved because those models may themselves be biased; red teaming is another evaluation approach.Some studies use sophisticated input generation for red-team evaluations.Source: D. Bias · p. 19

VII. FUTURE WORK

  • The section identifies LLM safety and security as topics for potential future research.Source: §VII · p. 19

A. Comprehensive Input Monitoring Approaches

  • Malicious-prompt classifiers may underperform with few examples and predefined classifier sets may miss new attacks, motivating semi-supervised or unsupervised detection and red-teaming models.Source: A. Comprehensive Input Monitoring… · p. 19

B. More Efficient and Effective Training Data Intervention

C. Interpretable Hallucination Mitigation

The discussion identifies unresolved challenges in hallucination interpretability, broad model defenses, agent security, watermarking, and evaluation, alongside practical mitigation directions. It concludes with a module-oriented taxonomy intended to help participants identify risks and choose mitigation strategies.

C. Interpretable Hallucination Mitigation

  • Interpreting how hidden-neuron activations relate to hallucinations may help explain why and how LLMs produce them, supporting mitigation research.The authors call for further exploration of this direction because hallucination remains an important issue.Source: C. Interpretable Hallucination… · p. 20

D. General Defense Framework against Model Attacks

  • A universal defense framework for conventional and emerging model attacks remains unsolved, despite safety training as a promising approach.The community is still constructing a comprehensive workflow for LLM security.Source: D. General Defense Framework against… · p. 20

E. Development of Defensive Tools for LLM Systems

  • CFI, provenance graphs, and attack-traffic detection can mitigate threats; proposed improvements include targeted system-call analysis, graph pruning, and anomaly detection.The passage also suggests lightweight instrumentation and graph summarization that preserves important dependencies.Source: E. Development of Defensive Tools for… · p. 20

F. Risks and Mitigation on LLM-based Agent

  • LLM agents face jailbreaks and risks from real-world actions, with potential malicious uses including cyberattacks and phishing.The discussion recommends regular robustness tests, real-time anomaly detection, ethical and legal oversight, and education and reskilling for workforce transitions.Source: F. Risks and Mitigation on LLM-based… · p. 20

G. Developing Robust Watermarking Methods

H. Improving the Evaluation of LLM Systems

  • Evaluation metrics are task-specific, while comprehensive hyperparameter searches are uncommon partly because validation-set inference is costly.The discussion calls for unified evaluation across scenarios and better ways to determine hyperparameter values.Source: H. Improving the Evaluation of LLM… · p. 20

VIII. CONCLUSIONS

  • The survey’s module-oriented taxonomy organizes risks by system module to help participants locate issues and select mitigation strategies.The paper presents this as guidance for future responsible LLM development.Source: §VIII · p. 20

Related papers

Loading 2401.05778v1…