PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation

Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, Kaiwen Cai, Yanghao Zhang, Sihao Wu, Peipei Xu, Dengyu Wu, Andre Freitas, Mustafa A. Mustafa

arXiv:2305.11391v2cs.AIcs.LG

TL;DR

LLMs are widely deployed despite hallucinations, safety weaknesses, privacy risks, and other trustworthiness problems. This survey categorizes vulnerabilities and examines how V&V techniques can support assurance across the LLM lifecycle, concluding that rigorous yet practical methods remain needed.

  • Problem

    Rapid LLM adoption has outpaced rigorous understanding and assurance of safety and trustworthiness across models and applications.

  • Method

    The survey reviews vulnerabilities and considers integrating falsification and evaluation, verification, runtime monitoring, and regulations and ethical use throughout the LLM lifecycle.

  • Results

    The survey identifies inherent issues, attacks, and unintended bugs while organizing V&V perspectives for analyzing LLM safety and trustworthiness.

  • Takeaways & Limitations

    The paper calls for novel, multidisciplinary V&V methods that address LLM nondeterminism, scale, and internet-derived training data.

  • Takeaways & Limitations

    The survey is not intended to be complete, especially regarding LLM applications across various domains.

Abstract

from arXiv · show

Large Language Models (LLMs) have exploded a new heatwave of AI for their ability to engage end-users in human-level conversations with detailed and articulate answers across many knowledge domains. In response to their fast adoption in many industrial applications, this survey concerns their safety and trustworthiness. First, we review known vulnerabilities and limitations of the LLMs, categorising them into inherent issues, attacks, and unintended bugs. Then, we consider if and how the Verification and Validation (V&V) techniques, which have been widely developed for traditional software and deep learning models such as convolutional neural networks as independent processes to check the alignment of their implementations against the specifications, can be integrated and further extended throughout the lifecycle of the LLMs to provide rigorous analysis to the safety and trustworthiness of LLMs and their applications. Specifically, we consider four complementary techniques: falsification and evaluation, verification, runtime monitoring, and regulations and ethical use. In total, 370+ references are considered to support the quick understanding of the safety and trustworthiness issues from the perspective of V&V. While intensive research has been conducted to identify the safety and trustworthiness issues, rigorous yet practical methods are called for to ensure the alignment of LLMs with safety and trustworthiness requirements.

1 Introduction

LLMs bring broad conversational capabilities but also serious safety and trustworthiness risks. The survey reviews these risks and examines how V&V can be extended across the LLM lifecycle.

  • LLMs use massive parameter counts and transformer-based architectures to learn universal language representations for varied NLP tasks.They commonly reach more than 10 billion learnable parameters.
  • Fluent LLM outputs can still hallucinate, state incorrect facts, lack safety mechanisms, and provide limited transparency or control.Reported consequences include suicide, fabricated legal precedents, and private-information leakage.
  • The survey reviews existing safety and trustworthiness vulnerabilities and investigates adapting V&V techniques to improve them.Its perspective is motivated by the wide adoption of LLMs in applications interacting directly or indirectly with end users.
  • V&V checks systems against design specifications and user operational needs, but LLM scale makes conventional white-box techniques difficult to apply.The paper therefore highlights black-box approaches as more practical for large models.
  • The survey organizes its discussion around falsification and evaluation, verification, runtime monitoring, and ethical use within a lifecycle framework.It includes more than 370 references while acknowledging that the survey is not complete.

2 Large Language Models

The paper situates LLMs within transformer-based language and image-synthesis developments, then maps their lifecycle, vulnerabilities, and safety-improvement techniques. It emphasizes that risks can arise from data collection through deployment and interaction.

  • Applications: LLMs support text generation, summarization, conversational AI, and image synthesis, while the survey focuses mainly on text-based conversational systems.Image-synthesis-specific concerns such as fake-image detection are outside its main focus.
  • Text-based Conversational AI: Transformer-based pretrained models achieve broad NLP capabilities through self-supervised learning and transfer learning over large unlabelled text corpora.Scaling model size has also been associated with improved performance.
  • Text-based Image Synthesis: Text-to-image systems commonly encode text with transformers and generate images using diffusion models or autoencoder decoders.The survey describes representative systems including DALL-E, StableFusion, and GLIDE.
  • Lifecycle of LLMs: The LLM lifecycle includes offline construction through pre-training, adaptation tuning, and utilisation improvement, followed by evaluation and deployment.Evaluation covers basic performance, application safety, and public benchmarks.
  • Lifecycle of LLMs: Vulnerabilities may arise throughout the lifecycle, including contaminated raw data early on and robustness gaps after deployment.The paper distinguishes performance, sustainability, and unintended-bug issues running across stages.
  • Safety and Trustworthiness Techniques: RLHF and guardrails are presented as safety techniques, with RLHF using human guidance and guardrails refusing requests about harmful or unsavoury topics.Guardrails improve social responsibility but remain vulnerable to circumvention such as prompt injection.

3 Vulnerabilities, Attacks, and Limitations

The survey classifies LLM vulnerabilities into inherent issues, intended attacks, and unintended bugs. These categories differ in origin, lifecycle location, and the kinds of mitigation they invite.

  • Inherent Issues: Inherent issues are vulnerabilities that LLMs cannot readily solve themselves but may gradually improve through more data and novel training methods.They include performance weaknesses, sustainability concerns, and trustworthiness or responsibility issues.
  • Inherent Issues: Performance weaknesses reflect LLMs’ failure to reach human-level intelligence, while sustainability issues arise from their substantially larger size and resource demands.Training and daily execution can have non-negligible sustainability implications.
  • Intended Attacks: Intended attacks are initiated by malicious actors against lifecycle stages and include robustness gaps, backdoors, poisoning, disinformation, privacy leakage, and unauthorised disclosure.The taxonomy separates attacks by deliberate attacker intent.
  • Unintended Bugs: Unintended bugs arise unconsciously during broader application integration and include bias, discrimination, and incidental exposure of user information.They may be addressed through high-quality training data and carefully designed APIs.
  • Lifecycle Placement: Inherent issues and unintended bugs may appear at any lifecycle stage, whereas attacks usually target particular stages such as pre-training, adaptation tuning, or alignment tuning.Backdoors are typically embedded during pre-training or adaptation tuning, while poisoning commonly occurs during training or alignment tuning.

3.1 Inherent Issues

LLMs have inherent correctness, sustainability, privacy, and responsibility issues that arise from their data-learned behaviour and lifecycle demands. These include factual and reasoning errors, substantial resource consumption, training-data risks, and misuse concerns.

  • Performance Issues: LLMs can produce both factual errors and reasoning errors, and similar errors occur across different models.Correctness is not guaranteed because behaviour is determined by complex models learned from data rather than deterministic rules.
  • Performance Issues: Factual errors can include fabricated affiliations and seriously wrong advice, while reliability may vary across domains.Such errors are also referred to as hallucinations in some literature.
  • Performance Issues: LLMs may answer calculation and logic questions incorrectly when prompts differ from patterns represented in training data.Carefully crafted prompt sequences can expose wrong answers because the models fit questions to prior experience rather than perform actual reasoning.
  • Sustainability Issues: LLMs impose substantial economic, energy, and carbon costs across their lifecycle.ChatGPT was trained with 30k A100 GPUs, and its monthly energy consumption cost was estimated at around $1.5M.
  • Other Inherent Trustworthiness and Responsibility Issues: Training-data risks include copyright, quality, privacy, absent consent, and potential privacy leakage.Internet crawling can involve personal information whose use in LLM training users may not know about or consent to.
  • Other Inherent Trustworthiness and Responsibility Issues: Final-model concerns include mimicking human and academic output, automated scam communications, malware generation, and the need to distinguish human from LLM-produced work.These issues motivate transparency in LLM development and technologies for verification and differentiation.

3.2 Attacks

LLM attacks exploit prompts, training data, model behavior, or deployment pathways to disclose information, activate malicious behavior, degrade performance, or produce harmful outputs. The surveyed attacks span adversarial perturbations, backdoors, poisoning, and disinformation across language, code, and image-generation tasks.

  • Prompt-based attacks: Prompt injection and prompt leaking can disclose sensitive LLM information, including system prompts and a model’s hidden codename.A simple conversation reportedly caused the new Bing to reveal “Sydney” and retrieve its prompt without proper authentication.
  • Adversarial attacks: Adversarial perturbations modify characters, words, or sentences to alter model outputs or reduce safety and toxicity scores.Reported perturbations include deletions, insertions, substitutions, spacing changes, and other character-, word-, and sentence-level transformations.
  • Robustness gaps: ChatGPT leads other models on the reported adversarial classification tasks but remains imperfect and underperforms commercial systems on some translation tasks.The passage also reports weaker biomedical-abstract and Reddit-comment translation robustness, alongside better spoken-language translation results.
  • Backdoor attacks: Backdoor attacks hide malicious knowledge in models through poisoned data or parameter modification, activating only when prompts contain attacker-controlled triggers.Triggers include character-, word-, and sentence-level patterns, including semantic-preserving and concealed variants such as homographs and dynamic sentences.
  • Backdoor attacks: Backdoors have been demonstrated in toxic-comment detection, translation, question answering, code suggestion, and text-based image synthesis.Reported effects include harmful responses, ignored toxic comments, concealed code payload completion, and image generation following attacker-specified descriptions or attributes.
  • Poisoning and disinformation: Poisoning attacks manipulate training data to induce wrong or biased outputs, exploiting LLM training on publicly accessible data from potentially unreliable sources.Examples include spam-filter failure after manipulating 1% of training data and attacks that perturb web or Wikipedia content before dataset collection.

3.3 Unintended Bugs

Unintended bugs can expose users’ private information without an attacker deliberately initiating an attack, while inherited bias can produce false or discriminatory outputs. These failures create security, privacy, and social risks in deployed LLMs.

  • Incidental exposure of user information: A ChatGPT chat-history bug exposed previous users’ conversations in sidebars and potentially disclosed personal data from ChatGPT Plus subscribers.OpenAI reported that the same bug may have revealed payment-related information for 1.2% of ChatGPT Plus subscribers.
  • Inherited bias and discrimination: LLMs can inherit bias and discrimination from training data, as illustrated by Galactica’s false and racist outputs and reported political bias in ChatGPT.Galactica was shut down after three days, while a political compass test reportedly associated ChatGPT with progressive and libertarian views.

4 General Verification Framework

The proposed verification framework places complementary V&V techniques across the LLM lifecycle. It combines failure discovery and explanation during evaluation with lifecycle-wide ethical principles and AI regulations, supported by a taxonomy of surveyed techniques.

  • Evaluation: The lifecycle framework introduces falsification and evaluation alongside explanation techniques during the evaluation stage.Falsification and evaluation seek diverse, non-exhaustive failure cases and statistical understanding of potential failures, while explanation techniques provide further analysis.
  • Ethical use: Ethical principles and AI regulations are imposed throughout the LLM lifecycle to support ethical use.
  • Lifecycle framework: Figure 5 illustrates how categories of V&V techniques are positioned across the LLM lifecycle.
  • Technique taxonomy: Figure 6 presents a taxonomy of the verification and validation techniques surveyed for large language models.The paper states that subsequent sections review these techniques in greater detail.

5 Falsification and Evaluation

This section surveys falsification and evaluation methods for finding LLM failures, assessing vulnerabilities, and comparing model performance with human experts. It also highlights the need for automated, lifelong assessment because human-led evaluation is costly and LLM services can change over time.

  • Falsification and evaluation: Red teaming actively interacts with LLMs to find failing examples and should involve people with diverse backgrounds and risk concerns.Unlike annotating pre-existing texts, red teams search for examples that fail through model interaction.
  • Prompt injection: Prompt injection can override instructions and safeguards, generate malicious content, or exploit poisoned web content retrieved through plugins and APIs.Attackers concatenate instructions with data or use tricky prompts to bypass ethical modules and filtering mechanisms.
  • Prompt injection: Adversarial-prompt research shows that black-box attacks can alter generated sentiment or targeted-letter frequency, while undesirable behaviour remains possible as prompt length increases.Token-space projection enables exploration of adversarial prompts, and one study identifies a positive probability of triggering undesirable behaviour with sufficiently long prompts.
  • Comparison with human experts: Across open-domain, financial, medical, legal, psychological, bibliometric, educational, ranking, and grammatical-correction comparisons, ChatGPT did not perform as well as expected.Studies also report that close collaboration between ChatGPT and human experts achieves the best results in safety analysis and logical consistency.
  • Comparison with human experts: LLMs can process massive medical-record collections, whereas human experts may better handle complicated reasoning and social or cultural contexts; the survey frames them as complements.The cited safety-analysis studies conclude that close collaboration between ChatGPT and human experts produces the best results.
  • Benchmarks: Benchmark evaluation covers adversarial robustness, out-of-distribution behaviour, safety scenarios, instruction attacks, and mathematical capability.Examples include AdvGLUE, ANLI, Flipkart reviews, DDXPlus, and GHOSTS.
  • Benchmarks: LLM services require lifelong assessment because behaviour can change substantially between releases, while continual fine-tuning and RLHF increase the risk of catastrophic forgetting.Providers are encouraged to release raw benchmark results so inspectors can examine model defects more thoroughly.
  • Testing and statistical evaluation: Human expertise is expensive and scarce, motivating automated testing and statistical evaluation to search for corner cases more intensively and fairly.Automated generation requires an oracle, coverage metric, and test-case generation method; statistical evaluation can assess properties probabilistically by aggregating samples.

6 Verification

This section reviews verification techniques for LLMs, emphasizing robustness certification and black-box approaches as models become too large for many traditional methods. It covers adaptations including interval bounds, abstract interpretation, randomized smoothing, and robustness verification for NLP models.

  • Overview: LLM verification remains an emerging area, with the survey reviewing NLP verification techniques and pioneering black-box methods for large-scale language models.The section distinguishes verifying LLMs from using LLMs to support specification formalization, code generation, assertion generation, or vulnerability repair.
  • Robustness verification: Adversarial examples can alter downstream implications such as sentiment or spam detection, while exhaustive exploration is challenged by exponential combinations of word changes.Verification methods therefore target robustness properties against structured perturbations rather than enumerating every unseen case.
  • Verification via Interval Bound Propagation: Interval Bound Propagation (IBP) certifies robustness by optimizing upper bounds over perturbations and keeping the resulting perturbation region on one side of the classification boundary.In NLP, IBP has been used to provide certified robustness under discrete word-vector perturbations.
  • Verification via abstract interpretation: Abstract interpretation approximates network behaviour with a simpler abstract domain that captures possible output ranges under input perturbations.Zonotopes can provide tighter bounds than boxes; Cert-RNN verifies output Zonotopes to determine certifiable robustness and improves speed over POPQORN.
  • Verification via randomized smoothing: Randomized smoothing constructs a classifier by injecting sampled noise into inputs, selecting the most frequent prediction, and certifying robustness when agreement exceeds a threshold.The certification applies within a specified radius around the input.
  • Verification via randomized smoothing: WordDP provides a robustness certificate for NLP models by treating a sentence as a database and its words as records under differential privacy.The method is presented as an early use of randomized smoothing for NLP robustness verification.
  • Black-box verification: Traditional layer-by-layer verification becomes computationally impractical at LLM scale, motivating black-box methods that query models without accessing their internal representations.Many existing techniques impose requirements on network categories or activation functions, limiting applicability to increasingly complex LLMs.
  • Black-box verification: Existing black-box word-substitution verification can guarantee unchanged predictions within a safe radius, but such methods are not scalable to LLMs.The limitation reflects the difficulty of extending current NLP verification approaches to large language models.

7 Runtime Monitor

Runtime monitoring is presented as a safeguard for LLM behavior when verification is impractical or insufficient, but practical monitoring remains underdeveloped. Current approaches focus especially on out-of-distribution detection, while broader output failures and attacks lack established monitoring methods.

  • Motivation: Runtime monitors safeguard LLM behavior against vulnerabilities and complement guardrails, especially when verification cannot fully address deployed-model behavior.The section motivates monitoring as a runtime safeguard rather than relying exclusively on pre-deployment verification.
  • Monitoring Out-Of-Distribution: Out-of-distribution inputs indicate possible risk because models may be unreliable on data that differs from their training distribution.Neural networks can make overconfident predictions on out-of-distribution samples, motivating runtime detection in safety-critical applications.
  • Monitoring Out-Of-Distribution: Current NLP out-of-distribution monitors use input-density estimation, feature or embedding-space approximation, or calibrated output confidence.Maximum softmax probability is identified as a classic baseline for output-confidence methods.
  • Monitoring Out-Of-Distribution: No single out-of-distribution method is currently superior, although their additional computation is considered negligible relative to one neural-network inference.The methods remain improvable, partly because out-of-distribution data are effectively unbounded and in-distribution boundaries are ambiguous.
  • Monitoring Out-Of-Distribution: NLP out-of-distribution detection needs a fine-grained problem definition and fair evaluation because most LLM training datasets are unavailable.This uncertainty makes it unclear whether evaluation datasets are genuinely out of distribution for the tested models.
  • Output Failures and Future Work: Generative outputs require application-specific monitoring, while systematic monitoring research remains absent for reasoning, coding, robustness, backdoor, and poisoning failures.The diversity of downstream tasks makes a general monitoring framework for generative outputs extremely difficult, if not impossible.
  • Future Directions: Designing reliable monitors remains difficult because low prediction uncertainty does not guarantee reliable outputs, motivating work on model behavior, decision boundaries, and explainability.The section frames runtime monitoring as a longstanding challenge for LLMs because they have many known vulnerabilities and remain in an early stage of development.

8 Regulations and Ethical Use

Technical V&V can support alignment between LLMs and human interests, but it may be insufficient against abuse without ethical measures, governance, and regulation. The survey highlights unresolved concerns involving misuse, privacy, copyright, transparency, explainability, education, and industrial practice.

  • Ethical Use: Technical V&V supports alignment with human interests but may be insufficient to prevent LLM abuse, requiring complementary ethical means.The survey positions ethical measures as supplements to technical safeguards rather than replacements for them.
  • Regulation: Debates over suspension versus regulation, national actions, and calls for government oversight reflect disagreement about AI development and human-interest alignment.The passage cites Italy’s reported ChatGPT ban and a May 2023 U.S. Senate hearing as examples.
  • Regulation: The proposed V&V framework is presented as one viable way to address regulatory requirements such as robustness and transparency with technical means.
  • Privacy and Copyright: ChatGPT raises unresolved copyright questions about internet-sourced training data and privacy concerns for users whose dialogues may be collected or used.The section emphasizes the need to inform end users about whether and how their dialogues are handled.
  • Governance: Responsible AI governance requires properties including transparency, explainability, fairness, robustness, security, and privacy, with consistent definitions still needed.The survey notes that a comprehensive framework comparison is outside its scope.
  • Ethical Use: LLM-specific ethical concerns include potential misuse, unintended bias, and fair access, motivating enterprise principles that ask whether an action should be done, not merely whether it can be done.
  • Education and Practice: Insufficient V&V education contributes to industrial difficulty in building AI systems with safety guarantees, especially when untrained engineers develop safety-critical applications.The proposed educational response includes formal verification, statistics, and explainable AI.
  • Education and Practice: Different terminologies and meanings of trustworthiness fragment collaboration between AI and rigorous-design communities, supporting a call for shared curricula.The survey recommends introducing AI students to systematic safety analysis and design of AI-critical applications.

9 Discussions

The discussion identifies a lag between LLM adoption and rigorous safety research, emphasizing data governance, consequence-oriented safety analysis, lifecycle engineering, provable guarantees, and implementable standards. These directions respond to LLM-specific risks from internet-scale data, broad deployment, and limited empirical assurance.

  • Research Directions: Safety and trustworthiness research has lagged behind LLM adoption, while LLMs make these issues more consequential through direct interaction with many end users.
  • Data Privacy: Internet-derived LLM training data intensify privacy and copyright concerns because they may include private information without authorization and can be exposed to prompt-injection techniques.
  • Safety and Trustworthiness: Safety analysis should systematically model the deployment environment, usage patterns, and consequences of possible outputs rather than focusing only on inducing unexpected behavior.The discussion calls for measuring the extent to which unexpected outcomes may lead to bad consequences.
  • Rigorous Engineering: Rigorous engineering should cover the entire LLM development cycle, including evaluation, to support assurance cases for safety-critical applications.This approach is intended to shift development from an extensive mode toward an intensive mode.
  • Verification with Provable Guarantees: Empirical evaluation supplies performance evidence but not rigorous justification, whereas mathematically founded proofs such as statistical guarantees can increase user confidence.
  • Regulations and Standards: Regulations are widely considered necessary, but workable measures aligned with industrial standards remain underdeveloped and are important to assurance cases.The discussion calls for standards to help realize the potential of LLMs and AI generally.

10 Conclusions

The survey reviews known LLM vulnerabilities and considers how V&V techniques might be adapted to address them. It concludes that deployment requires sufficient assurance processes and novel, multidisciplinary methods suited to LLM scale, nondeterminism, internet-sourced data, and end-user interaction.

  • Conclusion: The survey overviews known LLM vulnerabilities and discusses adapting V&V techniques to address safety and trustworthiness.
  • Conclusion: Deployed LLMs should undergo sufficient assurance processes because applications increasingly interact directly or indirectly with end users.
  • Conclusion: Novel V&V techniques are needed for LLM nondeterminism, much larger model sizes, and training data gathered from the internet rather than carefully collected datasets.
  • Conclusion: Multidisciplinary development is needed to ensure that LLM trustworthiness issues are fully considered and addressed.
Loading 2305.11391v2…