Source-linked AI summary
Safeguarding Large Language Models: A Survey
Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, Saddek Bensalem, Xiaowei Huang
TL;DR
LLM deployment creates ethical, privacy, bias, robustness, and misuse risks that motivate systematic safeguards. This survey reviews guardrail frameworks, property-specific evaluation and enhancement methods, attacks and defenses, and approaches toward comprehensive safeguards. It concludes that guardrails require multidisciplinary, neural-symbolic, and systems-development-lifecycle perspectives, while remaining context-dependent and unable to resolve every conflict universally.
Problem
LLM deployment raises risks involving ethical use, data bias, privacy, robustness, misinformation, criminal misuse, and scientific applications.
Method
The survey synthesizes guardrail frameworks, techniques for evaluating and enhancing safety properties, attacks and defenses, and systematic approaches for building comprehensive safeguards.
Results
The survey provides a holistic review of existing guardrails, their effectiveness, safety-related properties, attacks, defenses, and strategies for strengthening safeguards.
Takeaways & Limitations
Comprehensive guardrails should consider multiple requirements through multidisciplinary methods, neural-symbolic approaches, and a systems development lifecycle.
Takeaways & Limitations
Guardrails are not universal solutions because conflicts may remain unresolved across varied contexts, so they should target specific well-defined scenarios.
Abstract
from arXiv · showhide
In the burgeoning field of Large Language Models (LLMs), developing a robust safety mechanism, colloquially known as "safeguards" or "guardrails", has become imperative to ensure the ethical use of LLMs within prescribed boundaries. This article provides a systematic literature review on the current status of this critical mechanism. It discusses its major challenges and how it can be enhanced into a comprehensive mechanism dealing with ethical issues in various contexts. First, the paper elucidates the current landscape of safeguarding mechanisms that major LLM service providers and the open-source community employ. This is followed by the techniques to evaluate, analyze, and enhance some (un)desirable properties that a guardrail might want to enforce, such as hallucinations, fairness, privacy, and so on. Based on them, we review techniques to circumvent these controls (i.e., attacks), to defend the attacks, and to reinforce the guardrails. While the techniques mentioned above represent the current status and the active research trends, we also discuss several challenges that cannot be easily dealt with by the methods and present our vision on how to implement a comprehensive guardrail through the full consideration of multi-disciplinary approach, neural-symbolic method, and systems development lifecycle.
1 INTRODUCTION
LLM deployment raises risks involving ethics, bias, privacy, robustness, and misuse, motivating post-hoc guardrails that monitor inputs and outputs. The survey organizes existing frameworks, evaluation and enhancement techniques, attacks, defenses, and a systematic development process.
- Rapid LLM deployment raises concerns about ethical use, data bias, privacy, robustness, misinformation, criminal misuse, and scientific applications.
- Guardrails are black-box, post-hoc algorithms that monitor LLM inputs and outputs and enforce actions to reduce embedded risks.For harmful inputs, a guardrail may stop the input or adapt the output to become harmless.
- Guardrail requirements can span hallucination, fairness, privacy, robustness, toxicity, legality, out-of-distribution behavior, and uncertainty, with responsibilities sometimes overlapping those of LLMs.The survey excludes ordinary accuracy as primarily an LLM benchmark rather than a guardrail responsibility.
- The survey reviews existing guardrail frameworks, techniques for evaluating and enhancing desirable properties, attacks that overcome safeguards, defenses, and reinforcement strategies.
- The paper argues that guardrails require a safety-critical-style development cycle spanning specification, design, implementation, integration, verification, validation, and release.It relates this process to industrial standards including ISO-26262 and DO-178B/C.
2 BACKGROUND FOR LARGE LANGUAGE MODELS
LLMs use scaled Transformer-based architectures for diverse applications, and their assessment combines capability evaluation, safety analysis, and public benchmarks. These dimensions examine both core performance and alignment with human values in practical environments.
- LLMs primarily use Transformer architectures composed of repeated blocks containing self-attention and feedforward layers connected by residual links.Self-attention enables concentration on neighboring tokens while analyzing a specific token.
- LLMs support complex tasks including conversational AI, translation, and story generation while scaling model dimensions, data, application scope, and computation cost.
- LLM evaluation covers essential capabilities, safety analysis, and publicly available benchmark datasets.Safety analysis examines alignment with human values, external interactions, and integration into applications such as search engines.
3 TECHNIQUES ON DESIGN AND IMPLEMENTATION OF GUARDRAILS
The survey reviews guardrail techniques from service providers and the open-source community, alongside methods for evaluating, analyzing, and enhancing LLMs against desired properties.
- This section surveys guardrail techniques proposed by LLM service providers and the open-source community.
- It also reviews methods to evaluate, analyze, and enhance LLMs according to desirable properties.A comparison table summarizes platforms and properties across approaches.
3.1 Guardrail Frameworks and Supporting Software Packages
Guardrail frameworks regulate, evaluate, or constrain LLM interactions through distinct neural-symbolic and programming-based workflows. The surveyed systems range from classification and retrieval-mediated dialogue control to structured output validation, continuous evaluation, and constraint-driven decoding.
- Guardrail frameworks: Guardrails are programmable, rule-based systems positioned between users and foundational models that evaluate outputs against predefined rules and block or warn on unsafe content.Their lifecycle includes data processing, guardrail model training, customization or fine-tuning, deployment, and interaction-stage checking.
- Llama Guard: Llama Guard classifies victim-model inputs and outputs against user-specified safety categories, but its reliability depends on category understanding and predictive accuracy.Its adaptable taxonomy makes it applicable to different user and application guidelines.
- Nvidia NeMo: Nvidia NeMo uses Colang constraints, embedding similarity, KNN retrieval, and repeated LLM calls to control dialogue flows and support fact-checking, hallucination prevention, and content moderation.Its effectiveness is closely tied to the KNN method.
- Guardrails AI: Guardrails AI defines XML-based RAIL specifications, initializes a guard, wraps the LLM, and can prompt regeneration when outputs fail structural, type, quality, or classifier checks.The framework is limited to text-level checks and is described as a Type-2 neural-symbolic system.
- Evaluation and programming frameworks: TruLens evaluates and monitors LLMs against standards, while Guidance AI combines prompting, generation, logic control, regex, and CFG constraints in a Python workflow.TruLens emphasizes continuous refinement rather than constraining inputs and outputs directly; Guidance targets controlled, highly formatted generation.
- LMQL: LMQL extends prompt templates into a Python-superset language with logit masking, custom operators, scripted beam search, and constraint-driven decoding across placeholders.FollowMap evaluation excludes next tokens that would violate specified constraints, reducing ineffective model calls.
3.2 Techniques for (Un)desirable Properties in Guardrails
The survey examines guardrails for hallucination, fairness, and related properties, emphasizing evaluation, bias mitigation, and interaction-level controls. It covers both data/model interventions and safeguards applied directly to user inputs and outputs.
- Scope: The survey focuses on hallucination, fairness, privacy, robustness, toxicity, legality, out-of-distribution behavior, and uncertainty as guardrail properties.Accuracy is treated as an LLM benchmark rather than a typical guardrail responsibility.
- Hallucination: Hallucinations are nonexistent, illogical, or irrelevant responses arising across data sourcing, pre-training, alignment, and inference.The survey discusses detection and evaluation methods, including continuous integration for testing model updates.
- Fairness: Fairness risks arise when gender, cultural, dataset, and social biases become encoded in representations and affect downstream decisions.The survey reviews intrinsic debiasing through training data and extrinsic debiasing through task-specific model adaptation.
- Interaction controls: Guardrails can control interactions through designed prompts, cultural prompting, and other input- and output-level interventions.Cultural prompting may reduce cultural bias but can be ineffective or exacerbate bias in some countries.
- Integrated mitigation: Comprehensive bias mitigation combines diverse data curation with algorithmic adjustments and other complementary strategies.The survey presents this as a broader approach than relying on a single intervention.
3.2.3 Privacy (Copyright)
The survey frames privacy as a data-protection concern intensified by LLMs’ potential to disclose personally identifiable information. It reviews privacy defenses for deployed systems and privacy-by-design practices for systems still under development.
- Privacy risks: Privacy regulations establish data-sharing and retention standards, yet LLMs may still release personally identifiable information.The survey cites the EU AI Act, GDPR, and CCPA as relevant legislative frameworks.
- Privacy defenses: Differential Privacy-tuned LLMs are reviewed as a leading approach for protecting sensitive information and reducing privacy-violation risks.The survey also notes that directly applying DP-SGD to general NLP models may require additional techniques.
- Watermarking: Watermarking can support privacy and copyright protection by promoting randomized green tokens during generation.The mechanism embeds a watermark by selecting green tokens before sampling.
- Deployed applications: Privacy safeguards for deployed applications include robust testing, continuous monitoring, and blocklists, allowlists, or suppression lists.These controls target unsafe content generation that could compromise privacy.
- Development practices: Privacy-by-design practices include ethical risk assessment, selective memory, information filtering, and removal of personally identifiable information from training data.These measures are intended for applications still in development.
3.2.4 Robustness
The survey distinguishes adversarial robustness from out-of-distribution robustness and focuses here on maintaining model performance under intentionally altered inputs. It presents input preprocessing and output monitoring as guardrail responses while noting that defenses remain under investigation.
- Definition: Adversarial robustness concerns maintaining performance when inputs are intentionally altered to cause errors, including through malicious queries or small perturbations.The section treats adversarial robustness separately from out-of-distribution robustness.
- Definition: The survey characterizes robustness using performance on new test data whose distribution differs from the training distribution.Adversarial robustness is examined by comprehensively perturbing an input from x to x1.
- Open challenge: Defensive methods for deliberate disruptions remain under investigation, motivating robust safeguards during user interactions.The survey identifies this as an ongoing research problem rather than a settled solution.
- Guardrail responses: Guardrails typically preprocess inputs to remove or neutralize adversarial content and monitor outputs using thresholds or deviation flags.Examples include correcting typos, standardizing input formats, and flagging problematic responses.
3.2.5 Toxicity
The survey treats toxicity as a broad family of harmful linguistic behaviors and reviews evaluation, live-interaction protection, and legality-related safeguards. It emphasizes that low-toxicity training and evaluation do not eliminate the need for runtime controls.
- Definition and risks: Toxicity encompasses offensive, abusive, hateful, rude, disrespectful, or unreasonable responses that may harm users or specific downstream tasks.The survey highlights risks for younger or otherwise vulnerable individuals.
- Evaluation: Toxicity evaluation uses trigger prompts and metrics such as Toxicity Classifier Score and PerspectiveAPI, but evaluators and encoders may introduce bias.The survey therefore identifies limitations in typical toxicity metrics.
- Runtime safeguards: Runtime safety guardrails remain necessary even when training data has low toxicity and evaluations validate reduced toxic-text generation.They help protect privacy, prevent bias, and maintain user trust during live interactions.
- Illicit outputs: Illicit-output safeguards operate by rejecting inappropriate inputs and moderating model outputs for users or downstream tasks.The survey presents input rejection and output moderation as complementary fronts.
- Development lifecycle: Safety development includes training-data screening, human review during training, and red-team testing before release.Red teaming uses simulated attacks and adversarial testing to uncover vulnerabilities and ethical or legal pitfalls.
- Post-release monitoring: Post-release monitoring uses language processing and anomaly detection to identify issues and trigger filtering or human intervention.The survey describes these measures as real-time responses to inappropriate inputs and outputs.
- Legality: Legality sets the minimum legal boundary, but content can be non-illegal yet toxic in context; illegal content is treated as toxic.ConstitutionalChain is described as filtering and modifying content to align with predefined constitutional principles.
3.2.7 Out-of-Distribution
OOD data for LLMs is difficult to define generally, but becomes more tractable in specific real-world tasks. Even minor semantic shifts can substantially reduce performance, motivating task-specific filtering or detection.
- OOD data is difficult to define for LLMs in general but more feasible when the application context is precise.
- In practical scenarios, OOD instances can be irrelevant to the task or significantly different from normal inputs.
- Minor semantic shifts caused by small perturbations can significantly reduce performance across several language-model applications.
- Anomaly-input filters and task-specific OOD detectors can mitigate OOD impacts in practical tasks.
3.2.8 Uncertainty
Uncertainty quantification supports safer LLM use by estimating whether outputs are reliable enough to accept. The survey emphasizes semantic uncertainty, reproducible unsupervised methods, and responses that communicate uncertainty or seek assistance.
- Uncertainty quantification gauges LLM confidence so unreliable outputs can be rejected or subjected to additional scrutiny.
- Combining reasoning chains, integrating prompts, and eliciting confidence levels can improve LLM calibration relatively.
- Semantic entropy addresses the gap between token-level likelihoods and the semantic content that matters in many applications.
- An out-of-the-box, unsupervised uncertainty method improves reproducibility and simplifies deployment while addressing failures of supervised measures under distributional shifts.
- A guardrail should help models recognize queries beyond their expertise and respond cautiously, flag uncertainty, or direct users to reliable sources.
4 OVERCOME AND ENHANCE GUARDRAILS
The survey reviews jailbreaks that bypass LLM guardrails across access settings, prompt levels, and manipulation strategies. It also shows that attacks range from optimized suffixes and prefixes to encrypted or context-based prompts, with some methods improving success and stealth.
- External guardrails only marginally reduce the average success rate of jailbreak attacks in the examined systems.
- Jailbreak comparisons consider attacker access, manipulation prompt level, evaluation settings, and targeted properties such as toxicity, privacy, fairness, and hallucination.
- White-box attacks require internal model access, but many can transfer to black-box settings through surrogate models.
- GCG optimizes adversarial suffixes with greedy and gradient-based discrete search to increase the likelihood of affirmative harmful responses.
- AutoDAN-Zhu and COLD-Attack pursue stealthier or constrained jailbreak prompts, addressing the detectability and restriction problems of GCG-style suffixes.
- PRP uses universal adversarial prefixes and prefix propagation to attack LLMs equipped with guardrails.
- AutoDAN-Liu uses evolutionary prompt refinement to circumvent built-in safeguards with stealthy prompts.
- ProMan directly manipulates open-source generation to force specified tokens and produce harmful, sensitive, or private responses.
5 DISCUSSIONS: A COMPLETE GUARDRAIL
The survey advocates a complete guardrail that addresses multiple, conflicting requirements through multidisciplinary design, neural-symbolic methods, rigorous engineering, and scenario-specific deployment. It emphasizes that guardrails remain limited by unresolved conflicts, domain dependence, semantic change, and agent autonomy.
- 5 Discussions: A Complete Guardrail: A complete guardrail should systematically consider multiple requirements rather than treating each requirement in isolation.
- 5.1 Conflicting Requirements: Safety can conflict with intelligence, fairness, privacy, and robustness, producing more conservative responses and increased abstention.
- 5.1 Conflicting Requirements: Guardrail design must consider exploratory depth and the distinction between an LLM’s capabilities and its behavior.
- 5.2 Multidisciplinary Approach: Creativity assessment can evaluate fluency, flexibility, originality, and elaboration as part of guardrail development.
- 5.2 Multidisciplinary Approach: Current guardrails can detect harmful content yet still produce biased or misleading responses, while requirements vary by domain and may conflict.
- 5.2 Multidisciplinary Approach: Ethical development requires fairness, accountability, transparency, community standards, and ongoing evaluation with feedback from diverse users.
- 5.2 Multidisciplinary Approach: Socio-technical and whole-system approaches treat social and technical aspects as interdependent when addressing fairness, bias, and ethics.
- 5.3 Neural-Symbolic Approach for Implementation: Independent rule-based guardrail languages may not resolve conflicting rules or adapt reliably to semantic shifts across scenarios and datasets.
6 CONCLUSIONS
The survey reviews current LLM safeguarding challenges, mitigation methods, attacks and defenses, and argues that guardrails require systematic, multidisciplinary development.
- The survey categorizes guardrails, evaluates their effectiveness, and reviews techniques for bypassing and strengthening them.It covers risks including hallucinations, fairness, and privacy.
- A systematic approach supported by multidisciplinary teams can manage guardrail complexity and provide assurance for the final product.
APPENDIX
The appendix provides examples illustrating hallucination, fairness, privacy, toxicity, out-of-distribution behavior, and uncertainty.
- The appendix includes examples of hallucination and fairness.
- The appendix includes examples of privacy and toxicity.
- The appendix includes examples of out-of-distribution behavior and uncertainty.