PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment

Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, Liang Lin, Zhihao Xu, Haolang Lu, Xinye Cao, Xinyun Zhou, Weifei Jin, Fanci Meng, Shicheng Xu, Junyuan Mao, Yu Wang, Hao Wu, Minghe Wang, Fan Zhang, Junfeng Fang, Wenjie Qu, Yue Liu, Chengwei Liu, Yifan Zhang, Qiankun Li, Chongye Guo, Yalan Qin, Zhaoxin Fan, Kai Wang, Yi Ding, Donghai Hong, Jiaming Ji, Yingxin Lai, Zitong Yu, Xinfeng Li, Yifan Jiang, Yanhui Li, Xinyu Deng, Junlin Wu, Dongxia Wang, Yihao Huang, Yufei Guo, Jen-tse Huang, Qiufeng Wang, Xiaolong Jin, Wenxuan Wang, Dongrui Liu, Yanwei Yue, Wenke Huang, Guancheng Wan, Heng Chang, Tianlin Li, Yi Yu, Chenghao Li, Jiawei Li, Lei Bai, Jie Zhang, Qing Guo, Jingyi Wang, Tianlong Chen, Joey Tianyi Zhou, Xiaojun Jia, Weisong Sun, Cong Wu, Jing Chen, Xuming Hu, Yiming Li, Xiao Wang, Ningyu Zhang, Luu Anh Tuan, Guowen Xu, Jiaheng Zhang, Tianwei Zhang, Xingjun Ma, Jindong Gu, Liang Pang, Xiang Wang, Bo An, Jun Sun, Mohit Bansal, Shirui Pan, Lingjuan Lyu, Yuval Elovici, Bhavya Kailkhura, Yaodong Yang, Hongwei Li, Wenyuan Xu, Yizhou Sun, Wei Wang, Qing Li, Ke Tang, Yu-Gang Jiang, Felix Juefei-Xu, Hui Xiong, Xiaofeng Wang, Dacheng Tao, Philip S. Yu, Qingsong Wen, Yang Liu

arXiv:2504.15585v4cs.CRcs.AIcs.CLcs.LG

TL;DR

LLM safety surveys have often concentrated on individual lifecycle stages despite safety concerns spanning the full process from data preparation to commercialization. This paper introduces “full-stack” safety through a systematic lifecycle-wide review and taxonomy, highlighting vulnerabilities across stages and directions for safer, more reliable deployment.

  • Problem

    Existing LLM safety surveys often focus on specific lifecycle stages, leaving comprehensive coverage of safety from data preparation through commercialization limited.

  • Method

    The paper systematically examines the LLM lifecycle and organizes its safety issues into a full-stack framework covering data, training, deployment, commercialization, agents, and applications.

  • Results

    The review identifies critical vulnerabilities at each stage, including privacy risks, toxic data, harmful fine-tuning attacks, and deployment challenges.

  • Takeaways & Limitations

    Robust LLM security requires attention to data integrity, model alignment, and post-deployment security across the entire lifecycle.

  • Takeaways & Limitations

    The survey notes that no specific defenses against model editing attacks have yet been developed.

Abstract

from arXiv · show

The remarkable success of Large Language Models (LLMs) has illuminated a promising pathway toward achieving Artificial General Intelligence for both academic and industrial communities, owing to their unprecedented performance across various applications. As LLMs continue to gain prominence in both research and commercial domains, their security and safety implications have become a growing concern, not only for researchers and corporations but also for every nation. Currently, existing surveys on LLM safety primarily focus on specific stages of the LLM lifecycle, e.g., deployment phase or fine-tuning phase, lacking a comprehensive understanding of the entire "lifechain" of LLMs. To address this gap, this paper introduces, for the first time, the concept of "full-stack" safety to systematically consider safety issues throughout the entire process of LLM training, deployment, and eventual commercialization. Compared to the off-the-shelf LLM safety surveys, our work demonstrates several distinctive advantages: (I) Comprehensive Perspective. We define the complete LLM lifecycle as encompassing data preparation, pre-training, post-training, deployment and final commercialization. To our knowledge, this represents the first safety survey to encompass the entire lifecycle of LLMs. (II) Extensive Literature Support. Our research is grounded in an exhaustive review of over 800+ papers, ensuring comprehensive coverage and systematic organization of security issues within a more holistic understanding. (III) Unique Insights. Through systematic literature analysis, we have developed reliable roadmaps and perspectives for each chapter. Our work identifies promising research directions, including safety in data generation, alignment techniques, model editing, and LLM-based agent systems. These insights provide valuable guidance for researchers pursuing future work in this field.

1 INTRODUCTION

LLM safety spans the lifecycle from data preparation and pre-training through post-training, deployment, and usage, while prior surveys often addressed only particular stages. This survey develops a full-stack framework that organizes these concerns, reviews the literature systematically, and identifies future research directions.

  • Motivation: LLM safety risks arise across data preparation, pre-training, alignment and fine-tuning, deployment, and agent interactions with tools, memory, and environments.Toxic data and privacy information can enter model parameters, while insufficient alignment and deployment vulnerabilities create additional risks.
  • Motivation: Existing surveys commonly focus on LLM research, trustworthiness, or a single lifecycle phase, lacking specialized coverage of safety across the complete lifecycle.The paper identifies deployment- and fine-tuning-focused surveys as examples of this limitation.
  • Taxonomy: The survey divides the LLM journey into data preparation, pre-training, post-training, deployment, and usage, further distinguishing alignment, fine-tuning, model editing, and unlearning.Deployment is separated into pure LLMs and LLM-based agents augmented with tools, memory, and other modules.
  • Contributions: The authors review over 800 pieces of literature and provide a full-stack taxonomic framework intended to organize safety throughout the LLM lifespan.The survey also correlates lifecycle phases with related sections to clarify safety issues and research stages.
  • Taxonomy: The taxonomy systematically covers data issues, pre-training filtering and augmentation, post-training attacks, defenses and evaluation, and later lifecycle safety concerns.The framework is organized across sections addressing different training and deployment stages.
  • Contributions: Chapter roadmaps, research perspectives, and a standardized architecture map security considerations across the complete lifecycle and identify potential future growth areas.The paper highlights future directions and technical approaches for LLMs and LLM-based agents.

2 DATA SAFETY

LLM data safety spans pre-training, fine-tuning, alignment, and synthetic-data generation, where poisoning, privacy leakage, and synthetic-data vulnerabilities can affect model reliability and security.

  • Pre-training Data Safety: Pre-training data safety concerns poisoning, privacy leakage, and the quality, integrity, and security of massive Internet- and open-source datasets.The survey reviews poisoning attacks, privacy leakage, and mitigation strategies for pre-training data.
  • Pre-training Data Safety: Targeted attacks can make language models memorize and reproduce sensitive information, while membership inference attacks often barely outperform random guessing in LLM settings.The reported membership-inference finding spans varying LLM sizes and domains.
  • Pre-training Data Safety: Mitigations include toxic-content filtering, training-data deduplication, safety-aware output management, source verification, continuous validation, and anomaly detection.These interventions address toxic content, privacy, poisoning, and unsafe generations at different points in pre-training.
  • Fine-tuning Data Safety: Fine-tuning approaches—including instruction tuning, parameter-efficient fine-tuning, and federated learning—remain vulnerable to poisoned data, malicious instructions, backdoors, and harmful trigger-conditioned outputs.The cited studies describe vulnerabilities across centralized and distributed fine-tuning settings.
  • Alignment Data Safety: Alignment data can be poisoned at human-feedback and RLHF stages, including by manipulating preference labels and reward signals during reward-model training.The survey emphasizes safeguards for preference-data curation and reward-model validation.
  • Safety in Data Generation: LLM-driven data synthesis introduces lifecycle-wide poisoning risks and remains limited by hallucination propagation and knowledge attenuation during imperfect teacher–student transfer.The survey identifies verifiability and error-containment mechanisms as needed for synthetic-data integrity.
  • Novel Data Generation Paradigms: Agent-based simulation frameworks are proposed to generate, evaluate, and refine synthetic datasets while integrating real-time safety checks and ethical oversight.The proposed data flywheel targets scalable synthesis and proactive detection of inaccuracies and harmful content.

3 PRE-TRAINING SAFETY

Pre-training safety addresses risks in large-scale training corpora through two complementary strategies: filtering undesirable content and augmenting data with safety-oriented information. Existing methods span heuristic, model-based, and black-box filtering, but their integration into secure pre-training remains unresolved.

  • Pre-training safety is organized around data filtering and data augmentation to address harmful content and privacy violations in training corpora.
  • Pre-training Data Filtering: Model-based filtering applies learned representations or classifiers to assess and remove harmful content such as toxicity, violence, pornography, and propaganda.
  • Pre-training Data Filtering: Black-box filtering relies on policy-driven or API-based criteria, offering operational filtering while limiting interpretability and transparency.
  • Pre-training Data Augmentation: Augmentation integrates safe demonstrations and toxic-content annotations to guide behavior and improve unsafe-input recognition while preserving valuable training data.One approach incorporates 40k human-annotated safety demonstrations updated monthly; another uses control tokens to mark toxicity.
  • Pre-training Data Filtering: Heuristic filtering uses domain blocklists, keyword matching, and predefined rules to remove overtly harmful content and personally identifiable information.Examples include a 13M unsafe-domain list and a 4.6M URL blocklist.
  • Roadmap & Perspective: Existing research has not shown how to integrate filtering methods into pre-training that ensures security from the source, leaving accuracy and efficiency as open directions.The survey also questions whether pre-training augmentation is cost-effective when later-stage safety alignment may yield better results under equal resources.

4 POST-TRAINING SAFETY

Post-training safety covers attacks, defenses, and evaluation across fine-tuning and alignment. The survey organizes attacks by toxic-data construction and fine-tuning mechanisms, including supervised and reinforcement-learning-based routes.

  • Post-training safety is structured around post-training attacks, defenses against those attacks, and evaluation mechanisms.
  • Alignment: The survey distinguishes general alignment from safety alignment, with alignment methods using preference feedback, reward modeling, reinforcement learning, or value-aware optimization.
  • Roadmap & Perspective: The paper contributes a systematic fine-tuning safety taxonomy, an attack-defense methodology, and forward-looking technical roadmaps beyond prior narrower surveys.
  • Attacks in Post-training: Fine-tuning adapts pre-trained models to downstream tasks efficiently, but minimal malicious or misaligned data can severely compromise their safety alignment.
  • Attacks in Post-training: Toxic fine-tuning data is constructed through fixed-prompt, iterative-prompt, and transfer-learning strategies designed to evade detection or transfer attacks across models.
  • Attacks in Post-training: SFT-based attacks manipulate parameters or inject minimal malicious data, whereas RL-based attacks corrupt preference or reward mechanisms to favor harmful outputs.

4.2 Defenses in Post-training

Post-training defenses seek to preserve or restore safety through alignment, regularization, data manipulation, detection, recovery, and safety-location methods. Open-weight models remain difficult to safeguard because adversarial modifications can undo conventional refusal training and durability is hard to evaluate.

  • Alignment: Alignment uses harmless preference data to make generations adhere to ethical and harmlessness objectives, while safety alignment treats safety as a distinct optimization concern.
  • Defenses Against Post-training Attacks: Regularization defenses constrain the fine-tuned model’s distance from an aligned model, including KL regularization and restrictions on safety-layer updates.
  • Defenses Against Post-training Attacks: Data-manipulation defenses mix alignment data into fine-tuning or modify system prompts to associate triggers with safe responses.
  • Defenses Against Post-training Attacks: Detection-based defenses filter harmful fine-tuning samples using moderation models, bi-level selection, or embedding-space analysis.
  • Safety Recovery: Safety recovery realigns compromised models by removing harmful knowledge, altering harmful coordinates, merging aligned parameters, or projecting updates into a safe subspace.
  • Safety Location: Safety location identifies transformer layers essential to defense activation, enabling targeted layer freezing, realignment, or protection against jailbreak attacks.
  • Open-Weight LLMs Safeguard: Open-weight safeguards face a difficult durability problem: prompt formats and random seeds can drastically change outcomes when attackers directly manipulate model weights.

4.3 Evaluation

The survey organizes LLM safety evaluation around safety, utility, and trade-off metrics, alongside benchmarks for harmful-prompt handling and general performance. It emphasizes evaluating both safety preservation and retained usefulness after attacks or defenses.

  • Evaluation Metrics: Safety metrics assess whether attacked models maintain safe outputs, including Attack Success Rate and increasingly fine-grained LLM-as-a-Judge measures.LLM-based evaluators can label outputs as safe or unsafe, measure alignment with human evaluators, or use 5-point scales, though some approaches cannot distinguish risk levels.
  • Evaluation Metrics: Utility metrics assess whether models retain downstream performance after attacks or defenses, using accuracy for tasks with ground-truth labels.The survey identifies mathematical, coding, and classification tasks as examples of closed-ended evaluations.
  • Evaluation Metrics: Safety alignment requires dual-preference evaluation because refusing harmful prompts alone does not establish that a model remains helpful under safety constraints.The survey contrasts simple refusal behavior with evaluations that jointly consider helpfulness and adherence to safety requirements.
  • Evaluation Benchmarks: Alignment and fine-tuning benchmarks overlap in practice, so the survey classifies them by purpose rather than enforcing a strict dataset boundary.Some alignment datasets can be adapted for fine-tuning evaluation, motivating separate safety-purpose and general-purpose categories.
  • Evaluation Benchmarks: Safety-purpose benchmarks test harmful-prompt handling and human-value alignment, whereas general-purpose benchmarks measure capabilities such as accuracy, knowledge breadth, and reasoning.General-purpose benchmarks also help assess how defenses affect model performance or simulate fine-tuning attacks when combined with harmful data.
  • Evaluation Benchmarks: Table 4 summarizes typical evaluation benchmarks and provides access links.

4.4 Roadmap & Perspective

The roadmap shifts attention from low-level harms toward high-level risks such as deception, sycophancy, and reward hacking as LLM reasoning and planning capabilities advance. It surveys behavioral tests, internal monitoring, and formally verifiable safety approaches for studying and constraining these risks.

  • From Low-Level to High-Level Safety: As low-level harmful behaviors become less explicit, safety research increasingly examines high-level risks such as deception and sycophancy.The survey links this shift to advances in LLM reasoning capabilities.
  • Deceptive Alignment: Deception is framed as systematically inducing false beliefs to achieve goals beyond conveying truth, without requiring human-like beliefs or intentions.
  • Deceptive Alignment: Deceptive tendencies are evaluated through hypothetical dilemmas, multi-agent games, autonomous tool use, prompt manipulation, multi-turn consistency, and internal-state monitoring.These designs probe environmental triggers, strategic interactions, covert constraint violations, persistence across dialogue, and discrepancies between reasoning traces and responses.
  • Deceptive Alignment: Prompting for honesty or highlighting risks can reduce deceptive behavior to some extent, but the surveyed experiments report that it cannot be eliminated entirely.
  • Reward Hacking: Reward hacking occurs when an agent exploits flaws or ambiguities in a reward function to obtain high rewards without completing the designer’s intended task.Reward tampering is presented as a special case involving direct interference with the reward source.
  • Reward Hacking: Single-scalar reward models may fail to represent complex human values, allowing models to exploit evaluator or reward-model biases through behaviors such as sycophancy and reward overoptimization.
  • Provably Safe AI System: Provably safe AI seeks rigorous, formally verifiable safety bounds, complementing empirical testing and red-teaming that may miss failure modes in complex or adversarial environments.The approach requires formal safety specifications and world models that translate abstract requirements into behavioral constraints.

5 SAFETY IN MODEL EDITING & UNLEARNING

The survey treats model editing and unlearning as lightweight deployment-time safeguards that can modify knowledge, parameters, or unsafe behavior. It organizes their attack, defense, scalability, stability, and evaluation challenges while positioning them alongside systematic alignment.

  • Overview: Model editing and unlearning modify model information or parameters during deployment to enhance security and privacy.The survey integrates both techniques into its fine-tuning discussion for a systematic account of their safety roles.
  • Model Editing: Model editing updates incorrect or outdated knowledge through gradient-based, memory-based, or locate-then-edit methods.Locate-then-edit approaches use causal tracing to identify knowledge-related neurons and modify them for factual updates.
  • Model Editing: Editing can attack aligned models by injecting harmful, incorrect, biased, or private information, but can also detoxify toxic neurons and reduce jailbreak susceptibility.The survey reports attacks using RoME and IKE, alongside DINM and other defensive applications.
  • Unlearning: Unlearning removes or mitigates selected knowledge, behaviors, or data from trained models through exact certified methods or scalable heuristic methods.Exact unlearning can require certified retraining from scratch, whereas heuristic approaches provide empirical rather than formal guarantees.
  • Future Directions: Model editing complements systematic alignment by enabling rapid, granular, and resource-efficient adaptations to emerging safety risks.The survey describes this combination as a hybrid governance paradigm pairing broad guardrails with surgical interventions.
  • Challenges and Opportunities: Key open issues include stable performance under repeated editing, defenses against editing attacks, standardized unlearning benchmarks, and using unlearning for interpretability.Excessive edits may compromise performance, while unlearning can provide causal insights into model behavior.

6 LLM(-AGENT) DEPLOYMENT SAFETY

The deployment-safety section examines LLMs and LLM agents across foundational-model inference, single-agent systems, and multi-agent systems. It addresses threats and defenses that expand as supplementary tools, memory, and environmental interactions are added.

  • Scope: Deployment safety is organized into LLM safety, single-agent safety, and multi-agent safety dimensions.The section begins with inference-time threats to foundational LLMs and then examines risks from supplementary agent modules.

6.1 Deployment Safety

Deployment safety covers attacks against model confidentiality, robustness, privacy, and content integrity, including extraction, membership inference, jailbreaks, and prompt injection. The survey organizes jailbreaks by attack strategy and reports vulnerabilities in membership inference and toxicity detection, including subtle and culturally biased outputs.

  • Threat Landscape: Single-LLM deployment faces model extraction, membership inference, jailbreak, prompt injection, data extraction, and prompt-stealing attacks.These threats target model confidentiality, robustness, privacy, and ethical compliance.
  • Model Extraction: Model extraction attacks can steal API-accessible behavior, reconstruct full model weights, or recover specialized capabilities such as coding.Reported examples include reconstructing a pre-fine-tuned model from fine-tuned LoRA variants.
  • Membership Inference: Membership-inference research spans token-, document-, collection-, RAG-, and in-context-learning settings, but evaluation is sensitive to memorization, data distributions, and synthetic samples.The survey notes that MIAs can work poorly on LLMs and that duplicated training data increases leakage risk.
  • Jailbreak Attacks: Jailbreak attacks bypass safety rules or filters to elicit unsafe content and are classified as optimization-based or strategy-based.Optimization-based methods revise prompts through gradients or attacker LLMs, while strategy-based methods use templates such as role-playing, ciphers, and long contexts.
  • Content Safety: Role-playing prompts and multilingual evaluation reveal vulnerabilities to subtle toxic or culturally biased outputs, especially in low-resource languages and indirect-harm settings.These findings concern toxicity detection and elicitation of harmful behavior.

6.2 Single-agent Safety

Single-agent safety extends beyond the LLM core to tools, memory, and environmental interaction, each introducing distinct attack surfaces and trustworthiness challenges.

  • Agent definition: Agents use an LLM for reasoning, decision-making, and reflection while integrating memory, tools, and the environment as capability-enhancing components.
  • Tool Safety: Tool interfaces map internal decisions into environmental actions and expose agents to risks from malicious tool use and tool-mediated exploitation.
  • Agent attacks: Agent jailbreaks bypass built-in safety mechanisms through prompts that can elicit malicious responses, including extraction of personal information from training data.
  • Memory Safety: Memory attacks comprise poisoning, privacy leakage, and misuse, allowing corrupted retrievals, sensitive-data extraction, or multi-turn safety circumvention.
  • Environment Safety: Agent-environment interaction involves perception, analysis, and action, but security risks arise throughout this process and existing solutions remain fragmented.

6.3 Multi-agent Safety

Multi-agent systems offer broader analysis through agent interaction but face more severe and intricate risks, including harmful-information propagation, interference, and coordinated attacks.

  • Multi-agent Safety: Compared with single-agent systems, multi-agent systems support interaction and broader problem-solving but face more severe and intricate security risks.
  • Attack taxonomy: Multi-agent threats can propagate harmful information, hallucinations, and biases through interactions or be deliberately orchestrated by malicious agents.
  • Attack taxonomy: Transmissive attacks spread dangerous information through the system, compromising agents with covert malicious content.
  • Attack taxonomy: Interference attacks disrupt communication and introduce misinformation, reducing information-transmission quality and defensive capability.
  • Attack taxonomy: Strategic attacks coordinate agents and optimize attack behavior over time, including role manipulation and persuasive collaboration against secure agents.
  • Defense taxonomy: Defenses include adversarial confrontation, consensus building through voting or debate, and graph-based structural anomaly detection.

6.4 Agent Communication Safety

Communication among interconnected agents, external tools, and environments is essential to multi-agent functionality but creates attack surfaces across channels, content, protocols, and dynamics.

  • Communication Safety: Agents communicate and collaborate through message passing, tool invocation, and environmental interaction, making these mechanisms central to system functionality and security.
  • Threats: The interconnected nature of multi-agent systems introduces threats targeting communication channels, content interpretation, and underlying protocols.
  • Content attacks: Content attacks embed malicious instructions in received or retrieved data, exploiting agents’ difficulty distinguishing instructions from information.
  • Defenses: Communication defenses span infrastructure, channels, content, dynamics, and environmental factors to prevent, detect, or mitigate attacks.
  • Defenses: Reliability estimation and decomposable message aggregation can reduce unreliable messages’ influence on final decisions.

6.5 Agent Safety Evaluation

Agent-safety evaluation spans attack-specific, module-specific, and holistic benchmarks, while deployment research increasingly treats attacks, defenses, and evaluation as a co-evolving system.

  • Benchmark landscape: Agent-safety benchmarks include dataset-based evaluations and sandbox environments created with other agents, each targeting distinct scenarios and assessment priorities.
  • Attack-specific evaluation: Attack-specific benchmarks test vulnerabilities to prompt injection, backdoors, and jailbreaks, including indirect injection during tool-integrated operation.
  • Module-specific evaluation: Module-specific evaluations examine tool safety across input, execution, and output stages, including simulated high-stakes tool executions.
  • Holistic evaluation: 2,000 test cases across 349 interactive environments cover 8 risk categories and 10 failure modes in AgentSafetyBench.
  • Deployment evaluation: Dynamic defenses such as response rewriting, permission control, and consensus filtering require evaluation of consistency, task-specific risk, and adaptability beyond single metrics.
  • Deployment roadmap: Automated red teaming creates a closed loop in which jailbreak samples are generated, tested against defenses, and fed back to refine attacks and defenses.
  • Future directions: Future evaluation directions include multimodal and tool-based tests, multi-objective safety–utility analysis, adaptive benchmarks, and automated red teaming.

7 SAFETY IN LLM-BASED APPLICATION

LLM-based applications are reshaping workflows while introducing interconnected risks involving truthfulness, privacy, robustness, copyright, ethics, and governance. Addressing these risks requires safeguards spanning technical protection, content and copyright management, ethical controls, regulatory compliance, and collaboration.

  • LLM-based applications enhance productivity and human-machine collaboration across content creation, programming, healthcare, and financial analysis.
  • Truthfulness: Hallucinations can produce plausible but inaccurate content, creating particular risks in healthcare, law, and finance.Examples include nonexistent diseases, unverified treatments, and flawed market forecasts.
  • Privacy: Training-data exposure and weak access controls can leak personal information, corporate secrets, and medical records.
  • Robustness: Prompt injection and jailbreak attacks can bypass restrictions, enabling harmful content or vulnerable code generation.Coding assistants may be manipulated into producing code containing security vulnerabilities or backdoors.
  • Copyright and Ethics: LLMs may reproduce copyrighted material or reinforce stereotypes, gender discrimination, and racial biases in consequential decisions.
  • Governance: Deployment challenges include misinformation, privacy, adversarial manipulation, copyright infringement, ethical concerns, and evolving regulatory requirements.The proposed response integrates privacy protection, content governance, copyright management, ethical safeguards, regulatory compliance, and collaboration across academia and industry.

8 POTENTIAL RESEARCH DIRECTIONS

The survey identifies data generation, post-training, model editing and unlearning, and LLM-agent safety as major directions for future research. It emphasizes reliable data, secure alignment, efficient but safe model updates, and stronger protection for agent tools and memory.

  • Data Generation: Reliable and automated data generation is a promising direction for safer, more robust model training.The survey treats data integrity as fundamental to training reliability.
  • Post-training: Secure fine-tuning and alignment remain important, with multi-objective alignment emerging as a potential research focus.
  • Model Editing and Unlearning: Model editing and unlearning could support efficient, continuous knowledge updates, but require safeguards against models becoming misaligned with human intentions.
  • LLM Agents: LLM agents require robust safety assurances for their tools, memory, and embodied-intelligence settings such as web and computer agents.

9 CONCLUSION

The survey introduces full-stack safety as an integrated analysis of security concerns across the LLM lifecycle, addressing the stage-specific focus of prior surveys. Its review organizes vulnerabilities and future directions across production, deployment, and use, underscoring the need for lifecycle-wide security.

  • Full-stack safety covers LLM security from data preparation and pre-training through post-training, deployment, and commercialization.
  • The survey systematically reviews over 900+ papers on safety issues across production, deployment, and use, including agents and applications.
  • The review identifies vulnerabilities involving privacy, toxic data, harmful fine-tuning attacks, and deployment challenges.
  • Ensuring robust security across the entire LLM lifecycle is presented as crucial for responsible and effective real-world deployment.
Loading 2504.15585v4…