Safety Layers in Aligned Large Language Models: The Key to LLM Security

• Full-parameter fine-tuning can degrade aligned-model security even when datasets contain no harmful data, motivating secure fine-tuning methods.This vulnerability creates a deployment challenge for aligned LLMs.Source: §1 · p. 1
• The study identifies safety layers as a small fraction of middle model layers crucial for recognizing malicious questions.Their existence is attributed to the security alignment process.Source: §1 · pp. 1–2
• Cosine-similarity analysis compares normal-normal with normal-malicious hidden-layer vectors to verify safety-layer existence through distribution discrepancies.The discrepancy begins at a specific layer and later converges.Source: §1 · p. 2
• Parameter scaling and over-rejection are combined to precisely localize safety-layer boundaries across several aligned LLMs.The method uses changes in over-rejection to distinguish candidate layer ranges more clearly.Source: §1 · p. 2
• SPPFT updates only parameters beyond the safety layers, preserving their gradients during fine-tuning while allowing learning from fine-tuning data.The approach is evaluated with harmless normal data and potentially harmful data.Source: §1 · p. 2

2.1 BACKGROUND AND RELATED WORK

• Prior work describes RLHF, instruction tuning, and DPO for alignment, while security alignment can also cause over-rejection and fine-tuning can substantially degrade security.These findings motivate secure fine-tuning research.Source: §2.1 · p. 2

2.2 SETUP OF OUR STUDY

• The study aims to understand how aligned LLM parameters identify malicious problems and how that mechanism can defend against fine-tuning-induced security degradation.Source: §2.2 · p. 3
• Experiments include four aligned LLMs from different publishers to examine whether findings generalize across models.The models are Llama-3-8B-Instruct, Llama-2-7b-chat, gemma-2b-it, and Phi-3-mini-4k-instruct.Source: §2.2 · p. 3
• During inference, each instruction is inserted into a shared dialogue template, tokenized, and passed through the embedding layer to form initial input vectors.Source: §2.2 · p. 3

3 SAFETY LAYERS: EXISTENCE AND LOCALIZATION

• Safety layers are a small middle segment of aligned LLM parameters that is crucial for recognizing malicious queries.The motivation is that identical final-position input vectors can diverge across hidden layers into opposite response tendencies for malicious and normal instructions.Source: §3 · p. 3, §3.1 · p. 3

3.2 LAYER-WISE DIFFERENCES IN INFERENCE ON VARIOUS QUERY TYPES

• Normal-malicious layer-vector pairs diverge more than normal-normal or malicious-malicious pairs, indicating where aligned LLMs begin distinguishing malicious content.The analysis repeatedly samples query-pair vector sets and computes layer-wise cosine similarities during the first autoregressive step.Source: §3.2 · p. 4

3.3 EXISTENCE OF SAFETY LAYERS IN ALIGNED LLMS

• Aligned models show a widening angular gap between normal-normal and normal-malicious pairs in middle layers, whereas pre-trained models show no comparable gap across layers.The contrast supports the conclusion that safety layers emerge from security alignment rather than from ordinary vector-distribution differences.Source: §3.3 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5

3.4 LOCALIZATION OF SAFETY LAYERS

• Cosine similarity provides only an imprecise initial range, so localization combines an approximate gap-based range with parameter scaling and over-rejection measurements.Cosine similarity loses information through dimensionality reduction, while over-rejection is more sensitive when ordinary malicious-query refusal rates are already low.Source: §3.4 · pp. 5–6, §3.4.2 · p. 6

3.4.3 PROGRESSIVE SAFETY LAYERS LOCALIZATION ADJUSTING

• Scaling candidate layer ranges progressively and tracking refused non-malicious queries identifies the boundaries where added layers increase or decrease over-rejection.With α > 1, security-contributing layers raise refusals as they are added, while unrelated layers reduce them; with α < 1, the trend reverses.Source: §3.4.3 · p. 7

3.5 SAFETY LAYERS OF ALIGNED LLMS

• Across aligned LLMs, localization trends show that adding security-contributing layers increases refusals in the over-rejection dataset, while including unrelated layers decreases them.Table 1 uses the 721-problem over-rejection dataset to record scaled ranges and refused-query counts during boundary adjustment.Source: §3.5 · pp. 7–8, Table 1 · p. 7

3.6 DISCUSSION: THREE STAGES OF LLM HIDDEN LAYERS

• Attention patterns support three internal stages: early layers focus on syntactic words, safety layers detect malicious intent, and later layers emphasize semantic content.The later layers also show stronger semantic variation, and transferring them to a pre-trained model improved logical and semantic answering without enabling malicious-query refusal.Source: §3.6 · p. 8, Figure 4 · p. 8

4.1 SPPFT: SAFELY PARTIAL-PARAMETER FINE-TUNING

• SPPFT freezes the safety-layer parameters during fine-tuning so their gradients remain unchanged and security is preserved.The method updates parameters beyond the safety layers while leaving the safety-layer parameters fixed.Source: §4.1 · pp. 8–9

4.2 EXPERIMENT SETTINGS

• The experiments cover normal, implicit, backdoor, and harmful-data fine-tuning attacks using datasets with varying malicious-data rates.Harmful-data datasets contain 1,000 normal samples and 1,000 * p malicious samples for p values 0.05, 0.1, and 0.2.Source: §4.2 · p. 9, §4.3 · p. 10, Table 3 · p. 10

4.3 FREEZE SAFETY LAYERS DURING DIFFERENT FINE-TUNING ATTACK

• SPPFT significantly mitigates security degradation across normal, implicit, and backdoor fine-tuning scenarios compared with full fine-tuning.It limits increases in harmful rate and harmful score in all three settings.Source: §4.3 · p. 9, Table 2 · p. 10
• For implicit attacks, SPPFT raises harmful rate by 2.84% and harmful score by 0.10, compared with 58.03% and 2.17 under full fine-tuning.The corresponding SPPFT increases are 3.51% and 0.08 for normal data, and 4.42% and 0.18 for backdoor data.Source: §4.3 · p. 9
• SPPFT preserves fine-tuning effectiveness and overall performance, with minimal Rouge-L variation and stable MMLU scores relative to full fine-tuning.Freezing discrete security-critical neurons was ineffective, while SPPFT was the most effective parameter-freezing defense in the ablation comparisons.Source: §4.3 · pp. 9–10
• Under harmful-data attacks, SPPFT preserves security better than full fine-tuning across malicious-data rates and outperforms Lisa in the reported setup.The evaluation uses LLaMA-2-chat and Phi-3-mini-4k-instruct at p = 0.05, p = 0.1, and p = 0.2.Source: §4.3 · p. 10, Table 3 · p. 10

5 CONCLUSION

• The work identifies safety layers and generalized methods for locating their range across aligned LLMs.It describes this as the first revelation of security mechanisms within the layer-wise internal parameters of aligned LLMs.Source: §5 · p. 10
• SPPFT preserves the security mechanism by not updating safety-layer gradients during fine-tuning.The method builds directly on the identified layer-wise mechanism.Source: §5 · p. 10

A.1 TEMPLATE OF INSTRUCTION TUNING AND INFERENCE

• During inference, the output field is empty and the inserted question leaves a colon at the final token position for next-word prediction.The same template format is used while processing inference queries.Source: §A.1 · p. 14
• Instruction-tuning data use the keys “Instruction,” “Input,” and “Output,” with separate templates for examples with or without an input field.“Instruction” defines the task, “Input” complements it, and “Output” contains answers and explanations.Source: §A.1 · p. 14

Related defenses

• Existing defenses span alignment-stage robustness, fine-tuning safeguards, prompt-level methods, multimodal defenses, and post-fine-tuning recovery.These approaches include invariant embeddings, harmful-representation removal, data management, state-drift control, security hints, defense datasets, re-alignment, safe subspaces, and parameter pruning.Source: §A.2 · p. 14

Comparison with concurrent work

• Zhao et al. (2024) identifies safety layers by pruning layers and measuring security degradation, then applies knowledge editing through LED.Their method defines the top-k security-associated layers as safety layers and uses additional auxiliary layers during inference.Source: §A.2 · p. 15
• This paper instead identifies safety layers through layer-wise input-vector processing and targets fine-tuning jailbreaks without unrelated auxiliary layers.The safety-layer ranges reported by Zhao et al. (2024), including for LLaMA-2-Chat, also differ from those identified here.Source: §A.2 · p. 15

Robustness across inputs

• Replacing sentences with different semantics causes only minimal fluctuations, indicating that the three curve properties are widespread across aligned LLMs.The result suggests these patterns are inherent to the models rather than artifacts of particular sentence selections.Source: §A.3.1 · p. 15

Layer-wise analysis

• Figures 5–7 evaluate layer-wise final-position-vector cosine similarity for normal-normal, malicious-malicious, and normal-malicious question pairs across three aligned LLMs.The analyses cover Llama-3-8B-Instruct, Llama-2-7B-Chat, and gemma-2B-it, using P = 100, Q = 100, and r = 500.Source: Figure 5 · p. 15, Figure 6 · p. 15, Figure 7 · p. 16

Prompt-template robustness

• The analysis uses the Alpaca template for standardization, while noting that structurally similar templates yield comparable results.The method visualizes how internal layers distinguish normal from malicious instructions through the curve gap.Source: §A.3.2 · p. 16

Default prompt templates

• Default chat templates produce only minor cosine-value changes while preserving the overall curve trends and the layers where normal-normal and normal-malicious gaps emerge and converge.Figure 8 applies each aligned model’s default template and compares it with the Alpaca-template analysis.Source: §A.3.2 · p. 16, Figure 8 · p. 16

A.3.3 REJECTIVE OUTPUT FEATURES OF ALIGNED LLM

• Aligned LLMs begin refusals with publisher-specific fixed templates, including “I’m sorry, but I can’t assist with that” and variants refusing instructions or requests.These templates differ because publishers use different security-alignment data.Source: §A.3.3 · p. 17

A.3.4 HOW TO GET α FOR EACH LLM?

• At α = 1.05, new-layer expansion changed over-rejections only slightly, with an average standard deviation of 2.4, preventing reliable lower-bound confirmation.The appendix cautions that α should be neither too far from nor too close to 1.Source: §A.3.4 · pp. 17–18
• The alpha-selection procedure determines whether α should exceed or fall below 1, then adjusts it in β = 0.05 increments until over-rejection variability reaches a threshold Nu around 15–20.The procedure uses 721 over-rejection samples and seeks a value that preserves semantic comprehension while making each added layer’s security effect distinguishable.Source: §A.3.4 · p. 18
• For Phi-3-mini-4k-instruct, both α = 0.8 and α = 1.2 localized the strongest security effect to layers 11–15, supporting the algorithm’s applicability across α choices.Table 5 reports the α = 1.2 progressive localization experiment over 721 over-rejection problems.Source: §A.3.4 · p. 18, Table 5 · p. 18

A.4.1 SCENARIOS FOR SPPFT

• SPPFT targets benign local fine-tuning and fine-tuning-as-a-service by freezing localized safety-layer parameters to preserve alignment and security.For service providers, SPPFT requires identifying the safety-layer range and supports secure fine-tuning with user-provided data.Source: §A.4.1 · p. 19, §2 · p. 20

1. Attacker Identity and Goals

• The service threat model assumes an API user can upload carefully crafted or backdoor datasets but cannot bypass moderation or access the model’s internal safety-layer parameters.The provider can moderate uploads, access model structure, and freeze safety layers during fine-tuning.Source: §1 · p. 19, §2 · p. 19

1. Defender Identity and Goals

• The defender is the fine-tuning API provider, whose objective is to preserve security and prevent harmful content regardless of user-provided data.The provider’s capabilities include model-parameter access, malicious-content moderation, and secure fine-tuning control.Source: §1 · p. 19, §2 · p. 19

A.4.2 FINE-TUNING HYPERPARAMETERS SETTINGS

• The appendix specifies fine-tuning datasets and evaluation procedures: DI has 4,000 entries, DN has 1,000, DB has 1,500 backdoor plus 1,500 normal entries, and harmful-data runs use learning rate 1e-5.In harmful fine-tuning, SPPFT is compared with full-parameter fine-tuning and Lisa (Huang et al., 2024b), and demonstrates superior prevention of aligned LLM security degradation.Source: §A.4.2 · p. 20, §A.4.4 · p. 22

A.4.4 HARMFUL FINE-TUNING

• SPPFT significantly mitigates security degradation from harmful-data fine-tuning across malicious-data rates, outperforming FullFT and Lisa’s method.SPPFT freezes all safety-layer parameters.Source: §A.4.4 · p. 22

A.4.5 ABLATION EXPERIMENT

• Freezing layers outside the safety layers fails to preserve security and may worsen harmfulness versus FullFT, while freezing only part of the safety layers is weaker than SPPFT.These ablations support the identified safety-layer boundaries.Source: §A.4.5 · pp. 22–23, Table 8 · p. 23

A.4.6 DIFFERENT DATASET

• On the alpaca-clean dataset, SPPFT limits the average harmful-rate increase to 3.74% during backdoor fine-tuning, demonstrating generalizability across datasets.The main experiments used Alpaca-Finance, combining general conversational and domain-specific data.Source: §A.4.6 · p. 23

A.4.7 EVALUATION METRICS

• SPPFT significantly preserves aligned-LLM security during fine-tuning without compromising performance relative to FullFT according to AlpacaEval 2.0 comparisons.Table 9 reports AlpacaEval 2.0 scores for models fine-tuned with SPPFT and FullFT.Source: §A.4.7 · p. 23, Table 9 · p. 23

A.4.8 REVERSE DPO

• Freezing safety-layer parameters during reverse DPO mitigates aligned-LLM security degradation compared with standard reverse DPO.The comparison covers LLaMA-2-chat, Phi-3-mini-4k-instruct, Llama-3-8B-Instruct, and gemma-2b-it.Source: §A.4.8 · pp. 23–24, Table 10 · p. 24

A.5.1 ATTENTION SCORE EXTRACTING DETAILS

• Using the Alpaca-short template to isolate question tokens, heatmaps show attention shifting from syntax before safety layers to semantic words within and key phrases after them.This layer-wise pattern was consistent across tested semantics in Llama-2-7b-chat and Phi-3-mini-4k-instruct.Source: §A.5.2 · p. 24, Figure 9 · p. 24, §A.5.1 · p. 24

A.5.3 TRANSFERRING EXPERIMENT

• Replacing Llama-3-8B’s second-half parameters raises the average score on 100 malicious questions from 1.88 to 3.19, whereas replacing its first half yields 2.03.The results associate post-safety-layer parameters more closely with logical comprehension and output ability.Source: §A.5.3 · pp. 25–26

A.5.4 MORE DISCUSSION

• Cosine-similarity gaps emerge only in intermediate layers of aligned models, remain absent across pre-trained models, and suggest safety layers mediate malicious-versus-normal distinction.The findings also suggest security-related neurons concentrate in contiguous layers whose freezing preserves alignment.Source: §A.5.4 · pp. 26–27