PaperlayerView exact revision on arXiv

Source-linked AI summary

LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion

Guanghao Zhou, Panjia Qiu, Cen Chen, Hongyu Li, Mingyuan Chu, Xin Zhang, Jun Zhou

arXiv:2602.00038v1cs.CYcs.AI

TL;DR

Fine-tuning can undermine LLM safety, while alignment during training adds complexity and resources. LSSF performs post-hoc realignment by extracting low-rank safety components and combining them with fine-tuned models, with experiments showing restored safety alignment and minimal downstream-performance impact.

  • Problem

    Fine-tuning can compromise LLM safety, while existing alignment methods increase training complexity and computational requirements and may inhibit general capabilities.

  • Method

    LSSF constructs a low-rank projection matrix to extract safety-vector principal components, uses safety singular value entropy to determine safety-critical ranks, and combines the components with fine-tuned models through linear arithmetic.

  • Results

    LSSF restores safety alignment across fine-tuned models with minimal impact on downstream task performance.

  • Takeaways & Limitations

    Low-rank safety components can be used for post-hoc safety realignment while preserving downstream capabilities across the evaluated fine-tuning scenarios.

  • Takeaways & Limitations

    Budget constraints prevented evaluation of larger models such as Llama-3.1-405B-Instruct.

Abstract

from arXiv · show

The safety mechanisms of large language models (LLMs) exhibit notable fragility, as even fine-tuning on datasets without harmful content may still undermine their safety capabilities. Meanwhile, existing safety alignment methods predominantly rely on the fine-tuning process, which inadvertently leads to the increased complexity and computational resources required. To address these issues, we introduce LSSF, a novel safety re-alignment framework with \underline{L}ow-Rank \underline{S}afety \underline{S}ubspace \underline{F}usion. Our proposed method exploits the low-rank characteristics of safety information in LLMs by constructing a low-rank projection matrix to extract the principal components of safety vectors. Notably, this projection matrix represents the low-rank safety subspace of the LLMs, which we have observed to remain stable during fine-tuning process and is isolated from the model's general capabilities. These principal components are used to effectively restore safety alignment when combined with fine-tuned LLMs through linear arithmetic. Additionally, to account for the varying encoding densities of safety information across different layers of LLMs, we propose a novel metric called safety singular value entropy. This metric quantifies the encoding density and allows for the dynamic computation of the safety-critical rank for each safety vector. Extensive experiments demonstrate that our proposed post-hoc alignment method can effectively restore the safety alignment of fine-tuned models with minimal impact on their performance in downstream tasks.

1 Introduction

LLM safety can be weakened by fine-tuning, motivating LSSF, a post-hoc method that extracts low-rank safety components and recombines them to restore alignment while preserving downstream performance. The method also uses safety singular value entropy to select pruning ranks according to layer-specific safety-information density.

  • Fine-tuning can compromise aligned LLMs’ safety, while existing alignment methods add training complexity, computational cost, and potential capability suppression.
  • Safety drift directions exhibit low-rank structure and share a common subspace distinct from directions associated with general capabilities.
  • Experiments on Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct show that LSSF restores safety alignment with minimal impact on downstream task performance.
  • LSSF constructs a projection matrix that extracts low-rank principal components of safety vectors for safety realignment within the low-rank safety subspace.
  • Safety singular value entropy quantifies safety-information density and helps determine the appropriate pruning rank for each safety vector.

2 Related Work

Prior work addresses safety realignment during fine-tuning or by directly adding safety vectors, but these approaches can increase training complexity or suppress downstream capabilities. LSSF extends arithmetic-based methods with low-rank pruning to target safety components more selectively.

  • Safety realignment during fine-tuning increases process complexity, while direct safety-vector arithmetic can suppress downstream task performance.
  • The framework uses low-rank safety principal components extracted from safety vectors before combining them with fine-tuned models through linear arithmetic.
  • LSSF extends task arithmetic to safety fine-tuning and applies low-rank pruning to mitigate effects on downstream capabilities.

3 Methodology

LSSF restores safety after fine-tuning by extracting low-rank safety components and integrating them into downstream-task models. Its method derives safety vectors, identifies safety-critical ranks through singular-value information, and applies a projection-based correction while limiting interference with task directions.

  • Safety vector construction: LSSF derives an inverse safety vector by fine-tuning a safety-aligned model on toxic data, where harmful queries receive affirmative responses.This inverse vector is denoted −δsafe and represents the direction opposite to safety alignment.
  • Safety vector construction: The inverse safety vector contains low-rank safety drift and a general suppression component that impairs general-task performance.The low-rank safety component can be extracted through low-rank decomposition, separating it from general suppression effects.
  • Low-rank orthogonal decomposition: SVD of standardized linear-layer activations produces left singular vectors used to construct a low-rank projection matrix for extracting safety components.The decomposition uses a calibration dataset, and randomized SVD is used to efficiently approximate solutions for large matrices.
  • Low-rank orthogonal decomposition: Singular-value entropy measures principal-component information content from squared singular values and their relative distribution, guiding the retained rank through an information-retention threshold.The retained rank determines the number of principal components preserved by the orthogonal projection.
  • Low-rank projection matrix: The projection matrix retains the most significant left-singular subspaces and applies scaling factors to strengthen or diminish corresponding safety directions.The rank-r projection has rank r, while larger singular values receive proportionally greater weighting.
  • Linear arithmetic: For a downstream-task model, LSSF adds a low-rank safety component to counteract safety-direction drift while minimizing effects on task and redundant directions.The method decomposes downstream fine-tuning changes into task, safety, and redundant offsets; the safety and corrective components share a low-rank subspace.

4 Experiment

Experiments across Qwen2.5 and Llama3.1 evaluate LSSF under LoRA and full fine-tuning, measuring downstream utility and refusal-based safety. Across settings, low-rank safety components restore safety while largely preserving downstream performance, with entropy and singular-vector ablations clarifying the method’s behavior.

  • Experimental Setup: Experiments use Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct, with additional model-scale validation reported for Qwen2.5-3B and Qwen2.5-14B.The main experiments use two instruction-tuned LLMs; Appendix G evaluates robustness across parameter scales.
  • Experimental Setup: Safety is evaluated by refusal rates on AdvBench, HarmfulQA, and CATQA, while utility uses ACC for classification and BLEU and ROUGE-L for generation.Llama-Guard3-8B evaluates safety through refusal rates to harmful queries.
  • LoRA SFT Results: LSSF significantly improves safety without compromising classification performance in the AG’s News and Yahoo Answers LoRA SFT tasks.The reported comparisons appear in Tables 1 and 2; VLGuard provides limited safety improvement when harmful data is mixed into training.
  • LoRA SFT Results: Compared with RESTA, LSSF enhances safety while avoiding downstream-task impact, whereas Lisa’s proximal regularization can inhibit downstream performance.The comparison attributes Lisa’s performance limitation to parameter-change regularization.
  • Full SFT Results: Under full SFT, baselines reduce Llama3.1-8B-Instruct BLEU and ROUGE-L scores by more than 0.2, while LSSF minimizes downstream suppression and maintains safety alignment.Low-rank safety components remain effective across fine-tuning scenarios, whereas other baselines show substantial variation in safe rejection rates across harmful-query datasets.
  • Safety Components: For Llama3.1-8B-Instruct, linear operations with low-rank principal components improve safety robustness without significantly affecting general capabilities.The result supports a shared low-rank subspace between safety drift and extracted safety components that remains separate from general-capability directions.
  • Ablation Studies: As η decreases, low-rank component matrices become smoother and approach zero, indicating that safety singular value entropy removes redundant directional information while retaining structured subnetworks.The visualization uses random down-sampling of the safety-vector delta matrix and projection matrix components.
  • Ablation Studies: When η > 0.8, safe refusal approaches 1.0 while BLEU is only slightly affected; increasing left-singular-vector weight beyond 2.5 similarly brings safety close to the safety model.These ablations show how entropy threshold and projection weighting regulate the safety–utility trade-off.

5 Conclusion

LSSF addresses safety alignment issues caused by fine-tuning by exploiting a low-rank safety subspace that remains largely unchanged and isolated from general capabilities. The framework uses safety principal components to rectify safety drift while preserving downstream performance.

  • Fine-tuning can compromise LLM safety guardrails, motivating the LSSF safety realignment framework.
  • LSSF uses low-rank safety principal components to rectify fine-tuning-induced safety drift within the low-rank safety subspace.
  • The low-rank safety subspace remains largely unchanged during fine-tuning and is isolated from directions representing general capabilities.
  • LSSF restores safety alignment without compromising downstream-task performance.

6 Limitations and Ethics Statements

The authors identify limits related to model scale and the use of toxic data in experiments. They argue that studying harmful model behavior and mitigation can support progress in LLM safety.

  • Budget constraints prevented evaluation of larger models such as Llama-3.1-405B-Instruct.
  • The authors plan to extend LSSF to multimodal and mixture-of-experts models.
  • The experiments used toxic data to compromise model safety, which may negatively affect the safety of open-source models.
  • The authors state that analyzing harmful LLM aspects and exploring mitigation strategies can drive progress in enhancing LLM safety.

A Proof of Shared Low-Rank Subspace

The proof formulates a low-rank approximation of safety-related weight changes using calibration activations and Frobenius-norm minimization. Eckart–Young theory motivates retaining dominant singular components while excluding directional drift that suppresses general capabilities.

  • The input matrix X is computed from a weight matrix W using activations from the calibration dataset D_anchor.
  • The formulation assumes W is obtained by safety fine-tuning an unsafe model and separates safety-related shifts from directional drift.
  • Frobenius-norm minimization defines the low-rank approximation cW for the safety-related weight change.
  • Directional drift that suppresses general capabilities is excluded because it does not affect the model’s safety.
  • Eckart–Young theory selects the top r singular values and corresponding singular vectors as the optimal rank-r approximation.

B Proof of The Singular Value Entropy

This proof establishes the singular-value formulation of the Frobenius norm through SVD. It uses orthogonality and the trace of the squared singular-value matrix to derive the norm expression.

  • For an m × n matrix, the Frobenius norm is defined as the square root of the sum of the squared matrix elements.
  • SVD decomposes a matrix A into orthogonal matrices U and V and a diagonal singular-value matrix Σ.
  • The derivation concludes by expressing the Frobenius-norm quantity through the singular values.
  • The orthogonality of V simplifies the product A⊤A in the singular-value derivation.
  • Because Σ2 is diagonal, Tr(Σ2) equals the sum of its diagonal elements.

C Proof of The Optimality of SVD

The section derives a rank-constrained approximation by projecting a weight matrix onto the subspace spanned by its first r left singular vectors. It also lists alignment baselines and experimental hyperparameters used for comparison.

  • Optimality proof: SVD provides the optimal rank-r approximation of Z = WX_in under the Eckart–Young theorem.The decomposition uses the first r left singular vectors to construct the approximation.
  • Optimality proof: The solution sets Ŵ = UU⊤W, where U contains the first r left singular vectors.U is an orthogonal d_out×r matrix.
  • Optimality proof: Because UU⊤ is a rank-r projection matrix, rank(Ŵ) ≤ r, making Ŵ the optimal solution to the constrained minimization problem.The rank bound establishes feasibility while the SVD construction establishes optimality.
  • Baselines: The comparison includes NA-SFT, VLGuard, Lisa, and RESTA as fine-tuning or post-alignment baselines.These methods respectively omit safety enforcement, reinforce alignment during fine-tuning, separate alignment and task optimization, or merge safety and model weights.
  • Experimental setup: All tasks use 10 epochs and a 1 × 10−5 learning rate, while LoRA fine-tuning uses r = 16 on 8 Nvidia A100 GPUs.The listed baseline-specific settings include 500-sample alignment datasets for VLGuard and Lisa, and weights 1, 1, and −1 for RESTA.

D.2 Dataset

The experiments evaluate downstream utility, safety, and general proficiency across classification, generation, harmful-query detection, and broad capability benchmarks. They use Llama3.1-8B-Instruct with safety fine-tuning based on JailJudge and multiple evaluation datasets.

  • Downstream datasets: The downstream tasks include AG’s News and Yahoo Answers classification plus Chinese medical dialogue generation.AG’s News has four categories, while the medical dialogue training data is sampled from internal medicine.
  • Metrics: Classification uses accuracy, while generation uses BLEU and ROUGE-L to measure n-gram precision and sequence-level similarity.Accuracy is based on correctly predicted samples over total samples; ROUGE-L uses the longest common subsequence.
  • Safety evaluation: Safety evaluation uses AdvBench, HarmfulQA, CATQA, and Llama-Guard3-8B for harmful-request assessment.The first three datasets cover harmful requests or questions across malicious topics and safety themes.
  • Safety-model construction: Llama3.1-8B-Instruct is fine-tuned with sampled JailJudge data, while DoAnythingNow and AdvBench are used to evaluate safety.JailJudge includes synthetic, adversarial, in-the-wild, and multilingual risk scenarios.
  • General capability evaluation: General proficiency is assessed with MBPP, GSM8K, BBH, MMLU, and IFEval across programming, mathematics, reasoning, knowledge, and instruction following.These datasets collectively cover diverse domains and tasks.
  • Experimental setup: Training uses 3 epochs and a 1e-5 learning rate on 8 Nvidia A100 GPUs.The reported setup applies during the training process for the described evaluations.

E Robustness Against the Number of Calibration Datasets

The calibration analysis tests how dataset size and harmful-data proportions affect safety realignment during Yahoo Answers LoRA fine-tuning. The method reaches optimal performance at 64 calibration samples, while 128 is used by default.

  • Calibration-set size: 64 calibration samples achieve optimal performance without significant fluctuation in the Yahoo Answers LoRA safety-realignment experiment.The experiment varies samples from PKU-SafeRLHF on Llama3.1-8B-Instruct.
  • Calibration-set size: The default calibration dataset size is 128 samples to align with the referenced experimental setup.This choice follows the reported setup rather than the empirically optimal size identified in the sensitivity analysis.
  • Harmful-data robustness: Accuracy and rejection rates show no significant changes as the harmful-data proportion increases in mixed Yahoo Answers training data.The robustness test mixes harmful and safe PKU-SafeRLHF data during LoRA SFT on Llama3.1-8B-Instruct.

G Robustness Against Model Parameters

The robustness analyses examine model size, singular-value entropy, and layer-specific safety encoding. The method maintains high refusal rates across Qwen2.5 sizes, while entropy controls retained rank and reveals that denser safety encoding can be represented with lower rank.

  • Model-size robustness: A 0.99 safety refusal rate is achieved for Qwen2.5 models with 3B, 7B, and 14B parameters.The models are evaluated after LoRA SFT on AG’s News and Yahoo Answers followed by safety realignment.
  • Entropy and rank: Higher singular value entropy ratios retain more weight-matrix ranks, allowing entropy to control information retained in low-rank safety components.The relationship is analyzed for Llama3.1-8B-Instruct across layers.
  • Robustness analyses: Figure 6 evaluates robustness against varying proportions of harmful and safe data, while Figure 7 relates entropy thresholds to retained rank across layers.The figures provide complementary robustness and rank-sensitivity analyses.
  • Entropy and rank: Safety singular value entropy quantifies layer- and matrix-specific encoding density and enables dynamic rank selection for low-rank pruning.Different weight matrices within a layer can encode safety information at different densities.
  • Principal-component visualizations: Decreasing singular value entropy reduces the rank of low-rank safety principal components and produces progressively smoother visualizations.This behavior is shown for safety vectors and their principal components across multiple layers and projection matrices.
Loading 2602.00038v1…