Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection

Motivation

• LLMs’ strong general-purpose capabilities do not ensure safe behavior, motivating alignment for helpful, honest, and harmless deployment.The introduction cites risks including toxic or biased outputs, persuasive misinformation, and assistance enabling harmful actions.Source: §1 · p. 2

Related approaches

• Replay and reference-model proximity can improve capability retention, but add data, complexity, or hyperparameter burdens and do not explicitly remove interfering safety-gradient components.The introduction characterizes these approaches as soft constraints on updates or distributional deviation.Source: §1 · pp. 2–3

Problem framing

• The paper treats sequential alignment as objective-heterogeneous continual learning, where safety-gradient interference with capability-supporting directions is one testable source of alignment tax.The authors do not claim gradient interference explains all degradation; they identify it as a mechanism that admits a local intervention.Source: §1 · pp. 2–3

OGPSA

• OGPSA projects safety gradients onto the orthogonal complement of a low-rank general-capability reference subspace, removing components that would increase selected reference losses to first order.It estimates the subspace from a small, representative set of general data and is designed as a lightweight procedure for SFT or DPO.Source: §1 · p. 3

Evaluation

• Across model families and alignment stages, OGPSA improves the observed safety–utility trade-off over standard baselines, while the paper reports limitations of its first-order approximation.The stated evaluation includes SFT, DPO, and sequential SFT→DPO settings.Source: §1 · p. 3

LLM Safety Alignment

• Prior LLM safety alignment uses test-time guards [22] [30] [23] [58] or steering [29] [63], while safety-data post-training can degrade general capabilities [44] [3] [43].Test-time interventions add inference latency and system complexity.Source: §2 · pp. 3–4
• Replay or task-specific mitigations [44] [35] [70] add training cost or lack universality [59], whereas OGPSA characterizes constrained safety descent only to first order.It avoids large-scale replay but still requires periodic reference-gradient computation; it does not provide a global guarantee of safety or capability preservation.Source: §2 · p. 4

Continual Learning

• Continual learning targets sequential learning without forgetting through regularization, replay, or gradient optimization, but usually assumes homogeneous objectives unlike LLM stages spanning SFT and DPO.LLM training shifts both data distributions and optimization objectives; OGPSA builds its preserved subspace from general-capability reference gradients and projects within standard SFT/DPO-style updates.Source: §2 · p. 4

Positioning relative to gradient-projection CL

• Unlike projection-based CL methods such as GEM [39] and GPM [51], OGPSA adapts subspace construction and validation to preserve broad capabilities across SFT, DPO, and SFT→DPO.The contribution is its formulation for objective heterogeneity, not the projection operator itself.Source: §2 · p. 4

3.1 Sequential Safety Alignment and the Alignment Tax

• The alignment tax is Φ(θpre; Deval) − Φ(θsafe; Deval), measuring evaluation-suite metric loss after safety post-training.Directly constraining the evaluation metric Φ during training is difficult or expensive, motivating a differentiable capability surrogate.Source: §3.1 · p. 4

3.2 Heterogeneous Continual Learning Perspective

• Sequential safety alignment is heterogeneous continual learning: stages shift data distributions and objectives, so safety gradients can interfere with pretrained capability directions and reduce utility [44] [35].The stages may also change the risk functional, such as moving from likelihood-based supervision to preference or ranking optimization.Source: §3.2 · p. 5

3.3 First-Order Capability Preservation via Gradient Orthogonality

• A small reference collection spanning general-ability facets estimates a low-rank subspace; updates orthogonal to its gradients preserve each reference capability to first order.The resulting rule removes from each safety update the component lying in the local general-capability subspace.Source: §3.3 · p. 5

4.1 Overview

• OGPSA removes from each safety gradient its component in a low-rank general-capability subspace before updating parameters.The method is plug-and-play and leaves the underlying safety objective unchanged.Source: §4 · p. 5, Figure 3 · p. 6, §4.1 · p. 6

4.2 General-Capability Subspace Estimation

• Because general-utility metrics can be costly or non-differentiable, OGPSA estimates capability directions from small differentiable reference objectives.It periodically refreshes an orthonormal, low-rank basis from reference mini-batch gradients, filtering redundant directions with thresholded Gram–Schmidt.Source: §4.2 · p. 6

4.3 Projected Safety Optimization

• The projected update is the steepest local safety-descent direction satisfying first-order preservation constraints for the selected reference objectives.This claim applies to the local linearized reference losses; it does not guarantee preservation of all downstream capability metrics.Source: §4.3 · p. 7
• OGPSA adds periodic reference-gradient computations and low-rank projection operations, with overhead below large-scale replay in the reported experiments.The overhead is nonzero and should be reported alongside wall-clock time and token counts.Source: §4.1 · p. 6, §4.3 · p. 8

5.1 Experiments Setup

• Experiments covered LLaMA3.1-8B-Instruct and Qwen2.5-7B-Instruct across SFT, DPO, and sequential SFT-DPO, using 10k PKU-SafeRLHF-derived safety samples and comparing OGPSA with tuning, Merged, LoRA, and 10k-sample UltraFeedback replay.Individual benchmark scores are primary evidence; Avg. Gain is a compact summary relative to the instruct baseline.Source: §5.1 · p. 8

5.2 Overall Performance

• OGPSA achieved the highest Avg. Gain across both models and all three stages; in sequential SFT-DPO, Avg. Gain rose from 33.98% to 42.74% on Qwen2.5-7B-Instruct and from 19.74% to 32.98% on Llama3.1-8B-Instruct.Avg. Gain summarizes the trade-off against the instruct baseline; the paper cautions that it combines heterogeneous benchmarks and should be interpreted alongside individual metric results.Source: §5.2 · p. 8
• Standard alignment often improved safety while reducing some general-utility metrics, and OGPSA had the highest aggregate gain against GPM and STAIR in Qwen2.5-7B SFT.Existing replay, weight interpolation, and LoRA mitigations did not uniformly dominate the trade-off; the advanced-baseline comparison tests one projection-based continual-learning method and one stronger safety-alignment method, not all such baselines.Source: §5.2 · pp. 8–9, Table 2 · p. 9
• On Qwen2.5-7B-Instruct, OGPSA reduced I-GCG attack success rate versus corresponding SFT and DPO baselines while keeping optimization difficulty comparable or higher.This initial stress test does not establish robustness to broader adaptive or multi-turn attacks.Source: Table 3 · p. 9, §5.2 · p. 9

5.3 Ablations

• In DPO ablations, separate Helpful and Truthful directions in a rank-2 subspace gave the best reported balance: SimpleQA 3.35%, IFEval 63.40%, and HHH 90.68%.Single directions offered narrower protection, while a mixed single-gradient projection reached IFEval 61.00%; appendix SFT and math-domain experiments also supported matching reference directions to target capabilities.Source: §5.3 · p. 9, Table 4 · p. 9
• In DPO, 100–200 samples per reference dimension recovered several capability metrics more effectively than 10K-sample replay, while periodic subspace re-estimation outperformed a static projection.Periodic re-estimation every 5 DPO steps and 30 SFT steps balanced optimization empirically; OGPSA adds training time for reference gradients but was faster and more token-efficient than general-data mixing in this setup.Source: Table 5 · p. 9, §5.3 · pp. 9–10, Table 6 · p. 9
• Across tested Qwen2.5 sizes from 0.5B to 7B, OGPSA generally improved retained general-capability metrics under SFT and DPO while maintaining competitive safety scores.On Qwen2.5-0.5B, SimpleQA after SFT increased from 0.09% to 0.88%, and Stereotype increased from 79.31% to 96.74%; the evidence is limited to the tested Qwen2.5 family.Source: §5.3 · p. 10

Conclusion

• OGPSA mitigates capability regression by updating safety gradients along the orthogonal complement of a learned low-rank capability subspace.The paper frames the alignment tax as a heterogeneous continual-learning problem.Source: §6 · p. 10
• Across SFT, DPO, and sequential SFT→DPO pipelines, OGPSA consistently improves the observed safety–utility trade-off.The method is presented as a lightweight, plug-and-play solution.Source: §6 · p. 10

Limitations

• OGPSA’s efficacy is bounded by its first-order local approximation, reliance on reference data diversity, and minor computational overhead in distributed settings.Source: §6 · p. 10

Future Work

• Future work will investigate gradient-conflict diagnostics, stronger compute-matched baselines, comprehensive black-box adaptive safety evaluations, and scaling to larger architectures.Source: §6 · p. 10

7 Impact Statement

• OGPSA may reduce unnecessary capability degradation from safety alignment and preserve usefulness in benign settings.This is a potential positive impact of the method.Source: §7 · p. 11
• Improved post-alignment utility should not be mistaken for comprehensive safety; OGPSA does not guarantee robustness to all jailbreaks, misuse prompts, or deployment distributions.Source: §7 · p. 11
• The authors recommend using OGPSA alongside red-teaming, policy evaluation, refusal calibration, monitoring, and human oversight.Source: §7 · p. 11

• The authors used large language models to polish the manuscript, then reviewed and edited the content and took responsibility for the final paper.Source: §8 · p. 11

A Detailed Experimental Setups

• OGPSA builds its reference set from 200 UltraFeedback helpfulness samples and 200 HaluEval truthfulness samples, while safety and replay data each draw 10k samples.Safety labels use GPT-4omini refusals, and HaluEval hallucinated answers are format-matched to correct answers to prevent reward hacking.Source: A Detailed Experimental Setups · p. 17
• The comparisons cover SFT, DPO, and sequential SFT-DPO, with OGPSA evaluated against weight merging, LoRA, and replay using 10k UltraFeedback samples.The models are LLaMA3.1-8B-Instruct and Qwen2.5-7B-Instruct.Source: A Detailed Experimental Setups · p. 17
• Training uses three SFT epochs and one DPO epoch, with subspace refresh periods K=30 for SFT and K=5 for DPO.The experiments use LLaMA-Factory, batch size 128, and a cosine scheduler with warm-up ratio 0.1.Source: A Detailed Experimental Setups · p. 17
• The 10-benchmark evaluation covers harmlessness, truthfulness, helpfulness, instruction following, and adversarial robustness, using official metrics and default generation temperature.Safety benchmarks are StrongReject, XSTest, WildChat toxicity, and Do-Not-Answer stereotype; utility and robustness benchmarks include SimpleQA, GPQA, MMLU, BIG-bench HHH, IFEval, and AdvGLUE.Source: A Detailed Experimental Setups · p. 17
• StrongReject reports average goodness across PAIR and PAP jailbreaks, XSTest sums full and partial refusal rates, and the WildChat toxicity subset contains 219 English samples.StrongReject goodness is 1 minus the rubric score; XSTest uses its unsafe split, and stereotype refusal is evaluated on Do-Not-Answer’s fairness-related split.Source: A Detailed Experimental Setups · pp. 17–18

B Pseudo-code

• Every K steps, OGPSA samples reference datasets, computes reference gradients, and forms an orthonormal basis; it then removes the safety gradient’s basis-aligned component before updating parameters.The projected gradient is g_safe − U(U^T g_safe), and parameters are updated using learning rate η.Source: B Pseudo-code · p. 18

C Theoretical Derivations

• For any differentiable function, the steepest descent direction restricted to a linear subspace equals the negative orthogonal projection of its gradient onto that subspace.This section provides the mathematical foundation for Proposition 1.Source: C Theoretical Derivations · p. 18

C.1 Steepest Descent Direction in a Linear Subspace

• The derivation models the LLM parameter space as Euclidean, with a differentiable loss, its gradient, an allowable linear subspace, and the orthogonal projection onto that subspace.Source: §C.1 · p. 18
• When the projected gradient is nonzero, the derivation seeks a unit vector in the subspace that minimizes the loss’s directional derivative.Source: §C.1 · p. 18
• Orthogonal decomposition separates the gradient into components inside and outside the subspace, and the outside component has zero inner product with every in-subspace direction.Source: §C.1 · p. 19
• The Cauchy-Schwarz bound shows that maximum descent occurs when the unit direction is collinear with the projected gradient and points oppositely.Source: §C.1 · p. 19
• OGPSA takes the allowable subspace as the null space of general capabilities and the function as the safety loss, making its update optimal within the non-forgetting zone.Source: §C.1 · p. 19

D.1 Overall Performance of Lllama

• Figure 4 reports aggregate safety and general-capacity scores for alignment strategies on Llama3.1-8B-Instruct.The aggregates average four safety datasets and six general-capacity datasets; the caption does not state the scores.Source: Figure 4 · p. 20

D.2 Impact of Subspace Dimensionality and Diversity using SFT

• Table 9 examines how reference-data diversity and constraint-subspace dimensionality affect SFT alignment outcomes on Qwen.It compares Helpfulness versus Truthfulness reference data and one versus two constraint directions; the caption does not report which setting performs best.Source: Table 9 · p. 20

D.3 Generalization to the Mathematical Domain

• Table 10 compares Qwen2.5-7B-Instruct performance on mathematical benchmarks across alignment categories.The caption identifies the best result within each category but supplies no benchmark values.Source: Table 10 · p. 20

D.5 Impact of Subspace Update Frequency using SFT

• Table 11 investigates SFT alignment robustness when Qwen’s reference set is replaced with 200 GSM8K samples.The replacement set is math-only; the caption does not state the resulting alignment outcomes.Source: Table 11 · p. 21

D.4 Impact of Sample Size Budgets for Gradient Estimation

• Table 12 evaluates how DPO gradient-estimation sample budgets affect performance stability on Qwen.The caption says the method remains effective with limited budgets but gives no sample counts or performance values.Source: Table 12 · p. 21

D.5 Impact of Subspace Update Frequency using SFT

• Table 13 compares static subspaces with SFT subspace updates at varying intervals on Qwen.The caption frames the comparison in terms of optimization dynamics but does not specify the best update frequency or its outcome.Source: Table 13 · p. 21

• The authors report that the paper’s main claims accurately reflect its contributions and scope, and that its limitations are discussed.Source: NeurIPS Paper Checklist · p. 22
• The paper reports full assumptions and complete, correct proofs for each theoretical result.Source: NeurIPS Paper Checklist · pp. 22–23
• The authors report that the information needed to reproduce the main experiments is disclosed and that code and data are openly available with reproduction instructions.Source: NeurIPS Paper Checklist · pp. 23–24
• The paper reports specifying training and test details and providing sufficient computer-resource information to reproduce its experiments.Source: NeurIPS Paper Checklist · pp. 24–25
• The authors report that experimental results are not accompanied by error bars.Source: NeurIPS Paper Checklist · p. 24
• The checklist reports that the research conforms to the NeurIPS Code of Ethics and that the paper discusses positive and negative societal impacts.Source: NeurIPS Paper Checklist · p. 25
• The paper reports no crowdsourcing or human-subject research and discloses that LLMs were used to polish the manuscript, with the authors reviewing and taking responsibility for its content.Source: NeurIPS Paper Checklist · pp. 27–28