PaperlayerView exact revision on arXiv

Source-linked AI summary

DNAlign: Dynamic Null-Space Safe Alignment for LLMs

Jisheng Dang, Yushuo Zhao, Dewei Liu, Junfeng Fang, Bimei Wang, Tiantian Rao, Hong Peng, Bin Hu, Tat-Seng Chua

arXiv:2610.02844v1cs.AI

TL;DR

LLM safety alignment must contend with approaches that can be costly or disrupt knowledge needed for benign tasks. DNAlign combines dynamic control with null-space projection to target safe behavior while preserving model utility. Evaluations report reduced harmful outputs and competitive fluency, coherence, and factual utility across multiple backbones.

  • Problem

    Safety-alignment approaches can be costly or interfere with pre-trained knowledge, degrading fluency and factual accuracy on benign tasks.

    Source: §I · p. 1

  • Method

    DNAlign models LLMs as dynamic systems and combines control signals with null-space constraints to target harmful content while preserving neutral knowledge.

    Source: §I · p. 2, 1) System State Definition · p. 4

  • Results

    DNAlign outperforms unedited models and other alignment methods on safety reward and core performance, with reported gains on utility and generalization evaluations.

    Source: §I · p. 2, J. Summary and Key Findings · p. 12

  • Takeaways & Limitations

    The authors conclude that DNAlign balances safety alignment with core-knowledge preservation and performs across evaluations of safety, utility, generalization, and practicality.

    Source: §VI · p. 12

Abstract

from arXiv · show

Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign.

I. INTRODUCTION

DNAlign addresses the challenge of aligning LLM outputs with human safety values while preserving fluent, factual responses to benign queries. It combines dynamic control and null-space constraints, and is evaluated on multiple 7B-scale models for safety and core performance.

I. INTRODUCTION

  • LLMs must balance safety alignment with preserving factual, fluent responses to benign queries.The introduction frames these as two challenges for reliable deployment.Source: §I · p. 1

A. Safety Alignment for LLMs

  • Existing RLHF and DPO require extensive human labeling, while test-time interventions can be brittle or degrade benign-task fluency and factual accuracy.Prompt engineering and controlled decoding may weaken under changes in phrasing, domain, or prompt structure.Source: §I · p. 1

I. INTRODUCTION

  • DNAlign combines control signals and null-space constraints to suppress harmful content without interfering with pre-trained knowledge.The introduction describes this as a unified framework for safety alignment and knowledge preservation.Source: §I · p. 2
  • On Vicuna-7B and Qwen-7B, DNAlign outperforms unedited models and other alignment methods in average reward, coherence, and diversity, with generalization to new harmful scenarios.The introduction reports evaluations on multiple 7B-scale LLMs.Source: §I · p. 2

II. RELATED WORK

  • Prior research often addresses safety alignment and knowledge preservation in isolation, leaving their integration fragmented.The related-work overview identifies the separation of these goals as a trade-off limiting real-world applicability.Source: §II · p. 2

B. Preservation of LLMs’ Core Knowledge

Prior knowledge-preservation methods range from costly fine-tuning and continual-learning approaches to targeted editing and null-space projection, but existing methods have limitations for modern LLMs and behavioral alignment. The paper frames joint safety and knowledge preservation as a fragmented problem, then reviews LLM generation as next-token prediction from contextual hidden states.

B. Preservation of LLMs’ Core Knowledge

C. Integration of Safety and Knowledge Preservation

  • The paper identifies a gap in jointly optimizing safety and knowledge preservation: sequential pipelines can overwrite knowledge corrections, while prompt-based strategies are brittle and do not address harmful learned associations.Sequential approaches include knowledge editing followed by RLHF alignment,; prompt-based strategies are described in,.Source: C. Integration of Safety and Knowledge… · p. 3

III. PRELIMINARY

  • The preliminary section establishes LLM generation and knowledge retention as the two foundations for DNAlign’s safety-alignment design.The LLM account describes next-token prediction from preceding context through a Transformer and LMHead.Source: §III · p. 3, A. Large Language Models · p. 3

A. Large Language Models

  • Final-layer hidden states are presented as semantic anchors distinguishing neutral from safety-relevant prompts, while response quality is scored against human preferences.Neutral prompt–response pairs P0 represent general knowledge, and harmful pairs P1 require safety alignment.Source: A. Large Language Models · p. 3

B. Mathematical Framework for Knowledge Retention

DNAlign models LLM generation as a discrete-time dynamic system and uses transient interventions to steer safety behavior while preserving pre-trained generation logic. Its knowledge-retention framework constrains interventions to the approximate null space of neutral-prompt hidden states.

B. Mathematical Framework for Knowledge Retention

  • Interventions are constrained to approximately lie in the null space of K0, minimizing their impact on neutral knowledge representations.K0 contains final-layer hidden states for neutral prompts, and the intervention signal is u_t.Source: B. Mathematical Framework for… · p. 3

IV. METHOD

  • DNAlign combines control theory and null-space projection to target safety interventions while protecting pre-trained neutral knowledge.The framework treats the LLM as a discrete-time dynamic system.Source: §IV · p. 3

A. Dynamical System Formulation for LLMs

  • The dynamic-system formulation uses transient state adjustments rather than parameter changes, aiming to preserve contextual coherence, semantic fluency, and neutral knowledge encoding.The framework models sequential internal-state evolution and control-signal interactions during token generation.Source: A. Dynamical System Formulation for LLMs · pp. 3–4

1) System State Definition:

  • The state combines contextual KV memory h_t with the final-layer pre-logit vector o_t, which directly parameterizes the next-token distribution.The initial state is computed from the full prompt using the model’s original pre-trained forward pass.Source: 1) System State Definition · p. 4

2) State Transition and Control Signal Injection:

The transition model retains the LLM’s native state-update mechanism and injects task-specific control signals at each generation step. The signals adjust token probabilities and context while leaving model parameters unchanged.

2) State Transition and Control Signal Injection:

  • Without intervention, the model updates its KV context using the generated token and computes the next pre-logit state through its native Transformer transition, with fixed LMHead weights.The next token is sampled from the softmax distribution over the current pre-logit vector.Source: 2) State Transition and Control Signal… · p. 4
  • Control signals u_t are recomputed at each step without modifying model parameters, and combine a pre-logit adjustment with a structured KV-context adjustment.The pre-logit component biases token probabilities away from harmful candidates; the KV adjustment is intended to keep updated context semantically consistent.Source: 2) State Transition and Control Signal… · p. 4

B. Null-Space Constraints for Knowledge Preservation

DNAlign uses null-space constraints to limit safety interventions to directions orthogonal to representations of neutral knowledge. It constructs the protected subspace from neutral-prompt hidden states and projects control signals into its null space.

B. Null-Space Constraints for Knowledge Preservation

  • Null-space constraints restrict safety-control signals to directions orthogonal to subspaces encoding neutral knowledge.The stated aim is to target harmful content without altering the model’s native processing of neutral prompts.Source: B. Null-Space Constraints for… · p. 4

1) Construction of the Projection Matrix:

  • Neutral prompts from SHP are encoded as final-layer hidden states and aggregated into a matrix whose principal components define the protected subspace.The retained basis uses the first k left singular vectors from an SVD; cross-validation selects k to balance knowledge coverage and subspace compactness.Source: 1) Construction of the Projection Matrix · p. 4
  • The projection matrix P maps control signals into the null space of the protected subspace, making them orthogonal to it.The matrix is constructed from the retained protected-subspace basis.Source: 1) Construction of the Projection Matrix · pp. 4–5

2) Constraining Control Signals:

DNAlign projects control components into the null space of the protected neutral-knowledge subspace before injecting them into the LLM state. A human-preference-trained value function guides optimization of the signals.

2) Constraining Control Signals:

  • Before state injection, DNAlign projects both components of the control signal into the protected subspace’s null space.The KV-pair component is projected using P; the pre-logit component follows the same logic.Source: 2) Constraining Control Signals · p. 5
  • Projected signals have no components in the protected neutral-knowledge subspace, while retaining the intended ability to act on harmful-content subspaces.The paper presents preservation of fluent, factual benign responses as a theoretical consequence of this constraint.Source: 2) Constraining Control Signals · p. 5
  • The projection matrix is computed offline, and its signal multiplication is described as lightweight relative to Transformer operations.This design avoids additional computational overhead during online inference.Source: 2) Constraining Control Signals · p. 5

1) Value Function Training:

  • A value function trained on HH-RLHF safe/helpful pairs estimates state rewards to guide control-signal optimization toward human-preferred safe outputs.The function is a three-layer MLP trained with Bellman loss; the reward may use next-state estimates or a final UltraRM-13B reward prioritizing harmlessness.Source: 1) Value Function Training · p. 5

2) Gradient Ascent for Control Signal Update:

DNAlign uses gradient ascent to optimize control signals for expected reward while regularizing perturbations. Its end-to-end workflow loads an offline projection matrix, projects optimized signals, and uses them to update the LLM state for token generation.

2) Gradient Ascent for Control Signal Update:

  • DNAlign optimizes control signals by gradient ascent to maximize expected long-term reward while limiting perturbations with a regularization coefficient.The update rule uses step size α and 100 gradient-ascent iterations to refine the signal for harm suppression without incoherence.Source: 2) Gradient Ascent for Control Signal… · pp. 5–6

D. End-to-End Workflow of DNAlign

  • The workflow computes an initial state, loads the precomputed projection matrix, optimizes and projects control signals, then updates the state for token generation.The initial state is formed from attention KV pairs and a pre-logit vector; generation repeats until EOS or the 200-token limit.Source: Figure 3 · p. 6, D. End-to-End Workflow of DNAlign · p. 6
  • Loading the offline projection matrix avoids on-the-fly decomposition and consumes less than 0.1% of total inference time on Vicuna-7B.The workflow is described as avoiding full-model fine-tuning through offline preprocessing and a lightweight value function.Source: D. End-to-End Workflow of DNAlign · p. 6
  • Projected controls modify the state before the next token is sampled from the resulting pre-logit vector.The paper describes KV adjustment for coherence and pre-logit adjustment to suppress harmful tokens.Source: D. End-to-End Workflow of DNAlign · p. 6

V. EXPERIMENTS

The experiments are organized around safety, utility preservation, generalization, and deployment efficiency. The stated workflow prepares the projection and value function, runs models and baselines on multiple datasets, and analyzes performance and extensions.

V. EXPERIMENTS

  • The evaluation frames four questions: harmful-output reduction, preservation of benign-task quality, generalization to novel harmful scenarios, and practical efficiency.It also asks how step size and iteration count affect safety and utility metrics.Source: §V · pp. 6–7

A. Research Workflow

  • The preparation stage builds the projection matrix from SHP neutral prompts and trains the value function on HH-RLHF under reward-model supervision.Vicuna-7B, Qwen-7B, and baselines use unified random seeds and hyperparameters for comparison.Source: A. Research Workflow · p. 7
  • Methods are evaluated on HH-RLHF, SHP, harmful benchmarks, and harmless benchmarks, with outputs used to compute four core metrics.The workflow aggregates dataset results to examine the safety–utility trade-off and conduct sensitivity, inference-time, and projection-matrix ablation studies.Source: A. Research Workflow · p. 7

B. Experimental Setup

Experiments compare DNAlign on two 7B backbones across mixed, harmful, and benign prompts, using established baselines and four metrics to assess safety, utility, and quality.

B. Experimental Setup

  • DNAlign is evaluated on Qwen-7B and Vicuna-7B against unedited backbones, Static RE, Contrastive Decoding, and RE-Control.HH-RLHF and SHP cover mixed scenarios; RealToxicityPrompts, ToxiGen, and Categorical-HarmfulQA assess harmful prompts, while six datasets assess benign scenarios.Source: B. Experimental Setup · p. 7

C. Evaluation Metrics and Tools

  • Four metrics jointly assess safety, utility, practicality, and quality, with average reward measuring alignment with human preferences via UltraRM-13B.The reward averages scores over 1,000 test samples; higher values indicate better alignment with safe and helpful interactions.Source: C. Evaluation Metrics and Tools · p. 7
  • Coherence uses TF-IDF cosine similarity to assess semantic similarity, while diversity tracks repeated 2-, 3-, and 4-grams in generated text.Higher diversity scores indicate richer, more varied language.Source: C. Evaluation Metrics and Tools · pp. 7–8
  • The comprehensive quality score rates responses from 1 to 10 across helpfulness, accuracy, fluency, and safety using DeepSeek-R1-Distill-Qwen-32B.The evaluation model is reported to correlate 0.82 with human evaluator scores.Source: C. Evaluation Metrics and Tools · p. 8

D. Performance Comparison on HH-RLHF and SHP

On HH-RLHF and SHP, DNAlign reports improved safety and maintained utility across both backbones. Sensitivity tests identify a balanced configuration of 100 epochs and a 0.2 step size.

D. Performance Comparison on HH-RLHF and SHP

  • On HH-RLHF, Vicuna-7B average reward is 1.2258 vs. 0.9614 for RE-Control, while Qwen-7B average reward rises from 0.7218 to 0.9430.Vicuna-7B also improves coherence (0.4325 vs. 0.3887) and comprehensive quality (6.16 vs. 5.94); the Qwen-7B result has the highest comprehensive quality.Source: D. Performance Comparison on HH-RLHF… · pp. 8–9
  • On SHP, Vicuna-7B average reward (1.0756) and comprehensive quality (6.97) exceed RE-Control, with diversity and coherence stable; Qwen-7B shows the same positive pattern.The authors report that performance improvements span safety-oriented and utility-oriented metrics while preserving output diversity.Source: D. Performance Comparison on HH-RLHF… · p. 9

E. Hyperparameter Sensitivity Study

  • Coherence is most sensitive to training settings; diversity remains 0.933 to 0.942 except below 30 epochs or above a 0.8 step size.Comprehensive quality and average reward improve with more epochs but plateau or decline when the step size exceeds 0.5.Source: E. Hyperparameter Sensitivity Study · p. 9, Figure 4 · p. 9
  • At 100 epochs and a 0.2 step size, diversity is 0.9366, coherence is 0.4238, comprehensive quality is 7.24, and average reward is 0.943.This configuration is identified as the optimal balanced profile in the tested settings.Source: E. Hyperparameter Sensitivity Study · p. 9, Figure 4 · p. 9

F. Inference Time Analysis

DNAlign's added inference time narrows at larger batch sizes, while projection-matrix tests identify 20,000 samples as a stable, lower-cost choice for Vicuna-7B.

F. Inference Time Analysis

  • At 10 batches, DNAlign takes 12.447 minutes, 1.891 minutes longer than the base model and 1.83 minutes longer than RE-Control.At one batch, DNAlign takes 123.734 minutes; the precomputed projection matrix adds less than 0.1% of total inference time.Source: F. Inference Time Analysis · pp. 9–10

G. Research on Sample Parameters of Projection Matrix

  • Projection-matrix errors fluctuate sharply from 500 to 3,000 samples, reaching 2.923 at 3,000; samples from 1 to 100 are described as overfitting.The low errors at small sample sizes lack generalizability because of insufficient data diversity.Source: G. Research on Sample Parameters of… · p. 10
  • At 20,000 samples, projection-matrix error is 0.0094, comparable to 0.0100 at 50,000 samples while reducing computational costs.The authors identify 20,000 as a stable, low-error choice that also exceeds the 4,096 hidden dimension and supports cross-model comparisons.Source: G. Research on Sample Parameters of… · p. 10

H. Generalization Testing

DNAlign is reported to improve safety alignment while preserving utility across harmful and harmless datasets. The summary also reports negligible inference overhead at larger batch sizes.

H. Generalization Testing

  • On RealToxicityPrompts, DNAlign achieved 0.7999 average reward, a 137.1% increase over Qwen-7B, with diversity of 0.9241 and coherence of 0.4648.The passage describes RealToxicityPrompts as testing responses to model-generated toxic seeds.Source: H. Generalization Testing · p. 11
  • On ToxiGen, DNAlign achieved 0.5465 average reward, a 229.4% increase over the base model, with coherence of 0.2251 and diversity of 0.9615.Across the three external harmful-prompt datasets, the authors report strong generalization; comprehensive quality may be marginally lower than the unedited backbone on some datasets.Source: H. Generalization Testing · p. 11

I. Impact Analysis on Harmless Datasets

  • Across six harmless datasets, diversity ranged from 0.8464 to 0.8836 and coherence from 0.4077 to 0.5692, with performance closely competitive with Qwen-7B.DNAlign outperformed RE-Control on key utility and safety metrics; comprehensive quality was marginally lower than the base model in some instances.Source: §I · pp. 11–12

J. Summary and Key Findings

  • Across RQ1–RQ4, DNAlign reports highest HH-RLHF Average Reward and Comprehensive Quality (1.2258 with Vicuna-7B), competitive harmless-set Diversity and Coherence, and negligible overhead at larger batches.The summary also reports an Average Reward gain of up to 81.9% over baselines in benign scenarios.Source: J. Summary and Key Findings · p. 12

VI. CONCLUSION

  • The conclusion characterizes DNAlign as balancing safety alignment and core knowledge preservation, with experiments outperforming RE-Control on key metrics across multiple datasets.The authors describe it as a promising solution for reliable, high-quality LLM deployment.Source: §VI · p. 12

Related papers

Loading 2610.02844v1…