PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models

Miao Yu, Hao Huang, Lu Yuan, Yunpeng Li, Kun Wang, Zuming Jiang

arXiv:2610.09600v1cs.AI

TL;DR

Prior safety interpretability largely examines aligned models, leaving refusal mechanisms in pretrained models and their evolution underexplored. SafeEvo identifies and tracks refusal circuits, and SCA’s circuit-targeted updates reduce average ASR under SFT and DPO while retaining more utility.

  • Problem

    Safety interpretability has largely focused on aligned models, with limited investigation of safety mechanisms in pretrained base models and their evolution during post-training.

    Source: §1 · p. 2

  • Method

    SafeEvo uses optimization-based circuit extraction to identify refusal circuits in pretrained models and trace their evolution, while SCA confines safety updates to an identified refusal circuit.

    Source: §1 · p. 2

  • Results

    SCA reduces average ASR by 2.29% points (43.81% ↓) under SFT and 6.47% points (74.94% ↓) under DPO, while retaining more utility.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across three LLMs, the authors report that circuit-targeted alignment yields stronger safety, less over-refusal, and better utility preservation.

    Source: §5 · p. 10

    The refusal-suppression ablations require white-box weight access and are in principle an uncensoring recipe.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Safety interpretability advances the study of Large Language Model (LLM) alignment from behavioral constraints driven by data or algorithms towards a deeper understanding of internal mechanisms. However, existing works have focused primarily on safety-related representations, attention heads, or neurons after alignment, while largely overlooking the safety mechanisms in pretrained-only models and their evolution across alignment checkpoints. To address this, we propose SafeEvo, an interpretability framework from the circuit (sparse subgraphs of an LLM) perspective. SafeEvo first applies an optimization-based extraction algorithm to identify weak refusal circuits in pretrained base LLMs that can independently express refusal behavior. Causally ablating these circuits completely eliminates the base model's refusal of harmful inputs. SafeEvo then traces the evolution of refusal circuits across successive alignment checkpoints and finds that their structures change progressively, suggesting that the alignment tax may result from refusal-circuit updates affecting utility-related parameters. To validate this, SafeEvo introduces Safety Circuit Alignment (SCA), which confines safety updates to the refusal circuits. Experiments across three LLMs and two alignment algorithms show that, on average, SCA outperforms vanilla alignment in three aspects: \textbf{(1) stronger alignment}, lowering harmfulness score by 63.21\%; \textbf{(2) less over-refusal}, yielding a 58.44\% decrease in refusal rates for benign queries; and \textbf{(3) better utility}, retaining 99.58\% of the original model capabilities.

1 INTRODUCTION

Prior safety-interpretability work largely studies aligned models, leaving safety mechanisms in pretrained models and their evolution underexplored. SafeEvo identifies and tracks refusal circuits, then uses them to guide alignment updates.

1 INTRODUCTION

  • SafeEvo addresses the limited investigation of safety mechanisms in pretrained base models and their evolution during post-training.Its framework attributes alignment to refusal circuits and tracks them throughout safety post-training.Source: §1 · pp. 2–3
  • Ablating the identified circuits raises HarmBench ASR by 42.76% for Llama-3-8B, 55.98% for Qwen-2.5-7B, and 45.91% for Mistral-7B.The circuits are weak and effective only for a subset of harmful queries, yet can independently elicit refusal.Source: §1 · p. 2
  • Adjacent refusal circuits’ overlap falls from 98% at alignment onset to 78% after 20 checkpoints, while independent extractions share only 26% overlap.The authors suggest underconstrained circuit updates may spill over into utility-relevant parameters, offering a possible account of alignment tax.Source: §1 · p. 2
  • SCA confines safety updates to one identified refusal circuit, reducing average ASR by 2.29% points (43.81% ↓) under SFT and 6.47% points (74.94% ↓) under DPO while retaining more utility.Across three LLMs, it also outperforms safety-neuron and random-circuit baselines.Source: §1 · pp. 2–3

2 RELATED WORK

Related work approaches safety through behavioral alignment and mechanistic interpretability, but prior interpretability studies largely focus on post-alignment models and finer-grained components.

LLM Alignment

  • Alignment methods steer harmful-query responses toward refusal through SFT or DPO, while refusal-only optimization can induce over-refusal and erode general capability.Other approaches synthesize jointly helpful and harmless data or add explicit utility constraints to reward shaping.Source: §2 · p. 3
  • SafeEvo distinguishes its circuit-level target by identifying a subgraph that independently realizes refusal, a sufficiency property individual causal neurons do not exhibit.The paper argues that this circuit offers a more intrinsic locus for targeted tuning.Source: §2 · p. 3

Safety Interpretability

  • Prior safety interpretability studies locate safety in representations, attention heads, or neurons, but largely examine post-alignment models.SafeEvo instead traces circuit-level structures from pretrained base models across alignment checkpoints.Source: §2 · p. 3

3 REFUSAL CIRCUIT DISCOVERY AND EVOLUTION

This section tests whether pretrained models contain sparse circuits sufficient for refusal and tracks how their composition changes during safety alignment. It reports sub-1% circuits with 100% out-of-distribution refusal and progressive structural drift.

3.1 REFUSAL CIRCUIT EXTRACTION

  • Optimization-based extraction learns sparse neuron masks using refusal and complementary harmful-compliance objectives, selecting a circuit sufficient for refusal while its complement retains compliance.The model weights remain frozen during mask optimization, and an L1 penalty encourages compactness.Source: §3.1 · pp. 3, 5

3.2 REFUSAL CIRCUITS IN PRETRAINED-ONLY LLMS

  • Extracted circuits containing <1% of parameters reach 100% refusal on OOD BeaverTails queries, while ablating them drops full-model refusal to 0%.The authors conclude that the extraction identifies circuits that generalize to unseen harmful queries.Source: Table 1 · p. 5, §3.2 · p. 6
  • Circuit ablation leaves refusal rates of 0.00%–0.63% and raises HarmBench ASR from 48.43%, 32.70%, and 42.14% to 91.19%, 88.68%, and 88.05%; random-ablation increases are much smaller.The authors interpret these causal ablation results as evidence that safety capabilities may originate in refusal circuits.Source: Figure 3 · p. 5, §3.2 · p. 6

3.3 REFUSAL CIRCUIT EVOLUTION DURING SAFETY ALIGNMENT

  • Warm-started adjacent-checkpoint circuits decline from 1.00 to 0.78 Jaccard similarity, while base-to-later similarity reaches approximately 0.60.The authors suggest that safety updates to refusal circuits may offer a potential account of alignment tax if unconstrained updates affect utility-related parameters.Source: Figure 4 · p. 6, §3.3 · pp. 6–7
  • Independent extractions at one checkpoint overlap by only 0.26 on average, versus 0.78–0.98 for warm-started extractions, indicating non-unique refusal circuits.The authors suggest that multiple valid circuits implement refusal functionality while sharing only some components.Source: §3.3 · p. 6

4 SAFETY CIRCUIT ALIGNMENT

SCA restricts safety-alignment updates to pre-alignment refusal circuits. Across the reported evaluations, it improves safety while preserving utility and reducing over-refusal.

4.1 ALIGNMENT ALGORITHM (SCA)

  • SCA fixes the extracted refusal-circuit mask and restricts updates to circuit neurons, while leaving other neurons and attention modules unchanged; it supports SFT and DPO objectives.The experiments use three LLMs, two objectives, and held-out safety, utility, and over-refusal datasets.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 MAIN SAFETY–UTILITY RESULTS

  • On held-out evaluations, SCA outperforms baselines in alignment efficacy, capability preservation, and avoiding over-refusal.The evaluations use safety and utility datasets that are out of distribution from training.Source: §4.3 · p. 8
  • Across six model–objective blocks, SCA’s HarmBench and AdvBench attack success rates rank first or second among full-LoRA, random-support, and SCA.Under DPO, Llama-3-8B SCA reaches 0.63% on HarmBench and 0.38% on AdvBench, versus 16.98% and 12.69% for full-LoRA.Source: §4.3 · p. 8
  • Across six model–objective blocks, SCA’s largest utility decrease is only 1.44 percentage points across GSM8K, MMLU, and HellaSwag.Several benchmark scores improve over the starting model.Source: §4.3 · pp. 8–9
  • For benign XSTest prompts, SCA reduces over-refusal versus full-LoRA; for example, Llama-3-8B under SFT rises from a 0.8% base rate to 1.6%, versus 18.8%.Qwen-2.5-7B under SFT has rates of 6.8% for SCA and 28.4% for full-LoRA.Source: §4.3 · p. 9

4.4 SCA VERSUS RANDOM BASELINE

  • Across three models, SCA has lower HarmBench ASR than random updates at the same update parameterization and sparsity budget; all paired prompt-level 95% bootstrap intervals exclude zero.The reductions are 14.80, 8.49, and 7.23 percentage points for Llama-3-8B, Qwen-2.5-7B, and Mistral-7B-Instruct-v0.1, respectively.Source: §4.4 · pp. 9–10, Table 4 · p. 9

5 CONCLUSION

SafeEvo extracts sparse refusal circuits and tracks their evolution across alignment checkpoints, moving beyond analyses limited to already-aligned models. Experiments report that pretrained models contain refusal circuits and that SCA’s circuit-confined updates yield stronger safety, less over-refusal, and better utility preservation.

5 CONCLUSION

  • SafeEvo uses differentiable mask optimization to identify sparse, functionally complete refusal circuits and trace their evolution across alignment checkpoints.The framework moves beyond endpoint analyses of already-aligned models.Source: §5 · p. 10
  • Experiments on three LLMs find weak refusal circuits in pretrained-only models that causally govern refusal of harmful queries, and that are neither unique nor stable during safety post-training.Source: §5 · p. 10
  • Confining safety updates to pre-alignment refusal circuits with SCA yields stronger safety, less over-refusal, and better utility preservation.Source: §5 · p. 10

AI USE STATEMENT

The authors report limited use of AI tools for prose polishing and routine coding chores. They state that AI tools did not contribute to research design, experiments, or reported results, and that assisted outputs were checked against raw logs.

AI USE STATEMENT

  • A general-purpose language model polished grammar and phrasing in author-drafted prose and tightened captions for page limits.Source: AI USE STATEMENT · p. 10
  • An AI coding assistant handled routine plotting boilerplate and formatting result dictionaries into table LaTeX.Source: AI USE STATEMENT · p. 10
  • AI tools were not used for research ideas, method design, experiments, interpretation, or reported numbers; authors verified assisted text and scripts against raw logs.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work uses public safety resources and automatic output scoring, while acknowledging that its refusal-suppression ablations could serve as an uncensoring recipe and require white-box weight access.

  • All harmful prompts and completions come from established public benchmarks and alignment datasets used under their original licenses.Source: ETHICS STATEMENT · p. 10
  • The study collected no human-subject data and scored outputs automatically, avoiding exposure of annotators to model outputs.Source: ETHICS STATEMENT · p. 10
  • The authors acknowledge that refusal-suppressing ablations could be an uncensoring recipe, but require white-box weight access.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The authors report releasing anonymized code and making models, datasets, and judges public, with methods, baselines, metrics, and run protocols documented across the main text and appendices.

A PER-MODEL SPARSITY AND MASK PROVENANCE

The appendices describe per-model circuit and mask statistics, an external SN-Tune selector baseline, and recorded circuit-extraction configurations.

A PER-MODEL SPARSITY AND MASK PROVENANCE

A.1 EXTERNAL SN-TUNE SELECTOR BASELINE

  • The SN-Tune reimplementation intersects per-prompt top-1000 attention and top-2000 MLP selections, capped at 100 per layer and component, to gate LoRA updates.The selector gates specified attention and MLP projection slices; selected gradients were nonzero and unselected gradients exactly zero.Source: §A.1 · p. 16, Table 6 · p. 16

A.2 RECORDED EXTRACTION CONFIGURATION

  • Table 8 records the final three starting-model extractions used for ablation and instantiates Equation 6.Table 7 separately identifies the mask object used by each experiment.Source: §A.2 · p. 16, Table 7 · p. 16, Table 8 · p. 17

B ABLATION: EXTRACTED CIRCUIT VERSUS A FIXED MATCHED-RANDOM

The section reports starting-model circuit ablation measures and describes a 50-prompt sufficiency diagnostic, with a width-matched random control and an explicit-refusal criterion.

  • The ablation comparison reports explicit-refusal rate and raw LlamaGuard-3 attack success rate for the intact model, C1 ablation, and width-matched random ablation.Source: Table 9 · p. 17
  • Selected and random identities are both evaluated with zero-ablation, holding intervention type and lesion width fixed.The random control is a prespecified seed-42 draw matched to C1's active count in every Linear.Source: B ABLATION: EXTRACTED CIRCUIT VERSUS A… · p. 17
  • 16.00% of prompts were explicitly refused by the intact Mistral model in the 50-prompt sufficiency diagnostic.Explicit refusals include standard refusal phrases and direct negative formulations declining the harmful request.Source: B ABLATION: EXTRACTED CIRCUIT VERSUS A… · p. 17

C MULTI-SEED ROBUSTNESS AND PAIRED BOOTSTRAP

Across four training seeds, the HarmBench raw-ASR comparisons show paired prompt-level intervals excluding zero; XSTest results reveal a coverage–over-refusal trade-off, under specified training settings.

C MULTI-SEED ROBUSTNESS AND PAIRED BOOTSTRAP

  • All three 95% paired-bootstrap intervals exclude zero for width-matched random-minus-circuit HarmBench ASR differences across four training seeds.The bootstrap resamples 159 prompts; the separate Llama random ensemble is disjoint from circuit seeds.Source: C MULTI-SEED ROBUSTNESS AND PAIRED… · p. 17

D XSTEST COVERAGE–OVER-REFUSAL PROTOCOL

  • Circuit-SFT covers fewer unsafe-contrast prompts than Full-SFT but over-refuses less; under DPO, circuit tuning lowers unsafe-contrast ASR on Llama and Mistral.Qwen full-DPO’s near-total unsafe-contrast ASR reduction coincides with 34.8% benign-prompt over-refusal.Source: D XSTEST COVERAGE–OVER-REFUSAL PROTOCOL · p. 18

E HYPERPARAMETERS

  • Circuit and width-matched-random runs use rank-16 LoRA on MLP projections with frozen attention, while full-LoRA also patches attention Linears.SFT and DPO use different epoch, batch, and learning-rate settings; utility evaluation uses a 5-shot harness capped at 2,000 examples per task.Source: E HYPERPARAMETERS · p. 18

Related papers

Loading 2610.09600v1…