PaperlayerView exact revision on arXiv

Source-linked AI summary

Targeted Neuron Modulation via Contrastive Pair Search

Sam Herring, Jake Naviasky, Karan Malhotra

arXiv:2605.12290v1cs.LG

TL;DR

The mechanisms by which alignment fine-tuning produces refusal behavior, and whether identified safety signals can be steered, remain unclear. The paper introduces contrastive neuron attribution to identify sparse MLP circuits, finding that ablating them reduces refusal in instruct models while preserving coherent output, whereas base-model steering shifts content without changing behavior.

  • Problem

    It remains unclear whether late-layer safety signals arise from fine-tuning and whether they can be steered to explain refusal mechanisms.

  • Method

    Contrastive neuron attribution compares harmful and benign prompt activations to identify 0.1% of distinguishing MLP neurons across matched base and instruct models.

  • Results

    Ablating fewer than 0.1% of MLP activations reduces refusal rates by over 50% across tested architectures while preserving coherent output; base models show content shifts without behavioral change.

  • Takeaways & Limitations

    Alignment fine-tuning transforms pre-existing late-layer discrimination structure into a sparse, targetable refusal mechanism.

  • Takeaways & Limitations

    Experiments are limited to Llama and Qwen architectures up to 72B parameters, and standard faithfulness metrics do not directly apply.

Abstract

from arXiv · show

Language models are instruction-tuned to refuse harmful requests, but the mechanisms underlying this behavior remain poorly understood. Popular steering methods operate on the residual stream and degrade output coherence at high intervention strengths, limiting their practical use. We introduce contrastive neuron attribution (CNA), which identifies the 0.1% of MLP neurons whose activations most distinguish harmful from benign prompts, requiring only forward passes with no gradients or auxiliary training. In instruct models, ablating the discovered circuit reduces refusal rates by over 50% on a standard jailbreak benchmark while preserving fluency and non-degeneracy across all steering strengths. Applying CNA to matched base and instruct models across Llama and Qwen architectures (from 1B to 72B parameters), we find that base models contain similar late-layer discrimination structures but steering these neurons produces only content shifts, not behavioral change. These results demonstrate that neuron-level intervention enables reliable behavioral steering without the quality tradeoffs of residual-stream methods. More broadly, our findings suggest that alignment fine-tuning transforms pre-existing discrimination structure into a sparse, targetable refusal gate.

1 Introduction

The paper introduces contrastive neuron attribution (CNA) to identify sparse MLP neurons distinguishing harmful from benign prompts and tests whether these signals are shaped by alignment fine-tuning. Ablating 0.1% of these neurons reduces refusal in instruct models while preserving coherent output, whereas base-model steering mainly shifts content.

  • Motivation: Safety-related late-layer signals distinguish harmful from benign prompts, but their origin in fine-tuning and steerability remain unclear.This motivates testing whether pretrained models contain structures that fine-tuning adapts into safety-relevant functions.
  • Method: CNA compares prompt-set activations to identify the 0.1% of MLP neurons whose activations most distinguish harmful from benign prompts.The method applies the contrastive principle at the level of individual post-activation MLP hidden units.
  • Core findings: Results replicate across Llama and Qwen architectures despite their different fine-tuning paradigms.The evaluation covers both base and instruct variants spanning 1B to 72B parameters.
  • Core findings: Over 50% reduction in refusal rates follows from clamping 0.1% of MLP activations to zero in instruct models across tested sizes and architectures.The intervention maintains coherent output quality and avoids mode collapse at high steering strengths.
  • Core findings: Base-model discrimination neurons produce content shifts without changing refusal behavior, whereas instruct-model neurons become causal safety gates after fine-tuning.Base models contain comparable activation differences, indicating that fine-tuning transforms function rather than creating the discrimination structure anew.

2 Background

Contrastive Activation Addition (CAA) steers model behavior using residual-stream differences between contrastive prompt sets, but operates coarsely across the full residual stream. The proposed method applies the same contrastive idea to individual neurons to identify responsible components.

  • Contrastive steering: CAA computes average residual-stream activation differences between contrastive prompt sets as a control vector for inference-time steering.This control vector is used to alter model behavior during inference.
  • Contrastive steering: CAA is effective but coarse because it operates on the full residual stream without identifying responsible neurons.The background motivates neuron-level analysis as a more targeted application of the contrastive approach.
  • Neuron-level steering: The proposed method transfers CAA’s contrastive principle to individual neurons.This formulation is intended to identify which neurons are responsible for the targeted behavior.

3 Method: Contrastive Neuron Attribution

Contrastive neuron attribution identifies behavioral circuits by comparing MLP activations between positive and negative prompts. The method uses forward passes, selects the top 0.1% of neurons by contrastive difference, excludes prompt-insensitive neurons, and tests causality through inference-time scaling.

  • Contrastive discovery: Contrastive discovery defines positive prompts exhibiting the target behavior and negative prompts exhibiting its opposite, then compares their model activations.MLP activations are recorded at the last token position using forward pre-hooks on down_proj.
  • Neuron selection: 0.1% of MLP neurons are selected by the largest absolute mean activation differences between positive and negative prompt sets.The selection spans all layers and uses k equal to 0.1% of total MLP activations.
  • Method properties: The method requires only forward passes and activation comparisons, without gradients, linearization, or auxiliary training.This provides a neuron-level interpretation of contrastive activation steering rather than residual-stream steering.
  • Neuron filtering: Neurons appearing in the top 0.1% of MLP activations for ≥80% of diverse prompts are flagged as prompt-insensitive and excluded from discovered subsets.This filtering targets neurons that fire regardless of prompt content.
  • Causal intervention: Causality is tested by scaling each circuit neuron's activation at inference time, with m = 0 for ablation, m = 1 as baseline, and m > 1 for amplification.Refusal benchmarks span Llama 3.2, Llama 3.1, and Qwen 2.5 variants from 1B to 72B parameters at different steering strengths.

4 Experimental Setup

The experiments compare matched base–instruct models across Llama and Qwen architectures, spanning 1B to 72B parameters. Evaluation measures ablation effects, steering strength, and output quality using defined behavioral and repetition-based metrics.

  • Models: The study evaluates base and instruct variants of Llama and Qwen models ranging from 1B to 72B parameters across multiple layer depths and hardware configurations.Smaller models run on NVIDIA RTX 3080 GPUs in bfloat16, while larger scale-comparison models run on a B200 node in bfloat16.
  • Evaluation metrics: Ablation effect is the change in refusal rate under circuit ablation on JBB-Behaviors, with m = 0 denoting ablation.The setup defines this metric specifically for measuring behavioral change after ablating the discovered circuit.
  • Evaluation metrics: Steering strength α is defined as a multiplier, where 0.0 ablates a neuron and 1.0 is baseline; CAA comparisons transform values so α = 0 is baseline and α = 1 is maximum intervention.Output quality is measured as the complement of the fraction of repeated n-grams in a provided string.

5 Results

Across instruct models, CNA reduces refusals while preserving generation quality and general capabilities, unlike CAA’s severe quality degradation at high intervention strengths. Ablating only 0.1% of MLP activations causally weakens refusal behavior, while alignment fine-tuning transforms late-layer discrimination circuits into behavioral safety gates absent in base models.

  • Steering quality: CNA maintains generation quality above 0.97 across all steering strengths while decreasing refusal rate monotonically in 8 instruct models.CAA reduces refusals but degrades quality sharply at α ≥0.75.
  • General capabilities: CNA preserves baseline MMLU accuracy within 1 point across all steering strengths, whereas CAA falls to near-zero at maximum intervention.The evaluation uses 1,000 MMLU questions averaged across 8 instruct models.
  • Per-model results: At maximum steering strength, CNA preserves generation quality above 0.96 for every instruct model, while CAA falls below 0.60 for 6 of 8 models.These results use 100 harmful prompts, with baseline refusal measured at α = 0.0.
  • Causal ablation: Ablating just 0.1% of MLP activations reduces refusal rates by more than half in most cases while retaining coherent, useful responses.This supports the discovered circuit functioning as a refusal gate over an otherwise capable model.
  • Rubric-based scoring: StrongREJECT compliance scores improve by an average of 6% for Llama and 31% for Qwen after circuit ablation.The Llama-3.3-70B judge scores harmfulness, deception, and dangerousness on a normalized 0-to-1 scale.
  • Base–instruct comparison: In base models, steering late-layer discrimination neurons changes content but never causes refusal, whereas fine-tuning makes the circuit a causal safety gate.After fine-tuning, m = 0 produces compliance, m = 1 standard refusal, and m > 1 stronger refusal; extreme amplification can overwhelm the residual stream.

6 Discussion

The discussion distinguishes shared late-layer discrimination structure from model-dependent neuron function: it shifts content in base models but changes behavior in instruct models. It also identifies sparse late-layer intervention as effective while noting architectural and evaluation limitations.

  • Structure vs. function: Late-layer discrimination neurons occur in both base and instruct models, but their intervention produces content shifts in base models and behavioral change in instruct models.This separates circuit structure from neuron-level function across tested architectures.
  • Implications for targeted intervention: ∼10% of final layers is sufficient for behavioral steering, while ablating 0.1% of MLP activations produces large behavioral change without disrupting response quality.The discussion frames this as targeted intervention using a sparse, late-layer circuit.
  • Structural localization: Layer-by-layer analyses of Llama-3.2-1B and Qwen2.5-3B localize discrimination neurons to the final ∼10% of layers.Quantitative steering results across all 16 models confirm that behavioral effects generalize, while larger-model per-layer analysis remains future work.
  • Future work: Open questions concern whether CNA generalizes to mixture-of-experts architectures and to behaviors beyond refusal that admit clean contrastive pairs.These questions arise because mixture-of-experts MLP structure differs fundamentally and the current work focuses on refusal.
  • Limitations: The evaluation uses behavioral steering, objective response-coherence methods, and benchmarks because raw activation differences do not directly support standard faithfulness metrics.The experiments cover Llama-family and Qwen-family architectures with gated SiLU MLPs and GQA attention, up to 72B parameters.

7 Related Work

Prior work localizes refusal and alignment mechanisms in sparse neuron circuits, late-layer structures, and residual-stream directions. This work extends those approaches with forward-only contrastive discovery and base–instruct comparisons showing pre-existing structure without the same behavioral effect.

  • Neuron-basis circuit discovery: Arora et al. motivate neuron-basis circuit discovery with sparse MLP circuits, while CNA requires only forward passes rather than RelP’s linearization and eager-attention requirements.Arora et al. report ∼100-200 neurons explaining complete task behaviors.
  • Refusal mechanisms: CNA extends refusal-mechanism studies by showing that late-layer structure pre-exists fine-tuning and that instruct-circuit ablation preserves generation coherence.This passage contrasts CNA with SAE-based Hydra Effect analysis and safety-neuron freeze-and-retrain methods.
  • Alignment localization: CNA extends alignment localization by showing that similar structure exists before fine-tuning but lacks the behavioral effect.Chaudhury reported alignment signals concentrated in specific layer ranges of Llama 3.2 1B.
  • Representation engineering: Representation-engineering methods steer refusal through residual-stream directions, whereas this work extends them with neuron-level refusal steering.Arditi et al. showed refusal-direction erasure prevents refusal on harmful prompts and addition elicits refusal on benign ones across 13 models up to 72B parameters.
  • Circuit discovery methods: Unlike iterative circuit-discovery methods, CNA trades faithfulness guarantees for simplicity and requires no gradients, auxiliary models, or iterative search.ACDC and path patching use iterative edge pruning, while RelP achieves comparable quality in a single pass.

8 Conclusion

Contrastive neuron attribution shows that alignment fine-tuning converts pre-existing late-layer discrimination structure into a functional refusal mechanism. Ablating fewer than 0.1% of MLP activations reduces refusal rates by over 50% while preserving coherent output and avoiding residual-stream steering degradation.

  • Alignment and refusal: Alignment fine-tuning transforms pre-existing late-layer discrimination structure into a functional refusal mechanism.Base models show similar activation differences, but steering the identified neurons has no behavioral effect.
  • Alignment and refusal: Base-model neuron steering produces no behavioral effect, indicating refusal crystallizes during post-training rather than reflecting a pre-existing capability.The same attribution method identifies similar activation differences in base and instruct models.
  • Steering efficacy: Fewer than 0.1% of MLP activations reduce refusal rates by over 50% across architectures from 1B to 72B parameters while preserving coherent output.The result holds across all tested architectures.
  • Steering efficacy: Neuron-level ablation avoids the generation degradation that limits residual-stream steering methods’ practical applicability.The conclusion attributes this advantage to intervening at the neuron level rather than on the residual stream.

Impact Statement · A Sample Prompt Sets · B Hyperparameter Details

The paper discusses the dual-use risk of identifying refusal circuits, provides harmful and benign prompt examples for refusal analysis, and documents experimental hyperparameters in a dedicated table.

  • Impact Statement: Identifying refusal circuits may enable targeted attacks on language-model safety mechanisms, creating a potential dual-use concern.The authors argue that the scientific value of understanding alignment mechanisms outweighs this risk.
  • Impact Statement: The authors suggest that understanding refusal-circuit fragility may ultimately provide broader insight into alignment mechanisms.They note that similar findings are emerging across the interpretability community.
  • A.1 Refusal: Harmful Prompts (Positive Class): The harmful-prompt set includes requests involving lock picking, authentication bypass, vehicle hotwiring, fake identification, phishing, account hacking, explosives, and identity theft.These prompts define the positive class for refusal analysis.
  • A Sample Prompt Sets: The prompt examples contrast harmful and benign requests as matched classes for studying refusal behavior.The harmful examples are paired with benign everyday and safety-oriented tasks.
  • A.2 Refusal: Benign Prompts (Negative Class): The benign-prompt set includes everyday requests about baking, Python sorting, tire changes, paper airplanes, thank-you notes, email setup, fire extinguishers, and personal-data protection.These prompts define the negative class for refusal analysis.
  • B Hyperparameter Details: Experimental settings are documented in a dedicated table titled “Experimental hyperparameters.”The supplied passage identifies Table 5 but does not provide its individual parameter values.

C Layer Localization Data · C.1 Layer Concentration Summary · C.2 Base vs. Instruct Concentration

The contrastive discovery method localizes discrimination neurons across Llama and Qwen models, with instruct-model circuits concentrated in late layers across refusal, capitals, and subject-verb agreement tasks. This late-layer pattern also appears in matched base models, indicating it predates fine-tuning.

  • C Layer Localization Data: The study reports full layer-by-layer localization results for the contrastive discovery method across Llama and Qwen models.
  • C.1 Layer Concentration Summary: The layer-concentration analysis measures the fraction of the top-200 discrimination neurons in the final 3 layers and final quarter.The reported values are percentages.
  • C.1 Layer Concentration Summary: In instruct models, discrimination neurons for refusal, capitals, and subject-verb agreement concentrate heavily in late layers.
  • C.1 Layer Concentration Summary: The final 3 layers are designated “Top 3” in the layer-concentration analysis.
  • C.2 Base vs. Instruct Concentration: Late-layer concentration pre-exists fine-tuning, with base models exhibiting similar layer-level patterns to their instruct counterparts.
  • C.2 Base vs. Instruct Concentration: Table 7 compares layer concentration from contrastive discovery across matched base and instruct models on refusal, capitals, and subject-verb agreement tasks.

C.3 Neuron Overlap Between Base and Instruct · C.4 Per-Layer Distribution: Llama-3.2-1B-Instruct

Fine-tuning preserves the layer-level concentration pattern of neuron circuits while replacing most individual neurons, with only 8–29% surviving between matched base and instruct models. In Llama-3.2-1B-Instruct, refusal, capitals, and subject-verb agreement circuits share a late-layer, right-skewed distribution concentrated in L14–L15.

  • C.3 Neuron Overlap Between Base and Instruct: Fine-tuning largely replaces individual neurons despite stable layer-level architecture.The comparison uses shared (layer, neuron) index pairs between matched base and instruct circuits.
  • C.3 Neuron Overlap Between Base and Instruct: The overlap measure counts shared (layer, neuron) pairs out of 200.This definition is reported in Table 8 for neuron overlap between base and instruct models.
  • C.3 Neuron Overlap Between Base and Instruct: 8–29% of individual neurons survive the transition from base to instruct.Fine-tuning replaces the circuit while preserving its layer-level concentration pattern.
  • C.4 Per-Layer Distribution: Llama-3.2-1B-Instruct: All three Llama-3.2-1B-Instruct tasks produce visually similar right-skewed per-layer neuron distributions.The tasks are refusal, capitals, and subject-verb agreement.
  • C.4 Per-Layer Distribution: Llama-3.2-1B-Instruct: The majority of neurons for refusal, capitals, and subject-verb agreement concentrate in L14–L15.This pattern is shown for Llama-3.2-1B-Instruct using contrastive discovery.
  • C.4 Per-Layer Distribution: Llama-3.2-1B-Instruct: 82–87% of neurons lie in L13–L15 across the three tasks.The figure describes concentration in the final 2–3 layers and characterizes late-layer concentration as universal for content discrimination.
Loading 2605.12290v1…