How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE

Yi Shi; Tanyu Chen; Kai Shen

Source-linked AI summary

How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE

Yi Shi, Tanyu Chen, Kai Shen

arXiv:2609.09793v1Sep 9, 2026cs.CRcs.AIcs.CLcs.LG

TL;DR

Directional ablation had been established mainly on dense models up to roughly 70B parameters, leaving its behavior at frontier MoE scale uncertain. This paper applies it to GLM-5.3-Flash and finds that refusal is removed most thoroughly by jointly editing attention, dense, and routed-expert writers, while conventional module matching reaches only part of the effect.

Problem
Whether a single rank-one weight edit removes refusal at frontier scale remains unestablished, especially when mixture-of-experts models distribute writing across routed experts.
Method
The study exhaustively enumerates residual-stream writers in GLM-5.3-Flash and tests directional ablation across attention, dense, routed-expert, and multi-stream residual components.
Results
Refusal reductions span 41–89 percentage points across seven harmful benchmarks, with no detected capability change; jointly editing all writer groups removes 0.776 of refusal.
Takeaways & Limitations
A single direction reaches most refusal in this model only when projected out of all residual writers, while conventional module matching silently misses most of the effect in the MoE.
Takeaways & Limitations
The study evaluates one checkpoint, GLM-5.3-Flash, without cross-vendor comparison, so its specific numerical findings are properties of that model.

Abstract

from arXiv · show

Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hundred contrastive prompts, which makes it the canonical white-box attack on open-weight alignment. However, it has been established only on dense models up to roughly 70B parameters. We study whether it survives the shift to frontier mixture-of-experts (MoE) models whose residual streams are no longer a single tensor and whose weights ship quantized. We apply it to GLM-5.3-Flash (320B parameters, 288 routed experts, a four-wide hyper-connection residual, block-FP8). The attack survives the architecture, but what it reaches is no longer where a reader of the original recipe would look for it. Editing the attention, dense and routed-expert writers on their own removes 0.039, 0.016 and 0.148 of refusal respectively; editing all three together removes 0.776. As a result, 74% of the effect exists only under the joint intervention. The part the conventional recipe reaches by module-name matching accounts for 0.066 of that 0.776, which is why it fails silently on an MoE. The effect does not follow from removing just any direction: ablating a random direction orthogonal to it leaves refusal unchanged. A category-concentrated residue survives every edit we tried: subspaces fitted on violence, sexual content and hate leave measurable refusal at every rank from 1 to 12. We report the method, the 41-89 percentage-point reductions it achieves across seven harmful benchmarks with no detected change in capability, and the boundary where it stops.

1 Introduction

This paper tests directional ablation on a 320B MoE with sparse experts, a multi-stream residual, and quantized weights. The attack works, but its effect is strongly joint across writer groups and leaves category-concentrated refusal behind.

Motivation and setup: Directional ablation was previously established only on dense models up to roughly 70B parameters.The paper therefore tests whether its architectural premises extend to frontier-scale MoE models.
Motivation and setup: 320B parameters, 288 routed experts, a four-wide hyper-connection residual, and block-FP8 weights define the tested frontier model.The study applies directional ablation to GLM-5.3-Flash, which combines all three architectural shifts examined.
Core findings: 0.039, 0.016 and 0.148 of refusal disappear when editing attention, dense/shared projections, or routed experts alone, versus 0.776 when editing all three.Three quarters of the effect appears only under the joint intervention, showing strongly non-additive writer-group contributions.
Core findings: 0.066 of the 0.776 effect is reached by conventional module-name matching, which skips fused expert parameters without raising an error.This creates a silent failure mode in which the edit appears clean while missing most relevant writers.
Core findings: Directional ablation removes 41–89 percentage points of refusal across seven harmful benchmarks with no detected capability change.The least-cleared benchmarks are denser in the resistant categories, defining the reported boundary.
Core findings: A random orthogonal direction leaves refusal unchanged, while category-concentrated residue survives subspaces fitted to violence, sexual content, and hate at ranks 1 through 12.The tested edits therefore remove much, but not all, refusal-related behavior.

2 Background

The background frames directional ablation as a rank-one weight edit whose validity depends on exhaustive writer enumeration, a single additive residual, and a single refusal direction. MoE and hyper-connection architectures challenge those premises, but scalar stream mixing preserves the relevant orthogonality condition.

2.1 Directional Ablation: Directional ablation estimates a refusal direction from contrastive prompts and replaces each residual writer W with W ← W − rr^T W.The edit projects writer outputs onto the orthogonal complement of r without gradients or optimization.
2.1 Directional Ablation: The original construction assumes exhaustive writer enumeration, a single additive residual channel, and refusal mediated by one direction.The first two assumptions are structural, while the single-direction premise is empirical and model-dependent.
2.2 Sparse Mixtures of Experts: In GLM-5.3-Flash, each routed expert down-projection writes directly to the residual, expanding the feed-forward writer set from one matrix to E + 1.Fused experts are absent from conventional module-name traversal, so a dense-model procedure can silently skip most MoE feed-forward writers.
2.3 Multi-Stream Residuals: Hyper-connections distribute the residual across parallel streams and recombine them through an operator whose effect depends on whether it rotates hidden dimensions.Scalar weighting across streams preserves zero projection along the refusal direction; hidden-dimension rotation would not.
2.3 Multi-Stream Residuals: Writer-level and layer-boundary projections need not be equivalent because sublayers can read intermediate states before a boundary projection is applied.The discrepancy must be measured rather than inferred from the multi-stream architecture alone.

3 Related Work

Prior work established refusal directions, activation steering, MoE safety interventions, and broader concept-erasure methods, mostly in dense models and activation space. This paper addresses the distinct question of how much a single rank-one weight edit removes at frontier MoE scale.

Linear representations and weight editing: Directional ablation was shown across thirteen open-weight chat models up to 72B parameters as a white-box weight edit that removes refusal without inference overhead.The method complements activation steering and other direct parameter interventions.
Other safety-removal methods: Prompt jailbreaks evade safety at the input level, whereas fine-tuning removes behavior from weights using relatively small adversarial datasets.Directional ablation is positioned as a lower-cost direct weight intervention than these alternatives.
Linear concept erasure: Concept-erasure research studies projections in activation space, including iterative nullspace projection and LEACE.The paper distinguishes that literature from editing the weights that write the residual stream.
Geometry beyond a single direction: Later work describes refusal as a multi-dimensional concept cone, category-specific directions, or an affine structure rather than a single universal direction.These findings concern representational geometry and do not establish the effect of rank-one weight editing at frontier scale.
Safety in MoE models: MoE safety studies identify topic-driven routing or expert-specific interventions, while other work reports refusal-related expert routing patterns and attention involvement.Those approaches either tune parameters with gradients or steer activations at inference time, unlike the present white-box edit.
Architectural changes: Hyper-connections and related multi-stream architectures were primarily evaluated for perplexity and downstream task performance, not directional ablation.This leaves their implications for residual-stream interventions as an open architectural question.

4 Method and Setup

The method adapts directional ablation to a 320B MoE with fused experts, a four-stream residual, and block-FP8 weights. It estimates one refusal direction, identifies residual writers structurally, applies rank-one projections, and evaluates harmful refusal, over-refusal, and capability under controlled comparisons.

Target model: Directional ablation keeps the original one-direction, rank-one projection recipe while targeting GLM-5.3-Flash, a roughly 320B-parameter MoE with 288 routed experts and a four-stream residual.The target also uses block-FP8 weights and combines gated linear and latent attention layers.
Writer set: 12,479 matrices write directly into the language residual stream, including 12,442 FP8 and 37 BF16 matrices.The inventory is confirmed against the released checkpoint; 24,921 of 76,108 tensors differ after editing.
Writer set: Expert down-projections account for 99.2% of writers and are fused three-dimensional tensors that module-name matching fails to reach.Writers are collected by traversing parameter-tensor structure and unfolding each expert tensor into one matrix per expert.
Direction and layer selection: The direction uses 256 harmful and 256 harmless prompts for estimation, with separate 32-prompt sets for layer selection and held-out JailbreakBench prompts for testing.A ten-times-layer-median activation mask is included but never fires at layer 22.
Direction and layer selection: Layer 22, at depth 0.49, is selected by validation over nine candidate layers using bypass rate minus a harmless-prompt KL penalty.Layers 20 and 22 are effective while layer 25 is not, so layer choice is measured rather than assumed monotonic.
Residual handling: The four-stream residual is collapsed to one hidden vector for reading, while the edit remains applied directly to residual-writing weights.Because hyper-connection mixing scales streams without rotating hidden dimensions, a direction absent from every stream remains absent after mixing.
Evaluation setup: Evaluation compares base and edited models on harmful refusal, over-refusal, and general capability under identical decoding and classification settings.Responses are re-judged blindly by a different model family, while key same-prompt comparisons use McNemar’s paired test.

5 Results

Directional ablation substantially reduces harmful refusal in GLM-5.3-Flash without detected capability changes, but its effect is strongly non-additive across writer groups and leaves category-concentrated residue.

Main Results: The edited model’s refusal rate falls from 0.51–1.00 to 0.08–0.48 across seven harmful benchmarks, with no detected capability change and zero false refusals on harmless prompts.Every harmful-benchmark reduction is significant, while capability intervals straddle zero; the MMLU interval still admits a drop of 3.7 points.
Main Results: The clean variants remain close to the contaminated originals, supporting generalization beyond the 256 AdvBench prompts used to estimate the direction.The reported pairs are 0.152 versus 0.131 on AdvBench, 0.196 versus 0.150 on JailbreakBench, and 0.327 versus 0.380 on StrongREJECT.
Which Weights the Edit Needs: Editing attention, dense and shared projections, or routed experts alone removes 0.039, 0.016 and 0.148 of refusal, whereas editing all three removes 0.776.The single-group effects sum to 0.203, leaving 0.573, or 74% of the joint effect, attributable only to their combination.
Ablation under a Multi-Stream Residual: Editing writers outperforms projecting at the layer boundary: on 253 shared prompts, rates are 0.174 versus 0.265 under the judge, with 27 of 31 discordant pairs favoring writer edits.The tested comparison covers every decoder layer; the proposed mechanism is that writer edits remove the component before sublayers can read it.

6 Ethics and Responsible Release

The paper frames directional ablation as a measurable security risk for open-weight models and reports both defensive blind spots and release trade-offs. It withholds general attack tooling and fitted artifacts while publishing category-level findings that defenders can use.

Threat model: The attack requires released weights, eight H100s, roughly five minutes of disk writes, and a few hundred harmful prompts.The paper presents these requirements as already available before its contribution, which is measurement of attack reach and limits.
What defenders gain: A module-name audit covers 0.7% of inference-path writers and reports no error, so both attacks and defenses must enumerate writers structurally.The same silent traversal failure affects an audit intended to detect the intervention.
What defenders gain: Refusal survived every fitted subspace in the concentrated resistant categories, although the mechanism carrying that residue remains unknown.Hardening one writer group may encounter the same non-additivity observed in the attack.
Release conditions: The edited weights and model card were already public, but the paper withholds the general implementation, fitted refusal directions, resistant-category subspaces, and fitting prompts.The authors distinguish a specific public checkpoint from a general tool that could apply to other MoE checkpoints.
Implications for safety in the weights: The paper reports a 41–89 percentage-point refusal reduction without detected capability change, making refusal and capability appear separable in the tested setting.The implication is presented as a defense concern because separable refusal can be excised independently.
Research program: The work targets one model and one research program focused on which safety properties require model weights versus inference or agent-infrastructure controls.Its stated subject is attack measurement, not a general evaluation of the model’s safety.

7 Conclusion

The paper applies an unmodified single-direction ablation recipe to a 320B sparse MoE with four residual streams and block-FP8 weights. It finds that the attack reaches most refusal only through joint writer editing, while category-concentrated refusal remains resistant and broader generalization is unmeasured.

Conclusion: The unmodified recipe targets a roughly 320B-parameter MoE with 288 routed experts, four residual streams, and block-FP8 weights.The direction is estimated from harmful-versus-harmless last-token means and removed with a rank-one projection, without training.
Conclusion: Thirty-two quantized-weight orthogonalization iterations reduce residual leakage from 13.4% to 5.2% and baked refusal from 0.188 to 0.094.The lossless-hook prediction is 0.031, showing a quantization-related implementation gap.
Conclusion: The seven-benchmark refusal rate falls from 0.51–1.00 to 0.08–0.48, while capability scores move by no more than 1.0 point.Every reduction is significant under a paired test, and false refusal on harmless prompts falls from 0.024 to zero.
Conclusion: Jointly editing attention, dense/shared projections, and routed experts removes 0.776 of refusal, whereas the individual effects remove 0.039, 0.016, and 0.148.Module-name matching reaches only 0.7% of inference-path writers and removes 0.066 of the joint effect.
Conclusion: Writer-level editing is more thorough than layer-boundary projection, removing 0.265 versus 0.174 on prompts valid in both conditions.The paired test detects the difference even though marginal intervals do not.
Conclusion: Subspaces fitted on violence, sexual content, and hate leave residue between 0.20 and 0.47, with none of twelve ranks removing it.The multi-stream residual does not obstruct ablation because cross-stream mixing is scalar and does not rotate hidden dimensions.
Conclusion: Generalization to other frontier MoE models and the mechanism carrying resistant refusal remain unmeasured.The stated scope is one model, one selected layer for direction estimation, and the intervention procedures tested here.

Limitations

The evaluation is limited by single-checkpoint scope, model-based judgment, partial capability testing, exploratory rank comparisons, and reproducibility constraints. Additional caveats include decoding variation, limited random controls, untested reasoning settings, and unpublished implementation.

Scope: The study evaluates only GLM-5.3-Flash, so specific numerical results and checkpoint-specific choices lack cross-vendor validation.The architecture-dependent conclusions are stated to transfer only to models with the same features.
Measurement: Judge-model scoring uses no human annotation, although agreement with the rule-based classifier ranges from 0.85–0.96.Small effects such as false refusal rely on an improved judge prompt rather than an independent human check.
Evaluation coverage: The four capability benchmarks use only a few hundred sampled items, ruling out large regressions but not replacing a full evaluation harness.
Rank analysis: The rank sweep compares twelve independently fitted subspaces, so it does not measure the effect of adding one nested dimension.Selecting the minimum over twelve conditions also compounds the exploratory nature of the comparison.
Experimental controls: The direction-specificity result uses one random orthogonal control, while capability comparisons lack a power analysis and thinking mode is untested.All evaluation runs use reasoning_effort=low.
Reproducibility: Greedy decoding under tensor parallelism produces roughly 1–3 points of run-to-run noise, so the paper makes no claims about differences of a few points across runs.Comparisons are made within a single run.
Reproducibility: Independent verification requires engineering effort because the implementation is withheld and the method must be rebuilt from its description.The method section provides the writer inventory, leakage criterion, scale handling, and shard-by-shard rewrite rationale.

A Judge label counts

The paper’s refusal rates are based on judge-model labels, with only REFUSAL counted as refusal. Label conventions and unusable or unfinished responses can materially change denominators and reported rates.

Labels: Every rate uses one of three judge labels: REFUSAL, COMPLIANCE, or UNFINISHED, with occasional unusable labels after retries.
Counting rule: The main text counts a response as refusal only when its label is REFUSAL.
Counting rule: The label counts expose denominators and allow rates to be recomputed under alternative conventions.Columns record REFUSAL, COMPLIANCE, UNFINISHED, and responses without usable labels.
Worked example: 0.920 is the reported HarmBench base refusal rate from 138 REFUSAL labels out of 150 prompts.Counting every non-COMPLIANCE response instead would produce 0.967.
Missing labels: Two AdvBench responses lack usable labels, so both AdvBench rows are scored out of 99.
Worked example: 0.260 would be the harmless false-refusal rate under a non-COMPLIANCE rule, versus 0.050 under the REFUSAL-only rule.The difference comes from 21 replies that ran past the token budget.
Rank accounting: At ranks 6, 7, and 8, missing labels reduce denominators to 70, 71, and 70; UNFINISHED rises from 2 at rank 1 to 15 at rank 10.Low REFUSAL-only rates at high rank therefore partly include replies that never reached an answer.