A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

• Alignment algorithms aim to suppress unwanted behaviors such as toxicity and bias; PPO uses a learned reward model, while DPO derives reward signals directly from pairwise preferences.The introduction presents these methods as responses to undesirable behaviors learned from large-scale web data (Sheng et al., 2019; Gehman et al., 2020).Source: §1 · p. 1
• Because prior work hypothesized jailbreak explanations through empirical studies, this work instead provides a mechanistic account of how alignment alters behavior.The paper notes that alignments can be undone (Wallace et al., 2019; Zou et al., 2023).Source: §1 · p. 1
• Unlike Jain et al. (2023), which studies fine-tuning on synthetic tasks, this work examines mechanism changes from RLHF in a natural-language setting.Its approach compares language-model representation spaces before and after alignment.Source: §1 · p. 1
• The study applies DPO to GPT2 using a carefully crafted pairwise toxicity dataset of toxic and non-toxic samples generated with PPLM (Dathathri et al., 2019).The paper first examines how toxicity is represented and elicited in GPT2, then studies why it is no longer generated after DPO and how that mechanism can fail.Source: §1 · pp. 1–2
• Interventions with toxic vectors in GPT2 produce safer outputs, supporting their role in toxicity; SVD identifies vectors representing specific toxicity dimensions.The vectors promoting toxicity occur in multilayer perceptron blocks.Source: §1 · p. 1
• DPO minimally shifts model parameters yet learns a distributed offset that bypasses toxic vectors, which can be reactivated to restore toxic outputs.The authors argue this offers a mechanistic explanation for how aligned models can be jailbroken or un-aligned.Source: §1 · p. 2

• Transformer language models map tokens into a residual stream that is updated by attention heads and MLP blocks across successive layers.The described architecture has embedding and unembedding layers surrounding a series of transformer layers.Source: §2 · p. 2
• Each MLP block decomposes its output into value-vector contributions, writing once per value vector in a set of sub-updates.The MLP uses two linear transformations with point-wise activations between them; each contribution is scaled by a coefficient value.Source: §2 · p. 2

Interpreting Value Vectors in Vocabulary Space

• Interpreting Value Vectors in Vocabulary Space: a value vector promotes or suppresses a token’s likelihood according to the sign of its dot product with that token’s embedding.A positive dot product increases likelihood, while a negative dot product decreases it.Source: §2 · p. 2
• Interpreting Value Vectors in Vocabulary Space: value vectors induce a static ranking of tokens, while their realized effects depend on input-conditioned scaling by coefficients determined by corresponding key vectors and the residual stream.The paper uses projections into vocabulary space to show value vectors that promote toxicity in Section 3.Source: §2 · p. 2

3.1. Extracting Toxic Vectors

• The linear toxicity probe achieves 94% validation accuracy on the Jigsaw toxic comment classification task.It is trained on the last-layer residual stream averaged across timesteps, using a 90:10 training-validation split.Source: §3.1 · pp. 2–3
• The authors extract 128 toxic value vectors and use singular value decomposition to form basis vectors spanning GPT2’s toxicity representation space.The value vectors both promote toxic-token likelihood when triggered and contribute to the probe-vector direction.Source: §3.1 · p. 3

3.2. Toxic Vectors in Vocabulary space.

• Vocabulary projections show that individual toxic value and singular-value vectors encode distinct toxicity dimensions or contexts in pre-training data.The projected tokens are used to inspect what the value vectors promote.Source: §3.2 · p. 3

3.3. Interventions Using Toxic Vectors

• Subtracting toxic components from GPT2’s residual stream reduces toxicity during generation.The interventions use toxic vectors during generation, with their scales chosen to yield perplexity similar to the post-DPO model.Source: §3.3 · pp. 3–4

4.1. Background: DPO

• DPO promotes preferred nontoxic continuations and suppresses non-preferred toxic continuations, using the frozen original model as its reference.The preference pair is represented by positive and negative terms that compare the updated model’s likelihoods with the reference model’s likelihoods.Source: §4.1 · p. 4

4.2. Constructing Pairwise Toxic Data

• The 24,576 preference pairs use Wikitext-2 prompts, greedy GPT-2 positive samples, and PPLM negative toxic samples guided by the WToxic probe.Training stops when validation loss converges with patience 10, which occurs after approximately 6,000 sample pairs.Source: §4.2 · p. 4

5.1. Toxic Vectors Remain After DPO

• GPT2 and GPT2DPO parameters barely change: corresponding parameters have cosine similarity above 0.99 and average norm difference below 1e-5.5, while toxic MLP vectors remain unchanged.The unembedding layer is an exception, with norm difference below 1e-3; toxicity can still be elicited with the unchanged toxic value vectors.Source: §5.1 · p. 5, §5.2 · p. 5

5.2. GPT2DPO Avoids MLP.kToxic Regions

• DPO shifts residual streams out of toxic activation regions, lowering toxic-vector activations through an offset distributed across earlier MLP value vectors.Although individual weight changes are minimal, their accumulation produces the offset; sparse inactive neurons and GeLU's small negative activations help make oppositely shifted value vectors contribute in the offset direction.Source: §5.2 · pp. 5–6

5.3. Un-aligning GPT2DPO

• Scaling just 7 toxic key vectors by 10x reverts GPT2DPO to its pre-aligned toxic behavior without affecting perplexity.Table 4 reports toxicity, perplexity, and F1 scores after this intervention, but the supplied passage does not give their values.Source: §5.3 · p. 7, Table 4 · p. 7

6.1. On Designing Robust Alignment Algorithms

• GPT2DPO leaves regions that elicited toxic behavior unchanged, instead learning minimal changes across layers to avoid them and receive reward.The authors frame this bypass, rather than removal of toxic capabilities, as a mechanistic explanation for how aligned models can be undone or jailbroken.Source: §6.1 · p. 7
• The authors propose testing whether alignment can eliminate undesirable regions or update only their weights, bias terms, late layers, or added suppression layers.They also ask whether jailbreak-ability or unalignability can be characterized without test samples; these questions are left for future work.Source: §6.1 · p. 7

6.2. On the Role of KL-Divergence Regularization

• The authors hypothesize that KL-divergence regularization encourages minimal, distributed changes by discouraging large weight shifts that could disrupt pre-training capabilities.They suggest this term in RLHF losses may account for the observed pattern.Source: §6.2 · p. 8
• Jain et al. (2023) report late-layer task-specific wrappers, unlike this work’s distributed changes; the authors conjecture that fine-tuning without a KL term may enable drastic, localized changes.They suggest the KL-divergence term may help explain the difference, since fine-tuning typically lacks it.Source: §6.2 · p. 8

7.1. Alignment Algorithms

• Alignment choices depend on available data: human feedback supports PPO or DPO, while undesirable-behavior-only labels support unlikelihood training or Cringe; this study uses DPO.Human feedback is used in Stiennon et al. (2020), Ouyang et al. (2022), and Touvron et al. (2023), with PPO (Schulman et al., 2017) or DPO; unlikelihood training (Welleck et al., 2020) and Cringe (Adolphs et al., 2023; Xu et al., 2023) can use undesirable-only labels.Source: §7.1 · p. 8

7.2. Mechanistic Interpretability

• Mechanistic interpretability reverse engineers model behaviors and has identified interpretable, controllable representations of world models, truthfulness, knowledge, linguistic properties, and tasks.Examples are discussed in Olah et al. (2020), Elhage et al. (2021), Geva et al. (2021), Li et al. (2023a, 2023b), Nanda et al. (2023), Meng et al. (2022), Hernandez et al. (2023), Burns et al. (2023), Geva et al. (2023), Conneau et al. (2018), Tenney et al. (2019), Ilharco et al. (2022), Hendel et al. (2023), and Todd et al. (2023).Source: §7.2 · p. 8
• Geometric characterization has been used to extract language-model features that solve toxicity detection (Balestriero et al., 2023).This work contrasts that approach with probing for specific representations (Park et al., 2023).Source: §7.2 · p. 8
• Jain et al. (2023) study how synthetic-task fine-tuning alters capabilities; this work instead examines RLHF effects in a more realistic natural-language setting.Source: §7.2 · p. 8

7.3. Jailbreaking Aligned Models

• Aligned models can be jailbroken through prompts eliciting unlearned behaviors or personal information, and can be un-aligned with as few as 100 fine-tuning examples.Jailbreak studies include Wallace et al. (2019), Zou et al. (2023), Wei et al. (2023), and Carlini et al. (2023); Nasr et al. (2023) discuss personal information, and Yang et al. (2023) and Qi et al. (2023) show un-alignment. This work adds a mechanistic understanding of these phenomena.Source: §7.3 · p. 8

8. Conclusion

• MLP vectors in the pre-trained model promote toxicity, and subtracting them from the residual stream can suppress toxic outputs.Source: §8 · p. 8
• The authors applied DPO using PPLM to craft toxic and non-toxic continuation pairs for Wikipedia prompts.Source: §8 · p. 8
• GPT2DPO averts toxicity by learning an offset distributed across multiple value vectors, preserving pre-trained behavior with minimal weight changes in line with the DPO loss.Source: §8 · p. 8
• Scaling the key vectors corresponding to toxicity-eliciting regions breaks GPT2DPO’s alignment and reverts it to toxic behavior.Source: §8 · p. 8
• The authors present their findings as a mechanistic case study of why aligned models can be jailbroken, with the aim of informing more robust alignment algorithms.Source: §8 · p. 8

A. Projecting Value Vectors onto Vocabulary Space

• As described by Geva et al. (2022), a token embedding’s dot product with an MLP sub-update indicates whether that update increases or decreases the token’s likelihood.Source: §2 · p. 2, A. Projecting Value Vectors onto… · p. 12

B. Shift in Residual Streams

• The section provides additional examples of residual streams shifting out of toxic regions.Figure 6 illustrates the shifts.Source: B. Shift in Residual Streams · p. 12, Figure 6 · p. 12
• Figure 6 shows residual-stream shifts at layers 12, 18, and 13.The accompanying text identifies MLP.v12 771, MLP.v18 2669, and MLP.v13 668 as the next three vectors with highest cosine similarity with WToxic.Source: Figure 6 · p. 12, B. Shift in Residual Streams · p. 12
• The three vectors are identified as the next highest in cosine similarity with WToxic.They are MLP.v12 771, MLP.v18 2669, and MLP.v13 668.Source: B. Shift in Residual Streams · p. 12

C. Shifts in Residual Streams vs. Shifts in MLP Value Vectors.

• Across layers, examples show MLP value vectors contributing in the δx direction, with Figure 7 comparing residual-stream and MLP-value-vector shifts at layer 12.Source: C. Shifts in Residual Streams vs.… · p. 13, Figure 7 · p. 13

D. Hyperparameters

• Hyperparameters for the toxic probe, DPO, and PPLM are reported in Tables 5 and 6, with Table 6 covering PPLM.Source: D. Hyperparameters · p. 13, Table 6 · p. 14