Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

1 INTRODUCTION

• Pairwise preference data is used for language-model alignment; DPO avoids RLHF’s reward-model and reinforcement-learning pipeline, and the authors emphasize curating sufficiently distinct preferences.They suggest CHES may help with that curation.Source: §1 · pp. 1–2
• Although DPO aims to increase the preferred-versus-dispreferred log-probability margin, both response probabilities can decrease, a phenomenon the authors call likelihood displacement.The paper uses theory and experiments to characterize its mechanisms and implications and to offer preventative guidelines across several model families and scales.Source: §1 · pp. 1–2
• Even single-prompt, single-token training can shift probability toward responses opposite in meaning, such as Yes when No is preferred over Never.The authors contrast this with prior explanations based on complexities in the preference-learning pipeline (Tajwar et al., 2024; Pal et al., 2024; Rafailov et al., 2024b).Source: §1 · p. 2
• The theory links displacement to embedding geometry: greater similarity between preferred and dispreferred responses makes preferred-response log likelihood more likely to decrease.The analysis concerns static token unembeddings and contextual hidden embeddings.Source: §1 · p. 2
• CHES is a model-aware preference-similarity measure that identifies training samples contributing most to displacement in UltraFeedback and AlpacaFarm, unlike other tested similarity measures.Those comparisons use hidden embeddings or token-level cues.Source: §1 · p. 2
• Filtering high-CHES samples prevented unintentional unalignment more effectively than adding an SFT term; Llama-3-8B-Instruct’s refusal rate fell from 74.4% to 33.4%.The probability shift was from preferred refusals toward responses complying with unsafe prompts.Source: §1 · p. 2

2 PRELIMINARIES

• A language model maps token-prefix hidden embeddings through an unembedding matrix to logits and a token distribution, and assigns sequence probabilities autoregressively.The notation y<k denotes the first k − 1 tokens of y.Source: §2 · p. 3

2.1 DIRECT PREFERENCE LEARNING

• Preference data consists of prompt, preferred-response, and dispreferred-response triples, and training minimizes a differentiable margin loss under a local monotonicity assumption.Data may be on-policy or static; the setup includes an initial SFT phase and generalizes losses including DPO, IPO, SLiC, REBEL, and GPO.Source: §2.1 · p. 3

2.2 LIKELIHOOD DISPLACEMENT

• Likelihood displacement occurs when preference loss decreases but preferred-response log probability falls; it is catastrophic when probability shifts to substantially less-preferred responses.Displacement is benign when the responses gaining probability are as preferable as the preferred response.Source: §2.2 · pp. 3–4
• Table 1 reports the largest preferred-token probability decreases across ten Persona-prompt runs and categorizes probability-gaining tokens as benign or catastrophic.The caption notes that when preferred and dispreferred tokens are similar in meaning, the largest gains are often in tokens opposite in meaning to the preferred token.Source: Table 1 · p. 4

3 CATASTROPHIC LIKELIHOOD DISPLACEMENT IN SIMPLE SETTINGS

• Displacement and catastrophic shifts occurred independently of model-capacity limits, multiple samples or output tokens, and initial SFT, even with one prompt and single-token responses.These were factors attributed to displacement in prior work.Source: §3 · p. 4
• The Persona experiments used OLMo-1B, Gemma-2B, and Llama-3-8B after one SFT epoch, then applied DPO to one randomly selected prompt.Prompts asked for agreement with a statement using a single token selected from pairs including Yes, Sure, No, and Never.Source: §3 · p. 4
• Preferred-token probability fell by 0.21 to 0.96 in absolute value in every run, and similar-meaning preference pairs often shifted mass to tokens with opposite meanings.Similar findings held for base models without initial SFT and when using IPO instead of DPO.Source: §3 · p. 4
• Because minimizing the preference loss requires increasing the log-probability gap, it does not require the preferred response’s log probability to rise.This explains how likelihood displacement can occur while the loss is minimized.Source: §3 · p. 4

4.1 TECHNICAL APPROACH

• The analysis tracks response log-probability dynamics under gradient flow, using unconstrained features that treat hidden embeddings as trainable parameters.The paper reports that this simplification helps extract salient sources of displacement in practice.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2.1 SINGLE TRAINING SAMPLE AND OUTPUT TOKEN

• Even one sample with single-token responses can displace preferred likelihood: unembedding alignment governs its decline, while orthogonal directions can send probability to unrelated or opposite-meaning tokens.The analysis links this possibility to token unembeddings encoding semantics.Source: §4 · p. 5, §4.2.1 · p. 6

4.2.2 RESPONSES WITH MULTIPLE TOKENS

• With multi-token responses, preferred likelihood depends on preferred-dispreferred and preferred-preferred hidden-embedding inner products; their effects depend on the signs of their coefficients.The coefficients are mostly positive across the models and datasets examined.Source: §4.2.2 · pp. 6–7
• CHES measures hidden-embedding similarity between preferred and dispreferred responses; higher scores imply more severe displacement and better predict it than other plausible similarity measures.The analysis derives the score by setting coefficients in Theorem 3 to one.Source: §4 · p. 5, §4.2.2 · p. 7
• In general settings, it remains difficult to characterize whether responses gaining probability at the preferred response’s expense are benign or catastrophic.The authors identify studying where probability mass goes in additional settings as future work.Source: §4.2.2 · p. 7

4.2.3 MULTIPLE TRAINING SAMPLES

• Additional training samples can worsen preferred-response likelihood decreases when preferred responses recur as dispreferred responses, and even samples with distinct preferences can contribute negatively.The paper characterizes when additional samples have this effect.Source: §4.2.3 · p. 7

5 IDENTIFYING SOURCES OF LIKELIHOOD DISPLACEMENT

• Across models, higher CHES percentiles produce larger decreases in preferred-response log probability, while low-score subsets increase it.The experiment ranks samples from UltraFeedback and trains DPO on separate 512-sample subsets centered on each percentile.Source: §5 · p. 8, Figure 2 · p. 8
• Edit distance and last-hidden-embedding inner products fail to identify displacement: preferred-response log probability generally increases across their ranked subsets.Their few decreases are minor, unlike the severity associated with high-CHES samples.Source: §5 · p. 8, Figure 2 · p. 8
• Similar findings hold with AlpacaFarm, IPO, and OLMo-1B experiments, extending the reported pattern beyond UltraFeedback and DPO.The appendix reports analogous experiments for each of these dataset, objective, or model changes.Source: §5 · p. 8
• High-CHES samples often have longer dispreferred responses, whereas low-CHES samples often have longer preferred responses.For Llama-3-8B on UltraFeedback, over 99% of hidden-embedding inner products are positive; the authors connect this to longer dispreferred responses contributing more positive terms to CHES.Source: §5 · p. 8

6.1 SETTING

• The experiments use Gemma-2B-IT and Llama-3-8B-Instruct, generating and classifying two responses per SORRY-Bench unsafe prompt before an 85%/15% train-test split.Refusals are preferred over non-refusals, with ties broken using PairRM.Source: §6.1 · p. 9

6.2 CATASTROPHIC LIKELIHOOD DISPLACEMENT CAUSES UNINTENTIONAL UNALIGNMENT

• For over 70% of generated prompts, both candidate responses were refusals, creating many same-type preference pairs in this safety setting.The authors relate this pattern to earlier single-token experiments in which preferring No over Never often increased the probability of Yes.Source: §6.2 · p. 9
• DPO shifts probability away from preferred refusals and lowers refusal rates from 80.5% to 54.8% for Gemma-2B-IT and from 74.4% to 33.4% for Llama-3-8B-Instruct.Similar drops occur on the test sets; the models were trained with the purpose of refusing prompts, without an attempt to maintain helpfulness.Source: §6.2 · p. 9

6.3 FILTERING DATA VIA CHES SCORE MITIGATES UNINTENTIONAL UNALIGNMENT

• Filtering to the 5% lowest length-normalized-CHES samples mitigates the refusal-rate drop and boosts refusal rates more than adding an SFT term; gold preferences also avoid displacement.The figure compares the original dataset, SFT-augmented DPO, gold preferences, and CHES-filtered data; analogous results hold when retaining up to 15% of original samples.Source: §6.3 · p. 10, Figure 3 · p. 10
• High length-normalized-CHES scores tend to mark pairs with two refusals or two non-refusals rather than one of each; training on either same-type subset also lowers refusal rates.The figure’s sample ranking supports using the lowest-scoring 5% as the filtered set.Source: §6.3 · p. 10, Figure 4 · p. 10

Preference learning for language model alignment

• Direct preference learning, popularized by DPO, avoids RLHF’s reward-model-plus-RL pipeline, which can be computationally expensive and unstable.The paper’s analysis applies to methods maximizing the log-probability ratio of preferred and dispreferred responses.Source: §7 · pp. 10–11
• Whether other direct preference learning variants suffer from likelihood displacement remains a potential avenue for future work.The paper names variants proposed by Ethayarajh et al. (2024), Hong et al. (2024), Song et al. (2024a), and Wu et al. (2024).Source: §7 · p. 11

Analyses of direct preference learning

• Prior analyses studied DPO sample complexity and optimization, while Chen et al. (2024) examined ranking failures distinct from likelihood displacement.In likelihood displacement, preferred responses can remain more probable than dispreferred responses even as their probability declines.Source: §7 · p. 11

Analyses of likelihood displacement

• Earlier work linked likelihood displacement to similar preferences, small edit distance, dataset size, model capacity, or initial SFT; this paper refines those accounts with embedding geometry and broader settings.The paper reports displacement can occur with one sample, an 8B model, or without initial SFT, while multiple samples can still increase severity.Source: §7 · p. 11

Jailbreaking and unalignment

• Prior work reported unalignment from seemingly benign SFT data; this paper describes refusal-rate declines even when training is explicitly intended to refuse unsafe prompts.The related-work passage characterizes the Section 6 findings as a more extreme case.Source: §7 · p. 11

8 CONCLUSION

• Likelihood displacement can move probability from preferred responses to opposite-meaning responses and, in safety training, contribute to unalignment; CHES identifies contributing samples and filtering high scores can prevent this outcome in the reported experiments.The authors emphasize curating data with sufficiently distinct preferences and present CHES as a potentially useful aid.Source: §8 · p. 11

8.1 LIMITATIONS AND FUTURE WORK

• The theory analyzes instantaneous log-probability change while abstracting away architecture, and the authors do not claim catastrophic displacement occurs universally.They call for studying full training evolution, architecture effects, and additional settings where displacement may be catastrophic.Source: §8.1 · p. 11
• The broader utility of CHES filtering remains to be assessed, including for general instruction-following performance and more complex preference-curation pipelines.One proposed direction is selecting distinct preferences from candidate responses generated by diverse models.Source: §8.1 · pp. 11–12

A LENGTH-NORMALIZED CHES SCORE

• The appendix defines length-normalized CHES for preference samples using hidden embeddings, motivated by the correlation between CHES scores and response lengths.It is the variant applied to avoid introducing length bias when filtering data.Source: A LENGTH-NORMALIZED CHES SCORE · p. 17

B COMMON INSTANCES OF THE ANALYZED PREFERENCE LEARNING LOSS

• The appendix lists GPO, DPO, IPO, SLiC, and REBEL as instances of the analyzed preference-learning loss family.Their loss forms correspond to different choices of the per-sample loss function.Source: B COMMON INSTANCES OF THE ANALYZED… · p. 17

C RELATION TO EXISTING CLAIMS ON LIKELIHOOD DISPLACEMENT

• The paper formalizes prior intuition that similar preferences drive displacement, finds edit distance is not generally indicative, and distinguishes its findings from accounts based on capacity, sample count, or initial SFT alone.It also notes prior math and reasoning benchmark degradations as possible catastrophic cases, while emphasizing safety effects and settings with many acceptable responses.Source: C RELATION TO EXISTING CLAIMS ON… · p. 18

D.1 SINGLE TRAINING SAMPLE AND OUTPUT TOKEN (OVERVIEW IN SECTION 4.2.1)

• For a single sample with single-token responses, the rate of change in preferred-response log probability depends on preference alignment and token-unembedding alignment.Greater similarity between preferred and dispreferred responses can produce a larger decrease.Source: §D.1 · p. 19
• Alignment of other token unembeddings with the preferred–dispreferred unembedding difference determines which tokens gain log probability.Tokens already more likely under the model receive larger weight in this alignment term.Source: §D.1 · p. 19

D.2 RESPONSES WITH MULTIPLE TOKENS (OVERVIEW IN SECTION 4.2.2)

• For multi-token responses, displacement depends on initial-token geometry and inner products between preferred–dispreferred and preferred–preferred hidden embeddings.The analysis also assumes the preferred and dispreferred responses initially differ in their tokens.Source: §D.2 · pp. 19–20
• Because relevant coefficients are mostly positive empirically, a higher CHES score implies more severe likelihood displacement.The direction of an embedding-inner-product effect depends on the sign of its coefficient.Source: §D.2 · p. 20
• For general responses, recipient probabilities depend on coefficient signs set by next-token distributions, whose typical patterns remain difficult to assess.The analysis identifies this as a direction for future work.Source: §D.2 · p. 20

D.3 MULTIPLE TRAINING SAMPLES (OVERVIEW IN SECTION 4.2.3)

• With single-token responses and unique prompts, an additional sample affects displacement through a coefficient and the prompts’ hidden-embedding inner product.The no-duplicate-prompt condition is assumed for this analysis.Source: §D.3 · p. 21
• Even distinct preference pairs can exacerbate displacement when the cross-sample coefficient is negative; with positive prompt-embedding alignment, another sample can steer probability toward its preferred token.The negative-coefficient case includes certain non-contradicting samples as well as contradicting ones.Source: §D.3 · p. 22

E LOSSES INCLUDING SFT REGULARIZATION OR DIFFERENT WEIGHTS FOR THE PREFERRED AND DISPREFERRED RESPONSES

• The analysis extends to losses with SFT regularization or greater preferred-response weighting, but experiments find these modifications less effective than improving data curation for mitigating displacement’s adverse effects.Source: E LOSSES INCLUDING SFT REGULARIZATION… · p. 23

• The analysis studies gradient flow for losses with SFT regularization and with different preferred- and dispreferred-response weights.Source: §E.1 · p. 23
• With SFT regularization, preferred-response log probability evolves according to the original-loss contribution plus a non-negative gradient-norm term.The proposition states this for a single preference sample.Source: §E.1 · p. 23
• Adding SFT regularization or assigning the preferred response a larger weight encourages its log probability to increase.Source: §E.1 · p. 23

F LINEAR MODELS DO NOT SUFFER FROM LIKELIHOOD DISPLACEMENT

• The result highlights the importance of accounting for hidden-embedding evolution when analyzing likelihood displacement.Source: F LINEAR MODELS DO NOT SUFFER FROM… · p. 24
• The result applies to one prompt with single-token preferred and dispreferred responses, with the prompt hidden embedding fixed and only the unembedding matrix trainable.Source: F LINEAR MODELS DO NOT SUFFER FROM… · p. 24
• Under those conditions, the preferred response log probability is monotonically non-decreasing during training.Source: F LINEAR MODELS DO NOT SUFFER FROM… · p. 24

G.1–G.4 PROOFS OF THEOREMS 4–7

• A derivation analogous to Theorem 4’s proof combines intermediate results to obtain the desired expression for the derivative of ln πθ(t)(y+|x); Lemma 2 gives −ℓx,y+,y−(t) > 0.Source: §G.2 · p. 24, §G.3 · p. 26

G.5–G.6 PROOFS OF THEOREMS 8–9

• The proofs of Theorems 8 and 9 use the chain rule to isolate contributions from additional training samples.Source: §G.5 · p. 28, §G.6 · pp. 28–29

G.9 PROOF OF PROPOSITION 3; G.10 AUXILIARY LEMMAS

• For a fixed prompt embedding, the proof shows the preferred-response log probability cannot decrease; auxiliary lemmas establish derivative properties used in the analysis.Source: §G.9 · p. 30, §G.10 · pp. 30–31

H.1 CATASTROPHIC LIKELIHOOD DISPLACEMENT IN SIMPLE SETTINGS (SECTION 3)

• Across the examined models and preference pairs, the orthogonal component of the unembedding difference was larger than its projection onto the preferred-token unembedding.Source: §H.1 · p. 31

H.2 EMPIRICAL EVALUATION OF THE COEFFICIENTS FROM THEOREM 3

• Across OLMo-1B, Gemma-2B, and Llama-3-8B with UltraFeedback and AlpacaFarm, over 69% of Theorem 3 coefficients are positive on average.Experiments in Sections 5 and 6 also show that CHES accurately predicts how much samples contribute to likelihood displacement.Source: §H.2 · p. 31

H.3 IDENTIFYING SOURCES OF LIKELIHOOD DISPLACEMENT (SECTION 5)

• Additional experiments test whether the Section 5 findings hold on AlpacaFarm, with IPO, and with OLMo-1B using DPO and IPO.The appendix also provides representative high- and low-CHES UltraFeedback samples for three model families.Source: §H.3 · p. 32

H.4 UNINTENTIONAL UNALIGNMENT IN DIRECT PREFERENCE LEARNING (SECTION 6)

• Section 6 supplements its unalignment experiments with IPO results, mean preferred-response log-probability changes, and examples of unsafe prompts followed by compliance after DPO.The examples begin with models generating two refusal responses; after DPO, they comply with the prompts.Source: §H.4 · p. 32

I FURTHER IMPLEMENTATION DETAILS

• The authors provide implementation details and reproduction code based on PyTorch and Hugging Face.The code is linked from the paper’s GitHub repository.Source: I FURTHER IMPLEMENTATION DETAILS · p. 32

I.1 CATASTROPHIC LIKELIHOOD DISPLACEMENT IN SIMPLE SETTINGS (SECTION 3)

• Single-token experiments use Persona prompts for the “ends justify means” trait, with responses drawn from specified No/Never/Maybe/Perhaps and Yes/Yeah/Sure/Certainly/Absolutely sets.DPO uses one prompt per run for 100 steps; IPO uses a lower learning rate because higher rates caused unstable training, and reported runs exclude runs with increasing loss.Source: §I.1 · p. 32

I.2 IDENTIFYING SOURCES OF LIKELIHOOD DISPLACEMENT (SECTION 5)

• The displacement-identification experiments use 5,000 randomly selected UltraFeedback training samples and the 9,691-sample AlpacaFarm human-preferences subset, after specified prompt and response filtering.Models train for one epoch on preference-similarity percentile subsets using DPO or IPO; inputs are truncated to 512 tokens.Source: §I.2 · p. 33

I.3 UNINTENTIONAL UNALIGNMENT IN DIRECT PREFERENCE LEARNING (SECTION 6)

• Unsafe-prompt experiments used 435 SORRY-Bench prompts after filtering 15 of 450 for missing gold labels, split 85%/15%; human-labeled refusals were preferred over non-refusals, and DPO or IPO ran one epoch.Source: §I.3 · p. 33