Language Models Resist Alignment: Evidence From Data Compression

• LLMs can exhibit behaviors that deviate from human intentions, a problem the paper calls model misalignment.Source: §1 · p. 1
• Prior studies report that alignment can be superficial and that fine-tuning aligned models may weaken safety mechanisms (Qi et al., 2024a; Cohen et al., 2024; Wen et al., 2024; Jain et al., 2023).Source: §1 · p. 2
• Elasticity comprises resistance to leaving the pre-training distribution and faster rebound under reverse fine-tuning after deeper alignment.Source: §1 · p. 2
• The model’s compression-rate changes across datasets are inversely proportional to dataset sizes, which the authors analogize to coupled-spring deformation.They conjecture that elasticity helps explain resistance to alignment and makes inverse alignment possible.Source: §1 · pp. 1–2
• The authors model training and alignment through compression theory to investigate the mechanism underlying elasticity.Source: §1 · p. 2
• Experiments observe consistent resistance and rebound across various LLMs, supporting the universality of elasticity.The introduction describes rebound as a return toward the pre-training distribution under reverse fine-tuning.Source: §1 · p. 2

• Prior studies report that well-aligned models can be compromised and that non-malicious fine-tuning may impair safety mechanisms (Yang et al., 2023b; Hubinger et al., 2024; Dong et al., 2024).Source: §2 · p. 2
• Studies also report that models selectively adhere to training objectives to preserve inherent preferences (Greenblatt et al., 2024; Lang et al., 2024; Park et al., 2024).Source: §2 · p. 2
• Other work proposes separating safety-critical from utility-related model regions at neuron and rank levels.Source: §2 · p. 2
• Research on shallow safety alignment argues that alignment should shape internal mechanisms beyond surface-level tokens.Source: §2 · p. 2

3 What is Elasticity?

• The paper introduces elasticity as a phenomenon leading to resistance to alignment and uses compression theory tools to analyze it.Source: §3 · p. 2

3.1 Preliminaries

• Pre-training and supervised fine-tuning are framed as language-model training objectives, while minimizing log-likelihood is equivalent to minimizing compression rate.The section also invokes lossless compression and Shannon entropy to establish the compression-theory background.Source: §3.1 · pp. 2–3

3.2 The Compression Protocol of LLMs

• The protocol models training and alignment as joint compression of datasets, using token trees and Huffman coding after pruning the tree to a model-dependent depth.It treats compression rate as a proxy for training loss and assumes a binary token vocabulary.Source: §3.2 · pp. 3–4

3.3 The Formal Definition of Elasticity

• Inverse alignment is a transition back toward the original model after fine-tuning an aligned model on a much smaller dataset.Source: §3.3 · p. 4
• Elasticity is defined by a simple inverse operation on a much smaller dataset that brings the transformed model close to its original model under an evaluation metric.The section uses compression rate as that metric for analyzing elasticity during alignment.Source: §3.3 · p. 4

4.1 Formal Derivation of Elasticity

• The analysis treats Dp as larger pre-training data, Da as smaller alignment data, and Dt as a minimal perturbation, assuming Da shares a distribution with a subset of pre-training data.Source: §4.1 · pp. 4–5
• As perturbation grows, normalized compression rates decrease for Dp and increase for Da, indicating preference for the larger dataset and biased post-perturbation behavior.Source: §4.1 · p. 5

4.2 Elasticity and Inverse Alignment

• A substantial pre-training/alignment size gap makes alignment-dataset performance degrade rapidly under perturbations, often by several orders of magnitude, because normalized compression-rate changes are inversely proportional to dataset size.Theorem 4.2 states the inverse relationship between dataset size and normalized compression-rate change.Source: §4.2 · p. 5
• Elasticity makes inverse alignment possible: subsequent perturbations tend to move the model toward the pre-training rather than alignment distribution.The authors identify the data-volume disparity between these datasets as the relevant setting.Source: §4.2 · p. 5

4.3 Elasticity and the Hooke’s Law.

• Across datasets, compression-rate changes are inversely proportional to dataset sizes, and their absolute product remains constant, characterizing perturbation impact and model elasticity.Source: §4.3 · pp. 5–6

5 How Elasticity Resists Alignment?

• Pre-trained models resist alignment, while reverse fine-tuning causes post-trained models to return quickly toward their pre-training distribution.The paper names these patterns resistance and rebound, respectively.Source: §5 · p. 6

5.1 Existence of Language Models’ Resistance

• Across models and datasets, inverse-alignment training loss remains below forward-alignment loss across slice pairs, supporting the existence of resistance.The experiment trains an earlier SFT slice on responses from a later slice, then reverses that direction; tested base models include Llama2-7B, Llama2-13B, and Llama3-8B.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Existence of Rebound

• Models with more positive-data training initially perform better, then drop faster under negative-data fine-tuning before the decline slows and stabilizes.The result is reported for Gemma-2B and Llama2-7B; ablations across alignment algorithms, metrics, and fine-tuning directions also validate rebound.Source: Figure 4 · p. 7, §5.2 · pp. 7–8
• Rebound is tested by inversely fine-tuning positively trained slices with negative data on IMDb sentiment generation and Beavertails safe dialogue.Positive sample sizes range from 1,000 to 10,000 and negative sample sizes from 100 to 2,000; evaluation uses sentiment and safety scoring models.Source: §5.2 · p. 7

5.3 Internal Factor of Rebound

• As Qwen model size increases from 0.5B to 4B to 7B, performance declines faster initially and more slowly afterward under negative-data fine-tuning.The authors interpret this pattern as a positive correlation between model parameter scale and elasticity.Source: Figure 5 · p. 8, §5.3 · p. 8
• Across TinyLlama pre-training slices of 2.0T, 2.5T, and 3.0T, larger data volumes correspond to faster initial declines and slower subsequent declines.The authors say this pattern is consistent with the inference proposed in Theorem 4.2.Source: Figure 6 · p. 8, §5.3 · p. 8

6 Conclusion and Outlook

• The authors attribute alignment fragility to elasticity, reporting that normalized compression-rate changes inversely with dataset size and that elasticity strengthens with model size and pre-training data volume.They conclude that language models exhibit elasticity and thereby inherently resist alignment.Source: §6 · pp. 8–9

6.1 Limitations

• The theoretical analysis specifies the mass distribution in Assumption A.7, and experiments do not systematically validate elasticity across the full pre-training and alignment lifecycle.The authors identify empirical study of the distribution’s exact form as valuable.Source: §6.1 · p. 9

6.2 Broader Impacts

• The authors argue that open-source models carefully aligned and audited could be reverted toward pre-alignment states at minimal cost if advanced inverse-alignment techniques become feasible.They call for alignment algorithms with fine-tuning robustness.Source: §6.2 · p. 9

A Assumptions and Proofs

• The analysis assumes binary tokens and defines mass as each leaf probability multiplied by the number of leaves, with expected mass equal to 1.Source: A Assumptions and Proofs · p. 14
• Responses longer than the model’s perfect-compression depth must be compressed in segments, each no longer than that depth.Source: A Assumptions and Proofs · p. 14
• It assumes pruned-tree mass follows Pareto distributions with identical parameters across datasets; independence is treated as an approximation when the number of leaves is large.Source: A Assumptions and Proofs · p. 15
• Theorem A.8’s proof establishes d l < 0 for sufficiently large k.The theorem considers pre-training, alignment, and perturbation datasets.Source: A Assumptions and Proofs · pp. 15, 18

B Additional Experiment Results

• The displayed result gives d l < 0.Source: B Additional Experiment Results · p. 19

B.1 Additional Experiment of Language Models’ Resistance

• The experiment uses Llama2-7B and finer-grained forward-alignment snapshots across a wider range of k and l, measuring relative KL divergence after forward and inverse alignment.Source: §B.1 · p. 19
• Inverse alignment yields substantially smaller KL divergence than forward alignment in the snapshot comparison.Source: Table 2 · p. 19, §B.1 · p. 19

B.2 Additional Experiment of Language Models’ Rebound

• The algorithm comparison first applies SFT with varying amounts of positive data, then applies the corresponding RL algorithm using negative samples as selected responses and positive samples as rejected responses.Source: §B.2 · pp. 19–20
• Across PPO, DPO, KTO, and SimPO, performance declines after fine-tuning with negative data, with degradation accelerating as positive-data volume increases.The reported result extends the rebound pattern observed with SFT.Source: §B.2 · p. 20, Figure 7 · p. 20
• KL divergence between aligned and base models supplements task-specific scores as a distribution-level evaluation metric.Source: §B.2 · p. 20
• With negative-data SFT followed by positive-data inverse alignment, elasticity persists and larger models show greater elasticity.The reverse-setting figure covers models of 0.5B, 4B, and 7B parameters.Source: §B.2 · p. 20, Figure 8 · p. 21
• Only a small amount of unsafe data is needed to reduce aligned-to-base KL divergence below ϵ = 1 × 10^-2, and more safe fine-tuning reduces the unsafe-data amount required.Source: §B.2 · p. 20, Table 3 · p. 21

B.3 Additional Experiment of Internal Factor of Language Models’ Rebound

• Under DPO, rebound strengthens across models with 0.5B, 4B, and 7B parameters.The appendix reports that the trend is consistent with the results in Figure 5.Source: Figure 9 · p. 21, §B.3 · p. 21
• Across reported pre-training volumes from 0.1T to 3.0T, rebound becomes more pronounced; negative-data fine-tuning causes faster initial decline followed by slower decline.The appendix reports results at 0.1T, 0.5T, and 1.0T, alongside main-text results spanning 0.5T to 3.0T.Source: §B.3 · pp. 21–22, Figure 10 · p. 22

C.1 Transfer Learning and Language Models’ Elasticity

• In Alpaca, θ2’s greater SFT knowledge did not make it easier than θ1 to learn θ3, so transfer learning alone cannot explain resistance.The discussion distinguishes this autoregressive setting, where positive and negative samples follow conflicting input-output distributions, from classification over partitions of one dataset.Source: §C.1 · pp. 22–23

C.2 Discussions of Practical Steps to Mitigate Inverse Alignment Risks

• Matching data volumes across alignment targets may reduce resistance to later perturbations of targets with smaller training volumes.The paper also proposes using the elasticity theorem to quantify data needed for new targets while preserving previously aligned targets.Source: §C.2 · p. 23
• Implementing elasticity-based algorithms is limited by inadequate theory for characterizing dataset features, especially when real-world datasets contain fused features.The paper identifies finer characterization of fused-feature datasets as a possible research direction.Source: §C.2 · p. 23

C.3 Discussions of Quantitatively Characterize the Impact of Dataset Size on Elasticity

• For TinyLlama, resistance is nearly absent at 0.1T but notable at 0.5T, placing the possible change point between those volumes.The paper presents this interval as a suggested range based on observed continuity, not a precisely established threshold.Source: §C.3 · p. 23
• Without finely spaced pre-training-data models, the proposed 0.1T–0.5T change-point range cannot be pinned down accurately.The authors identify continuously varying model series and fine-grained pre-training datasets as needed for clearer quantitative characterization.Source: §C.3 · pp. 23–24