ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization

1 INTRODUCTION

• The offset shifts chosen and rejected hidden states together, so their relative margin barely changes and the ranking loss has little incentive to penalize its growth.The offset moves both sides’ scores by tens of nats while changing the average list margin by less than 3 nat.Source: §1 · p. 2
• When the offset raises one token’s logit above the base model’s top token, it creates a vocabulary well that can dictate decoding and cause repetitive generation collapse.The well can form on an ordinary token such as Conf, not only on a rare token with a large output-vector norm.Source: §1 · p. 2
• Causal interventions show the common-mode offset drives the vocabulary well: retaining it reconstructs the well, and adding it to a base model recreates collapse where norm-matched random directions do not.Centering the rejected-side loss to hide the offset prevents the well in the flagship setting.Source: §1 · p. 2
• The offset appears across objectives, but their demands differ: textbook DPO keeps it weakly aligned with vocabulary directions, while extended full-parameter SimPO from base models can drive collapse.The offset also grows without a well from a public SFT checkpoint under the official SimPO recipe.Source: §1 · p. 2
• Hiding and penalizing the offset prevents collapse up to three times the base budget in the four tested base models, showing collapse is not inevitable in preference optimization.The offset is active before generation visibly fails, exposing a blind spot in ranking metrics.Source: §1 · p. 3

Likelihood displacement, squeezing, and gradient entanglement

• Prior work links DPO to falling chosen-response likelihoods (Razin et al., 2025), entangled chosen–rejected gradients (Yuan et al., 2025), and probability squeezing toward base-favored tokens (Ren & Sutherland, 2025); this paper identifies a representation-space offset behind wells.The well tokens initially rank far below the base model’s preferred tokens, typically by 17 to 39 nat.Source: §2 · p. 3

Regularizing absolute likelihood

• DPO-Positive (Pal et al., 2024), explicit NLL regularization (Pang et al., 2024), and N-DPO (Im et al., 2026) constrain probabilities rather than the shared hidden-state direction; an NLL term does not prevent the well in these experiments.The paper presents this as a distinction between scalar-probability constraints and offset control.Source: §2 · p. 3

Unembedding geometry and representation anisotropy

• Unlike pretrained representation anisotropy (Mu & Viswanath, 2018; Ethayarajh, 2019; Bi´s et al., 2021; Gao et al., 2019; Puccetti et al., 2022; Macocco et al., 2025; Sun et al., 2024; Stolfo et al., 2024), the common-mode offset is built during preference training and can grow like a steering vector until generation collapses.Many severe wells emerge from under-trained tokens, while decoding-time corrections leave the training-time cause in place.Source: §2 · p. 3

3 SETUP

• The main objective ranks K responses sequentially with a Plackett–Luce loss, scoring each response by its average per-token policy log-probability without a reference model.For K = 2, the ranking term becomes a length-normalized, reference-free pairwise objective without SimPO’s target margin.Source: §3 · p. 3
• The experiments use 5,383 four-response driving-preference lists and UltraFeedback, while also testing textbook DPO and adapter or full-parameter fine-tuning.The setup studies per-token ranking, SimPO, and textbook DPO, whose loss demands span two orders of magnitude.Source: §3 · pp. 3–4
• The common-mode offset is the mean final-hidden-state displacement across response positions, averaged over rejected-response probes; subtracting it yields a position-specific residual.The decomposition requires two checkpoints and no access to training internals; the paper tracks offset norm, chosen–rejected cosine, and vocabulary projection loudness.Source: §3 · p. 4
• Likelihood displacement is not itself pathological; a vocabulary well concentrates released probability mass on a few tokens, while generation collapse occurs when those tokens dominate decoding.Diffuse degradation instead spreads released mass thinly and is measured by the base model’s per-token negative log-likelihood of samples; it is not counted as collapse.Source: §3 · p. 4
• Mass flow is measured under teacher forcing on rejected responses, with a well nucleated when the top token captures most released mass; held-out ranking uses accuracy and margin metrics.The setup also evaluates sampled-generation repetition and word-stem cycling, and generally uses rank-32 LoRA, learning rate 10^-5, effective batch size 32, and a three-pass base budget.Source: §3 · p. 4
• Table 1 reports collapse outcomes for untreated base-model runs at their onset budget, except Llama-3.2-3B, which is shown at 6×; the caption marks ring counts and their sample basis.The caption states that every base except Llama-3.2-3B collapses, onto one token or forms of one word.Source: Table 1 · p. 5

• Within 504 steps, 89% of released probability mass concentrates on textt, and almost every sampled generation repeats it despite top-1 accuracy of 0.995 and pairwise accuracy above 0.94.The experiment uses Llama-2-7B with sequential K = 4 Plackett–Luce ranking on driving lists.Source: §4 · p. 5
• 14 of 17 base-model lineages collapse within six times the base budget, while Llama-3.2-3B remains nearly as fluent as the base model despite over 30 nat per token of rejected-response decline.The lineages span five tokenizer families and 0.5B to 14B parameters; Phi-3-mini and Gemma-2-2B were tested only up to three times the budget.Source: §4 · p. 5
• Full-parameter fine-tuning of Llama-2-7B collapses onto textt by step 250 of the same 504-step budget.The authors report that onset tracks learning rate summed over training and predicted a step-200-to-300 collapse based on two earlier runs.Source: §4 · p. 5

• Across prompts, per-response offsets have median cosine similarity above 0.97 to their global mean, and chosen–rejected offsets have cosines of 0.83 to 0.996.For single-token wells, the well token has the largest entry of the projected shared vector.Source: §5 · p. 6
• The offset lowers chosen and rejected scores by 15 to 65 nat per token but shifts their average-list margin by only 0.2 to 2.7 nat.Its effect on both scores is 10 to 270 times its effect on their difference; a 2.5-fold z-loss still leaves 65 to 66% of released mass in the well.Source: §5 · p. 6
• Keeping only the common offset makes the well the top token at nearly every rejected position, while the residual alone rarely does.This pattern also holds under full-parameter fine-tuning, where fine-tuned unembedding rows alone almost never lift the well.Source: §5 · p. 6
• On the Llama-2 flagship, centering keeps most offset norm but turns its vocabulary projection away, and no well forms.Centering computes the rejected-side ranking loss on hidden states with the offset subtracted as a constant.Source: §5 · p. 6
• Injecting the trained offset into a base model makes the well token dominate generation, while norm-matched random or orthogonal directions do not form a well.Removing half of the offset's logit bias cuts degenerate greedy samples on 60 prompts from 100% to 0%.Source: §5 · p. 6

• In 510 snapshots, a well never tops a majority of rejected positions while median dose stays below one; dose above one is necessary in these runs but not sufficient.A quarter-strength injected offset fails to trigger the well, while a half-strength offset pushes dose above one and puts the well on top at nearly all rejected positions.Source: §6 · p. 6, Figure 3 · p. 7
• High unembedding-row norms make some tokens more exposed, but are not required: UltraFeedback Llama-2 forms a well on ordinary token Conf.In driving-data adapter runs, most single-token wells are in the top 1% of base-model row norms and never occur in WikiText-103.Source: §6 · p. 7
• At usual settings, only the per-token ranking loss forms a well; at three times the budget, full-parameter SimPO crosses the threshold on Llama-2 and Llama-3.1, while textbook DPO stays below it.Textbook DPO's offset stays short and spreads across the vocabulary; the reference model is not what protects it.Source: §6 · p. 7
• The per-token loss demands about 42 nat per token, versus about 2.4 nat for SimPO and about 0.3 nat per token for textbook DPO.The authors define demand as the per-token margin beyond which a ranking loss stops pushing; adding a reference model does not protect the per-token loss from forming a well.Source: §6 · p. 7
• Within its published range, SimPO grows a long but quiet offset without a well, yet generation degrades into unrelated-word strings that standard repetition counts miss.Reward models prefer the base-model sample on about nine in ten prompts; adding the SimPO offset alone to the base model does not reproduce this diffuse degradation.Source: §6 · p. 8
• At three times the budget, full-parameter SimPO passes dose one on both Llama-2 and Llama-3.1, with 80 to 86% of samples degenerating despite the best held-out ranking among UltraFeedback runs on those bases.Source: §6 · p. 8
• From the public Llama-3-8B SFT checkpoint, the official SimPO recipe grows the offset from norm 33 to 77 without forming a well, while sample NLL rises from 1.35 to 3.86.Two public reward models prefer the one-fold samples over the SFT model's on 63 to 65% of prompts, but the three-fold samples on only 33 to 34%.Source: §6 · p. 8

• A norm penalty shrinks the offset, while a loudness penalty flattens its vocabulary projection without necessarily shrinking its norm.The recipe retains centering; on driving lists it also keeps an NLL term of weight 1.0.Source: §7 · p. 8
• Listwise ranking preserves held-out ordering among rejected responses at 0.82 to 0.92, while pairwise ranking inverts that order.The authors identify offset control as most useful for objectives worth keeping despite high demand.Source: §7 · p. 9
• Across four base models at or beyond untreated onset budgets, offset-control runs form no well and keep sampled degeneration at or below about 12%, versus 71 to 81% of released mass in a single well for three Llama-2 baselines.Held-out pairwise accuracy is 0.94 to 0.96, within 2 points of runs without offset control on Llama-2, Mistral, and Qwen2.5-7B.Source: §7 · p. 9, Table 3 · p. 9

8 WHAT RANKING METRICS MISS

• ρ = −0.78 for Skywork-Reward-V2 and ρ = −0.79 for ArmoRM: across 17 UltraFeedback runs, higher held-out pairwise accuracy coincided with lower reward-model preference for the run’s samples over the base model’s.The comparison used one sample for each of 300 held-out prompts; because the base models were not instruction-tuned, the authors use win rate only to compare runs.Source: §8 · p. 9
• In seven of nine lineages with snapshots every 50 steps, the eventual well entered the eight largest entries of W¯b up to 1,200 steps before taking over, after held-out ranking had saturated.The authors describe tracking ∥¯b∥, its loudness, and leading entries of W¯b as an early signal, but note they have read it only from completed runs.Source: §8 · p. 9

9 CONCLUSION

• Across runs, held-out ranking accuracy anticorrelates with an external judge (ρ = −0.78), while hiding the offset from the loss and penalizing it prevents collapse up to three times the base budget.The offset shifts final hidden states across positions and chosen and rejected responses; it forms a well when its projection lifts a token above those the base model would predict.Source: §9 · p. 10
• Evidence spans 17 lineages, two datasets, and adapter and full-parameter training, but penalty strengths were calibrated on driving data and the six-fold result covers only OLMo.There were two further seeds for flagship settings and one seed elsewhere; apart from SFT-start runs, demand analysis starts from base models.Source: §9 · p. 10

REPRODUCIBILITY STATEMENT

• Configurations, measurement code, and the held-out ranking probe are documented in §§3–7 and Appendix B, with code and training and held-out preference data slated for release upon publication.Held-out ranking uses 600 lists, or 300 where stated; prompts and chosen responses do not appear in training splits.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

• The authors used generative AI tools across conceptual development, experiments, implementation, dataset cleaning, interpretation, translation, and writing, but state they did not generate synthetic datasets and reviewed the assisted work.They retain responsibility for the final content.Source: AI USE STATEMENT · p. 10

APPENDIX OVERVIEW

• Appendix A extends related work, while Appendix B collects setup, full results, supporting measurements, and a formal statement of common-shift invariance.The appendix overview maps its parts to the main-text sections they support.Source: APPENDIX OVERVIEW · p. 16, A EXTENDED RELATED WORK · pp. 16–17

B.2 SETUP AND MEASUREMENT DETAILS

• The setup specifies scene-dependent β values, defines exposure as the number of times a ranked response serves as a negative baseline, and evaluates runs across mass flow, degeneration, held-out ranking, and offset geometry.Held-out ranking uses 600 lists, or 300 for specified runs; computationally intensive analyses use representative subsets.Source: §B.2 · pp. 17–18

B.1 FULL CROSS-FAMILY ONSET TABLE

• Fourteen of 17 base-model lineages collapse within six times the base budget; the table records untreated runs at onset or, absent collapse, at the largest tested budget.Collapse is defined by a well or by most samples degenerating or forming a word ring; testing budgets vary by lineage.Source: §B.1 · p. 17, Table 4 · p. 18

B.3 TRAINING CURVES AND EXTENDED-BUDGET TRAJECTORIES

• Without offset control, the flagship runs reach the lowest ranking loss and highest training margin, with almost all extra margin coming from pushing rejected responses down.The appendix also presents six-fold-budget OLMo trajectories, including per-position loudness penalties.Source: §B.3 · p. 18, Figure 4 · p. 19

B.4 ROUTES OF THE RELEASED MASS AT EXTENDED BUDGET

• At six times the base budget, offset-control penalties redirect released mass through alternative routes; residual penalties close both routes on OLMo-3 but OLMo-2 still develops word-family rings and a deep tail.Across Llama-2 and Qwen, norm penalties keep the offset short, while loudness penalties can leave it large but limit its leading vocabulary entry.Source: §B.4 · p. 18, §B.5 · p. 19

B.6 A LISTWISE ADVANTAGE ORTHOGONAL TO COLLAPSE

• On driving lists, sequential listwise K=4 preserves graded ordering of unseen rejected responses at 0.82 to 0.92, whereas pairwise K=2 inverts it at 0.17 to 0.27; this benefit does not appear on UltraFeedback.The authors describe the listwise advantage as domain-dependent and orthogonal to collapse.Source: §B.6 · p. 19

B.7 REMOVING THE OFFSET FROM A COLLAPSED MODEL

• Removing half or all of the collapsed model’s offset bias at decoding stops collapse, but removing only the well token’s logit lets collapse move to another token; z-loss compresses scale without preventing well capture.The strongest z-loss lowers log Z from 76.0 to 30.5, yet the top token retains 65 to 66% of released mass, against 74% untreated.Source: §B.7 · p. 20, §B.8 · pp. 20–22, Figure 8 · p. 22

B.9 DOSE ALONG TRAINING AND THE OBJECTIVE’S DEMAND

• Across the β sweep, the dose passes one only for per-token averaging above the base-gap demand; textbook DPO and SimPO have lower doses within the base budget.Full-parameter SimPO crosses D = 1 at step 3000 under the three-fold budget.Source: §B.9 · p. 22, Figure 11 · p. 24
• In all nine tracked lineages, median dose exceeds one before takeover; the well reaches a majority of rejected positions within 100–300 steps.Across the β sweep, final dose exceeds one only for per-token averaging when demand is above the base gap; summed-score runs remain below 0.2 dose in the healthy cases.Source: Figure 9 · p. 22, §B.9 · p. 22, Figure 10 · p. 23

B.10 FULL-PARAMETER ONSET

• Under full-parameter per-token training, the offset alone makes the well top-ranked at 98–100% of positions, while unembedding rows alone do so at no more than 1.5%.Onset tracked integrated learning rate; a prespecified Llama-2 run nucleated at step 250.Source: §B.10 · p. 23

B.11 A SUPERVISED FINE-TUNING CONTROL

• Chosen-response NLL-only fine-tuning builds a shared offset but no well: its largest vocabulary entry is 3.4 and the top token captures 8.6% of released mass.The offset norm reaches 56 and chosen/rejected offsets have cosine 0.99; the ranking term decides where the offset points.Source: §B.11 · p. 23

B.12 SIMPO FROM A SUPERVISED FINE-TUNING CHECKPOINT

• From the public SFT checkpoint, SimPO improves held-out ranking without forming a well, even as the offset grows from 32.5 to 77.3 across budgets.No vocabulary token reaches median dose 0.4; at three-fold budget the SFT model’s per-token NLL on generated samples rises to 3.86.Source: §B.12 · pp. 23–24, Table 6 · p. 24

B.13 EXTERNAL REWARD MODELS

• Across 17 UltraFeedback runs, better held-out ranking coincides with public reward models preferring the run’s samples to the base model’s less often.In seven of nine collapsing lineages, the eventual well enters the eight largest entries of the base unembedding before taking over rejected positions.Source: Figure 12 · p. 25, §B.13 · p. 25, Table 7 · p. 25

B.14 SHAPES OF WELL THAT STANDARD RULERS MISS

• Standard n-gram repetition measures miss latent, non-repeating, and word-family-ring wells, as well as diffuse degradation detectable through base-model likelihood.The flagship repetition loop is flagged, but three other well shapes are missed.Source: §B.14 · p. 26

B.15 SUPPLEMENTARY MEASUREMENTS

• The flagship offset retains cosine 0.85 when recomputed on held-out text with unseen phrasing.Natural-run takeover rises from under 10% to a majority of rejected positions within 100–300 steps.Source: §B.15 · p. 26
• In 13 bases, the highest-norm unembedding row belongs to a rare token, whose gain can exceed base gaps of 17–39 nat.Under full-parameter UltraFeedback fine-tuning, the Llama family collapses onto the bare space token, with the offset alone making it top-ranked at nearly every position.Source: §B.15 · p. 26
• At matched margins of 12–16 nat, uncontrolled Llama-2 runs channel 42–51% of released mass into one well, versus 8–20% with offset control.Capture takes off at different margins across OLMo bases, so margin is not a stopping criterion.Source: §B.15 · p. 27

B.16 SEEDS AND SCOPE

• Across tested seeds, untreated flagship runs collapse onto the same well while treated seeds remain well-free; adapter rank changes onset timing rather than whether a well forms.The tested demand sweep covers per-token averaging, textbook DPO, and SimPO under adapter and full-parameter protocols.Source: §B.16 · p. 27

B.17 COMMON-SHIFT INVARIANCE AND THE TRAINED OFFSET

• Prompt-relative losses are invariant to adding a constant to all response scores, leaving the common score component unidentified by the objective.For a shared hidden shift, the first-order objective constraint lies along one direction; the orthogonal complement is constrained only by a second-order term.Source: §B.17 · p. 27
• The invariance claim is local and first-order, and shared hidden shifts arise through parameter updates and shared Jacobians rather than being model parameters themselves.An absolute-score term such as chosen-response NLL breaks the invariance and can delay displacement.Source: §B.17 · p. 28
• The common component of the trained offset exceeds its differential component by 20–530 times in four collapsing runs, with cosine to the base score-gradient direction at most 0.16.The offset carries the collapse-producing component but does not account for all likelihood displacement.Source: §B.17 · p. 28, Table 8 · p. 28