Learning Dynamics of LLM Finetuning

• The framework decomposes prediction changes during finetuning into interpretable influence terms and applies across SFT, DPO variants, and reinforcement-learning methods.It targets a uniform analysis of instruction tuning and preference tuning.Source: §1 · pp. 1–2
• It uses learning dynamics to explain phenomena including hallucination, repeated responses, and declining confidence during off-policy DPO.The proposed explanations concern how training examples influence predictions on other responses.Source: §1 · p. 2
• The analysis identifies a squeezing effect in off-policy DPO, where negative gradients can reduce confidence broadly and make desired outputs less likely.The effect is linked to probability mass shifting toward already likely outputs.Source: §1 · p. 2
• The paper proposes a simple method inspired by this analysis to improve alignment performance.Source: §1 · p. 2

• Learning dynamics ask how a gradient update on xu changes the model’s prediction on xo.The framework relates parameter changes to prediction changes after one GD step.Source: §2 · p. 2
• The framework tracks confidence through changes in log πt(y | xo) and decomposes each step using the model’s logit gradients and loss residual.Source: §2 · p. 2
• The analysis assumes that the relative influence of learning one example on other examples remains relatively stable during training.The common lazy-eNTK assumption is sufficient but not necessary.Source: §2 · p. 3
• The eNTK measures model-specific similarity between examples, while the loss-derived Gt determines the energy and direction of adaptation.Larger eNTK norm indicates stronger potential influence from the learned example.Source: §2 · p. 3
• In MNIST, learning an example pulls up the correct label for same-class inputs and can also raise the learned class probability for similar inputs from another class.The accumulated influence correlates strongly with the integral of per-step influence curves.Source: §2 · p. 3

• LLM finetuning is challenging because inputs and outputs are high-dimensional and sequential, algorithms differ, and pretrained models assign little mass to unlikely tokens.These properties complicate a uniform dynamical analysis and create conditions for squeezing effects.Source: §3 · p. 4

3.1 PER-STEP DECOMPOSITION OF THE SFT LOSS

• Causal masking lets the framework represent autoregressive token predictions with a decomposition close to multi-label classification, using sequence-conditioned eNTK terms.The eNTK depends on both input prompts and responses.Source: §3.1 · p. 4

• SFT applies strong upward pressure to the target response while slightly increasing confidence in similar responses through cross-example influence.Source: §3.1 · pp. 4–5
• DPO differs from SFT through large negative gradients, which can push down confidence across many responses rather than selectively raising the preferred response.The negative-gradient distinction is identified as central to their differing learning curves.Source: §3.3 · pp. 5–6

3.3 THE SQUEEZING EFFECT CAUSED BY NEGATIVE GRADIENT

• The squeezing effect occurs when a negative gradient is applied to an unlikely response: probability mass shifts toward the previously most likely output, while other probabilities decline.Peakier distributions squeeze more, and the effect can contribute to repeated phrase generation; practical dynamics can be more complicated when positive and negative pressures interact autoregressively.Source: §3.3 · p. 6

• DPO variants can partially mitigate squeezing by limiting negative-gradient strength or changing where negative gradients are applied.Source: §3.2 · p. 5, §3.3 · p. 7

4.1 LEARNING DYNAMICS OF SFT

• SFT can amplify hallucinations by increasing the probability of a preferred response for another question, causing answers to mix material across unrelated training questions.The authors connect this cross-question influence to sufficiently strong similarity between responses under the empirical neural tangent kernel.Source: §4.1 · pp. 7–8
• SFT steadily increases confidence in the preferred response while also increasing confidence in responses similar to it.The experiments probe 500 examples and multiple response types, including preferred responses, rephrases, and other generated responses.Source: §4.1 · p. 7, §4 · p. 7
• Similar responses initially gain confidence during SFT but later decline as diminishing pull-up pressure leaves global push-down pressure dominant.This pattern occurs even for responses unseen during SFT, including ChatGPT-generated responses, test-set responses, and random English sentences.Source: §4.1 · pp. 7–8
• ChatGPT-generated responses are treated as similar despite semantic differences, suggesting that preferred idioms or phrases act as model-specific fingerprints.Figure 3 visualizes influence among response types, with updating responses on the x-axis and observing responses on the y-axis.Source: §4.1 · p. 8

4.2 LEARNING DYNAMICS OF OFF-POLICY DPO

• DPO makes confidence decay faster than SFT because its large negative gradient pushes down many responses, especially when applied in a low-confidence region.The analysis attributes the rapid decay to the squeezing effect, while the greedy-decoding response becomes sharply more confident and can reinforce prior bias and repeated phrases.Source: §4.2 · p. 9

4.3 MITIGATING THE SQUEEZING EFFECT BY AUGMENTING THE TRAINING SET FOR SFT

• Training-set extension weakens squeezing: non-preferred responses decay more slowly, greedy-decoding confidence rises more slowly, and the extend method outperforms the baseline after several DPO epochs.The extend method is worse before DPO because it explicitly trains on less-preferred responses, but its post-DPO performance is better when squeezing is mitigated.Source: §4.3 · p. 10

5 CONCLUSION

• The analysis explains counter-intuitive effects including repeater behavior, hallucination, and confidence decay under off-policy DPO.It also offers a dynamical perspective on why some on-policy and RL-free methods outperform vanilla off-policy DPO.Source: §5 · p. 10
• The squeezing effect arises when gradient ascent after a softmax pushes down nearly all token probabilities and transfers mass toward the most-likely labels.The authors suggest this perspective may extend to other deep-learning systems with large negative gradients on already-unlikely outcomes.Source: §5 · p. 10
• The paper uses its analysis to inspire a simple method that improves model alignment by mitigating the squeezing effect.The conclusion characterizes the method as simple but counter-intuitive and effective within the studied setting.Source: §5 · p. 10
• The framework does not directly cover token-wise-supervised methods such as RLHF and PPO because it assumes teacher forcing and relatively stable eNTK behavior.The authors leave token-wise learning dynamics for future work.Source: §A.2 · p. 15

B.2.2 DIFFERENT PREFERENCE FINETUNING ALGORITHMS

• Across the preference objectives, the residual terms connect their learning signals to SFT-like behavior while differing in update strength and gradient direction.The derivations cover DPO, IPO, SLiC, and SPPO-style objectives.Source: §B.2.2 · pp. 19–21
• DPO and IPO have identical update directions, with a scalar controlling update strength through the model’s confidence gap.Larger confidence gaps produce a larger scalar and a smaller residual-gradient norm.Source: §B.2.2 · pp. 20–21
• DPO and IPO therefore exhibit a regularizing effect that discourages the confidence gap from becoming too large.SLiC provides a hard version of this effect by stopping its push when the gap exceeds δ.Source: §B.2.2 · p. 21
• SPPO removes the large negative gradient by applying two positive update vectors, with the preferred response receiving the longer vector.The authors connect this design to reduced squeezing, greater stability, and better performance.Source: §B.2.2 · p. 21
• The framework’s core assumption is tested by examining whether the relative influence of an updating example across observing examples remains stable during training.The appendix evaluates this assumption in MNIST and LLM finetuning settings using empirical neural tangent-kernel quantities.Source: C THE “RELATIVE STABLE” ENTK ASSUMPTION · p. 21, Figure 6 · p. 21

Experimental setup

• The pipeline directly computes the eNTK term for MNIST and visualizes these dynamics in Figure 7.Source: §C.1 · p. 21, Figure 7 · p. 22

MNIST findings

• The eNTK norm remains relatively stable across different target inputs, especially in the later training stages.The earliest epochs are comparatively noisy, while the final epochs behave similarly to finetuning.Source: §C.1 · p. 22
• Similar inputs, such as “4” and “9” or “5” and “8”, exhibit clear pairing effects in their influence dynamics.Source: §C.1 · p. 22
• Dissimilar inputs also show pairing effects, including the examples “6” and “7” or “2” and “5”.Source: §C.1 · p. 22
• The pairing effect is not strictly symmetric because the A and G terms are inconsistent.Source: §C.1 · p. 22

Computational scope

• Computing the LLM eNTK norm directly is expensive because each token requires multiplying a V × d matrix by a d × V matrix.The analysis therefore focuses on relative relationships between norms rather than their absolute values.Source: §C.2 · p. 22

Measurements

• The analysis introduces LBKuo and SignDeltauo to track the strength and direction of relative influence through Kuo.Source: §C.2 · p. 23

LLM experiments

• The relative influence metrics remain stable across SFT and DPO under different supervisory signals.The experiments track LBKuo and SignDeltauo to measure influence strength and direction.Source: §C.2 · p. 23, Figure 8 · p. 24

Influence direction

• Natural-language sequences strongly influence nonlanguage sequences in LBKuo, especially near the end of finetuning, but SignDeltauo indicates that this influence is negative.Source: §C.2 · p. 23
• SFT using GPT-generated chosen or rejected responses produces stronger influence on ChatGPT-generated sequences than on the original response, suggesting a similarity pattern.Source: §C.2 · p. 23

DPO dynamics

• DPO exhibits dominant push-down pressure, with almost all SignDeltauo terms strongly negative.The only positive term is associated with y+u and is roughly 0.5, smaller than positive values in the SFT cases.Source: §C.2 · p. 23

Probing dataset

• The probing dataset places responses in a two-dimensional space based on semantic relevance and format relative to the preferred response.Source: D MORE ABOUT EXPERIMENTS · p. 24, Figure 10 · p. 25

Probability dynamics

• During SFT and DPO, the supervisory signal increases fastest, while DPO changes the preferred-response probability and complementary probability mass faster than SFT.Source: D MORE ABOUT EXPERIMENTS · p. 25, Figure 9 · p. 25

Response categories

• The response space is divided into instruction-following, non-instruction-following human-language, and non-human token-sequence regions.Source: §D.1 · pp. 25–26
• The dataset includes the initial response, chosen and rejected responses, multiple rephrases, responses from other questions, test responses, random English, and randomly permuted token sequences.Source: §D.1 · p. 26

Evaluation design

• Dprobtest uses unseen test prompts and responses, enabling comparison between directly influenced training examples and indirectly influenced examples.Source: §D.1 · p. 26

Consistent learning dynamics

• Across five pretrained models with different sizes and training recipes, the learning-dynamics curves exhibit similar trends.Source: §D.2 · p. 26, Figure 14 · p. 29

Rephrased responses

• Chosen responses keep increasing rapidly after several updates, indicating stronger pull-up pressure than for their rephrases.Source: §D.2 · p. 26
• Rephrases of chosen responses decay more slowly than the initial-policy response, whereas rejected-response rephrases do not show the same pattern.Source: §D.2 · p. 27
• Format-preserving GPT rephrases usually receive higher predicted probabilities than semantics-preserving GPT rephrases.The authors interpret this as evidence that SFT may prioritize format over sentence semantics.Source: §D.2 · p. 27

Training exposure

• In Dprob, the pull-up pressure is generally stronger than in Dprobtest because many prompts and responses occur during training.Source: §D.2 · p. 28, Figure 16 · p. 30

Response-group dynamics

• The learning curves show that y+u first increases then decreases, random permutations remain small and decrease, and y+j̸=u increases more slowly than y+u.Source: Figure 13 · p. 28, §D.2 · p. 28
• Test responses have a later inflection point than random human-language responses because they are semantically more similar to y+u.Source: §D.2 · p. 28

Rejected-response training

• When training uses rejected rather than chosen responses, the trends for chosen and rejected rephrases reverse.Source: §D.2 · p. 30

Consistent learning dynamics for different models

• The observed relative trends remain consistent across Pythia-410M, Pythia-1B, Pythia-2.8B, and Qwen1.5-0.5B models.Source: §D.3 · p. 31
• Chosen-response confidence decays more slowly than rejected-response confidence, while rephrases of chosen responses also decay more slowly than rephrases of rejected responses.These relative trends are reported across different models and align with direct pull-up pressure on chosen responses and push-down pressure on rejected responses.Source: §D.3 · p. 31

Learning dynamics of conducting SFT first, then DPO

• DPO decays most response confidences faster than SFT because its large negative gradient pushes down nearly the whole prediction curve except the highest-confidence response.The authors name this non-trivial pattern the “squeezing effect.”Source: §D.3 · p. 31
• Longer SFT strengthens squeezing during subsequent DPO by producing a peakier starting distribution and faster decay of non-argmax responses.The effect is attributed to longer SFT pushing the rejected response probability lower before DPO.Source: §D.3 · p. 31
• DPO can eventually reduce even the confidence of chosen responses, despite its positive adaptation pressure on them.The paper notes that related works report this behavior, while the two direct pressures alone do not explain why all observed response probabilities decrease.Source: E THE SQUEEZING EFFECT INTRODUCED BY… · p. 33

Negative-gradient mechanism

• A negative gradient in softmax cross-entropy guarantees that the targeted class probability decreases while the highest-confidence other class increases.The resulting update redistributes probability mass away from the targeted class and toward the current maximum.Source: §E.1 · p. 35

Distribution-dependent behavior

• When the prediction distribution is uniform, removed probability mass is distributed equally across all non-target classes.Source: §E.1 · p. 35
• When the distribution is slightly non-uniform, lower-confidence non-target classes tend to decrease while higher-confidence ones tend to increase.This is described as a general trend rather than a guarantee.Source: §E.1 · p. 35
• When the distribution is very peaky, all probabilities other than the most-confident class decrease, squeezing probability mass into that class.This practical regime is common in LLM finetuning because only a small portion of the large vocabulary carries substantial probability.Source: §E.1 · pp. 35–36

Strength of the effect

• A larger absolute learning rate or feature norm makes the squeezing effect more serious by weakening the targeted-class contribution and making decreases across other classes more likely.Source: §E.1 · p. 36

Simple experiment

• For a flat distribution, the negative gradient pushes down the targeted class and reallocates its probability mass approximately evenly across the other classes.Source: Figure 19 · p. 36, §E.2 · p. 37
• In a non-uniform multimodal distribution, a negative gradient on a confident class squeezes probability mass toward other confident classes while leaving low-confidence classes nearly unchanged.This matches the predicted behavior when the negative gradient is imposed on a peaky region.Source: Figure 20 · p. 37, §E.2 · p. 38

Connection to DPO

• Off-policy DPO can apply large negative gradients to already unlikely outputs, reinforcing the model’s prior high-confidence predictions rather than necessarily forming preferred responses.The authors identify this as a potential drawback of off-policy DPO.Source: §E.2 · p. 38
• Longer SFT creates a peakier starting distribution and therefore accelerates decay of non-argmax responses during DPO.The same trend appears in the LLM finetuning experiments.Source: §E.2 · p. 38

Dataset extension

• The proposed dataset-extension method adds rejected-response pairs during SFT so DPO’s negative gradients act on a peaky rather than valley region, weakening squeezing.The method is presented as both a simple intervention and an ablation of squeezing’s influence.Source: §E.2 · p. 38, §F.2 · p. 39

Learning dynamics

• The proposed method’s learning dynamics show slower confidence decay during DPO because rejected-response regions are pulled up during SFT before negative gradients are applied.The resulting gradients act on a peaky rather than valley region, restraining squeezing.Source: §F.3 · p. 39

Pairwise comparisons

• After SFT alone, the proposed method and baseline are closely matched, with E1 vs B1 win rates of 0.4729 and 0.4679.Source: §F.3 · p. 40
• The proposed method beats the baseline after 2, 4, and 6 DPO epochs, with win rates of 0.6518 vs 0.5151, 0.6928 vs 0.6045, and 0.6667 vs 0.5432.These comparisons directly evaluate the proposed and baseline pipelines at matched DPO durations.Source: §F.3 · p. 40
• The best proposed checkpoint, E2-4, outperforms E2-2 and E2-6 with win rates of 0.6853 vs 0.5517 and 0.6324 vs 0.5316.The results indicate that both overly short and overly long DPO training are suboptimal.Source: §F.3 · p. 41

Qualitative evaluation

• The proposed method produces fewer degenerate responses than the baseline comparison despite both methods inevitably generating some such responses.Source: Figure 23 · p. 41