PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks

Haodong Liang, Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai

arXiv:2610.02437v1stat.MLcs.AIcs.LG

TL;DR

The paper asks how fine-tuning on semantically correct demonstrations can still cause forgetting of previously acquired capabilities. It analyzes a linear-softmax classification model that separates semantic and style updates, then shows that under specified population-update conditions SFT can forget while RFT retains zero semantic error.

  • Problem

    Although SFT and RFT can differ in retention even with correct supervision, existing accounts leave open how fitting a distribution of correct responses can produce errors on previously mastered tasks.

    Source: §1 · p. 2

  • Method

    The authors analyze a linear-softmax classification model with shared semantic rules and task-specific prompt distributions and style preferences, decomposing SFT and RFT updates into semantic and style components.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Under explicit conditions and population updates, SFT develops style drift and strictly positive semantic forgetting, while RFT preserves zero semantic error from a perfectly fitted, style-neutral checkpoint.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this model, fitting task-specific style preferences can shift the semantic decision boundary and disrupt prior predictions, while exact-gradient RFT preserves within-class symmetry from a symmetric initialization.

    Source: §5 · p. 11

    The analysis is limited to a single-token linear-softmax model with a shared semantic rule across tasks; nonlinear policies, multi-token outputs, and evolving rules remain future extensions.

    Source: §5 · p. 11

Abstract

from arXiv · show

Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.

1 Introduction

The paper asks how fitting semantically correct demonstrations can still impair previously learned capabilities, focusing on teachers’ preferences among equivalent answer styles. In a linear-softmax classification case study, it decomposes SFT and RFT updates and identifies conditions under which their style dynamics yield different retention outcomes.

1 Introduction

  • Prior work reports weaker retention under SFT than RFT even with correct supervision, while leaving open how fitting correct-response distributions can create errors on mastered tasks.The paper isolates preferences among semantically equivalent expressions; for example, different phrasings can express the same answer while a teacher favors one style.Source: §1 · pp. 1–2
  • The paper analyzes a linear-softmax classification model in which tasks share a semantic rule but differ in prompt distributions and teachers’ preferences among equivalent responses.Its exact update decomposition separates semantic and style components to trace how style fitting can affect previously mastered tasks.Source: §1 · p. 2
  • From a policy with no style preference, RFT preserves a zero style channel while SFT develops population style drift along the task mean when teachers have within-class preferences.At the same policy and prompt, the methods’ semantic updates are parallel, but their style updates differ.Source: §1 · p. 2
  • SFT has a strictly positive forgetting lower bound under population updates from a perfectly fitted, style-neutral checkpoint when the next task has nonzero prompt mean and class-asymmetric teacher preferences; RFT retains perfect semantic accuracy.The authors attribute the contrast to SFT style drift and RFT’s preservation of within-class symmetry under exact policy gradients.Source: §1 · pp. 2–3
  • Finite-sample analysis bounds off-axis parameter energy in both methods and gives RFT a smaller expected-energy upper bound than SFT.The bounds are consistent with the population analysis in the large-sample limit.Source: §1 · p. 2

2 Preliminaries

The preliminaries define a binary semantic classification setting in which prompts determine the rewarded class, while teachers may prefer particular styles within that class. They then specify the distinct SFT and RFT learning rules and the sequential-task retention measures used in the analysis.

2.1 Problem setup

  • Each task has prompts drawn from a Gaussian distribution and a teacher that selects only tokens in the correct semantic class, while allowing class-specific style preferences; prompt means are orthogonal to the shared semantic direction.A shared unit vector determines the correct binary class, and all tokens in a class express the same semantic answer in different styles.Source: §2.1 · p. 3

2.2 Learning rules

  • SFT minimizes cross-entropy to match the teacher’s token distribution, whereas RFT maximizes expected semantic reward using the policy’s own sampled responses.Both process one prompt at a time from zero parameters; the stated RFT update uses the exact policy gradient.Source: §2.2 · pp. 4–5

2.3 Continual learning

  • The policy is trained sequentially across tasks, and final evaluation uses average generalization error across tasks and average forgetting over earlier tasks.Forgetting compares an earlier task’s final accuracy with its best accuracy among preceding task checkpoints; negative values indicate positive backward transfer.Source: §2.3 · p. 5

3 Main Results

The section decomposes SFT and RFT updates into semantic and style channels, showing shared semantic directions but different style dynamics. Under explicit conditions, population SFT produces finite-time semantic errors while RFT preserves zero error; finite-sample bounds also favor RFT's off-axis energy.

3 Main Results

  • The section traces a progression from decomposing policy updates, through shared semantic and differing style dynamics, to population forgetting and finite-sample bounds.Source: §3 · p. 5

3.1 Semantic–style decomposition

  • The semantic–style decomposition separates class-level mean weights α from within-class token preferences β, clarifying how style can alter semantic predictions.A within-class style preference can change the prediction when ξβ(x) reverses the sign of ⟨∆α, x⟩.Source: §3.1 · p. 6

3.2 The semantic channel: a shared update direction

  • At a fixed policy and prompt, SFT and RFT update the semantic channel in the same direction, and both population updates increase the semantic margin along u.The SFT update does not depend on which style token the teacher sampled, while the RFT update is rescaled by the probability assigned to the correct class.Source: §3.2 · pp. 6–7

3.3 The style channel: RFT freezes, SFT drifts

  • From a class-uniform policy, RFT preserves zero style deviation, whereas SFT can drift along a nonzero task mean by fitting teacher style preferences.If teachers have no within-class preferences or task means are zero, both methods avoid off-axis population style drift; when drift occurs, it can shift the semantic decision boundary.Source: §3.3 · p. 7

3.4 From Style Drift to Semantic Error

  • SFT has strictly positive forgetting and generalization error over a finite training interval, while both remain zero for RFT under the stated population conditions.The separation starts from a common checkpoint that perfectly fits the previous task and has no style preference; the next task has a nonzero mean and a teacher preference stronger in one semantic class.Source: §3.4 · p. 8

3.5 Finite-sample off-axis energy

  • With finite prompt sampling, RFT has a smaller upper bound on expected off-axis parameter energy than SFT.When task means are bounded and nonzero, the RFT bound vanishes with the step size, whereas the SFT bound retains a nonzero task-mean term.Source: §3.5 · p. 9

4 Experiments

The experiments compare SFT and RFT across population and finite-sample task sequences, then test how task means, teacher preferences, and prompt distributions affect their differences.

4 Experiments

  • Teacher preferences assign higher probability to expressions with larger compatibility scores, while a zero preference vector gives a uniform within-class teacher.Preferences and style embeddings vary across tasks and classes, with the simulation setup choosing strongly nonuniform teachers.Source: §4 · p. 10
  • The setup trains SFT and RFT from W = 0 across eight sequential tasks, with two semantic classes of six tokens and three style dimensions.Finite-sample runs use 25 prompts per task and 50 seeds; population runs average updates over 10,000 fresh prompts and 10 seeds.Source: §4 · p. 10
  • Population RFT has zero generalization error and forgetting across tested learning rates, while population SFT has strictly positive values; finite-sample RFT also beats SFT at every matched rate.Population SFT develops off-axis energy while population RFT remains near zero; finite-sample off-axis energy is one to two orders of magnitude larger for SFT.Source: §4 · p. 10, Figure 2 · p. 11

Control and ablation experiments.

  • With zero task means or uniform teachers, the population SFT–RFT gap vanishes, but finite-sample SFT still has greater off-axis energy and generalization error; Laplace prompts leave results nearly unchanged.Population RFT also retains low forgetting with corrupted rewards and finite-action REINFORCE.Source: Control and ablation experiments · p. 11

5 Conclusion

The conclusion attributes forgetting in this classification case study to fitting task-specific style preferences, while the literature review situates SFT–RFT comparisons among distributional and continual-learning accounts.

5 Conclusion

  • Fitting style preferences can shift the semantic decision boundary and disrupt previously learned predictions, while exact-gradient RFT preserves within-class symmetry from a symmetric initialization.The analysis is a case study using a single-token linear-softmax policy.Source: §5 · p. 11
  • The theory is limited to a single-token linear-softmax model with a shared semantic rule across tasks; nonlinear policies, multi-token outputs, and evolving rules remain future extensions.Source: §5 · p. 11

A Literature Review

  • Prior empirical studies compare SFT and RFT retention, on-policy supervision, model-generated trajectories, and self-distillation, including reports of reduced forgetting under several RFT or on-policy settings.The reviewed reports include Chu et al. (2025), Lai et al. (2026), Chen et al. (2026), Zhang et al. (2026), and Shenfeld et al. (2026a).Source: A Literature Review · p. 16
  • Distributional accounts relate forgetting to KL divergence, policy-change bias, and target distributions, while continual-learning theories examine task structure, optimization, and architecture.Replay, parameter regularization, and parameter isolation are reviewed as complementary mitigation approaches.Source: A Literature Review · p. 16

B Supporting lemmas

These supporting results establish update identities and symmetry properties used to analyze semantic and style dynamics, alongside conditions for correct classification and a softmax continuity bound.

B Supporting lemmas

  • The expected teacher-draw gradient is an unbiased stochastic gradient of the SFT loss, and gradient descent on the SFT and RFT losses yields their respective update rules.Source: B Supporting lemmas · p. 17
  • When the class channels align with the semantic direction and γ > 2B, the decision boundary is ⟨u, x⟩ = 0 and every task has zero loss.Source: B Supporting lemmas · p. 18
  • Under zero task means and zero initialization, the update trajectories are rotation-equivariant in directions orthogonal to u, so their expected token vectors lie in span(u).Source: B Supporting lemmas · pp. 19, 21
  • The softmax map is 1-Lipschitz in Euclidean norm because its Jacobian is positive semidefinite with operator norm at most 1.Source: B Supporting lemmas · p. 21

C.1 Proof of Theorem 1

  • With β = 0 and an on-axis semantic difference, the population semantic update reduces to a one-dimensional calculation along u.The proof uses the symmetry of the projected Gaussian prompt component to eliminate the orthogonal and task-mean terms.Source: §C.1 · pp. 22–23

C.2 Proof of Theorem 2

The proof shows that exact RFT updates preserve within-class symmetry, whereas SFT can create off-axis style updates when teacher preferences and task means are nonuniform.

C.2 Proof of Theorem 2

  • When β = 0, exact RFT gives identical update vectors to tokens in the same semantic class and therefore preserves β = 0 at every step.Source: §C.2 · p. 24
  • At β = 0, SFT is uniform within each semantic class, but its style-coordinate update reflects the teacher’s within-class probability deviation.Source: §C.2 · pp. 24–25
  • The SFT population update’s off-axis component is proportional to the task mean, so nonzero task means can turn style fitting into off-axis drift.Source: §C.2 · p. 25

C.3 Proof of Theorem 3

The proof tracks task-2 updates in semantic and style coordinates from a perfectly fitted, style-neutral checkpoint. RFT preserves zero semantic error, while SFT’s style drift yields positive error on both tasks.

C.3 Proof of Theorem 3

  • RFT keeps both tasks correctly classified at every iterate because its style channel remains zero and its semantic increments stay on-axis.The proof applies the shared semantic rule to each RFT iterate.Source: §C.3 · p. 26
  • SFT’s parameter rows remain in the span of the semantic direction and task-2 mean, reducing the population dynamics to two scalar coefficients per token.The orthogonal residual has zero mean and is independent of the coordinates that determine the update.Source: §C.3 · pp. 26–27
  • At the first task-2 step, the semantic-channel update has no component along the task mean, while individual token rows can move along it through within-class deviations.The proof uses the fact that the class-average update is parallel to u, then tracks token-specific style coefficients.Source: §C.3 · p. 28
  • SFT’s style drift creates a region misclassified under both task distributions, giving each task strictly positive error after the specified updates.The proof establishes the error by showing that the highest-scoring token is wrong throughout that region and that its probability is positive.Source: §C.3 · pp. 30–31

C.4 Proof of Theorem 4

The proof separates semantic and off-axis parameter energy under fresh prompt sampling. It bounds finite-sample off-axis energy for both methods, while RFT retains its semantic channel under exact per-prompt gradients.

C.4 Proof of Theorem 4

  • The proof decomposes each update into semantic and off-axis components and tracks their energy along the sampled training trajectory.It conditions on the training history before analyzing the fresh prompt at each step.Source: §C.4 · p. 32
  • The conditional SFT energy recursion controls off-axis energy using the task mean and prompt-sampling fluctuations.The proof derives the bound by expanding the energy update and taking conditional expectations.Source: §C.4 · p. 34
  • RFT preserves a zero style channel pathwise, and its semantic coefficient remains nonnegative and nondecreasing along the trajectory.The proof reduces RFT to the semantic channel and shows the conditional radial component of its off-axis drift is nonpositive.Source: §C.4 · p. 35
  • The finite-sample bounds rely on fresh prompts; the proof does not directly cover reuse of a fixed prompt sample.For RFT, pathwise style invariance also requires exact per-prompt gradients rather than sampled action gradients.Source: §C.4 · p. 36

D.1 Exact and sampled policy gradients

The paper distinguishes exact RFT gradients from REINFORCE estimates based on sampled actions. Sampling preserves the update in conditional mean but can add style noise that breaks pathwise invariance.

D.1 Exact and sampled policy gradients

  • A sampled-action REINFORCE update is an unbiased estimate of the exact RFT update conditional on the current policy and prompt.The estimator’s conditional mean agrees with the RFT update.Source: §D.1 · p. 37
  • When the style channel is initially zero, a rewarded single-action sample can create nonzero style deviations even though the conditional mean style increment is zero.An unrewarded sample produces no update for this estimator.Source: §D.1 · p. 37
  • Averaging K independent action-gradient estimates reduces style noise but does not guarantee that every update preserves a zero style channel.The estimates are computed at the same policy and prompt.Source: §D.1 · p. 37

D.2 Limiting behavior under continued task-2 training

Continued unregularized population SFT can eventually eliminate semantic errors, even though its finite-time behavior includes forgetting. In the numerical illustration, errors later decline but style drift remains, whereas RFT stays error-free.

D.2 Limiting behavior under continued task-2 training

  • With a sufficiently small fixed step size, continued unregularized population SFT drives semantic errors on both tasks to zero, despite positive finite-time errors.This conclusion concerns semantic errors, not convergence to a finite parameter matrix.Source: §D.2 · pp. 37–39
  • The population objective approaches its infimum without a finite minimizer, and exact matching of teacher style preferences is unnecessary for vanishing semantic error.The parameters must diverge in norm along any sequence approaching the infimum.Source: §D.2 · p. 39

D.3 Numerical illustration of finite-time forgetting

  • In the illustration, training starts from a perfectly fitted, style-neutral checkpoint and uses η = 10^-3 for 10^6 steps; the theorem’s guaranteed interval ends at τ0 ≈ 0.00334.The task means are (0, 2) and (0, 3), and the task-2 teachers have within-class preferences differing by Δρ = 0.49.Source: §D.3 · p. 40
  • Near τ = 0.8, SFT old- and new-task errors reach 13.4% and 19.4%, then fall to 1.25% and 1.81% at τ = 1000; RFT stays zero.The inset compares early SFT old-task error with Theorem 3’s lower bound; the later times extend beyond the guaranteed interval.Source: Figure D.1 · p. 40, §D.3 · p. 40

E.1 Control experiments

Control experiments show that removing either task-mean shifts or teacher style preferences eliminates the population SFT–RFT performance gap. Finite-sample SFT nevertheless retains worse off-axis and generalization–forgetting behavior in these controls.

E.1 Control experiments

  • With zero task means, both population methods have negligible generalization error and forgetting, while finite-sample SFT has greater off-axis energy and a worse generalization–forgetting tradeoff than RFT.The residual population off-axis energy reflects Monte Carlo prompt sampling.Source: §E.1 · p. 41, Figure E.1 · p. 41
  • With uniform teachers and nonzero task means, the population performance gap disappears, but finite-sample SFT still introduces style fluctuations by sampling individual teacher tokens.Population SFT preserves β = 0 because its updates are constant within each semantic class.Source: §E.1 · p. 41, Figure E.2 · p. 41

E.2 Ablation experiments

Ablations test prompt distributions, reward correctness, and exact action expectations. The findings show that the main population separation is robust to Laplace prompts, while departures from exact RFT assumptions introduce distinct finite-sample effects.

E.2 Ablation experiments

  • With Laplace prompts, population SFT’s off-axis energy and forgetting and the finite-sample curves are nearly unchanged; population RFT keeps forgetting below 0.3%.At η = 0.5, RFT off-axis energy is about 3 × 10^-2 versus 2.5 × 10^-4 with Gaussian prompts, and error reaches 1.5%.Source: §E.2 · p. 42, Figure E.3 · p. 42
  • Class-flip reward corruption preserves RFT’s zero-style invariance, whereas token flips introduce style fluctuations and more forgetting than class flips in finite-sample runs.Monte Carlo RFT exhibits little forgetting across the tested noise levels.Source: §E.2 · p. 42, Figure E.4 · p. 43
  • At the reference learning rate, REINFORCE forgetting moves closer to the exact RFT baseline as the number of sampled actions increases in both population and finite-sample settings.Prompt and update counts are matched within each protocol, but action-query and computation budgets differ across action counts and the exact baseline.Source: §E.2 · p. 42
Loading 2610.02437v1…