PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features

Cunchun Li, Haonan He, Yifan Gao, Minglei Li, Jingqi Ye, Qingyu Yang, Peng Ye

arXiv:2609.33463v1cs.CL

TL;DR

SFT’s strongest corrections target unlikely demonstrated tokens, which may be informative or noisy, while nonnegative token reweighting cannot reverse a harmful learned feature. SCALE freezes the pretrained model and SFT delta and uses predictive entropy to learn gates that suppress, reverse, or extrapolate it. Across three backbones, it improves mathematical reasoning and code-generation performance while remaining competitive on general retention.

  • Problem

    SFT strongly corrects unlikely demonstrated tokens, although these can represent either useful novelty or noisy supervision, and nonnegative token weights cannot reverse a learned update.

    Source: §1 · p. 2

  • Method

    SCALE freezes the pretrained model and SFT delta, then learns bounded token- and module-specific gates by minimizing predictive entropy alone.

    Source: §1 · p. 3

  • Results

    Across three backbones, SCALE improves mathematical reasoning and code generation while remaining competitive on general retention; its mathematical-reasoning Avg@16 scores are 37.84%, 43.60%, and 36.57%.

    Source: §1 · p. 3, §6.2 · p. 9, §7 · p. 11

  • Takeaways & Limitations

    The results support controlling how a completed SFT delta is used, rather than only modifying how it is learned.

    Source: §1 · p. 3, §7 · p. 11

    Entropy serves as a local selection signal, not a correctness oracle, and task-beneficial entropy descent depends on alignment within the restricted gate subspace.

    Source: §1 · p. 3

Abstract

from arXiv · show

Supervised fine-tuning (SFT) learns most aggressively from tokens that the model deems least likely. This helps acquire new behaviors, but also amplifies noisy or conflicting supervision and can overwrite useful pretrained knowledge. Through a unified policy-loss view, we revisit existing token-reweighting methods and show that they assign nonnegative coefficients to demonstrated tokens. Consequently, they can suppress or amplify supervised updates, but cannot reverse harmful features once learned. Moreover, larger training weights do not amount to feature extrapolation, since they change the optimization trajectory rather than scale a fixed SFT direction. We argue that reversal and extrapolation require a stable reference frame defined by a fixed SFT delta. Motivated by this, we propose SCALE (Selective Control of Adaptation via Local Entropy), an entropy-guided adaptation-strength-control method that freezes the pretrained model and the SFT delta and learns bounded token- and module-specific gates by minimizing predictive entropy alone. These gates suppress, reverse, or extrapolate frozen SFT features according to their alignment with entropy reduction. Across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base, SCALE achieves mathematical-reasoning averages of 37.84, 43.60, and 36.57, exceeding the strongest corresponding baselines while remaining competitive on general-retention benchmarks. It also attains the best average code-generation performance across HumanEval, HumanEval+, and MBPP for all three models. These results suggest that effective SFT correction can benefit from controlling how already learned residuals are used, rather than only modifying how they are learned.

1. Introduction

SFT strongly corrects tokens the model finds unlikely, which can acquire useful behaviors but also amplify noisy or conflicting demonstrations. SCALE instead controls a frozen SFT delta with entropy-guided gates that can suppress, reverse, or extrapolate its contribution.

Motivation

  • SFT gives stronger corrections to less probable demonstrated tokens, making its useful acquisition of novel patterns inseparable from amplification of potentially noisy or conflicting supervision.A low-probability reference may provide valuable information or conflict with a coherent alternative the pretrained model prefers.Source: §1 · p. 2

Gap

  • Representative token-reweighting methods use nonnegative demonstrated-token coefficients, so they can suppress or amplify supervised updates but cannot reverse their direction.The paper distinguishes this training-time reweighting from post-SFT control of a completed update.Source: §1 · pp. 2–3

Method

  • Entropy is a local selection signal rather than a correctness oracle; its task-beneficial effect depends on alignment between entropy advantage and task value in the restricted gate subspace.The pretrained model and SFT parameters remain fixed while the gates are optimized.Source: §1 · p. 3
  • SCALE freezes the pretrained model and learned SFT delta, then learns bounded token- and module-specific gates using predictive entropy alone.Gate values below zero reverse the delta, values from zero to one suppress it, and values above one extrapolate it.Source: §1 · pp. 2–3

Results

  • 37.84%, 43.60%, and 36.57% are SCALE’s mathematical-reasoning Avg@16 results on the three backbones, exceeding DFT by 5.03, 6.08, and 1.91 points.SCALE remains competitive on general retention and has the best code-generation averages across HumanEval, HumanEval+, and MBPP for all three models.Source: §1 · p. 3

2. Related Work

Prior token-reweighting work changes token influence while learning an update, whereas post-training delta methods select or combine learned updates. SCALE instead modulates one completed SFT delta for each input and module.

Token-level reweighting for SFT

  • Token-cleaning and reweighting methods intervene while the task update is being learned; SCALE freezes a completed SFT delta and learns how strongly it acts per input and module.The cited token methods include influence-based cleaning, probability-based DFT and ProFit, entropy-based EAFT, and other adaptive or regularized approaches.Source: §2 · p. 3

Post-training model editing and delta manipulation

  • Task Arithmetic, TIES-Merging, AdaMerging, and WEMoE manipulate or combine learned updates, while SCALE controls a single SFT delta with token-conditioned gates.SCALE’s gates support suppression, reversal, and extrapolation of that same delta.Source: §2 · p. 4

3. Preliminaries

The policy-loss view treats a prefix as a state and a vocabulary token as an action, while keeping demonstrated contexts teacher-forced. Predictive entropy supplies signed action-level feedback, but does not identify correct actions.

3. Preliminaries

  • The policy-loss formulation treats each prefix as a state and each vocabulary token as an action, with action distributions and coefficients defining the gradient surrogate.Coefficients are evaluated at the current iterate and held fixed during the backward pass.Source: §3 · p. 4
  • An on-policy action expectation does not make the demonstrated contexts or trajectories on-policy.The formulation keeps contexts teacher-forced.Source: §3 · p. 4

3.1. Predictive entropy as a signed policy loss

  • Predictive entropy yields signed action-level gradient feedback, but the signal does not identify correct actions and is used with an explicit feature-alignment assumption.The entropy advantage is centered and positive when an action probability exceeds exp(-H_t).Source: §3.1 · p. 4

4. Revisiting Token Correction Methods for SFT

Nonnegative token weights control the magnitude of demonstrated-token learning, not the application of a completed update. A frozen SFT delta supplies a reference frame for suppression, reversal, and extrapolation, while entropy-based selection relies on a conditional alignment premise.

4.1. What nonnegative token weights control

  • Nonnegative weighted cross-entropy raises the demonstrated logit, lowers competing logits, or vanishes, so it cannot reverse that direct CE direction.This is a claim about the weighted CE component, not an impossibility theorem for shared-parameter training.Source: §4.1 · p. 5

4.2. A fixed delta defines suppression, reversal, and extrapolation

  • After SFT, a gate scales the same frozen module-level delta: zero disables it, values between zero and one suppress it, negatives reverse it, and values above one extrapolate it.The delta operator is frozen, though hidden states and final logit directions can still change with the gates.Source: §4.2 · p. 5

4.3. Why entropy can select fixed features

  • Under a positive correct-versus-incorrect margin, entropy descent strengthens helpful fixed contributions and weakens harmful ones; when the margin is negative, the conclusion reverses.The premise is local to task-supporting deltas reducing uncertainty and interfering deltas increasing it, not unrestricted feature learning.Source: §4.3 · pp. 5–6

5. Method

SCALE separates learning an SFT delta from calibrating its use: after freezing the pretrained model and learned delta, it trains bounded gates using predictive entropy. The gates can suppress, reverse, or extrapolate the frozen delta, with local task improvement conditional on entropy–task alignment in the gate subspace.

5.1. Stage I: learning the task delta

  • Stage I learns a LoRA task delta; Stage II freezes the base, output head, and learned factors instead of relearning the delta direction.The section frames the method as separating delta acquisition from calibration.Source: §5.1 · p. 6, §5 · p. 6

5.2. Stage II: identity-centered signed gates

  • Each targeted module receives an independent scalar gate that applies a bounded coefficient to its frozen delta, allowing suppression, reversal, or extrapolation.Zero gate weights and biases recover the Stage-I model; the bounded range limits intervention magnitude, not output divergence.Source: §5.2 · p. 6

5.3. Parameter efficiency

  • Stage II adds only d_m + 1 trainable gate parameters per targeted module, while the base model and LoRA adapter remain frozen.This is a small fraction of the parameters used to learn the original SFT delta.Source: §5.3 · p. 7

5.4. Entropy-only training on response prefixes

  • Stage II minimizes top-100 conditional entropy at response positions only, without CE, KL, or a budget term, using demonstration prefixes.For k below vocabulary size, the objective is conditional rather than full-vocabulary entropy.Source: §5.4 · p. 7

5.5. Policy credit assignment within the frozen-delta family

  • Under gate-subspace entropy–task consistency, differentiability, and nonzero task-value gradient, a sufficiently small complete-gradient entropy step locally increases task value.The condition requires directional consistency in accessible gate directions, not pointwise agreement between confidence and correctness.Source: §5.5 · pp. 7–8

5.6. Why allow reversal and extrapolation after SFT?

  • The analysis finds that expanding gates beyond [0, 1] can remove attenuation-boundary obstructions, but does not guarantee the optimizer reaches the nonlinear optimum.At a zero gate, a positive derivative calls for reversal; at one, a negative derivative calls for extrapolation.Source: §5.6 · p. 8
  • Freezing the learned delta preserves the meaning of its coefficients during calibration; interleaving CE would change the operators being controlled.SCALE’s distinctive intervention is token- and module-conditioned signed scaling of one completed SFT delta.Source: §5.6 · p. 8

6. Experiments

Across three backbones, SCALE leads the compared methods on mathematical reasoning and achieves the highest code-generation Avg on each backbone. The experiments also show that preferred gate ranges vary by task and backbone.

6.1. Setup

  • Experiments compare SCALE with parameter-efficient and token-level adaptation baselines across three backbones, evaluating math, general-capability retention, and code generation.The setup uses NuminaMath-CoT for math fine-tuning, Magicoder-OSS for code fine-tuning, and specified benchmark suites for evaluation.Source: §6.1 · pp. 8–9
  • The study trains separate gates for all 42 lower- and upper-bound combinations; reported maxima are oracle sweeps, while fixed ranges serve controlled comparisons.Stage II freezes the base model and LoRA adapter while training only the lightweight gate.Source: §6.1 · p. 9, Figure 3 · p. 11

6.3. Discussion

  • High-scoring math ranges favor negative lower bounds, while coding ranges cluster near L = 0 with larger positive upper bounds; preferred ranges also vary by backbone.Qwen2.5-Math models favor upper bounds above one for math, whereas Qwen3-4B-Base peaks at a smaller upper bound.Source: §6.3 · p. 9
  • At fixed U = 1, negative lower bounds improve math on all three backbones; expanding [0, 1] to [0, 6] improves code pass@1 on all three.The reported patterns support reversal for math and extrapolation for code; [0, 1] also exceeds LoRA in math.Source: §6.3 · p. 9

6.2. Main results

  • SCALE achieves the best mathematical-reasoning results on every benchmark across all three backbones, with the highest retention Avg on both Qwen2.5-Math models and second-highest on Qwen3-4B-Base.Table 1 reports mathematical-reasoning Avg@16 and general-capability retention results.Source: §6.2 · p. 9, Table 1 · p. 10
  • SCALE obtains the highest code-generation Avg on Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base.Table 2 reports code-generation performance.Source: §6.2 · p. 9, Table 2 · p. 11

7. Conclusion

The conclusion presents SCALE as calibration of a completed SFT adaptation, while the appendices clarify the gradient interpretation and a reference-token entropy threshold. The notation appendix points readers to a consolidated notation table.

7. Conclusion

  • SCALE freezes the SFT delta and learns an entropy-guided signed gate to suppress, reverse, or extrapolate it, improving math and code while retaining general capability across three backbones.The conclusion frames this as calibrating a completed adaptation rather than only changing how it is learned.Source: §7 · p. 11

A. Notation

B.1. An exact gradient identity for SFT

  • The exact gradient identity represents the weighted CE gradient; its importance-sampling factor cancels in expectation and does not make the deterministic CE logit gradient unbounded.The coefficient is stopped in the surrogate, and the representation is related to DFT’s RL analysis.Source: §B.1 · pp. 15–16

B.2. Reference-token entropy threshold

  • A reference token above exp[−H(p)] identifies entropy decrease along a particular probability-mixture path, but is not an optimal weighting rule for ordinary SFT.The resulting CE logit direction generally differs from that mixture-path direction.Source: §B.2 · p. 16

C. Token Correction Objectives

The section distinguishes one-sided weighted-CE objectives from post-hoc delta correction, then examines entropy-based selection and the limits of the implemented gate analysis.

C. Token Correction Objectives

  • Nonnegative weighted-CE components are one-sided; EKSFT's full objective and Taheri et al.'s negative-token loss fall outside that class.The EAFT row reports its published entropy weight without specifying a stop-gradient convention, and the practical TALR rule retains the nonnegative-ray conclusion.Source: C. Token Correction Objectives · p. 16, Table 4 · p. 17

D. Empirical Motivation: Delta Entropy and Local Feature Selection

  • 22.59 Math Avg@16 follows suppression of entropy-increasing contributions, versus 12.59 for entropy-decreasing suppression and 15.47 for LoRA.The frozen-model diagnostic motivates using entropy change to select weakened delta contributions and learning a continuous correction gate.Source: D. Empirical Motivation: Delta Entropy… · p. 17

E.1. Intervention coordinates versus shared gate parameters

  • Scalar gates rescale an entire module delta at a position, so they cannot independently separate its semantic components or express every advantageous intervention.The analysis assumes no globally affine base-to-SFT logit model, and a chain-rule pullback is not generally an orthogonal projection.Source: §E.1 · p. 18

E.2. Exact policy loss for the implemented conditional entropy

  • For unchanged top-k support, the implemented entropy objective is conditional entropy over retained actions, and lowering it does not by itself lower other terms.When k is less than the vocabulary size, the objective is not full-vocabulary entropy, and the conditional action value need not equal the full-vocabulary expected value.Source: §E.2 · p. 18

E.3. What detaching a gate input changes

  • Detaching gate inputs yields a semi-gradient rather than the ordinary derivative, so task-improvement guarantees require an additional error condition.Freezing parameters alone does not remove this difference; the experiments do not directly measure the terms in the sufficient condition.Source: §E.3 · pp. 18–19

F. Post-hoc Calibration and a Fixed delta Reference

The section fixes the completed delta as a reference for post-hoc gate optimization, while distinguishing that design choice from a guarantee of better optimization or performance. It also specifies the two-stage training and evaluation setup.

F. Post-hoc Calibration and a Fixed delta Reference

  • Freezing the delta removes an optimization term with no fixed sign, isolating how to use a completed delta without proving post-hoc optimization beats joint or alternating optimization.Setting every gate to one identifies the same Stage-I model, while setting every gate to zero recovers the base when other parameters are unchanged.Source: F. Post-hoc Calibration and a Fixed… · p. 19
  • Monotone loss descent still requires a genuine descent direction and suitable step size; minibatch noise, optimizer state, top-k changes, and stopped inputs require separate conditions.The post-hoc design fixes the intervention reference, not these optimization issues.Source: F. Post-hoc Calibration and a Fixed… · p. 19

G.1. More Implementation Details

  • Stage I trains separate one-epoch LoRA deltas on 100k NuminaMath-CoT examples for math and 75,197 Magicoder-OSS examples for code; Stage II calibrates rather than relearns their factors.Stage II uses 20k-example subsets and 157 optimizer steps; gates are the only trainable components.Source: §G.1 · p. 20
  • Stage II uses semi-gradient updates and normalized top-k conditional entropy, so complete-gradient task-improvement claims require an additional error condition.Evaluations use task-native zero-shot prompts for mathematical reasoning and general-capability retention, with all methods run under the same vLLM decoding configuration.Source: §G.1 · p. 20

G.2. Ablation studies

The ablations and analyses examine SCALE’s gate behavior, training cost, and theoretical conditions for entropy-guided correction. They report contributions from reversal and extrapolation while qualifying when entropy descent supports task improvement.

G.2. Ablation studies

  • 37.84 Math Avg@16 for entropy calibration exceeds 16.27 for continued LoRA and 16.17 for cross-entropy calibration.The comparison uses the same 20k examples; Table 6 reports the ablation.Source: §G.2 · pp. 20–21

G.3. Realized gate behavior

  • 39.36% and 44.34% of generated-token coefficients are below zero, while 42.82% and 36.55% exceed one on the two Qwen2.5-Math backbones.Thus, most generated-token coefficients fall outside the unit interval in both models.Source: Table 8 · p. 21, §G.3 · p. 21
  • Removing reversal or extrapolation lowers mathematical-reasoning accuracy, and restricting gates to [0, 1] produces the largest reduction for both models.Removing reversal lowers Math Avg@16 by 16.95 and 9.15 points on the 1.5B and 7B models; removing extrapolation lowers it by 6.99 and 12.46 points.Source: Table 9 · p. 21, §G.3 · p. 21

G.4. Training Time

  • 20.0–46.0 minutes for Stage-II gate training compares with 53.0–305.1 minutes for baseline runs across four settings.Full SCALE training also includes its preceding LoRA stage; the timing excludes queueing and evaluation.Source: §G.4 · p. 22, Figure 4 · p. 22

H.1. Proof of the entropy-advantage identity

  • The entropy gradient identity links entropy descent to a gate’s directional effect, while task alignment requires the delta error to have small effects on accessible directions.The aggregate shared-gate condition need not give the correct task-contribution sign for every coordinate separately.Source: §H.1 · p. 23, §H.3 · pp. 23–24

H.2. Local one-sidedness under weighted CE

  • A nonnegative weighted cross-entropy coefficient rescales one direct CE direction, but shared-network propagation does not preserve coordinatewise sign guarantees.The result does not rule out unlearning through parameter sharing or a different loss.Source: §H.2 · p. 23

H.4. Proof of local task improvement and a finite step bound

  • A sufficiently small step gives local task improvement under the stated assumptions, but the bound applies only within a differentiable neighborhood and is not unconditional for momentum or minibatch noise.A low numerical entropy alone does not certify task improvement for an implemented update.Source: §H.4 · p. 24

H.5. A margin model for helpful and harmful features

  • Entropy descent identifies helpful versus harmful contributions only with a reliable semantic reference; the signed parameterization permits delta scaling beyond attenuation, without guaranteeing improved final predictions.In the margin model, the conclusion reverses when m < 0; the expanded parameter range does not necessarily enlarge the final prediction set.Source: §H.5 · p. 25, §H.6 · p. 25

H.7. Proof of the attenuation-boundary result

The attenuation-boundary result shows that extending the gate range can lower the objective near a constrained local minimum with nonzero gradient. Its shared-gate realization requires an accessible descent direction.

H.7. Proof of the attenuation-boundary result

  • A nonzero gradient at a local minimum on [0, 1]^M guarantees an arbitrarily close point in the extended box (L, U)^M with strictly lower objective.For shared gates, realizing that improvement additionally requires a parameter direction whose mapped change has negative inner product with the objective gradient.Source: §H.7 · pp. 25–26
  • If the constrained point is a global minimum, the extended box attains a lower objective, but this does not imply endpoint optimality for nonlinear entropy.The descent argument uses a nonzero gradient to identify a blocked outward direction at zero or one.Source: §H.7 · p. 25

I. Limitation

The evaluation is limited to models up to 7B parameters and to the Qwen family, and the gate range is selected empirically. The authors identify larger-scale, cross-family, and more adaptive gate selection evaluations as future work.

I. Limitation

  • Experiments cover models up to 7B parameters; SCALE’s effectiveness and scalability on larger models, including 30B+ models, remain unvalidated.The authors describe the gated-delta mechanism as parameter-efficient and theoretically independent of model size, while leaving larger-scale effectiveness to future evaluation.Source: §I · p. 26
  • Experiments use Qwen-family models, so generality and robustness across other architectures and pretraining distributions remain to be assessed.The paper calls for evaluation beyond Qwen.Source: §I · p. 26
  • Gate ranges are selected through empirical evaluation; automatically determining the optimal intervention range remains open.The authors identify adaptive range selection as future work.Source: §I · p. 26
Loading 2609.33463v1…