Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning

• Exact probability matching penalizes a student for strengthening a reference preference, while preserving ordering alone can let the gap between tokens approach zero.Source: §1 · pp. 1–2
• SRP-SFT applies one-sided hinge penalties to selected reference-scaled non-target margins, allowing further separation once a threshold is met while CE fits the observed target.Source: §1 · p. 2
• Across three matched seeds, SRP-SFT improves reasoning and code-generation scores over CE and LP-SFT, with less margin contraction and fewer ranking inversions than LP-SFT.Source: §1 · p. 2
• The paper analyzes KL costs, feasible geometry, conditional utility, and finite-step conditions, alongside ranking controls, diagnostics, sensitivity analyses, and additional backbones.Source: §1 · p. 2

• Prior SFT auxiliaries include output matching, parameter anchoring (Kirkpatrick et al., 2017; Li et al., 2018), and target reweighting (Wu et al., 2026; Diao et al., 2026).Source: §2 · p. 2
• Relational supervision transfers information through candidate rankings (Peng and Zhang, 2025), prediction correlations (Huang et al., 2022), or logit differences (Hofstätter et al., 2020; Li et al., 2025a).Source: §2 · p. 2
• Unlike exact distribution matching, relational supervision need not reproduce teacher probabilities; SRP-SFT uses one-sided lower bounds because ordering alone permits shrinking gaps and exact gap matching penalizes expansion.Source: §2 · p. 2

• At identical teacher-forced prefixes, student and reference logits can be compared without changes in generated history confounding local preference changes.Source: §3 · p. 3
• A margin’s sign gives token ordering and its magnitude gives log-odds strength; a reference-preferred pair becomes an inversion, contraction, or expansion depending on the student margin.A negative student margin is an inversion, a smaller nonnegative margin is a contraction, and a larger margin is an expansion.Source: §3 · p. 3
• The scale ρ sets individual selected-margin bounds: ρ = 0 requests weak order, 0 < ρ < 1 permits bounded contraction, ρ = 1 requests the reference margin, and ρ > 1 requests sharpening.The trained soft penalty need not satisfy every bound.Source: §3 · p. 3
• In free-logit coordinates, CE contracts a non-target margin toward zero without finite-time inversion, while shared parameters can produce contraction or expansion.Source: §3 · p. 3

4 Method: Selective Margin Constraints

• The method applies its auxiliary to selected non-target relations while CE supervises the observed token at the same fixed prefix.Source: §4 · p. 3

4.1 Selecting Non-Target Alternatives

• At each valid supervised position, the reference’s top-K non-target alternatives define plausible candidates; increasing K broadens coverage and raises the maximum constraint count quadratically.Source: §4.1 · p. 3

4.2 Orienting Selected Pairs

• The frozen reference orients each selected pair, ignores teacher ties, and fixes candidate identities and orientations per prefix; neither outside relations nor total candidate-set mass receive direct supervision.Source: §4.2 · p. 3

4.3 Penalizing Only Margin Shortfalls

• A pair receives hinge penalty only below its requested bound; at ρ = 1, contraction and inversion are penalized but further expansion is not, and finite µ can leave residual violations.The zero-loss region describes the target, not an invariant set for training updates.Source: §4.3 · pp. 3–4, Figure 2 · p. 4

4.4 Gradients and Cached Implementation

• Active-pair steps raise preferred logits, but other pairs or shared parameters can reactivate satisfied hinges; cached reference IDs and logits let updates recompute hinges without another reference pass.Source: §4.4 · p. 4

5.1 The KL Cost of Losing a Margin

• For a teacher-preferred pair, the minimum KL cost of reducing its margin depends on the pair’s local probability mass and gap, unlike SRP’s hinge cost.Scaled student logits satisfy every SRP constraint when the scale is at least max{1, ρ}, even though local KL can diverge as the scale grows.Source: §5.1 · p. 5

5.2 Admissible Geometry and Conditional Utility

• The constraints form an order-cone feasible region; at zero violation and ρ ≥ 1, teacher-monotone conditional utility is not lowered relative to the reference.The guarantee assumes teacher–utility alignment and concerns choices within the selected set, not its probability mass, the excluded target, or sequence-level correctness.Source: §5.2 · p. 5

5.3 When a Soft Penalty Repairs Violations

• The finite-step analysis assumes Lipschitz margin gradients along the update segment and uses the actual fixed reduction weights.The bound also applies to optimizer steps such as AdamW.Source: §5.3 · p. 5
• A finite-step bound can certify a strict decrease in aggregate violation when its right-hand side is negative, accounting for CE interference, hinge crossings, and curvature.The result requires neither convexity nor a frozen neural tangent kernel, but it certifies aggregate—not pairwise—repair.Source: §5.3 · p. 5

6 Experimental Setup

• Fine-tuning uses Qwen3-4B-Base on UltraFeedback’s train_sft split for one epoch, with shared initialization, per-seed data order, and optimizer across three seeds.The protocol uses 478 AdamW steps, a 2 × 10−5 learning rate, 15 warmup steps, and effective batch size 128.Source: §6 · p. 5
• The matched comparison includes CE, GEM, DFT, ASFT, EAFT, LP-SFT, and SRP-SFT; SRP-SFT uses top-10 non-target tokens and fixed coefficients across all three seeds.LP-SFT matches the reference distribution over top-10 non-target tokens; SRP-SFT adds a hinge on strictly ordered pairs.Source: §6 · p. 5
• Reasoning results average twelve metrics from one checkpoint, while the code mean averages HumanEval+ and MBPP+ pass@1/pass@4.Held-out margin diagnostics use the 998-example test_sft cache.Source: §6 · p. 6

7.1 Task Utility of Margin Constraints

• 69.48 ± 0.02 code mean and 44.96 ± 0.55 twelve-metric reasoning mean exceed LP-SFT's 66.17 ± 0.27 and 43.10±0.46 across three seeds.At seed 1234, MATH pass@1 rises from 49.56 for LP-SFT to 54.56, and code mean rises from 65.91 to 69.50.Source: Table 3 · p. 6, §7.1 · p. 6

7.2 Comparing Preservation Targets

• Across 14 losses, task mean falls as rank flips and contraction rise (Spearman −0.87 and −0.84); SRP-SFT leads both code columns, while Margin-MSE scores 54.82.RankOnly scores 49.07; the ρ = 1 setting cuts contraction to 0.11 and gives the best mean, 57.88.Source: §7.2 · p. 6, Table 4 · p. 7

7.3 Distributional Fidelity versus Margin Preservation

• On the 998-example cache, SRP-SFT cuts contraction from 0.268 to 0.110 and inversions from 7.41% to 5.65% versus LP-SFT, despite higher local KL (0.075 versus 0.049).The lower local KL does not identify the checkpoint with less contraction and fewer inversions.Source: Figure 3 · p. 7, §7.3 · p. 7

7.4 Preservation Strength versus Task Utility

• ρ = 2.5 scores 61.56 versus 57.05 at ρ = 1, while Flip rises from 0.0565 to 0.0937 and contraction from 0.1098 to 0.1891; one-factor peaks are µ = 3 and K = 5.The score rises from 47.05 at ρ = 0 to 61.56 at ρ = 2.5, then declines; increasing µ beyond 3 lowers the score while reducing Flip.Source: Figure 4 · p. 7, §7.4 · p. 7

7.5 Can Global Scaling Recover the Gains?

• Entropy matching raises LP-SFT MATH pass@1 from 49.56 to 51.96 and code mean from 65.91 to 67.53, closing 48% and 45% of the gaps to SRP-SFT.The fitted scalar sharpens the output distribution but leaves each token's candidate ranking unchanged.Source: §7.5 · pp. 7–8

7.6 Transfer Across Backbones

• SRP-SFT leads LP-SFT in 18 of 24 backbone–benchmark pairs, including 11 of 12 code pairs; HumanEval+ rises on every backbone by 0.55 to 4.09.Mathematics is more backbone-dependent: SRP-SFT leads MATH-500 on both Qwen bases, Ministral, and SmolLM3, but trails on Gemma and Granite.Source: Figure 5 · p. 8, §7.6 · p. 8

7.7 Convergence Behavior

• With ρ = 0.5, the ten-epoch run declines early, then keeps MATH near Base while maintaining later code gains; it does not lead every early checkpoint.The one-epoch headline uses ρ = 1, and the final MATH and code measurements occur at epochs 10.0 and 9.5, respectively.Source: Figure 6 · p. 8, §7.7 · p. 8

8 Conclusion

• SRP-SFT uses a soft hinge on selected non-target margins at fixed teacher-forced prefixes.The method is presented as a margin target rather than full distribution matching.Source: §8 · p. 8
• Matched-seed comparisons show reasoning and code score improvements over CE and LP-SFT.The evaluation suite includes loss screens, sensitivity sweeps, entropy matching, six additional backbones, and ten-epoch training.Source: §8 · p. 8
• The evaluations map where the margin target improves scores across training and model settings.Source: §8 · p. 8

Limitations

• SRP-SFT was tested only with chat-formatted models that already had usable downstream ability, and all trained models were at most 4B parameters.When a base release lacked a chat template, the study used its instruction-tuned release; no larger model was trained, and reported math and code scores use a one-epoch schedule at this scale.Source: Limitations · p. 9

A Additional Background and Method Details

• SRP-SFT permits nonconstant logit changes within its zero-loss feasible region, while cached reference logits let training add pairwise terms without another reference forward pass.Beyond the scaled margin boundary, expansion is unpenalized unless it is also teacher-relative expansion with ρ > 1.Source: A Additional Background and Method… · p. 12

B Proofs and Theoretical Extensions

• The appendix characterizes the KL cost of margin violations, but small selected-pair mass can make inversion cost approach zero, while SRP can remain zero as KL diverges.At ρ = 1, the unweighted SRP pair term equals δ, independent of pair mass; its position-level contribution is δ/|P|.Source: §B.1 · p. 13

B.2 Order-Cone Geometry and Conditional Utility

• The order-cone analysis bounds feasible logit distance by V/2 and gives utility guarantees only for teacher-ordered utilities; arbitrary utilities lack a uniform guarantee.The appendix also bounds softmax total-variation change by tanh(D/4) for logit-change range D, with the bound attained in a two-probability case.Source: §B.2 · p. 14

B.3 Full and Adjacent Pair Graphs

• Adjacent-pair and full-pair penalties share a zero set, but their mean values can differ; teacher ties require edges across distinct teacher levels or paths for every strict comparison.The examples show neither penalty mean uniformly dominates the other.Source: §B.3 · p. 15

B.4 Finite-Step Repair with Shared Parameters

• A sufficient finite-step condition certifies aggregate repair, but active pairs can cancel in shared parameter space.In one-prefix free-logit coordinates, nonempty teacher-oriented active sets cannot cancel completely.Source: §B.4 · pp. 15–16

B.5 From an Objective Budget to Sorting Risk

• The objective-budget sorting-risk bound is conditional on a fixed weighted training collection; for ρ > 1, matched initialization incurs an explicit sharpening debt.With matched initialization, 0 < ρ ≤ 1, and CE f ≥ 0, the stated risk rate is conditionally O(μ^-β/(β+1)) under the specified margin-distribution assumptions.Source: §B.5 · p. 16

B.6 Proof of Proposition 6

• In free-logit coordinates, CE dynamics preserve the sign of nonzero non-target margins at finite time, and discrete logit-gradient steps with 0 < η ≤ 2 decrease their magnitude.An initial tie remains a tie, while an initially strict order keeps its sign.Source: §B.6 · pp. 16–17
• For an unscaled, unit-temperature free-logit step, 0 < η ≤ 2 guarantees sign preservation and strict margin-magnitude reduction.The bound is specific to that objective and setting.Source: §B.6 · p. 17

B.7 Proof of Proposition 7

• In parameter space, margin effects can have either sign: same-state CE can expand a non-target margin, and auxiliary-pair interactions may be negative.A sufficient condition for contraction is Iu→a > ηLa∥g∥^2/2.Source: §B.7 · p. 17

B.8 Proof of Lemma 8

• For an already positive student margin dS_a, the exact ℓ∞ distance to a tie is dS_a/2; parameter-space preservation additionally requires a local Lipschitz bound.The parameter-space condition applies only if the perturbation remains in the stated neighborhood.Source: §B.8 · p. 18

B.9 Proof of Corollary 9

• The average-violation-to-reversal bound requires ρ > 0 and ργ > 2ϵ; it counts ties as failures, so it also bounds strict rank inversions.The bound applies under the perturbation conditions of Lemma 8.Source: §B.9 · p. 18

B.10 Weighted Penalties: Exactness and Approximate Feasibility

• Weighted hinge penalties recover the constrained optimum only under the stated conditional exactness assumptions, including a feasible solution and finite multipliers.Under those assumptions, global minimizers of the penalty objective are exactly the global constrained solutions.Source: §B.10 · p. 18

B.11 State Coverage, Normalization, and Independent-Holdout Certificates

• Transfer and holdout certificates depend on state–pair coverage and an independent i.i.d. holdout with the student, reference, and thresholds fixed in advance.The resulting upper bound applies to admissible logit perturbations and can be clipped at one.Source: §B.11 · p. 19

B.12 Finite-Precision References and Ties

• Finite-precision guarantees use stored reference logits; exact ties are excluded, and candidate membership can change near top-K ties.Relating stored logits to a higher-precision reference requires an additional numerical-error bound.Source: §B.12 · p. 19

C SRP-SFT Training Algorithm

• A controlled eight-step audit found exact CE–K=1 agreement in losses, gradient norms, and parameter hashes under its specified deterministic configuration.The audit matched initialization, data order, RNG state, optimizer, and BF16 precision across eight ranks.Source: C SRP-SFT Training Algorithm · p. 20

D.4 Loss Configurations

• The regularizer penalizes only margins below the reference-scaled target; it does not penalize further expansion, and for ρ > 1 even margins above the teacher can violate the bound.Reference logits are generated once with bfloat16 inference and cached as float16 values for training.Source: §D.4 · p. 21

D.5 Evaluation Protocol

• MATH-500 uses temperature 0.6, top-p 1.0 and 16 samples per problem; HumanEval+ and MBPP+ use temperature 0.8, top-p 1 and 20 samples per task.EvalPlus samples count as correct only if both base and additional tests pass. AIME uses 32 samples per problem; GSM-Plus uses five-shot prompting.Source: §D.5 · pp. 21–22, §D.7 · p. 23
• The broader reasoning screen includes LogiQA2, BBH-5, ARC-Challenge, HellaSwag, MMLU, and a text-only physics subset.The BBH screen uses the five-object split, and Phy-Oly uses the specified English competition-physics split.Source: §D.5 · p. 22

D.6 Software and Hardware

• Training comparisons use Python 3.12 and PyTorch 2.6.0 on eight A800-80GB GPUs; evaluation uses PyTorch 2.9.0 and vLLM.The evaluation also uses lm-evaluation-harness for likelihood and broad-reasoning tasks.Source: §D.6 · p. 23

D.7 Reporting Scope

• Matched EvalPlus results cover Base, CE, LP-SFT, and SRP-SFT, with 20 samples per task and shared data, generation, and evaluation settings.Table 6 adds pass@16 to the code metrics reported in Table 2.Source: §D.7 · p. 23, Table 6 · p. 23

D.8 Equal Benchmark Weighting

• The robustness check equally weights nine benchmarks after averaging metrics within MATH-500, AIME, and GSM-Plus.Both this aggregate and the headline reasoning mean require all twelve metrics.Source: §D.8 · p. 23, Table 7 · p. 23

E Independent-Seed Robustness Checks

• At seed 42, SRP-SFT scores 59.50/81.60 on MATH pass@1/pass@4 and 69.47 on code mean, versus LP-SFT’s 49.45/77.00 and 66.14.The seed-42 and seed-3407 results contribute to the independent-seed comparison.Source: E Independent-Seed Robustness Checks · p. 24, Table 10 · p. 25
• Across seeds 1234, 42, and 3407, SRP-SFT has higher MATH pass@1, reasoning mean, and code mean than CE and LP-SFT.Its three-seed code mean is 69.48 ± 0.02, and its reasoning mean is 44.96 ± 0.55.Source: Table 8 · p. 23, E Independent-Seed Robustness Checks · p. 24

F Cross-Backbone High-Quality Robustness

• The six-backbone robustness study uses separate mathematics and code corpora, with domain-specific checkpoints and a one-epoch protocol.The study uses OpenMathReasoning for mathematics and rStar-Coder for code; its 4,096-token sequences distinguish it from the main 2,048-token screen.Source: F Cross-Backbone High-Quality Robustness · p. 24

F.1 Reproduction Protocol

• For the cross-backbone runs, CE, LP-SFT, and SRP-SFT share each backbone/domain cache and fixed coefficients; evaluation uses separate final checkpoints for mathematics and code.The protocol records filtered training subsets, cache details, optimization settings, and evaluation manifests.Source: §F.1 · pp. 24–26

F.2 Cross-Backbone Results

• On instruction-tuned backbones, SRP-SFT leads on Ministral, CE on SmolLM3, LP-SFT on Granite, and the untouched Base on Gemma.These comparisons use all pre-specified arms for each backbone.Source: §F.2 · p. 27
• For Qwen3.5-2B, SRP-SFT exceeds Base on both MATH-500 metrics but not GSM8K; on Granite, it leads GSM8K pass@1 and MATH-500 pass@16, while LP-SFT leads MATH-500 pass@1.The reported comparisons keep the four metrics visible rather than collapsing them into one model ranking.Source: §F.2 · p. 27, Table 16 · p. 27
• Reasoning results also differ across Ministral, SmolLM3, and Gemma: the leading method changes between GSM8K and MATH-500 metrics.For example, the untouched Base leads Gemma on GSM8K and MATH-500 pass@1, while LP-SFT leads MATH-500 pass@16.Source: §F.2 · p. 27, Table 17 · p. 27
• Code comparisons are task-dependent: SRP-SFT leads HumanEval+ on Ministral but the Base leads MBPP+, while several other backbones show mixed or near-Base outcomes.Granite’s SRP-SFT HumanEval+ score is 76.22 versus 76.25 for Base; its MBPP+ score remains lower.Source: §F.2 · pp. 27–28

G Parameter Sensitivity and Ablation

• The ρ = 2.5 setting scores 61.56 overall, while ρ = 1.5 scores 60.89 with lower contraction; neither performance nor preservation changes monotonically with ρ.Relative to ρ = 1, ρ = 2.5 raises Avg. from 57.05 to 61.56 while Flip rises from 0.0565 to 0.0937 and contraction from 0.1098 to 0.1891.Source: G Parameter Sensitivity and Ablation · pp. 28–29, Figure 7 · p. 29, Table 21 · p. 29

G.1 Headline Pair-Graph Controls

• Adjacent-pair training gives higher MATH scores and slightly lower code mean than full-pair training, with lower Flip and KL but higher contraction.The comparison changes the pair set while holding the common training implementation, environment, and hyperparameters fixed.Source: Table 23 · p. 29, §G.1 · p. 29

G.2 LP-SFT/ListNet Equivalence and Training Audit

• LP-SFT and ListNet have matching gradients under matched settings, and a controlled eight-update training audit verifies implementation identity.Across the audit, parameter hashes match after every update, and the maximum residual after subtracting teacher entropy is 7.15 × 10^-7.Source: §G.2 · p. 30, Table 25 · p. 30

G.3 Ranking-Loss Form Ablation

• SRP-full has the strongest three-task mean in the ranking-loss screen, while Margin-MSE records fewer inversions than SRP-SFT.Margin-MSE has 0.0380 inversions versus 0.0565 for SRP-SFT.Source: §G.3 · p. 30, Table 26 · p. 31, Table 27 · p. 31

H Full Ranking-Loss Ablation

• The screen compares pointwise CE, pairwise relation losses, and listwise objectives; SRP’s hinge becomes inactive once its requested margin bound is met.At matched settings, LP-SFT and ListNet have the same student gradient and distribution-matching geometry; RankOnly enforces order without a teacher-margin target.Source: H Full Ranking-Loss Ablation · pp. 31–32
• Margin-MSE has the lowest displayed Flip rate, while SRP-SFT (ρ = 1) has the highest three-task mean; Table 28 reports relation and directional diagnostics.The appendix notes that performance is reported separately in Table 4.Source: H Full Ranking-Loss Ablation · p. 32, Table 28 · p. 32

I.1 Sharpness and the Fitted Scale

• The downstream entropy-matched control fits an interior LP scale of α∗ = 1.0993 and an interior CE scale of α∗ = 1.081 against SRP-SFT entropy.The logits are multiplied by the fitted scale before sampling; the geometry panel instead uses α∗ = 1.116.Source: §I.1 · p. 32, §I.2 · p. 32

I.2 Margin Geometry

• For teacher-preferred pairs with reference margin above 0.1, LP-TempMatch scaling changes the mean margin ratio from 0.934 to 1.043, while SRP-SFT’s mean ratio is 1.244.The geometry panel uses α∗ = 1.116; SRP-SFT has a below-margin rate of 0.229, compared with 0.447 for LP-TempMatch and 0.539 for unscaled LP-SFT.Source: §I.2 · pp. 32–33, Table 30 · p. 33

I.3 Violation-Conditioned Diagnostics

• In the highest CE-violation bin, SRP-SFT’s full-vocabulary entropy is 0.165 below CE, while LP-SFT’s is 0.016 below CE.The diagnostics group positions by the CE checkpoint’s violation rate and omit 22 zero-violation positions.Source: §I.3 · p. 33, Table 31 · p. 33

J Convergence Data

• The convergence trajectory uses a separate MATH-500 evaluation draw: pass@1/pass@4 are 48.43/72.06 here (mean 60.24), versus 48.11/71.23 in the main table (mean 59.67).The trajectory’s code mean is 59.17 under its stated sampling protocol, while Table 2 averages to 58.92 with different context lengths.Source: J Convergence Data · p. 33