PaperlayerView exact revision on arXiv

Source-linked AI summary

ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection

Tao Liu, Taiqiang Wu, Runming Yang, Shaoning Sun, Junjie Wang, Yujiu Yang

arXiv:2601.09195v3cs.CLcs.AI

TL;DR

Traditional SFT overfits single-reference surface expressions because it does not accommodate language’s one-to-many nature, while multiple references are costly and difficult to optimize. ProFit uses token probability to mask low-probability, non-core tokens and retain high-value semantic signals. Across diverse benchmarks and model families, it consistently outperforms conventional SFT, with scope limitations for creative generation and a static threshold.

  • Problem

    Traditional SFT rigidly aligns with one reference and can overfit non-core expressions, while multiple references impose expensive data construction and convergence challenges.

  • Method

    ProFit uses model-predicted token probabilities to retain high-probability semantic signals and mask low-probability tokens during SFT.

  • Results

    ProFit consistently outperforms conventional SFT and other data-selection methods across diverse reasoning, mathematics, and instruction-following benchmarks and multiple LLM families.

  • Takeaways & Limitations

    Low-probability tokens are identified as a primary source of optimization interference, while probability-guided masking offers a robust approach to improving model generalization.

  • Takeaways & Limitations

    The assumption that low-probability tokens are non-core primarily holds for logic-intensive tasks, while creative generation may use them for stylistic diversity.

Abstract

from arXiv · show

Supervised fine-tuning (SFT) is a fundamental post-training strategy to align Large Language Models (LLMs) with human intent. However, traditional SFT often ignores the one-to-many nature of language by forcing alignment with a single reference answer, leading to the model overfitting to non-core expressions. Although our empirical analysis suggests that introducing multiple reference answers can mitigate this issue, the prohibitive data and computational costs necessitate a strategic shift: prioritizing the mitigation of single-reference overfitting over the costly pursuit of answer diversity. To achieve this, we reveal the intrinsic connection between token probability and semantic importance: high-probability tokens carry the core logical framework, while low-probability tokens are mostly replaceable expressions. Based on this insight, we propose ProFit, which selectively masks low-probability tokens to prevent surface-level overfitting. Extensive experiments confirm that ProFit consistently outperforms traditional SFT baselines on general reasoning and mathematical benchmarks.

1 Introduction

Traditional SFT can overfit a single reference’s surface wording, while multiple references impose costly data and convergence burdens. ProFit instead uses token probabilities to retain core semantic signals and consistently improves over standard SFT.

  • Motivation: Traditional SFT forces token-level alignment with one reference, despite language’s one-to-many expressive nature.This can encourage memorization of specific samples and overfitting to non-core expressions.
  • Motivation: Multiple references can alleviate single-reference overfitting but require expensive data construction and can hinder training convergence.The paper therefore prioritizes avoiding overfitting rather than comprehensively covering answer diversity.
  • Key insight: High-probability tokens tend to encode core reasoning logic and key semantics, whereas low-probability tokens more often represent replaceable expressions.The paper identifies prediction probability as an efficient proxy for semantic importance.
  • Method: ProFit selectively retains and trains high-probability tokens while masking low-probability tokens in single-reference supervision.The method uses online probabilities from the model being trained and derives that low-probability gradients can overshadow crucial-token optimization.
  • Results: ProFit consistently outperforms standard SFT across general reasoning and mathematics benchmarks.The reported evaluations cover GPQA-Diamond, MATH-500, AIME’24, GSM8K, and IFEval.

2 Related Work

Prior work has progressed from sample-level data selection to token-level optimization, but existing approaches either overlook low-information segments or depend on costly external reference models. ProFit is positioned within intrinsic, probability-guided token selection.

  • Sample-level selection: Recent SFT data selection evolved from complexity-based filtering toward importance-aware and in-context-learning-based sample metrics.These methods operate at the sample level and treat each input-response pair as an atomic unit.
  • Sample-level selection: Sample-level methods can overlook intra-sample low-information segments and stylistic noise, limiting focus on dense logical signals.The related-work discussion motivates finer-grained token-level optimization.
  • Token-level optimization: Classical token-level methods target hard tokens, while recent LLM methods include external-reference approaches such as Rho-1 and TIS-DPO.The paper contrasts these methods with more intrinsic probability-driven strategies.
  • Token-level optimization: DFT uses probability-driven soft reweighting, and CFT supports supervising critical regions without the same coarse sample-level granularity.Prior theory that suppressing low-probability tokens benefits strong-prior domains such as mathematics supports ProFit’s masking strategy.

3 Preliminaries and Motivation

Single-reference SFT penalizes valid paraphrases, while multi-reference training introduces annotation costs and optimization conflicts. ProFit addresses this dilemma by selectively supervising high-value signals from one reference.

  • Supervised fine-tuning: SFT optimizes a policy on input-reference pairs using an autoregressive negative-log-likelihood objective.The reference response is represented as a token sequence, with per-token probabilities derived from logits through softmax.
  • Motivation: Strict single-reference alignment overpenalizes valid semantic equivalents and can drive surface-form overfitting.The issue arises because paraphrastic variants are treated as incorrect token predictions.
  • Multi-reference limitations: Collecting multiple references scales annotation burden with the number of references and often requires expert annotators for complex reasoning or mathematics.This makes answer diversity costly to construct at high quality.
  • Multi-reference limitations: 34.1% to 33.5%: GPQA-Diamond performance decreased when moving from single-answer to three-answer SFT.The passage reports marginal gains on some tasks but inconsistent generalization and convergence difficulties for multi-reference training.
  • Selective alignment: ProFit replaces indiscriminate full-token fitting and expensive multi-reference data with selective alignment of core reasoning signals from one reference.The proposed strategy aims to achieve robust performance across diverse benchmarks while preserving the low-cost data configuration.

4 Methodology

ProFit identifies token probability as a proxy for semantic importance and uses threshold-based masking to train on high-value tokens while suppressing low-probability expressions. The method is supported by semantic-distribution analysis and a gradient-based argument.

  • Semantic Analysis: Core tokens concentrate in high-probability regions, whereas trivial tokens dominate the low-probability tail.The reported hypothesis test finds a significant distributional difference with p = 1 × 10−6.
  • ProFit: ProFit uses model prediction probabilities to retain high-probability tokens and mask low-probability, non-essential tokens during training.The method applies a threshold-based masking operator to isolate core semantic signals.
  • ProFit: The binary mask is computed from detached probabilities using a static threshold τ and an indicator function.Stop-gradient makes the mask a fixed gate during backpropagation and avoids step-function differentiability issues.
  • ProFit: ProFit defines its objective by applying the token mask to the sequence-level log-likelihood over T target tokens.Here, T = |y∗| denotes sequence length.
  • Gradient Analysis: Under a local non-degeneracy assumption, lower-probability tokens induce larger parameter-gradient lower bounds.The theorem assumes the logit Jacobian is full row-rank with σmin(Jθ(z)) ≥ γ > 0.

5 Experiments

Experiments evaluate ProFit across multiple model families and reasoning, mathematics, and instruction-following benchmarks. ProFit consistently exceeds standard SFT and other selective fine-tuning baselines, including substantial average-accuracy gains on Qwen3.

  • Experimental Setup: Evaluations cover Qwen3, Llama 3, and OLMo 2 models across GPQA-Diamond, MATH-500, GSM8K, AIME’24, and IFEval.The study uses 2,000 curated BAAI-InfinityInstruct samples and OpenCompass-based evaluation.
  • Threshold Ablation: Training only on low-probability tokens performs below the standard SFT baseline, while masking them consistently outperforms the baseline across tasks.The ablation contrasts low-probability-only training with the proposed high-probability-token strategy.
  • Main Results: ProFit consistently achieves superior performance across evaluated model families and scales.Table 1 compares ProFit with Vanilla, standard SFT, Entropy, and DFT across five benchmarks.
  • Main Results: 10.94%: ProFit reaches 52.33% average accuracy versus standard SFT’s 41.39% on Qwen3-4B-Base.On Qwen3-14B-Base, ProFit also reverses standard SFT’s 1.88% decline relative to Vanilla into a +5.64% gain.
  • Main Results: 31.49%: ProFit records the highest average accuracy on Qwen3-0.6B, exceeding DFT at 30.20% and Entropy at 28.86%.The same advantage over entropy-based filtering and dynamic reweighting appears across OLMo-2 and Llama-3.1.
  • Main Results: ProFit boosts performance across model sizes and architectures without the high data costs of multi-reference fine-tuning.This conclusion describes the method’s reported scalability and universality.

6 Extensive Analysis

Extensive analyses examine thresholding, training dynamics, LoRA capacity, reinforcement-learning initialization, and robustness across model settings. ProFit generally converges faster, achieves stronger benchmark performance, and reduces optimization interference relative to fitting low-probability or all tokens.

  • LoRA rank: Core-token training improves monotonically with LoRA rank, whereas non-core-token training and standard SFT show U-shaped trends caused by optimization interference.At rank 1024, LoRA regularization can outperform full fine-tuning for trivial tokens by reducing overfitting to non-essential statements.
  • Training trajectories: 60.1% accuracy in the first epoch lets ProFit surpass the Baseline’s peak performance of 54.9%, while low-probability-token training stagnates at 40%–50%.The low-probability setting also exhibits training instability over five epochs.
  • Reinforcement-learning initialization: Across MATH-500, OlympiadBench, and Minerva, ProFit provides a superior reinforcement-learning initialization with more robust training stability.On MATH-500, ProFit reaches 57.3% Avg@4 and 76.4% Pass@4, versus 53.1% and 70.6% for the Baseline.
  • Preserving aligned-model capabilities: ProFit mitigates alignment tax on Qwen2.5-Instruct, improving average scores by +1.24 on 0.5B and +0.17 on 7B models.Standard SFT shows a -1.83 average degradation on the 7B model.

7 Conclusion

The conclusion presents ProFit as probability-guided masking that targets non-core expressions while preserving core logic. Experiments and analyses report consistent gains, stable reinforcement-learning initialization, and reduced optimization interference across diverse settings.

  • Conclusion: ProFit uses token prediction probabilities as a proxy for semantic importance and selectively masks low-probability tokens.The method aims to reduce overfitting to surface-level phrasing and capture underlying logic.
  • Conclusion: Across Qwen, Llama, and OLMo2 models, ProFit consistently outperforms conventional full-parameter SFT and other data-selection methods on diverse benchmarks.The evaluations cover general reasoning, mathematics, and instruction following.
  • Conclusion: ProFit also serves as a superior reinforcement-learning initialization, promoting stable convergence and deeper reasoning exploration.The conclusion attributes optimization interference primarily to low-probability tokens.

Limitation

The paper identifies two principal limitations: its low-probability-token assumption is task-dependent, and its threshold is static across samples. The supplied proof passages additionally state the gradient setup and a non-degeneracy assumption.

  • Scope boundary: The assumption that low-probability tokens are non-core expressions primarily holds for logic-intensive tasks such as reasoning and mathematics.Creative-generation settings may use such tokens for stylistic diversity.
  • Thresholding: ProFit uses a static probability threshold across all samples, leaving instance-specific adaptive thresholding for future work.The authors associate this design with implementation simplicity and training stability.
  • Theoretical analysis: The proof assumes a non-degenerate Jacobian with σ_min(J_θ(z)) ≥ γ when relating logit gradients to parameter gradients.The derivation applies a spectral inequality to the Jacobian-based chain rule.

C Detailed Analysis of LoRA Rank across Datasets

Across GSM8K, MATH-500, GPQA, and IFEval, core-token training scales positively with LoRA rank, while non-core-token training and standard SFT exhibit instability or overfitting.

  • Universal Monotonicity for Core Tokens: From rank 4 to 1024, core-token performance grows monotonically across reasoning, knowledge-recall, and instruction-following datasets.The authors interpret this as evidence that core task semantics have high intrinsic dimensionality.
  • Interference from Non-Core Expressions: At high ranks such as r = 1024, non-core-token training sharply degrades performance on GSM8K and MATH-500 through overfitting.The excess capacity amplifies sensitivity to non-core expression noise in reasoning tasks.
  • Interference from Non-Core Expressions: On GPQA and IFEval, standard SFT and non-core-token training fluctuate or stagnate at medium-to-high ranks instead of maintaining core-token scaling.The reported pattern supports filtering non-core expressions for stable scaling.

D Detailed Analysis of Training Dynamics across Epochs

Across benchmarks, ProFit converges rapidly and remains stable, while low-probability-token training can degrade performance through overfitting to non-core expressions. The contrast is especially pronounced in reasoning and instruction-following tasks.

  • Rapid Convergence and Stability: ProFit settings typically reach near-optimal performance within 2 epochs and remain stable across benchmarks.The SFT baseline often requires more steps to plateau or exhibits fluctuations.
  • Overfitting to Non-Core Expressions: In GSM8K and MATH-500, p < 0.1 training peaks early but degrades during Epochs 3-5.The inverted-U pattern is attributed to overfitting to non-core expressions, impairing underlying reasoning logic.
  • The Gap in Instruction Following: In IFEval, high-probability-token training stabilizes around 50% accuracy, while low-probability-token training stagnates below 30%.The passage links this gap to core instruction-following semantics being encoded primarily in high-probability tokens.
  • Logical Coherence: Table 4 shows SFT producing an incorrect polynomial-factorization solution by hallucinating constraints, whereas ProFit derives the correct ordered triple through coherent coefficient comparison.The comparison illustrates how masking non-core expressions safeguards the reasoning chain against superficial errors.

G Training Dynamics Analysis in RL Stage

In the RL stage, ProFit is evaluated against Base and DFT using KL divergence, entropy, and response length. ProFit maintains stable, confident training while developing the longest reasoning responses, and its output comparison preserves a coherent solution chain.

  • KL Divergence Stability: ProFit and DFT maintain low, stable KL divergence, whereas the Base model rises to approximately 0.17 by the end of training.The Base-model rise is associated with policy drift from the reference model and potential reward hacking or language degeneration.
  • Entropy: ProFit ends with entropy around 0.09, indicating confident reasoning paths while retaining a slightly higher entropy margin than DFT’s saturation near 0.03.The passage contrasts this with the Base model’s collapse and rebound above 6.0.
  • Response Length: ProFit’s average response length grows from approximately 1,400 tokens to over 8,000 tokens, eventually exceeding the Base model around step 150.Response length is used as a proxy for the depth of Chain-of-Thought reasoning.
  • Output Comparison: On polynomial factorization, SFT hallucinates independent constraints and reaches an incorrect solution, while ProFit derives the correct ordered triple through coefficient comparison.The table compares model outputs rather than training curves.
Loading 2601.09195v3…