PaperlayerView exact revision on arXiv

Source-linked AI summary

Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning

Wenhao Yu, Shaohang Wei, Jiahong Liu, Yifan Li, Minda Hu, Aiwei Liu, Hao Zhang, Irwin King

arXiv:2602.01745v2cs.LGcs.AI

TL;DR

One-dimensional token weighting can confuse intrinsic uncertainty with downstream misalignment, especially for noisy or replaceable tokens. RankTuner calibrates these signals through relative ranks and uses the resulting scale for supervised fine-tuning, with consistent gains over probability-only and entropy-only baselines across the reported reasoning evaluations. Its strongest evidence is currently in supervised fine-tuning, while broader RL-style and multimodal applications remain future work.

  • Problem

    Probability-only and entropy-only weighting treat downstream alignment and intrinsic uncertainty in isolation, which can misidentify noisy or replaceable tokens as learning-critical.

  • Method

    RankTuner compares the ground-truth token rank with its expected rank and uses the inverse Relative Rank Indicator as a token-wise Relative Scale for fine-tuning.

  • Results

    Across multiple backbones and reasoning benchmarks, RANKTUNER achieves consistent gains over probability- or entropy-only reweighting baselines, with ablations highlighting complementary roles of both components.

  • Takeaways & Limitations

    Probability–entropy calibration focuses updates on under-learned critical tokens while down-weighting noisy or replaceable positions without collapsing diversity.

  • Takeaways & Limitations

    Empirical evidence is strongest in supervised fine-tuning; broader RL-style post-training and multimodal fine-tuning are outside the paper’s core claims.

Abstract

from arXiv · show

Token-level reweighting is a simple yet effective mechanism for controlling supervised fine-tuning, but common indicators are largely one-dimensional: the ground-truth probability reflects downstream alignment, while token entropy reflects intrinsic uncertainty induced by the pre-training prior. Ignoring entropy can misidentify noisy or easily replaceable tokens as learning-critical, while ignoring probability fails to reflect target-specific alignment. RankTuner introduces a probability--entropy calibration signal, the Relative Rank Indicator, which compares the rank of the ground-truth token with its expected rank under the prediction distribution. The inverse indicator is used as a token-wise Relative Scale to reweight the fine-tuning objective, focusing updates on truly under-learned tokens without over-penalizing intrinsically uncertain positions. Experiments on multiple backbones show consistent improvements on mathematical reasoning benchmarks, transfer gains on out-of-distribution reasoning, and pre code generation performance over probability-only or entropy-only reweighting baselines.

1. Introduction

Existing token-reweighting methods emphasize either ground-truth probability or entropy, but RankTuner combines both through a rank-based, uncertainty-aware signal. The paper validates this approach across models and reasoning benchmarks.

  • Prior methods use either ground-truth probability or predictive entropy as their primary token-reweighting statistic.
  • One-dimensional weighting can over-emphasize noisy or replaceable tokens because probability and entropy capture different aspects of token prediction.
  • RankTuner compares the ground-truth token’s rank with its expected rank to produce uncertainty-aware adaptive token reweighting.
  • RankTuner is evaluated across base models and reasoning benchmarks, with ablations examining complementary probability- and entropy-aware components.

2. Preliminaries

The paper formulates fine-tuning as weighted token-level negative log-likelihood and introduces probability, entropy, and guessing-based concepts for characterizing token importance. Standard supervised fine-tuning uses uniform token weights despite differing token difficulty and information density.

  • A language model assigns each vocabulary token a conditional probability, with p_t denoting the probability of the ground-truth token at step t.
  • Weighted fine-tuning objectives use w_t to determine each token’s relative importance during optimization.
  • Standard supervised fine-tuning assigns every target token the uniform weight w_t = 1.
  • Entropy H_t measures predictive uncertainty: high entropy indicates a flat distribution, whereas low entropy indicates a sharp distribution.
  • The guessing problem orders candidates by decreasing probability and uses expected guesses as an intuitive measure of uncertainty.

3. Token Reweighting Paradigms: A Joint View of Probability and Entropy

The paper contrasts probability-dominant and entropy-dominant weighting, arguing that their signals capture orthogonal aspects of token prediction. It therefore motivates joint calibration that suppresses noise and replaceable ambiguity while emphasizing critical low-entropy errors.

  • Prob-Dominant Weighting: Prob-Dominant weighting uses ground-truth probability as a token-level signal for downstream task alignment.
  • Entropy-Dominant Weighting: Entropy-Dominant weighting increases the fine-tuning signal for tokens with high predictive uncertainty.
  • Deeper Insights into the Different Finetuning Paradigms: Probability and entropy capture orthogonal aspects of generation, so meaningful fine-tuning signals should depend on both dimensions jointly.
  • Deeper Insights into the Different Finetuning Paradigms: Entropy-Dominant methods may up-weight noisy filler tokens such as “umm,” amplifying uninformative gradients.
  • Deeper Insights into the Different Finetuning Paradigms: Prob-Dominant methods may over-penalize low-probability but replaceable tokens such as “essentially,” risking distortion of linguistic flexibility.
  • Deeper Insights into the Different Finetuning Paradigms: A joint scheme should down-weight high-entropy replaceable tokens while emphasizing low-entropy positions where mistakes are critical.

4. Methodology: Bridging the Gap via Rank-Based Discretization

RankTuner makes probability and entropy comparable by discretizing them through the ground-truth rank and expected rank, then uses their relative discrepancy to construct token-wise fine-tuning weights.

  • Relative Rank Indicator: The Relative Rank Indicator I_t compares the realized ground-truth rank R_t with the context-conditioned expected rank E[R_t] to calibrate token difficulty and alignment.Both quantities measure guessing cost, providing a direct bridge between target probability and predictive entropy.
  • Relative Scale: The Relative Scale S_t = I_t^-1 is integrated into fine-tuning objectives to emphasize under-learned tokens while avoiding uniform treatment of all positions.This joint treatment is intended to down-weight high-entropy, replaceable tokens and retain emphasis on low-entropy critical positions.
  • Relative Rank Indicator: The indicator decreases as ground-truth rank worsens but increases with expected rank, rewarding accurate predictions more in high-uncertainty contexts.Its logarithmic compression and exponential rescaling make it approximately saturate near 1 when both ranks are large.
  • Empirical Validation: Empirical token distributions separate replaceable tokens near I ≈ 1 from critical computation tokens in the low-I region, while alternative monotone transformations remain broadly stable.The critical examples include “frac”, “0”, and “{”; replaceable examples include “them” and “all”.
  • Empirical Validation: On Minerva Math tokens predicted by Qwen3-8B, empirical rank–probability and expected-rank–entropy relationships closely follow their theoretical bounds.The rank plot follows R = 1/p, while expected rank aligns with the entropy-derived lower bound.
  • Probability–Entropy Calibration: The rank-based construction approximates a relative competence ratio that treats probability as alignment and entropy as context-dependent prior support.A small competence score indicates insufficient alignment relative to context difficulty, whereas a large score supports down-weighting the position.

5. Experiments

Experiments evaluate RANKTUNER across mathematical reasoning benchmarks, out-of-distribution reasoning tasks, ablations, and inference-entropy behavior. Results show broad benchmark gains, stronger transfer, complementary probability and entropy components, and entropy behavior closer to the original model.

  • Experimental setup: RANKTUNER is evaluated across multiple backbones, five mathematical reasoning benchmarks, Pass@1/Pass@16, and out-of-distribution ARC-C and GPQA tasks.Training uses the first 10k NuminaMath-CoT instances and includes Qwen2.5-Math-7B and Qwen3-8B.
  • Mathematical reasoning: Across five mathematical reasoning benchmarks, RANKTUNER consistently improves over original models and representative probability- and entropy-based fine-tuning baselines.It especially improves Pass@1 on MATH-OAI, Minerva Math, and OlympiadBench, while improving Pass@16 on most benchmarks.
  • Out-of-distribution generalization: RANKTUNER achieves the best performance on both ARC-C and GPQA, indicating out-of-distribution transfer beyond mathematical reasoning.The comparison reports Pass@1 performance on Qwen2.5-Math-7B.
  • Ablations: Removing entropy degrades performance across all Pass@k values, whereas removing probability mainly weakens gains as k grows, indicating complementary component roles.The w/o Prob variant can slightly improve Pass@1 but has weaker sample diversity or coverage at larger k.
  • Inference-time entropy: Inference entropy separates methods into P-decay, P-neutral, and P-boost signatures, while RANKTUNER stays closest to the original baseline.DFT shows the most aggressive entropy collapse, whereas OverTone produces the highest entropy.

6. Discussion

The strongest empirical support is currently for supervised fine-tuning. Extensions to RL-style post-training, multimodal adaptation, personalization, recommendation, and information extraction remain future work.

  • Scope: Empirical evidence is strongest in supervised fine-tuning, while broader RL-style post-training and multimodal fine-tuning are outside the paper’s core claims.The paper leaves systematic studies of these settings to future work.

7. Conclusion

RANKTUNER uses rank-guided token reweighting to calibrate downstream alignment by intrinsic uncertainty. Across reasoning benchmarks, it improves over one-signal baselines while preserving diversity, with both probability and entropy contributing to the gains.

  • Conclusion: RANKTUNER compares ground-truth rank with expected rank and uses the inverse Relative Rank Indicator to focus updates on under-learned critical tokens.Noisy or replaceable positions are down-weighted.
  • Conclusion: Across multiple backbones and reasoning benchmarks, RANKTUNER consistently improves over probability- or entropy-only reweighting baselines.Ablations and entropy analysis support complementary probability- and entropy-aware components.

Impact Statement

The paper describes a supervised fine-tuning method and supplies theoretical, algorithmic, and experimental supplementary analyses. Its broader technical scope includes rank–probability and rank–entropy relationships, implementation details, diagnostics, and additional experiments.

  • Impact Statement: The proposed method calibrates probability- and entropy-based signals for token-level supervised fine-tuning.The stated aim is improved training stability and downstream reasoning performance.
  • Supplementary analysis: The appendix contains theoretical analyses connecting rank, probability, and entropy, including rank–probability and expected rank–entropy bounds.It also includes derivations, coefficient analysis, and boundedness results.
  • Supplementary analysis: Supplementary materials provide pseudocode, complexity analysis, noise diagnostics, token-level visualizations, and an illustrative RL-style extension.The RL-style extension is identified as potential rather than validated core evidence.
  • Supplementary experiments: Additional experiments cover dataset statistics, baseline details, metrics, cross-architecture results, function sensitivity, ξ approximations, fixed-versus-dynamic K, and code fine-tuning.These materials extend the empirical and implementation analysis.

A.4. Coefficient Analysis Across Different Regimes

The coefficient K(ξ) is approximately 0.5 for typical low-rank reasoning tokens but decreases as uncertainty grows, supporting uncertainty-aware rank weighting. The implementation computes token-wise scales from rank, entropy, and probability signals without asymptotic overhead beyond SFT.

  • For ξ≈1, K(ξ)=0.5, justifying the approximation used for typical low-rank reasoning tokens.
  • K(ξ) decreases monotonically from approximately 0.265 at ξ=2 to 0.076 at ξ=10.
  • For most chain-of-thought tokens with R,E[R]∈[1,5], K(ξ) lies in [0.2,0.5], while high uncertainty further dampens rank differences.
  • The rank transformation is approximately 0.5 log2(E[R]/R) for typical tokens, with a smaller coefficient in high-uncertainty contexts.
  • Replacing rank quantities with probability and entropy bounds yields a conservative lower-bounding surrogate of the Relative Rank Indicator.
  • RankTuner computes token-wise scales in a single vectorized forward pass with O(|V|) operations per token and O(1) additional space per position.
  • The noise diagnostic ranks response tokens by probability, entropy, and relative-rank scores after inserting irrelevant sentences into selected examples.
  • Relatively replaceable or noisy tokens are down-weighted, whereas genuinely incorrect result-critical tokens receive stronger emphasis.

B.5. Experimental Validation of Tightness of Bounds

The experiments test approximation tightness and the scope of entropy-aware calibration across datasets and fine-tuning settings. They also identify RL-style post-training as an unvalidated extension rather than an evaluated result.

  • The validation measures rank–probability and inverse expected-rank approximation gaps on chain-of-thought tokens from Minerva Math predicted by Qwen3-8B.
  • Table 5 reports token counts and shares below and above Ht=2 across five mathematical reasoning benchmarks.
  • Entropy mass is not confined below Ht=2; AIME24, OlympiadBench, and AMC23 retain non-trivial high-entropy fractions.
  • The high-entropy tail remains relevant because minority high-entropy tokens can exert disproportionate training impact.
  • RankTuner initializes mathematical-reasoning weights with wt=pt but uses uniform wt=1 for general fine-tuning tasks.
  • Under the importance-sampling view, multiplying by pt neutralizes the potentially large inverse-probability factor and reduces gradient variance.
  • The method can be interpreted as an adaptive power loss with entropy-dependent scaling across model-strong and model-weak regimes.
  • The study evaluates supervised fine-tuning objectives, while PPO/GRPO composition remains a future direction without validated RL experiments.

C.3. Metrics

The paper evaluates generated-solution performance with Pass@k, using combinations of sampled responses to measure whether at least one solution is correct. Pass@1 reflects one-shot accuracy, whereas Pass@16 also reflects solution-path coverage.

  • Pass@k measures the probability that at least one correct solution appears among k sampled attempts.
  • For each problem, the evaluation generates n=16 independent samples and computes Pass@k for k∈{1,2,4,8,16}.
  • A sample combination passes when at least one member has binary correctness score ci=1.
  • Pass@1 represents expected one-shot accuracy, while Pass@16 is more sensitive to diverse reasoning paths and solution coverage.

C.4. Supplementary Cross-Architecture Results for Mathematical Reasoning

RANKTUNER improves mathematical reasoning across multiple model architectures, while ablations show robustness to alternative rank approximations and transformation choices. Dynamic exponent adaptation further contributes beyond simply combining probability and entropy.

  • Cross-architecture results: RANKTUNER consistently improves mathematical reasoning across Qwen2.5-Math-1.5B, Qwen3-4B, and Llama-3.1-8B.The gains often concentrate on Pass@16, suggesting broader coverage of diverse reasoning paths, although some benchmark-specific regressions occur.
  • Transformation ablation: Alternative monotone choices of f and g produce broadly stable results, supporting the rank-based probability–entropy calibration principle.Different variants are best on different datasets, but overall performance remains close.
  • ξ approximation ablation: The default max approximation for ξ achieves the best Pass@1 on MATH-OAI, Minerva Math, OlympiadBench, and AMC23.Arithmetic and logarithmic means improve some Pass@16 results, while the geometric mean is strongest on AIME24 Pass@1.
  • Dynamic exponent ablation: Dynamic K(ξt) outperforms all fixed-exponent controls, including K = 0.9 near its average training-time value.This indicates that adapting the exponent to token-level uncertainty contributes beyond combining probability and entropy.

C.8. Code Fine-tuning and Evaluation

The code fine-tuning study evaluates RANKTUNER and related weighting choices across coder backbones, using functional-correctness benchmarks. Results show a capacity-dependent pattern, with RANKTUNER strongest on the smaller model and competitive on the larger one.

  • Evaluation setup: The study fine-tunes Qwen2.5-Coder-3B and Qwen2.5-Coder-7B on Evol-Instruct-Code-80k, evaluating Pass@1 and Pass@10 on HumanEval and HumanEval+.For code generation, the general-task setting w_t = 1 is used as the starting token weight.
  • Results by model capacity: On Qwen2.5-Coder-3B, fine-tuning methods can noticeably underperform the original model, indicating a capacity effect.The passage attributes this to difficulty absorbing new code-style supervision without degrading general coding competence.
  • Results by model capacity: RANKTUNER is consistently the strongest fine-tuning baseline on Qwen2.5-Coder-3B and best preserves performance in that regime.
  • Results by model capacity: On Qwen2.5-Coder-7B, RANKTUNER achieves the best results on three of four metrics and remains competitive on the remaining metric.
  • Weighting rationale: The paper contrasts probability-only weighting with probability–entropy calibration: probability identifies the knowledge region in math tasks but can misdelimit low-probability knowledge in general tasks.The broader visualization includes ground-truth probability, entropy, and the Relative Rank Indicator, with the indicator normalized around 1.
Loading 2602.01745v2…