PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting

Muxi Diao, Lele Yang, Wuxuan Gong, Yutong Zhang, Zhonghao Yan, Yufei Han, Kongming Liang, Weiran Xu, Zhanyu Ma

arXiv:2601.02151v1cs.LGcs.AIcs.CL

TL;DR

SFT commonly causes catastrophic forgetting because external supervision can conflict with the model’s confident priors, whereas on-policy RL preserves general capabilities. The paper proposes EAFT, which gates loss by token-level entropy to suppress conflicting updates while retaining useful learning. Across domains and model scales, EAFT matches target-task performance while mitigating general-capability degradation.

  • Problem

    SFT is effective for domain adaptation but often causes catastrophic forgetting, motivating analysis of its discrepancy from capability-preserving on-policy RL.

  • Method

    EAFT dynamically modulates training loss with token-level entropy, down-weighting low-entropy conflicting data while preserving supervision for uncertain samples.

  • Results

    EAFT achieves a Pareto improvement by matching or exceeding target-task baselines while significantly mitigating catastrophic forgetting on general benchmarks.

  • Takeaways & Limitations

    EAFT provides a simple, robust approach for balancing domain adaptation with preservation of general capabilities.

  • Takeaways & Limitations

    EAFT is designed for domain adaptation and continual learning, not knowledge editing or counterfactual training that requires overriding prior beliefs.

Abstract

from arXiv · show

Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.

1 Introduction

SFT can degrade general capabilities because external supervision forces updates on low-entropy conflicts with the model’s strong priors. EAFT uses entropy-aware loss modulation to preserve general capabilities while retaining domain-task improvements.

  • SFT adapts models to specific domains but often causes catastrophic forgetting of general capabilities.
  • On-policy RL preserves robustness because its self-generated data aligns with the model’s probability landscape, unlike externally supervised SFT data.
  • Masking the bottom 15% of tokens in both entropy and probability significantly mitigated general-capability degradation relative to standard SFT.
  • EAFT dynamically modulates training loss using token-level entropy, down-weighting conflicting tokens while concentrating supervision on uncertain ones.
  • EAFT was evaluated across mathematical, medical, and agent domains using Qwen and GLM models from 4B to 32B parameters.
  • EAFT consistently matches or exceeds target-task baselines while significantly mitigating catastrophic forgetting on general benchmarks.
  • The method also targets Confident Conflicts empirically and is reported as robust to hyperparameter variation and computationally efficient.
  • “Confident Conflicts” are low-entropy, low-probability tokens that contradict strong model priors and drive destructive updates.

2 Related Work

Related work contrasts off-policy SFT, which can memorize specific demonstrations, with on-policy RL, which is more robust. Existing token-level methods mainly use probability or KL divergence, motivating entropy as a better signal for distinguishing uncertainty from conflict.

  • SFT maximizes likelihood on ground-truth demonstrations, whereas RL updates the model from its own generated responses using reward signals.
  • SFT is efficient but prone to memorization and reduced generalization, while RL can benefit from limited examples without severe overfitting.
  • This work attributes SFT instability to indiscriminate fitting of low-entropy samples that contradict pretrained knowledge.
  • Prior mitigation strategies include parameter-constraint methods, token-level learning-rate scaling, probability-based loss reweighting, and KL regularization.
  • Probability alone cannot distinguish epistemic uncertainty from Confident Conflicts, so entropy is introduced as the gating signal.

3 Empirical Analysis & Methodology

The analysis identifies Confident Conflicts—low-entropy, low-probability tokens—as a primary source of SFT-induced forgetting, then introduces EAFT to adapt supervision using entropy-based gating. EAFT suppresses updates on conflicting tokens while preserving learning from uncertain ones.

  • 3.1 Preliminaries: SFT’s uniform cross-entropy objective updates every token regardless of the model’s prior knowledge or uncertainty.This can force the model to fit targets that conflict with its internal state.
  • 3.1 Preliminaries: Token probability measures confidence, while predictive entropy measures uncertainty over the vocabulary.These metrics provide the basis for distinguishing uncertainty from conflict.
  • 3.2 Empirical Analysis: SFT data contains low-entropy, low-probability Confident Conflicts, unlike on-policy data, which lies in high-probability or high-entropy regions.The comparison uses token-level probability and entropy distributions for SFT data and model-generated rollouts.
  • 3.2 Empirical Analysis: Masking the bottom 15% of tokens by both entropy and probability significantly mitigated general-capability degradation relative to standard SFT.The pilot intervention nearly eliminated catastrophic forgetting on the reported benchmarks.
  • 3.2 Empirical Analysis: Low-entropy conflicts induce large cross-entropy gradients because fitting the target requires substantial updates against the model’s strong prior.These updates can overwrite general representations in the base model.
  • 3.3 Entropy-Adaptive Fine-Tuning (EAFT): EAFT replaces hard masking with soft entropy gating, scaling supervision to suppress confident-conflict gradients while retaining learning from uncertain tokens.Its normalized top-K entropy signal uses K = 20 and ln(K) as the maximum-entropy normalization factor.

4 Experiments

Experiments evaluate EAFT’s performance, mechanism, and domain generality across mathematical, medical, and agentic tasks. EAFT preserves target-task performance while reducing catastrophic forgetting by suppressing updates on confident conflicts and continuing to optimize uncertain tokens.

  • Performance: EAFT maintains competitive Math performance while significantly mitigating catastrophic forgetting across model families and scales.The experiments compare target-domain results with general-capability benchmarks across diverse settings.
  • Performance: EAFT achieves the highest average score across general tasks, whereas standard SFT causes a 10.7 point drop on CLUEWSC for Qwen3-4B.This supports stronger capability retention than the evaluated baselines.
  • Performance: The gap between EAFT and the best-performing method on math score remains consistently less than 1 point.EAFT also achieves the best performance on several math sub-benchmarks.
  • Mechanism: EAFT suppresses strong optimization pressure in the Confident Conflict region, protecting established representations from conflicting updates.The gradient landscape contrasts dark, high-magnitude SFT gradients with lighter EAFT gradients in that region.
  • Mechanism: EAFT rapidly reduces loss on high-entropy tokens comparably to SFT while preventing low-entropy conflicts from being over-optimized.Figure 4 defines high entropy as ≥2.0 and low entropy as ≤0.5.
  • Universality: Across biomedical and agent tool-use evaluations, EAFT consistently mitigates catastrophic forgetting, supporting domain-agnostic effectiveness.The medical and agentic experiments extend the evaluation beyond mathematics.

5 Analysis and Discussion

The analysis shows that entropy awareness, rather than a particular gating formula, reduces forgetting while preserving target-task performance. Soft gating retains adaptation signals that hard masking removes, and Top-20 entropy approximation provides near-exact fidelity with negligible overhead.

  • Analysis Design: The study tests robustness across linear, polynomial, sigmoid, and hard-threshold gating to determine whether entropy awareness is the primary driver.The robustness analysis compares alternative functions of normalized entropy, while the efficiency analysis evaluates approximate entropy computation.
  • Pareto Trade-off: EAFT variants occupy the optimal top-right Pareto frontier, unlike Masked SFT’s target-score drop and Standard SFT’s severe forgetting.The frontier compares target-domain performance with preservation of general capabilities.
  • Universality of Entropy Awareness: All entropy-aware variants consistently outperform SFT in general capabilities, indicating that entropy monitoring—not a specific mathematical form—drives reduced forgetting.The variants evaluated are EAFT, EAFT2, EAFT3, and EAFTsig.
  • The Necessity of Soft Gating: 65.60 versus 69.27: Masked SFT harms Math Score relative to EAFT by discarding confident tokens that carry essential adaptation signals.EAFT softens the impact of these tokens instead of removing them entirely.
  • Efficiency of Top-K Approximation: At K = 20, Top-K entropy reaches a Pearson correlation of 0.999 with exact entropy while memory overhead remains below 0.4 KB.The authors therefore adopt Top-20 approximation for entropy estimation.

6 Conclusion

The paper identifies Confident Conflicts as a primary driver of catastrophic forgetting and introduces EAFT to modulate training loss using token-level entropy. Across diverse domains and model scales, EAFT suppresses destructive updates while preserving efficient adaptation and general capabilities.

  • Core Finding: EAFT identifies Confident Conflicts as a primary driver of catastrophic forgetting in SFT.These conflicts are addressed through entropy-based loss modulation.
  • Method: EAFT dynamically modulates training loss using token-level entropy to suppress gradients from conflicting data.The method is presented as a simple mechanism for balancing adaptation with preservation of existing capabilities.
  • Evidence and Implication: Experiments across diverse domains and model scales validate EAFT’s ability to prevent destructive updates while maintaining learning efficiency.The conclusion characterizes EAFT as a robust solution for balancing domain adaptation and general-capability preservation.

7 Limitations

The paper limits EAFT’s intended use to domain adaptation and continual learning rather than universal SFT replacement. Its behavior can also preserve erroneous priors and may trade peak target specialization for general-capability retention.

  • Scope of Applicability: EAFT is designed for domain adaptation and continual learning, not knowledge editing or counterfactual training that must override prior beliefs.Suppressing Confident Conflicts can hinder intended updates in those settings.
  • Target Performance Trade-off: EAFT may closely approach but not exceed standard SFT’s peak target-domain specialization, making standard SFT preferable when target performance alone is paramount.The reported advantage is a Pareto improvement that preserves general capabilities.
  • Dependence on Base Model Quality: If the base model is confidently wrong or hallucinatory, EAFT may protect those erroneous behaviors because it assumes high-confidence priors are valuable knowledge.The paper suggests uncertainty calibration as future work.

A Qualitative Analysis: What are "Confident Conflicts"?

The qualitative analysis separates high-entropy reasoning tokens, low-entropy deterministic tokens, and low-entropy low-probability Confident Conflicts. This distinction motivates suppressing conflicting updates while retaining uncertain reasoning signals.

  • Branching Points (High Entropy): High-entropy tokens often represent semantic branching points, reasoning connectors, and complex reasoning steps that remain important for SFT.Examples include abstract verbs, general nouns, and logical transitions.
  • Confident (Low Entropy): Low-entropy, high-probability tokens mainly encode deterministic syntax, mathematical symbols, functional words, or rote patterns.These tokens are generally easier for the model to master.
  • Confident Conflicts: Confident Conflicts are low-entropy, low-probability tokens involving specific entities, rare domain terms, or noise where model priors mismatch SFT data.Examples include names and specialized terms such as Jayden and modpacks.
  • Interpretation: The distinction motivates entropy-adaptive training that protects the original distribution while learning critical reasoning patterns.The method targets low-entropy conflicts without suppressing high-entropy reasoning steps.

B.1 Mathematical Datasets

The paper uses curated datasets for mathematical, medical, and agentic-domain adaptation, with selected subsets supporting domain-specific experiments.

  • B.1 Mathematical Datasets: The mathematics training data uses a verified subset of NuminaMath selected for mathematical reasoning and precision in derivation.The final training set randomly sampled 19k instances whose model-generated outputs were verified as correct.
  • B.1 Mathematical Datasets: Huatuo-o1 transfers reasoning chains generated by DeepSeek-R1 from verifiable medical problems to target models.
  • B.1 Mathematical Datasets: Nemotron-Agentic-Tool-Use-v1 provides multi-turn trajectories in which models decompose user goals and execute tool calls.The experiments randomly sampled 20,000 trajectories from its non-thinking subset.

C Benchmarks

The evaluation combines general-capability benchmarks with mathematics, medical, and tool-use benchmarks, alongside the fine-tuning baselines used for comparison.

  • C Benchmarks: General capability is evaluated with MMLU, IFEval, and CLUEWSC, covering multitask knowledge, instruction following, and coreference reasoning.
  • C Benchmarks: Mathematical reasoning is measured with AIME24, AIME25, and GSM8K, spanning challenging contest problems and elementary multistep problems.
  • C Benchmarks: Medical evaluation uses MedMCQA, PubMedQA, and MedQA to test medical knowledge, scientific-literature reasoning, and clinical reasoning.
  • C Benchmarks: BFCL v3 evaluates function calling and tool use across programming languages, including parallel, nested, and multi-turn interactions.
  • C Benchmarks: The evaluated model descriptions include Qwen3-4B-Instruct-2507 and GLM-9B-0414, both lightweight models designed for efficient deployment and task execution.
  • C Benchmarks: The compared approaches include SFT, SFTKL, FLOW, DFT, and TALR, representing standard training, KL regularization, dynamic reweighting, probability scaling, and token-adaptive weighting.

F.3 Evaluation Protocol

The evaluation protocol reports average performance over repeated runs and follows established implementations for mathematical and medical benchmarks.

  • F.3 Evaluation Protocol: All evaluations use three independent runs and report average performance.
  • F.3 Evaluation Protocol: Mathematical benchmarks follow the Qwen2.5-Math evaluation implementation, while medical tasks follow the MedEvalKit framework.
  • F.3 Evaluation Protocol: The paper positions EAFT within a broader comparison with alignment techniques including RL and RFT.

G.1 Comparison with EAFTsig

The comparison favors linear EAFT over sigmoid gating because it is more robust to hyperparameters and easier to deploy, while also avoiding reference-model and rollout costs associated with RL.

  • G.1 Comparison with EAFTsig: EAFTsig achieves competitive performance, but the paper advocates linear gating for its hyperparameter robustness and ease of deployment.
  • G.1 Comparison with EAFTsig: Sigmoid gating is sensitive to its steepness α and centering threshold β, with high steepness creating a near-binary decision boundary.
  • G.1 Comparison with EAFTsig: Linear EAFT makes loss weight directly proportional to uncertainty, reducing the need for hyperparameter tuning and calibration across domains.
  • G.1 Comparison with EAFTsig: Compared with RL, EAFT avoids reference models and rollout generation while retaining a memory footprint and computational graph close to standard SFT.
  • G.1 Comparison with EAFTsig: Unlike static RFT data curation, EAFT updates its gating dynamically as the model’s uncertainty changes during training.
  • G.1 Comparison with EAFTsig: The manuscript states that large language models were used to improve clarity and grammatical correctness, followed by author review and editing.
Loading 2601.02151v1…