Source-linked AI summary
What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training
Yuxuan Gu, Kequan Niu, Xiaocheng Feng, Yun Li, Yiqiao Yang, Bing Qin
TL;DR
Full-context support can conflate task information with response-prefix predictability, complicating token-level SFT amid the need to learn target behaviors while preserving pretrained capabilities. CIG-SFT estimates task-conditioned information per demonstrated token and uses it for policy-aware supervision; across math and code, it outperforms SFT and recent adaptive methods while remaining competitive with reinforcement learning.
Problem
Full-context model support conflates task evidence with response-prefix predictability, complicating token selection amid the tension between learning target behaviors and preserving pretrained capabilities.
Source: §1 · p. 2, §3.1 · p. 4Method
CIG-SFT measures each demonstrated token’s task-conditioned information using frozen-reference comparisons with and without task evidence, then maps it to a policy-aware proximal, gradient-equivalent weighted SFT objective.
Source: §1 · pp. 2–3, §3.2 · p. 4Results
Takeaways & Limitations
Analyses suggest its gains reflect more selective learning-pressure allocation, alongside lower average supervision weights and smaller relative parameter updates.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Supervised fine-tuning (SFT) remains a widely used approach for post-training large language models, because fixed demonstrations can directly introduce desired reasoning strategies and behavioral patterns. Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution. Recent dynamic SFT methods mitigate this issue by reweighting token-level supervision according to model-side signals such as predictive probability, prior support, or distributional compatibility. However, model compatibility is not equivalent to learning value: a token that is weakly supported by the current model may either be an undesirable mismatch or precisely the task-specific decision that the demonstration is intended to teach. We propose CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token. Under teacher forcing, we compare the token likelihood from a frozen reference model with and without access to the task input while keeping the gold solution prefix fixed. The resulting conditional information gain quantifies how much the task itself reduces uncertainty about the demonstrated decision. We then convert this information into a KL-regularized projection of the current policy and derive an equivalent scalar-weighted SFT objective whose logit gradient exactly matches that of the projected target. Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines, with particularly strong improvements in generalization to challenging evaluation distributions, while requiring neither additional training data nor online rollouts.
1 INTRODUCTION
Post-training must teach desired behaviors without sacrificing capabilities already supported by pretraining, but model compatibility alone does not identify which demonstrated tokens are worth learning. CIG-SFT uses task-conditioned information to guide token-level updates and reports consistent gains over standard and adaptive SFT baselines.
1 INTRODUCTION
- Post-training adapts pretrained models to express capabilities in task-specific forms, including instruction following, reasoning decisions, and response conventions.SFT and reinforcement learning are presented as central, complementary post-training paradigms.Source: §1 · p. 1
- Fixed external demonstrations offer behavioral coverage but can mismatch model predictions and risk forgetting, whereas on-policy methods may miss useful behaviors the current policy rarely visits.The paper frames this as a trade-off between coverage and compatibility.Source: §1 · p. 2
- Predictive support alone cannot distinguish a weakly supported task-specific decision worth teaching from an undesirable mismatch, or a useful behavior from a trivial continuation.Existing dynamic SFT methods modulate token-level supervision using model-side support or compatibility signals.Source: §1 · p. 2
- CIG-SFT estimates each demonstrated token’s task-conditioned information by comparing frozen-model likelihoods with and without the task input while holding the preceding response fixed.It uses this information in a KL-regularized projection of the current policy and an exactly gradient-equivalent scalar-weighted SFT objective.Source: §1 · p. 2
- CIG-SFT consistently improves over standard SFT and strong adaptive fine-tuning baselines across mathematical reasoning and code generation, while remaining competitive with reinforcement-learning methods.The reported gains span model scales and domains, persist across training-set sizes and hyperparameter settings, and require neither additional data nor online rollouts.Source: §1 · pp. 2–3
2 RELATED WORK
Related work adapts fixed-demonstration SFT through non-uniform token weighting and contrasts it with on-policy post-training. CIG-SFT differs by measuring task-input information beyond what the preceding context already reveals.
Supervised fine-tuning and adaptive objectives
- Adaptive SFT methods use predictive confidence, prior support, or distributional typicality to set token weights; CIG-SFT instead measures the task input’s contribution beyond the preceding context.The reviewed approaches include DFT, ASFT, PriFT, IDFT, ProFit, and InfoSFT.Source: §2 · p. 3
RELATED WORK
- Non-uniform supervision is used to improve generalization from fixed demonstrations, while on-policy approaches do not guarantee exposure to the behaviors post-training aims to introduce.The two lines address different sides of the trade-off between model compatibility and behavioral coverage.Source: §2 · p. 3
Reinforcement learning and on-policy post-training
- On-policy methods train on trajectories from the evolving policy, improving compatibility with its generative distribution but limiting exposure to behaviors it rarely discovers.The cited reasoning studies report missing informative successful rollouts on difficult prompts and declining exploration diversity during training.Source: §2 · p. 3
3 METHOD
CIG-SFT separates how much task-specific information a demonstrated token carries from how far the current policy should move toward it. It estimates token information under a frozen reference policy, then uses a proximity-constrained target and gradient-equivalent weighted SFT objective.
3.1 PRELIMINARIES
- Full-context support signals can conflate task-input information with predictability from preceding response tokens.CIG-SFT separates these sources when deciding how strongly to learn from each demonstrated token.Source: §3.1 · p. 4
3.2 POINTWISE CONDITIONAL INFORMATION GAIN
- For each demonstrated token, CIG-SFT compares frozen-policy likelihoods with and without the task input while holding the gold response prefix fixed.The normalized surprisal reduction measures how much of the task-free surprisal is explained by the task and can be cached before fine-tuning.Source: §3.2 · pp. 4–5
3.3 INFORMATION-PROXIMAL PROJECTION
- The information score acts as utility for the demonstrated decision, while KL regularization keeps its projected target close to the current policy.A zero score leaves the policy unchanged; a higher score or weaker proximity constraint yields a larger shift toward the demonstrated decision.Source: §3.3 · p. 5
3.4 GRADIENT-EQUIVALENT OBJECTIVE REALIZATION
- A scalar-weighted hard-target SFT objective reproduces the projected-target update’s logit gradient when targets or weights are detached.The objectives need not have equal values; their gradients with respect to the trainable policy parameters are equivalent.Source: §3.4 · p. 6
4 EXPERIMENTS
CIG-SFT leads aggregate mathematical-reasoning results at both tested scales and extends its advantage over supervised baselines to code generation. Analyses also report data efficiency and more selective supervision.
4.1 EXPERIMENTAL SETUP
- Experiments cover mathematical reasoning and code generation, comparing supervised, preference-optimization, and reinforcement-learning baselines under fixed training and evaluation settings.Mathematical-reasoning methods train for one epoch; evaluation samples 16 responses per problem and reports Avg@16 and Pass@16.Source: §4.1 · p. 6
4.2 MAIN RESULTS
- 38.5% Avg@16 and 63.5% Pass@16 on Qwen2.5-Math-7B exceed PriFT-mass by 2.1/0.3 points and DFT by 6.8/12.4 points.CIG-SFT achieves the strongest aggregate performance across both mathematical-reasoning scales.Source: Table 1 · p. 7, §4.2 · p. 7
- 31.5% Avg@16 and 58.6% Pass@16 on Qwen2.5-Math-1.5B exceed PriFT-prob by 1.2 and 2.4 points, respectively.These results also improve over standard SFT by 17.7/16.3 and DFT by 5.2/9.6.Source: Table 2 · p. 7, §4.2 · p. 7
- 35.4% Avg@16 and 50.5% Pass@16 on Qwen2.5-Coder-3B are the best among supervised methods, exceeding the strongest aggregate baseline by 1.1/0.8.Gains span HumanEval+, MBPP+, LiveCodeBench, BigCodeBench, and DS-1000.Source: §4.2 · pp. 7–8, Table 3 · p. 8
- Removing information-ratio supervision causes the largest component-ablation drop, while online information estimation sharply degrades performance.On Qwen2.5-Math-1.5B, removing the information ratio lowers Avg@16/Pass@16 from 31.5/58.6 to 21.9/48.4; online ct reaches 6.9/21.9.Source: Table 4 · p. 8, §4.2 · p. 8
4.3 ANALYSIS
- With 1K demonstrations, CIG-SFT reaches 28.9% Avg@16 and 55.9% Pass@16, outperforming SFT, DFT, ASFT, and IDFT even at 100K.Its scores remain within 28.9–31.5% Avg@16 and 55.2–58.5% Pass@16 from 1K to 100K demonstrations.Source: §4.3 · p. 8, Figure 2 · p. 9
- CIG-SFT beats SFT and DFT across tested learning rates; performance is stable at moderate λ but drops when the proximity constraint vanishes.The main experiments use a learning rate of 5×10−5 and λ = 0.3.Source: Figure 3 · p. 9, §4.3 · p. 9
- CIG-SFT assigns smaller average token weights and induces smaller relative parameter updates than DFT, while achieving stronger downstream performance.Its average weights are approximately 0.4–0.46, compared with DFT’s approximately 0.8–0.86.Source: Figure 4 · p. 9, §4.3 · p. 9
5 CONCLUSION
CIG-SFT uses task-conditioned information to guide token-level supervision, combining information attribution with policy-aware proximal adaptation. The paper reports consistent gains across mathematical reasoning and code generation, alongside more selective supervision and smaller parameter updates.
5 CONCLUSION
- CIG-SFT combines conditional information attribution with policy-aware proximal adaptation in a gradient-equivalent objective.The authors present task-conditioned information as a complement to model-side compatibility for learning from fixed demonstrations.Source: §5 · p. 9
- Across mathematical reasoning and code generation, CIG-SFT consistently outperforms recent dynamic fine-tuning methods while remaining competitive with reinforcement-learning baselines.The reported gains persist across learning rates and data scales, and strong performance is obtained with substantially fewer demonstrations.Source: §5 · p. 9
- CIG-SFT uses lower average supervision weights and induces smaller relative parameter updates, suggesting gains from more selective learning pressure rather than more aggressive fitting.Source: §5 · p. 9
AI USE STATEMENT
The authors used generative AI as a research and writing assistant and independently reviewed and verified AI-assisted technical content.
AI USE STATEMENT
- Generative AI assisted with mathematical derivations, experimental design and interpretation, literature identification, manuscript editing, and scientific-figure design.Source: AI USE STATEMENT · p. 10
- The authors independently reviewed AI-assisted technical content, including checking derivations and verifying cited literature against original sources.Source: AI USE STATEMENT · p. 10
- The authors validated reported experimental results and conclusions against their own experiments and take responsibility for the paper’s final content.Source: AI USE STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
The paper describes its training objective, experimental setup, and code and configuration availability as resources for reproducing CIG-SFT and its evaluation.
REPRODUCIBILITY STATEMENT
- Section 3 specifies the complete training objective and derivation, including the conditional information score, information-proximal projection, and gradient-equivalent objective.Source: REPRODUCIBILITY STATEMENT · p. 10
- The paper reports the models, datasets, baselines, evaluation metrics, and main hyperparameters used in its experiments.Source: REPRODUCIBILITY STATEMENT · p. 10
- Training and evaluation code, along with configuration files required to reproduce the reported experiments, is available in the repository.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 CASE STUDY: TASK-CONDITIONED CREDIT AND MODEL PROBABILITY
The case study compares frozen-model token likelihoods with the task input and with that input erased, while holding the gold response prefix fixed. CIG-SFT emphasizes task-relevant decisions and suppresses predictable continuation or formatting tokens.
A.1 CASE STUDY: TASK-CONDITIONED CREDIT AND MODEL PROBABILITY
- The frozen base model evaluates each demonstrated answer token with the original task input and an erased-input counterfactual, holding the gold response prefix fixed.Source: §A.1 · p. 14
- CIG-SFT assigns larger weights to task-relevant decisions, including substitutions, comparisons, problem entities, and arithmetic choices.The case study contrasts these tokens with generic continuation and formatting tokens.Source: §A.1 · p. 14
- Predictable continuation tokens such as “is” and “boxed” can receive near-unit DFT weights but zero task gain, so CIG-SFT suppresses them.Source: §A.1 · p. 14
A.2 COMPUTATIONAL COST
The section separates training passes from method-specific preprocessing and measures CIG-SFT’s online and end-to-end costs. CIG-SFT’s online cost matches SFT, with additional one-time scoring and storage for cached information scores.
A.2 COMPUTATIONAL COST
- DFT and CIG-SFT reuse target probabilities from the trainable forward pass, adding only token-wise scalar operations during training.The section accounts separately for backward passes, optimization, and method-specific preprocessing.Source: §A.2 · p. 14
- CIG-SFT’s one-time frozen-reference scoring evaluates each demonstration with the task input and with an empty task input, without gradients, then caches the resulting information scores.The cached scores are reused throughout fine-tuning.Source: §A.2 · p. 15
- CIG-SFT’s online training cost matches SFT at 28.93 GiB peak GPU memory; offline scoring adds 0.48 hours, for approximately 2.06 hours end to end.The profile uses Qwen2.5-Math-1.5B, 30K examples, and 233 optimizer steps.Source: §A.2 · p. 15, Table 5 · p. 15
- Cached information scores increase offline storage from 41.5 MiB to 254.2 MiB, an additional 212.7 MiB, without increasing peak training GPU memory.The profile uses the same fixed training setup as the online-cost measurement.Source: §A.2 · p. 15, Table 5 · p. 15
A.3 STABILITY ACROSS RANDOM SEEDS
The appendix reports stability, data-scale, sensitivity, capability-retention, and in-domain generalization analyses. CIG-SFT’s reported results include performance across random seeds, learning rates, and evaluation settings.
A.3 STABILITY ACROSS RANDOM SEEDS
- 31.00 ± 0.46 Avg@16 across five seeds, with a 95% confidence interval of [30.42, 31.57], above PriFT-prob’s [28.89, 30.03].CIG-SFT ranges from 30.5% to 31.5% Avg@16 across seeds 42–46.Source: §A.3 · p. 15
A.4 DETAILED DATA-SCALE RESULTS
- Table 7 reports Avg@16 and Pass@16 results for each method across training-set sizes from 1K to 100K demonstrations.These values provide the detailed results underlying the training-scale analysis in Figure 2.Source: §A.4 · p. 15, Table 7 · p. 16
A.5 HYPERPARAMETER SENSITIVITY
- CIG-SFT remains strongest across tested learning rates, while performance drops sharply when projection coefficient λ is nearly zero and stays relatively stable across moderate values.The appendix gives detailed Avg@16/Pass@16 results for the learning-rate analysis and CIG-SFT results for projection-coefficient settings.Source: §A.5 · p. 16, Table 8 · p. 16, Table 9 · p. 17
A.6 GENERAL CAPABILITY RETENTION ANALYSIS
- CIG-SFT’s average accuracy changes from 56.1% before fine-tuning to 55.9% after fine-tuning, with small differences across four general-purpose benchmarks.DFT and IDFT average 54.9% and 54.4%, respectively, while ASFT reaches 55.7%.Source: §A.6 · p. 16, Table 10 · p. 17
A.7 IN-DOMAIN GENERALIZATION
- On 1,000 unseen NuminaMath-CoT examples, CIG-SFT achieves the best Avg@16 at 51.8%, ahead of DFT (50.8%), ASFT (50.5%), and SFT (42.4%).The sampled examples were excluded from training and drawn from the training data source.Source: §A.7 · p. 16, Table 11 · p. 17