PaperlayerView exact revision on arXiv

Source-linked AI summary

Finetuning with Sampling: SFT Learns Better Than You Think

Aayush Karan, Sitan Chen, Yilun Du

arXiv:2610.02140v1cs.LGcs.AIcs.CL

TL;DR

Posttraining seeks to add capabilities without losing existing ones, but RL relies on successful model-generated trajectories while SFT can use off-policy expert traces. This paper uses MCMC sampling to make those traces more aligned with a base model before SFT. Across several task areas, sampled-data SFT often generalizes better and forgets less than strong on-policy baselines.

  • Problem

    Posttraining aims to add new capabilities without losing existing ones, while RL is associated with stronger generalization and retention than SFT, which can suffer weak generalization and catastrophic forgetting.

    Source: §1 · p. 1

  • Method

    The paper uses MCMC-based sampling to progressively transform off-policy expert traces toward a base model distribution while retaining their information for finetuning.

    Source: §1 · p. 2

  • Results

    Across science, math reasoning, and open-ended expertise, SFT on sampled data can generalize better and forget less than vanilla SFT and strong on-policy baselines, including RL and self-distillation.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Sampling can serve as a model-native operator for shaping data for learnability, with proposed uses in on-policy distillation and RL on hard tasks.

    Source: §6 · p. 12

    A direct Metropolis-Hastings implementation for LLMs is infeasible and may mix slowly, requiring exponentially many steps in some settings.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting. At the same time, SFT can learn from off-policy expert data, whereas RL must rely on a model's ability to find successful trajectories with repeated sampling. In our work, we seek to leverage the strength of on-policy learning while utilizing the privileged information contained in off-policy data. However, rather than modifying the learning objective to accommodate this data, we instead tailor the data distribution to better suit the learner. We introduce a Markov chain Monte Carlo (MCMC) sampling algorithm that progressively transforms off-policy traces to be more on-policy given a reference model for finetuning. Across tasks like scientific skill acquisition, mathematical reasoning, and open-ended expertise, our sampling algorithm enables SFT to rival prevailing posttraining techniques, often generalizing better and forgetting less than strong on-policy baselines. In addition, the resulting finetuned models exhibit strong distributional performance and are capable of learning beyond sharpening the base model distribution. At a higher level, our approach presents sampling as a model-native operator that shapes data for learnability, offering broader utility as a general-purpose primitive throughout the posttraining stack.

1 Introduction

The introduction frames posttraining’s challenge as adding new capabilities without sacrificing existing ones: RL is associated with stronger generalization and retention, while SFT can use expert traces but is vulnerable to weak generalization and forgetting. The paper proposes shaping off-policy data toward the base model distribution so SFT can retain privileged supervision while becoming more on-policy.

1 Introduction

  • Figure 1 reports that sampling-composed SFT performs best on chemistry accuracy and prior-task retention among the compared approaches.The figure compares it with SFT on the original dataset and OPSD, and plots MMLU and AMC prior capabilities.Source: Figure 1 · p. 2
  • RL depends on the model finding successful trajectories through repeated sampling, which can leave hard training samples without a learning signal.When a correct rollout is not generated, there is no collective reward variance; SFT can instead learn from expert trajectories without an explicit reward.Source: §1 · pp. 1–2
  • The paper proposes progressively transforming off-policy expert traces toward a given base model distribution rather than modifying the learning algorithm to accommodate off-policy data.The stated aim is to retain privileged information while making the data distribution more suitable for learning.Source: §1 · p. 2
  • Across science, math reasoning, and open-ended expertise, SFT on sampled data can generalize better and forget less than vanilla SFT and strong on-policy baselines, including RL and self-distillation.The authors also report strong performance across multiple samples and learning capabilities not present in the base model.Source: §1 · p. 2

2 Related Works

Related work discusses the SFT–RL generalization and retention gap, approaches that combine on- and off-policy learning, self-distillation, and MCMC sampling for language models. The paper distinguishes its approach from methods that change the learning objective by leaving SFT intact and modulating the data.

SFT vs. RL

  • Prior work attributes SFT’s generalization and retention gap with RL to off-policy training, including unstable updates and greater forgetting, while on-policy data is associated with less forgetting.Xiong et al. [2025] suggest off-policy gradients can encourage overfitting; Shenfeld et al. [2025] connect forgetting with KL divergence from the base model, and Chen et al. [2025] offer a mode-seeking rationale.Source: §2 · p. 3

Bridging SFT and RL

  • Bridging approaches use expert data in RL or modify SFT objectives, whereas this paper leaves SFT intact and makes off-policy data more on-policy.The cited SFT approaches use importance-sampling weights or a KL term; RL variants incorporate expert data in context.Source: §2 · p. 3

Self-Distillation

  • Self-distillation conditions a teacher model on privileged information and distills it with a KL-divergence loss; OPSD is cited as a continual-learning baseline.The passage names Shenfeld et al. [2026], Ye et al. [2026], and Zhao et al. [2026] as variations on self-distillation.Source: §2 · p. 3

MCMC Sampling with LLMs

  • Prior LLM MCMC work tilts sampling toward rewards or probabilistic programs, or sharpens reasoning distributions, while this approach transforms off-policy data for later finetuning.The cited examples include Zhao et al. [2024] and Faria et al. [2024] for reward tilting, and Karan and Du [2025] for sharpened reasoning distributions.Source: §2 · p. 3

3 Preliminaries

The preliminaries define token sequences and describe language models as distributions over them, built from autoregressive conditional token probabilities. They then introduce sampling from that distribution and SFT’s cross-entropy training setup.

3 Preliminaries

  • A finite vocabulary X defines finite token sequences x_0:T, and X* denotes token sequences of finite but arbitrary length.The sequence notation uses x_<t and x_>t for tokens before and after position t.Source: §3 · p. 3
  • An LLM defines a distribution over token sequences by learning conditional token distributions p(x_t|x_<t), which determine the joint distribution.The joint distribution follows from the autoregressive conditionals.Source: §3 · p. 3
  • Sampling from an LLM proceeds token by token using those conditional distributions, thereby sampling from the joint distribution.Source: §3 · p. 3
  • SFT uses expert trajectories in response to queries and optimizes a cross-entropy loss through gradient updates to the LLM parameterization.Source: §3 · p. 3

4 Boosting Off-Policy Data with MCMC Sampling

This section formalizes how to transform off-policy expert data into samples that preserve its information while moving closer to the base model distribution. It develops an MCMC procedure for sampling from that target and adapts it for LLMs.

4.1 Targeting the Information Projection

  • The target is the base model distribution restricted to trajectories equivalent to the expert trace, uniquely minimizing KL divergence while preserving the expert information.When the base model can reliably generate correct trajectories, rejection sampling suffices; the method instead addresses cases where that is infeasible.Source: §4.1 · pp. 4–5

4.2 The Metropolis-Hastings Algorithm

  • Information-preserving Metropolis-Hastings proposals keep candidates within the equivalence set, and each MCMC step progressively reduces the policy gap to the base model.The method initializes from an expert trajectory and accepts or rejects proposed candidates; with irreducible proposals, the chain converges to the target.Source: §4.2 · pp. 5–6

4.3 Projection Sampling for LLMs

  • Projection sampling adapts Metropolis-Hastings with increasingly long restricted targets, using expert-conditioned prompts to resample completions while preserving correctness.It runs on the training dataset before finetuning, so its cost is one-time; increasing block size can lower cost but raises approximation error.Source: §4.3 · pp. 6–7

5 Experiments

The experiments test whether sampling-shaped expert traces let SFT acquire new capabilities while retaining prior ones. Across chemistry, math, and medical tasks, the reported results and analyses show strong generalization, retention, and learning beyond sharpening the base model.

5.1 Experimental Setup

  • The setup tests chemistry skill acquisition, hard math reasoning, and open-ended medical QA with expert trajectories, assessing task generalization and prior-capability retention against SFT and on-policy baselines.Math additionally includes GRPO and UFT; medical correctness is evaluated with an automated LLM grader.Source: §5.1 · pp. 8–9

5.2 Main Results

  • Across chemistry, math, and medical evaluations, sampling enables SFT to generalize and retain prior capabilities at least as well as, or better than, on-policy posttraining counterparts.The table covers Qwen2.5-7B-Instruct for chemistry and medical tasks and Qwen2.5-3B for math.Source: Table 1 · p. 9
  • Chemistry sampling SFT improves performance by +31.7%, exceeding OPSD’s improvement by +4.20%; simple zero-shot rewrites perform less strongly.The comparison is reported across pretrained base models and tasks; the authors say this suggests the full MCMC process matters.Source: §5.2 · p. 9
  • Math sampling SFT gains +18.0% on hardest MATH, +14.4% on AMC, +20.3% on GSM8K, and +33.7% on MATH500, beating the next-best RL baseline there by +26.9%.The MATH500 gain is relative to the base model; the other reported gains extend to adjacent out-of-distribution benchmarks.Source: §5.2 · pp. 9–10
  • The sampling procedure is tailored to the base model: Qwen finetuned on data boosted for Olmo severely underperforms all baselines.This comparison supports the authors’ claim that the sampling procedure’s customization to the base-model distribution is crucial.Source: §5.2 · p. 10
  • Sampling SFT avoids MMLU loss in chemistry and limits average prior-task accuracy loss to -1.10%; in medicine, it recovers 16.3% of vanilla SFT’s average prior-capability loss and forgets least among baselines.The chemistry and medical retention findings are reported alongside comparisons with other baselines, including OPSD.Source: §5.2 · p. 10

5.3 Analysis

  • On Olmo chemistry, sampling’s pass@k curve exceeds both the base model and OPSD, while some problems move from zero base-model pass rate to 67.2% or 53.1%.The persistent gaps at large k are presented as evidence of stronger learned capabilities rather than merely sharpening existing ones.Source: Figure 4 · p. 11, §5.3 · p. 11
  • As MCMC steps increase from 0 to 10, the boosted-data KL gap to the base model decreases while science-task finetuning accuracy improves.The analysis frames additional sampling steps as a way to scale sampling compute and generate more on-policy learning data.Source: Figure 5 · p. 11, §5.3 · p. 11

6 Conclusion

Boosted-data SFT generalizes better and forgets less than vanilla SFT and strong on-policy baselines across science, math reasoning, and open-ended expertise. The paper also presents sampling as a model-native way to shape data for learnability and describes possible uses beyond SFT.

6 Conclusion

  • Across science, math reasoning, and open-ended expertise, SFT on boosted data generalizes better and forgets less than vanilla SFT and strong on-policy baselines, including RL and self-distillation.The resulting models also show strong distributional performance across multiple samples and can learn capabilities not present in the base model.Source: §6 · p. 12
  • For on-policy distillation, the MCMC process induces a teacher distribution closer to the student in KL divergence, facilitating more stable updates.The authors also describe simulating on-policy RL rollouts on hard tasks where the base model cannot generate a learning signal.Source: §6 · p. 12
  • The authors frame sampling as a model-native operator that shapes data for learnability and may serve as a general-purpose primitive throughout posttraining.Their approach exchanges sampling-time compute for stronger, more on-policy trace distributions built from privileged information.Source: §6 · p. 12

A Examples of Boosted Off-Policy Data

The examples pair original and boosted responses across chemistry, math, and medical tasks.

A Examples of Boosted Off-Policy Data

  • In the chemistry balancing example, both the original and boosted responses give 3 CO2 + 3 H2O → C3H6O3 + 3 O2.The boosted response explains balancing carbon, hydrogen, and oxygen in sequence.Source: A Examples of Boosted Off-Policy Data · p. 15
  • In the travel-and-weakness medical example, both responses identify a patent foramen ovale as the likely cardiac abnormality.The boosted response connects the leg DVT to a possible clot passage into systemic circulation.Source: A Examples of Boosted Off-Policy Data · p. 16
  • For the inverse-function problem, both responses derive f(f(10)) = 1 from the given values of h.The original uses h(2) = 10 and h(1) = 2; the boosted response spells out the corresponding values of f.Source: A Examples of Boosted Off-Policy Data · p. 16

B Additional Results

Additional chemistry experiments test how the boosted data's alignment with the training model affects accuracy and report results for Olmo-3-7B-Instruct.

B Additional Results

  • The additional chemistry results include Olmo-3-7B-Instruct, which the authors use to demonstrate applicability beyond one pretrained-model family.The supplied table caption identifies Chemistry as the new-task generalization measure and MMLU, GPQA, AMC, MATH500, and GSM8K as prior-capability retention measures.Source: B Additional Results · p. 17, Table 2 · p. 18
  • 57.33% chemistry accuracy resulted when Qwen2.5-7B-Instruct trained on data boosted for Olmo-3-7B, below vanilla SFT's 61.8%.A 50/50 mix of boosted and original SFT data reached 62.14%, between standard SFT and SFT on boosted data.Source: B Additional Results · p. 18

C.1 Information-Preservation

The section checks information preservation by measuring accuracy on the boosted datasets.

C.1 Information-Preservation

  • Boosted-dataset accuracy is 94.33% with Qwen and 93.94% with Olmo for chemistry, 95.33% for math, and 95.86% for medical.The authors present these values as a check of the MCMC process's information-preservation validity.Source: §C.1 · p. 18

C.2 Sampling

The sampling implementation can reduce inference overhead by swapping in higher-likelihood candidates, while still monotonically reducing KL divergence between the data-generation policy and base distribution.

C.2 Sampling

  • The full Algorithm 1 can require two LLM forward passes per iteration: one to generate a candidate and another to estimate its transition probability.The passes correspond to proposing x′ via κC and estimating κC(x|x′).Source: §C.2 · p. 18
  • To reduce inference overhead, the implementation swaps whenever a generated candidate has higher likelihood than the current one.This approximates the swapping condition.Source: §C.2 · p. 18
  • These more aggressive shifts toward high-likelihood regions still monotonically reduce KL divergence between the data-generation policy and base distribution.Figure 5 demonstrates this behavior.Source: §C.2 · p. 18

C.3 Proposal and Grading Prompts

The chemistry proposal prompt asks a model to continue a partial solution using a supplied solution, while the medical grading prompt classifies responses against a reference answer for correctness.

C.3 Proposal and Grading Prompts

  • For chemistry, the proposal prompt supplies a problem, solution, and partial response, then asks an expert model to continue the response consistently toward a complete answer.The continuation is requested in the model’s own words and must end with an answer exactly matching the supplied solution.Source: §C.3 · p. 18
  • For medical grading, the evaluator compares a model response with a reference answer and judges medical accuracy and completeness rather than style or verbosity.The prompt treats a response as correct if it contains the reference’s key medical information, even with different phrasing or additional correct details.Source: §C.3 · p. 19
  • The medical grader must output only one of two labels: CORRECT or INCORRECT.Source: §C.3 · p. 19

Related papers

Loading 2610.02140v1…