PaperlayerView exact revision on arXiv

Source-linked AI summary

SoFT: Soft Targets for Generalizable LLM Fine-Tuning

Huihao Jing, Wenbin Hu, Shaojin Chen, Haochen Shi, Zhongwei Xie, Guijia Zhang, Yuxuan Liu, Haoyu Huang, Haoran Li, Yangqiu Song

arXiv:2609.32493v2cs.LGcs.AI

TL;DR

Multi-teacher, multi-domain fine-tuning must acquire diverse capabilities while maintaining generalization beyond training tasks. SoFT uses soft targets to couple learning from demonstrations with preserving the Base model, and achieves the best aggregate score across reasoning and agentic settings. Its results suggest that jointly controlling learning and preservation can improve mixed-data fine-tuning.

  • Problem

    Mixed-domain SFT must transfer diverse behaviors while retaining useful Base capabilities, but existing methods show varying trade-offs between in-distribution learning and out-of-distribution generalization.

    Source: §1 · p. 2

  • Method

    SoFT sets a minimum probability for each demonstrated token using the smallest KL change to Base, coupling demonstration learning with preservation through calibrated soft targets.

    Source: §1 · p. 2

  • Results

    SoFT achieves the best aggregate score in all three settings and improves 22 of 24 benchmark scores over Base.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results suggest that controlling learning and preservation together can improve mixed-data fine-tuning without teacher logits or online rollouts.

    Source: §7 · p. 9

    The experiments use published datasets, benchmark environments, and generated teacher trajectories, and aggregate scores do not establish that every capability improves.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We study supervised fine-tuning (SFT) in this setting, where students must acquire diverse capabilities while maintaining generalization beyond the training tasks. Our experiments reveal varying trade-offs between in-distribution learning and out-of-distribution generalization across SFT methods, motivating more explicit control over this balance. To this end, we propose soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model's existing capabilities. SoFT sets a minimum target probability for each demonstrated token while making the smallest KL change to the Base distribution. The resulting objective couples learning from demonstrations with adaptively weighted regularization toward the Base model. We further use domain-specific gradient budgets to control this balance and determine a probability threshold for each trajectory. Experiments on mixed-domain reasoning and agentic tasks show that SoFT achieves the best overall performance among the compared methods, with improvements in both in-distribution capability acquisition and out-of-distribution generalization.

1 INTRODUCTION

Multi-teacher, multi-domain fine-tuning must acquire diverse capabilities while preserving useful pretrained behavior. The introduction motivates SoFT as a way to couple those aims and reports gains across benchmark scores and aggregate performance.

1 INTRODUCTION

  • Mixed-domain demonstrations can include reasoning, tool decisions, and long interactions, so students must acquire diverse behaviors without losing useful pretrained behavior.Standard SFT trains toward one-hot targets on observed tokens, despite alternative valid continuations and differences in Base’s initial probabilities.Source: §1 · p. 2
  • SoFT replaces one-hot targets with minimum-KL soft targets that enforce a probability floor for demonstrated tokens while minimally changing Base.Its target couples demonstration learning with Base preservation, uses a gradient budget for control, and requires neither teacher logits nor online rollouts.Source: §1 · p. 2
  • At 7B, four of six fine-tuning baselines fall below Base in aggregate score, while SFT gains in-distribution but loses out-of-distribution.These results illustrate the acquisition–retention trade-off in mixed-data SFT.Source: §1 · p. 2
  • SoFT achieves the best aggregate score in all three settings, improves 22 of 24 benchmark scores over Base, and has about one tenth of SFT’s policy drift.The introduction also reports that stronger imitation of demonstrated tokens alone does not explain these gains.Source: §1 · p. 2

2 PRELIMINARY

The preliminary section frames SFT as learning from complete expert responses, surveys its reasoning and agentic applications, and distinguishes methods that reweight learning from those that preserve the Base policy.

2.1 SUPERVISED FINE-TUNING

  • SFT minimizes the negative log-likelihood of the complete expert response y⋆ for each query x in demonstration corpus D.The paper defines D as pairs of queries and complete reference responses.Source: §2.1 · p. 2

2.2 APPLICATIONS OF SFT

  • SFT transfers reasoning and agentic behaviors from fixed demonstrations, including intermediate reasoning, tool use, environment interaction, recovery, and task completion.Because it uses teacher outputs, trajectory SFT supports black-box distillation and reuses verified trajectories across updates.Source: §2.2 · pp. 2–3

2.3 SFT IMPROVEMENT

  • Prior improvements either reweight demonstrated-token learning or regularize movement from Base; SoFT couples these as complementary weights in one adaptive target.Earlier preservation methods generally specify regularization strength separately from the SFT signal.Source: §2.3 · p. 3

3 METHOD

SoFT builds an adaptive target from the Base distribution and demonstrated sequence, coupling demonstration learning with Base preservation. A global gradient budget sets sequence-specific probability floors, which determine token-level targets.

3 METHOD

  • SoFT constructs an adaptive target for each token from the demonstrated sequence and Base distribution, without teacher logits or online rollouts.This is an offline objective coupling demonstration learning with preservation of the Base policy.Source: §3 · p. 3

3.1 FROM A GLOBAL BUDGET TO TOKEN TARGETS

  • A global gradient budget R⋆ becomes a sequence-specific probability floor τx, which determines token-level SoFT targets.Figure 2 presents this progression from shared budget to sequence floor to token targets; its values are illustrative.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 SOFT TARGET CONSTRUCTION

  • At each teacher-forced state, SoFT chooses the distribution closest to Base that assigns the demonstrated token at least τx.The KL term discourages unnecessary changes to Base, and the floor constraint enforces the minimum probability.Source: §3.2 · p. 4

3.3 SOFT TARGET DECOMPOSITION

  • The decomposition makes demonstration-learning and Base-preservation weights complementary; tokens already above the floor keep their Base target.Other tokens receive only the extra probability needed to reach τx, and the resulting objective uses forward KL with coupled weights.Source: §3.3 · p. 4

3.4 ADAPTIVE CONTROL WITH R⋆

  • Because Base probabilities vary across demonstrations, SoFT solves for a sequence-specific floor that meets the shared gradient budget R⋆.R⋆ is the fraction of SFT’s initial gold-logit gradient retained by SoFT; the floor is obtained by scalar bisection and can be precomputed.Source: §3.4 · pp. 4–5

4 EXPERIMENTAL SETUP

The experiments compare fine-tuning methods on mixed-domain reasoning and agentic tasks under shared training conditions. Domain-specific budgets set SoFT’s learning–preservation balance, while evaluations distinguish in-distribution performance from out-of-distribution generalization.

4 EXPERIMENTAL SETUP

  • All fine-tuned methods share initialization, data, and training budget within each setting, with hyperparameters selected on held-out validation data.Source: §4 · p. 5

4.1 DOMAIN-AWARE R⋆

  • Domain-specific R⋆d budgets set the balance by domain, and each sequence uses its domain’s budget.Qwen2.5 budgets cover General, Math, Science, and Code; Qwen3-8B budgets cover Customer, Coding, Search, and Tool Interaction.Source: §4.1 · p. 5

4.2 TASKS AND EVALUATION

  • Reasoning uses 36,000 trajectories across four domains at two Qwen2.5 scales, while agentic training uses 8,252 trajectories across four capabilities with Qwen3-8B.Evaluation separates training-related ID benchmarks from cross-benchmark OOD evaluation; Table 1 summarizes suites and case counts.Source: §4.2 · p. 5, Table 1 · p. 6

5 EXPERIMENTAL RESULTS

Across the three settings, SoFT attains the highest aggregate performance while improving both ID and OOD means. Results also show that mixed-data training can trade gains in some tasks for losses elsewhere, whereas SoFT’s gains are broad but not universal at the benchmark level.

5.1 MAIN RESULTS ON REASONING TASKS

  • Mixed-data reasoning training can strengthen some tasks while weakening others; four 7B baselines and two 1.5B baselines finish below Base in GM.At 7B, Confidence’s GM is 60.24 versus Base’s 62.91 despite leading APPS and tying for the best Science score.Source: §5.1 · p. 6
  • At 1.5B and 7B, SoFT has the highest reasoning GM, improving all eight scores over Base at 1.5B and seven at 7B.At 7B, DFT remains strongest on Math and Confidence on APPS, while SoFT’s broader gains yield the highest aggregate score.Source: §5.1 · p. 6

5.2 RESULTS ON AGENTIC TASKS

  • Agentic mixed-data training also produces trade-offs: SFT leads on TMax and BigCodeBench but falls below Base on τ 2 ID and BFCL V4.Confidence and L2-SP also finish below Base in GM.Source: §5.2 · p. 7
  • SoFT improves seven of eight agentic scores over Base and reaches the highest GM, 36.93.Its τ 2 ID score rises from 48.33 to 65.00 and OOD from 20.67 to 48.33; LiveCodeBench dips from 39.05 to 38.04, while SFT leads on TMax and BigCodeBench.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 SHARED PATTERNS ACROSS TASK FAMILIES

  • At 8B, SFT raises the four-benchmark ID geometric mean from 23.95 to 27.20 but lowers the OOD mean from 37.84 to 36.67.Source: §5.3 · p. 7
  • SoFT achieves the highest GM in all three settings and raises both ID and OOD means in every setting.It improves 22 of 24 scores over Base, matches one, and falls slightly below on one.Source: §5 · p. 6, §5.3 · p. 7

6 EXPLORATORY ANALYSIS

Across three scales, SoFT achieves the largest benchmark gain with about one tenth of SFT’s policy drift. The analyses show that stronger demonstrated-token uptake does not guarantee better task performance, while a shared gradient budget produces similar relative target adjustments across domains.

6.1 RQ1: PERFORMANCE GAIN VERSUS POLICY DRIFT

  • SoFT has the largest GM gain in each setting, with relative policy drift about one tenth of SFT’s and its peak gain at 8B.Policy drift is measured as KL(π0∥πθ) on held-out teacher-forced prefixes and normalized by SFT’s KL in the same setting.Source: §6.1 · p. 7, Figure 3 · p. 8
  • SoFT’s output-token frequencies are closer to Base than SFT’s at both reasoning scales, and its response length is closer on 15 of 16 benchmarks.Mean Jensen–Shannon divergence is 0.044 versus 0.090 at 1.5B and 0.065 versus 0.169 at 7B; lexical similarity and task-success retention can differ, as L2-SP illustrates at 7B.Source: §6.1 · pp. 7–8, Figure 4 · p. 8

6.2 RQ2: REALIZED DEMONSTRATION LEARNING

  • At 1.5B, SoFT has the smallest realized acquisition (7.2%) and highest GM (54.17), while DFT has the largest acquisition (41.2%) and lower GM (49.40).At 7B, PriFT-mass, Confidence, and Rho-1 have acquisition ratios near 36%, while their GMs span 48.34 to 64.64; larger acquisition therefore does not imply better performance.Source: §6.2 · p. 8, Figure 5 · p. 9

6.3 RQ3: EFFECT OF THE GRADIENT BUDGET

  • At R⋆=0.5 on 1.5B, Science’s absolute target change is 0.739 nats per token versus Math’s 0.363, but domain averages cluster within 41.6–42.6% of available change.At 8B, the four agentic groups cluster within 48.4–49.9% at the same budget, making R⋆ an interpretable relative control across domains.Source: §6.3 · p. 8, Figure 6 · p. 9
  • At R⋆=0.3, positive demonstration weights apply to 16.9% of Search tokens and 19.8% of Customer tokens, with 9.3% versus 25.5% across Tool Interaction sources.As the budget rises, the solved floor activates additional positions; tokens already above the sequence floor retain their Base target, so activation reflects Base confidence rather than uniformly raising weights.Source: §6.3 · p. 9
  • The coding domain receives the largest selected budget in every setting; validation sets each domain’s learning strength, while the sequence-specific floor adapts it to Base confidence.Figure 6 presents target KL from Base as the gradient budget varies, with stars marking selected budgets.Source: §6.3 · p. 9, Figure 6 · p. 9

7 CONCLUSION

SoFT couples learning from mixed-domain demonstrations with preservation of the Base distribution. It achieves the best aggregate score across reasoning and agentic tasks in all three settings, though other methods lead on some benchmarks.

7 CONCLUSION

  • Across all three settings, SoFT has the best aggregate score, although other methods lead on some benchmarks.The results suggest that controlling learning and preservation together can improve mixed-data fine-tuning without teacher logits or online rollouts; future work will test whether this helps subsequent reinforcement learning.Source: §7 · p. 9

AI USE STATEMENT

The authors used language models as research subjects and to generate teacher trajectories, and used generative AI only to assist with parts of experimental-code implementation.

AI USE STATEMENT

  • Generative AI assisted with parts of the experimental code, but not with manuscript drafting, editing, revision, or table and figure text.The authors reviewed all experimental code and take responsibility for the methodology, code, results, claims, and manuscript.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The experiments use published datasets, benchmark environments, and generated teacher trajectories, without direct interaction with human participants. The authors note that these sources may contain errors, biases, or sensitive content.

ETHICS STATEMENT

  • Aggregate benchmark scores are not evidence that every capability improves, and source outputs or trajectories may contain errors, biases, or sensitive content.The work uses published datasets and benchmark environments and generated teacher trajectories, without direct interaction with human participants.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix details SoFT’s constrained KL target, its relationship to other objectives, gradient-budget interpretation, and practical implementation. It also specifies the training setup and release plans.

REPRODUCIBILITY STATEMENT

  • The statement describes fixed data splits and seeds, shared evaluation tasks within model-size comparisons, and planned release of code, configurations, manifests, and redistributable processed data.For third-party data that cannot be redistributed, source links and reconstruction scripts are planned.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 CONSTRAINED KL PROJECTION

  • SoFT chooses the target closest to Base that raises the demonstrated token to a probability floor, preserving Base’s relative preferences among alternatives.When Base already meets the floor, the target remains Base; otherwise, remaining probability mass is rescaled proportionally.Source: §A.1 · p. 15

A.2 SOFT TARGET DECOMPOSITION

  • The soft-target loss combines demonstrated-token learning with Base-policy preservation through complementary weights set by the probability floor.The decomposition is token-level, with the learning and preservation weights summing to one.Source: §A.2 · p. 15

A.3 RELATIONSHIP TO DFT

  • At initialization, SoFT’s demonstration-learning gradient matches token-weighted SFT, while its restoring gradient pulls the student toward Base.DFT instead uses the Base probability of the demonstrated token as its initial weight.Source: §A.3 · p. 16

A.4 INTERPRETATION OF R⋆

  • R⋆ is the fraction of the initial summed SFT gold-logit update retained by SoFT, and the same ratio holds for summed logit-gradient ℓ1 norms.The full parameter-gradient ratio can change during training.Source: §A.4 · p. 16

B.2.1 PRACTICAL APPROXIMATION

  • The implementation approximates the full-vocabulary target by retaining Base top-K tokens and the demonstrated token, then merging the remaining vocabulary into one tail bucket.This reduces target storage to O(BTK); cached Base probabilities avoid repeated reference passes, and chunking limits activation memory.Source: §B.2.1 · p. 17

B.2 FIXED-R⋆SOFT

  • The Base probabilities define each target, while bisection converts the fixed R⋆ budget into a sequence-specific floor before cross-entropy is computed.The implementation uses a frozen copy of the initial student as the reference model.Source: §B.2 · p. 17, §B.2.1 · p. 18

C TRAINING DATA DETAILS

  • Training uses one epoch and seed 42, with maximum sequence lengths of 8,192 tokens for reasoning and 32,768 for agentic tasks.Methods within each setting share data order; selected SoFT learning rates vary by model scale.Source: C TRAINING DATA DETAILS · p. 18

C.1 REASONING TASKS

The appendix documents mixed reasoning and agentic training and evaluation data, token-allocation diagnostics, and analyses of policy retention and normalized target-KL curves. It explains how data composition and Base-confidence profiles shape these diagnostics.

C.1 REASONING TASKS

  • The reasoning corpus uses 36,000 training trajectories and 800 validation examples for each Qwen2.5 student, with four task families in each ID and OOD suite.The ID suite includes math, programming, science, and commonsense riddles; the OOD suite pairs those capabilities with different question sources.Source: §C.1 · p. 18, §D.1 · p. 20
  • Reasoning-source answer verification varies: the manifest includes verified, upstream-verified, unverified, answer-agreement, and GPT-OSS rows without verified references.Length filtering checks context-window fit, while answer-quality labels are recorded separately.Source: §C.1 · p. 18

C.2 AGENTIC TASKS

  • The agentic training corpus contains 8,252 training and 824 validation trajectories, while its ID and OOD suites cover customer interaction, coding, search, and tool interaction.OOD sources include telecom and banking support, BigCodeBench, BFCL V4, and FRAMES.Source: §C.2 · p. 19, §D.2 · p. 20

E TOKEN-LEVEL ALLOCATION DIAGNOSTICS

  • The same gradient budget yields different adjustment rates across domains: non-coding agentic groups range from 16.9% on Search to 19.8% on Customer.Within Tool Interaction, Terminal and Tool Calling have adjustment rates of 9.3% and 25.5%, respectively; Base-confidence profiles shape these rates.Source: E TOKEN-LEVEL ALLOCATION DIAGNOSTICS · p. 20

F POLICY DRIFT AND BEHAVIORAL RETENTION

  • SoFT’s policy drift is lowest and Base-success retention among the highest at both reasoning scales, while drift and retention broadly move in opposite directions at 1.5B.At 7B, DFT moves far from Base while retaining many successes, whereas Rho-1 moves less and retains fewer; the drift diagnostic and benchmark retention use separate evaluations.Source: F POLICY DRIFT AND BEHAVIORAL RETENTION · p. 21, Figure 8 · p. 21

G.2 AN EXACT CANCELLATION OF TOKEN PROPORTIONS

  • With a fixed distribution of uncertain-token Base probabilities, the normalized target-KL curve is independent of the proportion of uncertain tokens, although absolute KL changes with that proportion.Increasing the uncertain-token proportion from 10% to 50% multiplies absolute KL by five while leaving the normalized curve unchanged when the distribution stays fixed.Source: §G.2 · pp. 22–23

G.3 A CLOSED FORM AND A LOW-PROBABILITY LIMIT

  • In the low-probability special case, the normalized curve approaches the budget fraction as confidence becomes sufficiently low, but remains sensitive to the probability profile.At r = 0.5, Cϵ(r) is about 0.189 for ϵ = 0.5 and 0.356 for ϵ = 0.01.Source: §G.3 · p. 23

G.4 NEARLY CERTAIN TOKENS

  • Nearly certain tokens can restore dependence on the uncertain-token proportion: even small residual contributions from many such tokens can affect the normalized curve.The high-confidence component contributes residual gradient mass and endpoint KL under the stated active-set condition.Source: §G.4 · p. 23

H.1 TRAINING OBJECTIVES AND PREDICTIVE ENTROPY

The appendix reports training-objective, predictive-entropy, gradient-norm, and learning-rate traces, then relates teacher-forced probability gains to benchmark outcomes. It cautions that optimization diagnostics do not replace task-performance comparisons.

H.1 TRAINING OBJECTIVES AND PREDICTIVE ENTROPY

  • SoFT’s reasoning loss changes little because its objective includes the entropy of the soft target as a constant offset, while objective scales differ across methods.The curves characterize optimization within runs; benchmark scores provide the performance comparison.Source: §H.1 · p. 24
  • Final reasoning entropy is higher for SoFT than SFT: 1.228 versus 1.066 at 1.5B and 1.009 versus 0.713 at 7B.Over the final 10% of agentic steps, their means are similar at 0.395 versus 0.386; entropy varies by setting and does not itself measure correctness or Base preservation.Source: §H.1 · p. 24, Figure 10 · p. 25

H.2 GRADIENT NORMS AND LEARNING-RATE SCHEDULES

  • SoFT’s mean logged gradient norm falls from 0.770 to 0.555 at 1.5B, 1.659 to 0.644 at 7B, and 1.014 to 0.178 at 8B.These logged magnitudes depend on loss scaling, trainer conventions, and fluctuations hidden by window averages.Source: §H.2 · pp. 24–25, Figure 11 · p. 25
  • SoFT uses peak learning rates of 10−5, 5×10−6, and 10−6 across the three model scales, followed by decay.Other methods use their own selected rates, and identical schedules overlap.Source: §H.2 · p. 25, Figure 12 · p. 25

I REALIZED DEMONSTRATION LEARNING

  • At 7B, methods with nearly identical aggregate acquisition ratios occupy widely separated benchmark-performance positions, so similar teacher-forced gains can accompany different task outcomes.The scalar summarizes teacher-forced probability gain; retention analysis and benchmark scores assess behavior on complete responses.Source: I REALIZED DEMONSTRATION LEARNING · p. 26, Table 6 · p. 26
  • DFT > SFT > SoFT in realized gain across all four reasoning domains at both scales; within SoFT, Code has the largest gain and the largest validation-selected budget.The domain budget and Base probability profile jointly shape the target used for these trajectories.Source: I REALIZED DEMONSTRATION LEARNING · p. 26

Related papers

Loading 2609.32493v2…