PaperlayerView exact revision on arXiv

Source-linked AI summary

GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification

Wangjie Gan, Miao Pan, Linbo Xi, Wenqi Zhang, Jintao Chen, Jianwei Yin, Xuhong Zhang

arXiv:2604.14258v3cs.AIcs.LG

TL;DR

SFT efficiently injects knowledge but can suffer from single-path supervision, entropy collapse, catastrophic forgetting, and gradient instability, motivating a better balance with robust generalization. The paper analyzes these dynamics and proposes GFT, which combines normalized group advantages with dynamic coefficient rectification. Across multiple model families and math-reasoning benchmarks, GFT consistently outperforms SFT-based baselines, improves data efficiency, and provides a stronger initialization for subsequent RL.

  • Problem

    SFT efficiently injects knowledge but is limited by single-path dependency, reduced exploration, catastrophic forgetting, and unstable inverse-probability weighting.

  • Method

    GFT combines Group Advantage Learning over diverse response groups with Dynamic Coefficient Rectification that bounds extreme inverse-probability weights.

  • Results

    GFT consistently outperforms standard SFT and strong SFT-based baselines across multiple model families and math-reasoning benchmarks, with higher data efficiency and stronger subsequent-RL initialization.

  • Takeaways & Limitations

    GFT offers a unified post-training paradigm that balances efficient knowledge injection with preserved exploration and robust generalization.

  • Takeaways & Limitations

    Evaluation focuses on mathematical reasoning with objective correctness, experiments cover models up to 8B parameters, and response-group construction adds overhead over standard SFT.

Abstract

from arXiv · show

Large language models are typically post-trained using supervised fine-tuning (SFT) and reinforcement learning (RL), yet effectively unifying efficient knowledge injection with robust generalization remains challenging. In this work, we provide a training-dynamics analysis showing that SFT can be interpreted as a special case of policy gradient optimization with an extremely sparse implicit reward and unstable inverse-probability weighting, which together lead to single-path dependency, entropy collapse, and gradient explosion. Motivated by this diagnosis, we propose Group Fine-Tuning (GFT), a unified post-training framework that addresses these intrinsic limitations through two mechanisms: Group Advantage Learning, which constructs diverse response groups and derives normalized contrastive supervision to alleviate reward sparsity, and Dynamic Coefficient Rectification, which adaptively bounds inverse-probability weights to stabilize optimization while preserving efficient knowledge injection. Experiments demonstrate that GFT consistently surpasses SFT-based methods and yields policies that integrate more smoothly with subsequent RL training.

1 Introduction

The paper diagnoses SFT’s weak generalization as arising from single-path supervision and unstable optimization, then proposes GFT to combine diverse group-based learning with stabilized updates. Across benchmarks, GFT consistently outperforms SFT and strong SFT-based baselines while improving subsequent RL initialization.

  • 1 Introduction: SFT’s strict imitation can overwrite pretrained representations, causing catastrophic forgetting and degraded out-of-distribution generalization.The paper links this pattern to systematic regressions relative to the base model.
  • 1 Introduction: SFT also narrows the policy to a demonstration manifold, reducing entropy, solution diversity, and exploration.This restriction contributes to diminished gains when RL follows SFT.
  • 1 Introduction: SFT can be interpreted as RL with a sparse reward restricted to the exact expert trajectory and an unstable inverse-probability weight 1/πθ(y|x).These dynamics produce single-path dependency, entropy collapse, gradient explosion, mechanical memorization, and overfitting.
  • 1 Introduction: GFT combines Group Advantage Learning over diverse response groups with Dynamic Coefficient Rectification for adaptive importance-weight stabilization.The response groups combine model samples, expert demonstrations, and teacher outputs; rectification uses a dynamic threshold to suppress extreme weights.
  • 1 Introduction: GFT consistently outperforms standard SFT and strong SFT-based baselines across multiple model families and math-reasoning benchmarks.It also provides a stronger initialization for subsequent RL, mitigates catastrophic forgetting, and preserves output diversity.

2 Preliminaries

The paper reframes SFT as a sparse-reward, importance-weighted RL special case and addresses its exploration and optimization limitations with group advantages and coefficient rectification. GFT uses diverse candidate trajectories, normalized relative rewards, and bounded low-probability-token updates.

  • 2 Preliminaries: SFT increases the likelihood of expert responses without explicitly considering alternative outputs, unlike RL, which evaluates samples generated by the current policy.This distinction motivates the paper’s reinterpretation of SFT through an on-policy expectation with importance sampling.
  • 2 Preliminaries: Group Advantage Learning constructs multiple candidate trajectories per query from expert, teacher, and self-generated responses.Each response receives a rule-consistent scalar reward, enabling learning from diverse reasoning paths.
  • 2 Preliminaries: Within-group reward normalization creates a relative contrastive signal that unifies imitation, distillation, and self-improvement.The mean and standard deviation center and scale rewards, while ϵ > 0 ensures numerical stability.
  • 2 Preliminaries: Dynamic Coefficient Rectification bounds inverse-probability weights when token probabilities are low, suppressing gradient explosion while preserving updates for confident predictions.The threshold τ distinguishes low-confidence tokens from predictions whose gradients remain unchanged.
  • 2 Preliminaries: The final GFT gradient combines group-wise advantage weighting with coefficient rectification to provide contrastive supervision and bounded updates.This directly targets SFT’s single-path dependency and low-probability-token instability.

4 Experiments

Experiments show that GFT improves data efficiency, preserves general capabilities, supports downstream RL, and benefits from the complementary roles of GAL and DCR.

  • Main Results: With only 10k training examples, GFT matches or surpasses baselines trained with 100k examples, demonstrating strong data efficiency.The gains change only marginally when distillation data are mixed in, indicating they are not primarily driven by additional distilled traces.
  • Main Results: GFT consistently improves performance across model scales and families, with more uniform gains across diverse math evaluations.For smaller heterogeneous models such as Llama-3.2-3B, GFT without mixing surpasses mixing strategies.
  • Ablation Study: Removing GAL most sharply reduces performance on Olympiad, whereas removing DCR primarily harms Minerva robustness, confirming their distinct contributions.GAL supplies group-based contrastive feedback for complex reasoning, while DCR rectifies gradient explosion and stabilizes optimization.
  • Ablation Study: Removing DCR causes severe training volatility, while removing GAL produces slow, suboptimal convergence; combining both yields efficient and stable optimization.These learning-dynamics patterns corroborate the component-specific performance effects observed in the ablations.
  • Compatibility with SFT and RL: GFT improves compatibility with subsequent RL by preserving rollout support and stabilizing updates, helping GFT + GRPO outperform SFT + GRPO on harder benchmarks.SFT remains useful for alignment and formatting initialization, while GFT maintains higher-entropy behavior for stronger downstream exploration.
  • Catastrophic Forgetting Analysis: GFT largely preserves base-model reasoning capabilities, whereas SFT causes substantial forgetting and the largest policy shift from the base model.Average KL divergence is used as a proxy for distributional drift and forgetting; GFT remains closer to the base policy than SFT.
  • Diversity of GFT: GFT achieves the highest Pass@128 and Pass@256 across benchmarks by combining diverse teacher and student trajectories with within-group reward comparisons.This teacher–student gap correction preserves diversity while encouraging higher-reward reasoning modes.

5 Conclusion

The paper analyzes SFT as a special case of RL and proposes GFT to address its exploration and optimization limitations. GFT combines contrastive diversity with bounded updates to balance knowledge injection and generalization.

  • SFT’s single-path dependency restricts exploration, while gradient explosion causes optimization instability.
  • GFT uses Group Advantage Learning to enhance response diversity through contrastive supervision.
  • GFT uses Dynamic Coefficient Rectification to prevent extreme weight updates and stabilize optimization.
  • Experiments show that GFT balances efficient knowledge injection with robust generalization.

6 Limitations

The evaluation and implementation of GFT remain bounded by task type, preparation cost, and model scale. The authors identify open-ended rewards, overhead, and larger models as areas requiring further study.

  • Evaluation focuses on mathematical reasoning with objective correctness, leaving extension to open-ended tasks with subjective rewards unresolved.
  • Response-group construction adds marginal data-preparation overhead compared with standard SFT, although it costs substantially less than online RL.
  • Experiments cover models up to 8B parameters, so validation on 70B+ models remains future work.

A Derivation: Viewing SFT as a Special Case of On-Policy RL

This derivation formulates SFT over expert demonstrations and shows how its objective and gradient relate to the current policy. The key distinction is that SFT’s expectation is taken over expert data rather than policy-generated samples.

  • The derivation treats SFT as training a policy model to imitate expert demonstrations from a dataset D = {(x, y∗)}.
  • The standard SFT objective is defined as the negative log-likelihood of the expert-provided output.
  • The SFT gradient increases the likelihood of expert responses without explicitly considering alternative outputs.
  • Unlike on-policy RL, the SFT gradient expectation is taken over the expert data distribution rather than samples generated by the current policy.

A.2 Importance Sampling Reformulation

The appendix reformulates the SFT gradient using importance sampling by relating the expert and policy-induced joint distributions. This recovers an equivalent on-policy formulation.

  • The expert data distribution and the current policy induce joint distributions that share the same marginal distribution over inputs.
  • Importance sampling rewrites the SFT expectation under the policy-induced distribution Q(x, y).
  • For deterministic demonstrations, the expert conditional distribution reduces to a Dirac delta before substitution into the gradient expression.
  • The reformulation recovers the equivalent on-policy formulation presented in the main text.

A.3 Reinforcement Learning Interpretation

The paper derives a reinforcement-learning interpretation of SFT by rewriting its gradient under the model policy. This shows SFT as an on-policy update with an extremely sparse expert-matching reward and high-variance importance weighting.

  • The implicit reward is one only when a sampled output exactly matches the expert demonstration.Learning therefore depends on trajectories coinciding exactly with expert responses.
  • The importance weight corrects for sampling from the model policy instead of the expert distribution.This correction introduces inverse-probability weighting into the SFT-as-RL formulation.
  • Under this interpretation, SFT is a degenerate RL setting with extremely sparse rewards and high variance.The formulation provides a unified perspective on supervised and reinforcement-based post-training.
  • SFT’s gradient can be rewritten as an on-policy policy-gradient objective through importance sampling and a deterministic expert distribution.The derivation first expresses the SFT gradient over expert data, then changes the expectation to the model policy.

B Formulation of Group Fine-Tuning

GFT formulates post-training around response groups, standardized advantages, and dynamically rectified coefficients. Its sequence- and token-level objectives connect SFT and policy-gradient learning, with SFT recovered as a degenerate group case.

  • B.1 Sequence-Level Objective: For each query, GFT constructs a response group whose candidates receive scalar rewards and standardized group advantages.These quantities define the sequence-level objective.
  • B.1 Sequence-Level Objective: The sequence-level GFT loss uses the group responses and their advantages together with dynamic coefficient rectification.The rectification function is defined separately and incorporated into the objective.
  • B.1 Sequence-Level Objective: Taking the gradient of the sequence-level objective yields a sequence-level policy-gradient formulation.This connects the loss directly to policy-gradient optimization.
  • B.2 Token-Level Decomposition: Each response is generated autoregressively, so its sequence log-probability decomposes into token-level prediction probabilities.The token-level formulation follows from this autoregressive decomposition.
  • B.2 Token-Level Decomposition: Substituting the decomposition into the sequence objective produces the token-level GFT loss and its corresponding policy gradient.The appendix provides both the loss and gradient forms.
  • B.3 Relation to SFT and RL Objectives: When a group contains one expert demonstration, the advantage is constant and the GFT gradient reduces to the standard SFT gradient.This establishes the single-demonstration case as a limiting form of GFT.
  • B.3 Relation to SFT and RL Objectives: With diverse sampled trajectories and non-trivial advantages, GFT recovers on-policy reinforcement learning with group-normalized advantages and bounded importance coefficients.The formulation is therefore presented as a strict generalization of SFT and a stabilized contrastive policy-gradient variant.

C Evaluation Settings

The evaluation uses a broad benchmark suite to examine the trade-off between SFT’s efficient knowledge injection and RL’s robust strategy discovery. The section motivates unified post-training methods by describing limitations in both paradigms and their hybrids.

  • C Evaluation Settings: Evaluation covers 11 benchmarks spanning mathematical reasoning tasks and datasets.The suite includes AMC23, College Math, Gaokao, Math, Minerva Math, TabMWP, OlympiadBench, Mmlu Stem, Sat Math, Mawps, and Svamp.
  • Trade-off Between SFT and RL: SFT is recognized for efficient knowledge injection and cold-starting but is prone to mechanical memorization and weak out-of-distribution generalization.These properties motivate combining imitation with exploration.
  • Trade-off Between SFT and RL: RL can discover robust strategies and optimize long-term objectives, yet it is computationally expensive and struggles to learn complex reasoning from scratch without guidance.The section presents this as the complementary limitation to SFT’s generalization weakness.
  • The Synergy Dilemma in Hybrid Post-Training: Standard SFT-then-RL hybrids face a synergy dilemma because SFT overfitting creates a rigid policy that constrains exploration for subsequent RL.The stated connection is between SFT-induced rigidity and the exploration space needed by RL.
  • Single-Stage Hybrids: Mixing Imitation and Exploration: Single-stage hybrid methods attempt to balance imitation and exploration through dynamic weighting, interleaved updates, or dense verification signals.The section cites SRFT and UFT as examples of this direction.
  • Gradient-Level Stabilization and Its New Trade-offs: Gradient-level analyses motivate rectification strategies aimed at addressing instability in post-training objectives.Dynamic Fine-Tuning is cited as an example that counteracts gradient explosion by reweighting the loss.
Loading 2604.14258v3…