PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models

Haitao Jiang, Wenbo Zhang, Jiarui Yao, Hengrui Cai, Sheng Wang, Rui Song

arXiv:2603.13985v1cs.AIcs.CL

TL;DR

Pre-trained LLMs require post-training for higher accuracy and more reliable reasoning, but SFT and RL have often been studied as separate methods. This study synthesizes their objectives, algorithms, data requirements, relationships, and applications, finding a shift toward hybrid pipelines that integrate both approaches. It further identifies open-weight, model-generated rollouts as an emerging scalable workflow.

  • Problem

    LLMs often need post-training for task-specific accuracy and reliable reasoning, while the relationship between SFT and RL remains insufficiently systematized.

  • Method

    The study unifies theoretical analysis, methodological comparison, and application evidence on SFT, RL, and hybrid post-training frameworks.

  • Results

    Applications from 2023 to 2025 show growing adoption of integrated SFT–RL training and a continued shift from API-based labeling to open-weight–generated datasets.

  • Takeaways & Limitations

    SFT provides stable offline learning, RL supports reward-driven generalization and exploration, and hybrid approaches balance training stability with adaptive behavior.

  • Takeaways & Limitations

    The survey may omit recent advances, and its paper-filtering strategy may introduce approximation bias because common benchmarks do not cover all real-world scenarios or methods.

Abstract

from arXiv · show

Pre-trained Large Language Model (LLM) exhibits broad capabilities, yet, for specific tasks or domains their attainment of higher accuracy and more reliable reasoning generally depends on post-training through Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). Although often treated as distinct methodologies, recent theoretical and empirical developments demonstrate that SFT and RL are closely connected. This study presents a comprehensive and unified perspective on LLM post-training with SFT and RL. We first provide an in-depth overview of both techniques, examining their objectives, algorithmic structures, and data requirements. We then systematically analyze their interplay, highlighting frameworks that integrate SFT and RL, hybrid training pipelines, and methods that leverage their complementary strengths. Drawing on a representative set of recent application studies from 2023 to 2025, we identify emerging trends, characterize the rapid shift toward hybrid post-training paradigms, and distill key takeaways that clarify when and why each method is most effective. By synthesizing theoretical insights, practical methodologies, and empirical evidence, this study establishes a coherent understanding of SFT and RL within a unified framework and outlines promising directions for future research in scalable, efficient, and generalizable LLM post-training.

1 Introduction

LLMs often need task-specific post-training because pre-training alone does not reliably provide accurate reasoning or complex interaction skills. This study compares SFT and RL, unifies their complementary roles, and identifies a rapid shift toward integrated post-training.

  • Pre-trained LLMs often require task-specific post-training to improve accuracy, reduce erroneous outputs, and handle new tasks.
  • SFT maximizes conditional token likelihood, whereas RL optimizes rewards derived from human or automated preference feedback.
  • SFT can teach basic Chain-of-Thought, while RL can improve step-wise correctness; combining them supports more reliable and robust reasoning.
  • The study systematically compares SFT and RL from algorithm-centric and data-centric perspectives and develops a unified framework for their integration.
  • Applications from 2023 to 2025 show expanding task domains, growing integrated SFT–RL training, and increased use of open-weight–generated datasets.

2 Background: SFT and RL

SFT adapts LLMs through expert demonstrations and language-modeling objectives, while RL updates models using reward signals from their own generations. The two paradigms differ in supervision and optimization, with SFT closely related to behavior cloning.

  • SFT adapts LLMs to specific tasks or domains using high-quality prompt–response pairs and standard language-modeling objectives.
  • SFT trains models to imitate expert behavior without explicit reward signals, making demonstration quality central to performance.
  • RL uses a manually specified or learned reward function to optimize model behavior.
  • SFT learns from expert-annotated prompt–response pairs, whereas RL updates the model from its own generations.

3 SFT and RL: Distinct Methodological Landscapes

SFT and RL research spans algorithm-centric improvements and data-centric strategies. SFT emphasizes demonstration-efficient adaptation, while RL develops alternative policy optimization, regularization, rollout selection, prompt selection, and curriculum methods.

  • LLM post-training research organizes SFT and RL into algorithm-centric training methods and data-centric data selection or synthesis.
  • SFT: SFT innovations address distribution matching, supervision noise, token-level policy signals, information gain, data mixing, and efficient instruction tuning.
  • RL: RL algorithm research modifies policy optimization through critic-free objectives, REINFORCE-style updates, and entropy or KL-based regularization.
  • RL: Data-centric RL selects informative rollouts or prompts and uses curriculum learning to reduce computation while maintaining performance.

4 Comparison and Combination of SFT and RL

SFT and RL are commonly used sequentially but share an underlying optimization structure that supports unified objectives and hybrid training. Recent methods combine offline demonstrations, online rollouts, and modified losses to balance imitation, exploration, memorization, and generalization.

  • Motivation: SFT and RL remain relatively disentangled in practice, despite growing interest in systematically comparing and combining them.RLHF typically applies SFT before RL, while methodological and theoretical integration remains comparatively underexplored.
  • Unified Objective: SFT can be formulated as a special case of RL using an indicator reward for reproducing an offline ground-truth trajectory.The indicator equals one when a policy sample exactly matches the dataset response.
  • Unified Objective: A unified post-training objective can include a proxy reward and KL regularization that limits deviation from a reference policy for stability.The reference policy is the base model, while β controls the KL penalty.
  • Hybrid Training: Hybrid methods combine offline demonstrations with online rollouts or interleave SFT and RL to balance imitation, exploration, and sample efficiency.Examples include off-policy guidance, single-stage objectives, negative-sample reinforcement, and performance-triggered switching between RL and SFT.
  • Trends: The surveyed trend from 2023 to 2025 shows increasing convergence toward hybrid SFT–RL pipelines and a shift toward open-weight-generated datasets.The reported proportions and 2025 projections derive from surveyed publications.

5 Applications

Application studies span question answering, mathematics, agentic behavior, and coding, using SFT, RL, or combinations tailored to each domain’s challenges. Common patterns include supervised reasoning demonstrations, task-specific rewards, rollout-based training, and iterative evaluation.

  • Question Answering: Question-answering methods augment reasoning with Chain-of-Thought, integrate external knowledge through RAG, and mitigate hallucinations.Approaches include SFT or RL on generated demonstrations, task-specific search rewards, response corrections, and uncertainty-aware outputs.
  • Mathematical Reasoning: Mathematical reasoning research uses SFT on curated or synthetic corpora and rollout-based methods to improve step-wise solution training.Researchers use self-generated solutions, majority voting, multiple reasoning paths, and approximations of process-level rewards.
  • Agentic Capabilities: Agentic post-training focuses on multi-step action selection and long-horizon planning using trajectory SFT, environment rewards, hierarchical RL, and surrogate outcomes.Some methods assign sentence-level rewards while updating token-level policies, whereas planning frameworks use seed plans and outcome-based rewards.
  • Coding: Coding studies improve code generation through self-instruction, modified problem constraints, open-source snippets, and multi-agent RL frameworks.Code-editing methods use human pull requests or iterative rollouts evaluated with unit tests to drive optimization.

6 Common Practice and Takeaways

The paper recommends choosing SFT and RL according to data, reward-model, and exploration conditions rather than treating them as interchangeable stages. In practice, SFT often initializes training, while RL supplies exploration and reward-driven improvement when reliable rewards are available.

  • Common Practice: Joint SFT–RL objectives can improve over standalone losses because SFT is expressible within an RL optimization framework.This relationship allows methods and techniques to transfer across paradigms.
  • Choosing Between SFT and RL: High-quality expert data generally favor SFT as the initial stage, while reliable reward models commonly motivate subsequent RL.The paper states that this SFT-then-RL practice typically achieves the highest reported performance.
  • Training Trade-offs: SFT is essential when models lack downstream-task familiarity or cannot generate enough positive samples, whereas RL may better preserve exploration and generalization.Direct RL remains vulnerable to entropy collapse and reward hacking.

7 Future Directions and Open Problems

The paper identifies sample- and compute-efficient post-training and learning under sparse or indirect rewards as unresolved challenges requiring further research.

  • Sample- and compute-efficient methodologies: State-of-the-art SFT and RL pipelines require substantial computation, high-quality data, and extensive rollout generation, creating practical and environmental concerns.Proposed responses include information-theoretic data-efficient SFT, quantization-aware training, and selective or partial RL rollouts.
  • SFT and RL under sparse or indirect reward signals: Many real-world tasks provide sparse, inconsistent, costly, or unavailable ground-truth reward signals, especially in safety-critical settings.Recent work explores verbal or otherwise indirect feedback for alignment.
  • SFT and RL under sparse or indirect reward signals: Future work should identify natural implicit supervision sources, including self-evaluation signals and user-churn behaviors, to guide SFT and RL in low-feedback regimes.

8 Conclusion

The study unifies SFT and RL as LLM post-training strategies, comparing their foundations and implementations while emphasizing their close relationship and complementary strengths.

  • SFT provides stable and efficient learning from high-quality offline datasets, whereas RL performs reward-driven optimization that enhances generalization and exploration.
  • Recent research characterizes SFT as a special case of RL, motivating hybrid approaches that balance training stability with adaptive behavior.
  • The study examines theoretical foundations, practical implementations, and applications of SFT and RL within a unified post-training perspective.

Limitations

The study’s scope is constrained by rapidly expanding literature, benchmark-oriented paper filtering, and a text-only focus that excludes some multimodal uncertainty.

  • The survey may omit recent SFT/RL advances because the field is progressing rapidly and producing a large volume of literature.
  • Benchmark-oriented paper filtering may introduce approximation bias because common benchmarks do not fully represent real-world scenarios or methodologies.
  • The study categorizes professional-domain research using benchmark-oriented searches and summarizes differences in input complexity, reasoning demands, and output characteristics.
  • The application analysis focuses on text-only tasks because multimodal settings introduce additional uncertainty in model selection and fusion techniques.

B.2 Benchmark-Oriented Paper Search

The benchmark-oriented search estimates LLM application trends from arXiv metadata using dataset mentions to assign papers across four domains, while acknowledging recall and classification limitations.

  • The search covers Computer Science arXiv preprints published from 1 January 2023 to 30 June 2025 using benchmark-oriented keyword queries.The metadata dataset contains approximately 195K papers across 2023, 2024, and the first half of 2025.
  • Twenty-six datasets spanning four domains serve as query keys, with papers assigned when any associated dataset appears at least five times.Assignments are not mutually exclusive, so cross-domain papers can count in multiple categories.
  • The method is transparent, scalable, and reproducible, using dataset mentions as a practical proxy for domain-specific research activity.
  • Higher keyword thresholds increase specificity but reduce recall, while low thresholds may inflate counts and high thresholds may exclude sparsely referenced legitimate work.Naming variations can also cause undercounting unless normalized.

B.3.1 LLMs for General QA Tasks

LLM post-training research for general QA spans sequential reasoning, hallucination management, retrieval, and mathematical reasoning, while agentic and code-oriented methods emphasize feedback, verification, and executable signals.

  • Step-by-Step Reasoning: General QA methods decompose questions into sequential sub-questions or use model rollouts to improve chain-of-thought quality.Some approaches generate sub-questions from ground-truth answers, while others create preference data from rollout quality.
  • Hallucination Management: Hallucination-focused methods use positive generations, conservative rewards, and alternative data pipelines to improve uncertainty handling during RLHF.Conservative RL assigns low scores when the model is unfamiliar.
  • LLM-based Retrieval: Retrieval-oriented post-training applies retrieval-based rewards to query generation and uses alternative criteria to extract key information from documents.Some work converts long documents into hierarchical graphs and applies RL to retrieval-related behavior.
  • Math-Based Capabilities: Mathematical reasoning methods combine mathematics-specific fine-tuning, data augmentation, distilled solution trajectories, and rollout verification.Verification models and filtered successful solutions support selection and iterative training.
  • Agentic Tasks: Agentic-task pipelines clone trajectories, filter erroneous actions, and combine SFT with RL or related objectives for multi-step execution and planning.Reported approaches include actor–reflector outputs, LLM judges, MCTS corrections, hierarchical rewards, and jointly trained objectives.
  • Code Capabilities: Code-focused methods use structured generation data, refinement rollouts, unit tests, code execution, and executable workflows to make outputs more verifiable.Code is also used as a reward signal for instruction following and as augmented data for mathematical or data-oriented tasks.

B.4 Trend Analysis Details

The trend analysis tracks publication growth, methodological convergence, model-choice changes, and practical or methodological context across LLM application studies from 2023 to 2025.

  • Rapid Growth Across Domains: Research activity increased across all major domains, with code-related studies rising from 115 in 2023 to 786 projected in 2025.QA increased from 292 to 983 projected studies, mathematics from 492 to 2,399, and agent-focused work from 100 to 261.
  • Convergence Toward Hybrid Training: Hybrid training became the most common methodology, increasing from 20.0% in 2023 to 73.8% in 2024 and 70.6% in 2025.Pure SFT fell from 73.3% to 19.1%, while RL-only methods reached 11.8% in 2025.
  • Shift from Proprietary to Open Models: Proprietary API-based model use declined from 32.2% in 2023 to 11.1% in 2025, while open-weight usage reached 25.0%.Benchmark use increased from 48.9% to 61.1%, whereas human-curated or web-scraped dataset use declined from 6.7% to 2.8%.
  • Practical Considerations: Hardware requirements vary with model size, training approach, optimizer, and other implementation choices, so community guidance should be treated as approximate.RL generally requires more memory than SFT because of rollout generation, policy gradients, and reward models.
  • Scope and Related Work: The analysis extends beyond text-only tasks through related work on multimodal learning, retrieval augmentation, hallucination, domain-specific datasets, and planning.The main analysis focuses on text-only tasks to enable clearer side-by-side comparisons and avoid additional uncertainty from multimodal model selection and fusion.
  • Analysis Procedure: Publication classification used LLM-assisted filtering followed by human verification of publication status and training-paradigm relevance.The study used a benchmark-oriented categorization procedure and specified the target training-type labels for classification.
Loading 2603.13985v1…