PaperlayerView exact revision on arXiv

Source-linked AI summary

Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning

Mohit Raghavendra, Junmo Kang, Alan Ritter

arXiv:2502.11284v1cs.LG

TL;DR

The paper asks how to allocate a fixed annotation budget between supervised and preference finetuning in LLM post-training. Across tasks, models, and budgets, it finds that pure SFT is strongest with little data, while larger budgets favor SFT followed by more preference data, with a small SFT allocation helping avoid cold starts.

  • Problem

    The paper studies how to allocate limited annotated data between SFT and preference finetuning, whose data structures and annotation costs differ.

  • Method

    The authors compare SFT and offline preference finetuning with DPO across four tasks, multiple model sizes, budgets up to 20,000 examples, and several SFT allocation ratios.

  • Results

    Below 1,000 examples, pure SFT performs best; at larger budgets, SFT followed by a higher proportion of preference data performs best, while direct PFT has a cold-start problem.

  • Takeaways & Limitations

    A small amount of high-quality SFT data combined with larger amounts of preference data may be an effective budget-conscious post-training strategy.

  • Takeaways & Limitations

    The study focuses on text-only models under 10 billion parameters and uses offline DPO and KTO as its RL-based methods.

Abstract

from arXiv · show

Post-training of Large Language Models often involves a pipeline of Supervised Finetuning (SFT) followed by Preference Finetuning (PFT) using methods like Direct Preference Optimization. Both stages require annotated data that are very different in structure and costs. We study how to optimally allocate a fixed training data budget between the two stages, through extensive experiments spanning four diverse tasks, multiple model sizes and various data annotation costs. Our findings reveal that just SFT on the base model dominates performance in low-data regimes ($<1,000$ annotated examples). With larger data-budgets, we observe that a combination of SFT and PFT, often with increasing portions allocated towards preference data yields optimal performance. However, completely eliminating SFT and running PFT directly on the base model yields suboptimal performance, described as the cold start problem on tasks like mathematics. We observe that this is due to the distribution shift arising from using DPO directly on the base model to elicit step-by-step reasoning. This limitation can be effectively addressed by allocating even a small portion ($<10$%) of the budget to SFT first, resulting in performance improvements of $15-20$% on analytical benchmarks like GSM8k. These results provide actionable insights for researchers and practitioners optimizing model development under budget constraints, where high-quality data curation often represents a significant portion of the total costs of model development.

1 Introduction

LLM post-training commonly combines SFT and preference-based methods, but fixed annotation budgets force a choice about how much data to allocate to each stage. The paper studies this allocation problem because SFT demonstrations and preference judgments differ in cost and supervision.

  • SFT and RL-based methods are commonly applied sequentially after pre-training, while recent work raises questions about whether extensive SFT is necessary.Preference finetuning can use offline preference data without training a reward model.
  • Fixed annotation budgets create a trade-off between collecting SFT demonstrations and preference-annotated data for LLM post-training.SFT provides direct instruction-response supervision, whereas preference data uses comparative judgments.
  • The study asks what SFT-to-PFT ratio is optimal for a given data budget.

2. Why is SFT a necessary precursor to PFT?

Across four tasks, model sizes, budgets, and annotation-cost configurations, the study evaluates how to allocate resources between SFT and PFT. It finds that SFT is especially important before preference tuning in low-data and cold-start settings, while larger budgets favor hybrid strategies emphasizing preference data.

  • The analysis spans Summarization, Helpfulness, Instruction Following, and Grade School Math across budgets from 100 to 20,000 examples and more than 1,000 finetuning runs.
  • SFT outperforms alternative allocations below 1,000 annotated examples, while combined SFT and preference tuning becomes more beneficial with larger datasets.
  • Allocating a higher preference-to-SFT ratio, such as 3:1, yields the best performance while remaining cost effective across tasks, models, and budgets.
  • Even minimal SFT data—less than 10% of the budget—substantially benefits preference finetuning applied directly to the base model.
  • Under budget constraints, the supported strategy is a hybrid pipeline combining a small amount of high-quality SFT data with a larger preference-data allocation.

2 Experimental Design

The study evaluates how to allocate fixed annotation budgets between supervised and preference finetuning across tasks, models, and data sources. It formalizes both objectives and compares their use in a controlled multi-task experimental setup.

  • 2.1 Problem formulation: The study compares SFT and PFT under fixed data-annotation budgets, seeking cost-effective allocation ratios while restricting analysis to models below 10 billion parameters.It assumes prompts are available, annotation consists of generating responses, and training labor costs are equal across methods.
  • 2.1 Problem formulation: The task-specific datasets and evaluation benchmarks are documented separately to support comparisons across the four post-training goals.Table 1 summarizes datasets, while Table 2 summarizes benchmarks and evaluation metrics.
  • 2.2 Supervised Finetuning: SFT minimizes negative log-likelihood for prompt-response pairs sampled from the supervised dataset.The model is trained to increase the probability of response y given prompt x.
  • 2.3 Preference Finetuning: PFT uses offline preference data and DPO to compare chosen and rejected responses without iterative reward-model training.The DPO objective uses a policy, reference policy, preference-strength parameter β, and prompt-response preference triplets.
  • 2.4 Models and Post-Training Data: The evaluation covers Helpfulness, Summarization, Instruction Following, and Grade School Mathematics using Qwen2.5-7B model families, with SFT and PFT each run for two epochs using LoRA.When comparing methods on a task, training data were matched as either both human-generated or both synthetically generated.

3 Analysis of SFT and PFT

Across fixed data budgets, pure SFT is strongest below 1,000 examples, while larger budgets favor hybrid training with more preference data after initial SFT. Direct PFT on the base model is generally weaker, but a small SFT allocation mitigates cold-start failures, especially for mathematical reasoning; cost ratios further shift the preferred allocation.

  • 3.1 For a given data-budget, what ratio should we allocate towards SFT and PFT?: 5,000 examples with 25% SFT matched 20,000 examples with 75% SFT on Summarization, Helpfulness, and Grade School Math.Optimal allocation can therefore outperform substantially larger but suboptimally allocated budgets.
  • 3.1 For a given data-budget, what ratio should we allocate towards SFT and PFT?: Below 1,000 examples, pure SFT performed best, whereas larger budgets consistently favored increasing the preference-data share.The study evaluates SFT ratios from 1.0 to 0.0 across budgets up to 20,000 examples.
  • 3.2 Why is SFT a necessary precursor to PFT?: SFT improved consistently with more data across model sizes, while direct PFT improved only in large-data regimes and mainly for larger models.Direct preference finetuning showed little or no improvement on smaller models across tasks.
  • 3.2 Why is SFT a necessary precursor to PFT?: Less than 10% SFT before preference finetuning substantially improved mathematical performance, with smaller gains on stylistic tasks for both DPO and KTO.The experiments compared SFT ratios of 0.1, 0.01, and 0.0 under equal total budgets.
  • 3.2 Why is SFT a necessary precursor to PFT?: Direct PFT encouraged some multistep reasoning on GSM8k but failed to teach the exact required format or meaningfully improve performance.The base model often produced short answers without the structured reasoning format required by the training data and system message.
  • 3.2 Why is SFT a necessary precursor to PFT?: A small SFT stage aligns the reference model’s response style, making it more compatible with subsequent DPO finetuning.The authors attribute the benefit to reducing the mismatch between the base model’s behavior and the required step-by-step format.
  • 3.3 How would the optimal allocation change under different costs of annotating SFT and PFT data?: Under most annotation-cost structures, allocating more budget to preference data after initial SFT was beneficial.SFT was especially beneficial at smaller budgets or when SFT annotation was much cheaper than preference annotation.

4 Related Work

Prior work studies training trade-offs under fixed budgets, but this paper focuses on allocating constrained post-training data between SFT and preference finetuning.

  • Fixed-budget studies have examined pretraining versus finetuning and finetuning versus distillation, while this work examines the SFT-PFT allocation trade-off.Post-training data requirements create significant costs, motivating allocation analysis under constrained budgets.
  • Related research also analyzes isolated data or compute costs, preference-data quality, and SFT-PFT effects on forgetting, generalization, alignment, and reference-model strength.

5 Conclusion

The paper studies SFT and PFT under fixed training-data budgets across tasks, model sizes, and algorithm choices. It finds that optimal allocation depends on budget regime, with small SFT allocations helping prevent direct-PFT cold starts.

  • For budgets below 1,000 examples, SFT is superior, whereas budgets above 10,000 favor SFT followed by PFT with more preference-data allocation.
  • Direct PFT on the base model exhibits a cold-start problem across every studied task, while as little as 10% SFT data can sometimes provide substantial benefits.
  • On some analytical reasoning tasks such as mathematics, even the best SFT-PFT allocation yields only modest improvements over SFT alone.

6 Limitations

The study uses offline DPO and KTO as simplified RL-based baselines and relies partly on GPT4o-generated data and evaluations. It also restricts models to under 10 billion parameters.

  • Offline DPO and KTO avoid reward modeling and iterative finetuning, simplifying comparison with SFT in implementation effort, compute costs, and annotation effort.
  • GPT4o is used for synthetic data generation and for evaluating Summarization and Helpfulness, which may introduce inherited model biases.
  • Models are limited to under 10 billion parameters because larger-scale experiments would be extremely compute intensive and could introduce compute-data trade-offs.The authors leave scaling trends for larger models and compute-data trade-offs for future work.

A.1 Ablation studies

The ablation studies test whether findings hold across model families and sizes, covering Llama3 and Qwen2.5 variants from approximately 1B to 8B parameters.

  • Experiments span Llama3 models at 8B, 3B, and 1B parameters and Qwen2.5 models at 7B, 3B, and 1.5B parameters.

A.2 Data Processing and Benchmarks

The experiments use separate datasets and prompts for SFT and PFT, with standardized training and evaluation settings across model families.

  • Prompts were not reused between SFT and PFT datasets because early experiments showed diminishing performance with reuse.
  • Fine-tuning uses the Unsloth framework and the chat-ml chat template.
  • Figure 9 compares performance across different SFT-PFT ratios for models in the Llama3 family.
  • Experiments use single-turn dialogues and 0-shot evaluations.
  • The study subsamples provided training splits across datasets and benchmarks for different data budgets.

Summarization

The summarization experiments use separate Reddit datasets for supervised fine-tuning and preference fine-tuning.

  • SFT uses the Reddit TL/DR dataset.
  • PFT uses a Reddit comparison dataset containing summarization preferences.

Instruction Following

This section details the datasets, synthetic-data construction, evaluation procedures, annotation and finetuning costs, and training settings used for instruction-following experiments. It also examines how minimal SFT affects cold-start DPO response length and reasoning quality, including an illustrative GSM8k comparison.

  • Grade School Math: The study evaluates grade-school math models on the GSM8k test set and instruction-following models on IFEval, using task-specific datasets and prompts.GSM8k prompts are drawn from the original training set for synthetic data generation, while evaluation uses the GSM8k test set.
  • A.2.1 Summarization: Summarization experiments use filtered Reddit TL;DR and Reddit Summarization Comparison datasets, evaluating win-rate against reference summaries on 500 SFT examples with GPT4o.The evaluation follows Rafailov et al. (2024).
  • A.2.2 Helpfulness: Helpfulness experiments mix HelpSteer1 and HelpSteer2 prompts, selecting highest-average responses for SFT and highest- versus lowest-average responses for preference pairs.Evaluation uses win-rate on 500 randomly selected Anthropic Helpfulness and Harmlessness test prompts with GPT4o.
  • A.2.3 Instruction Following: Instruction-following models use Tulu3 Persona-guided SFT and Preference datasets and are evaluated on IFEval.The datasets are attributed to Lambert et al. (2025), and evaluation uses IFEval (Zhou et al., 2023).
  • A.2.4 Grade School Math: Grade-school math lacks suitable in-distribution preference data, so the study synthetically generates SFT and preference data from GSM8k-related prompts.Synthetic SFT retains responses whose final answers were verified as correct, while preference responses come from a llama3.1-8B checkpoint finetuned on 1,000 SFT math responses and scored with GPT4o.
  • A.2.4 Grade School Math: The synthetic and original GSM8k datasets show similarly scaling performance, despite a small performance difference.The comparison evaluates whether synthetic data causes drastic degradation relative to the original dataset.
  • A.2.4 Grade School Math: Math preference annotation uses a style-and-correctness rubric that scores step-by-step solutions from 1 to 5, rewarding both correct answers and exact formatting.A score of 5 requires correct steps and a final line containing only the final numerical answer.
  • A.3 Finteuning compute v/s data annotation costs: Annotation costs substantially exceed finetuning costs: L40S finetuning costs about $0.00001 per example, versus $0.5-$1.0 for human annotation and $0.01-$0.001 for LLM annotation.These estimates imply that obtaining high-quality human or LLM annotations is significantly more expensive than finetuning a capable model.
  • A.4 Training hyperparameters: The appendix reports the SFT and PFT hyperparameter settings in Tables 3 and 4, alongside LoRA settings in Table 5.The supplied passages identify these tables but do not provide their individual parameter values.
  • A.5 Average model response length from cold-start models: Direct DPO on the base model produces much longer responses that often become multi-step rambling reasoning attempts.This behavior is observed for cold-start models and illustrated in the appendix’s example responses.
  • A.5 Average model response length from cold-start models: Minimal SFT before DPO yields more focused reasoning aligned with the expected format and average training-response length.The passage contrasts this behavior with direct DPO on the base model.
  • A.6 Illustrative example responses of models: In the GSM8k example, the ground-truth boots answer is 104 dollars.The problem states that one pair of heels costs $33, the other costs twice as much, and the boots cost five dollars more than both heels together.
  • A.6 Illustrative example responses of models: Base Model + SFT 1000 examples produces a structured four-step solution but incorrectly answers 94 instead of 104.Its response follows multistep reasoning and the exact format, yet uses the wrong final arithmetic sign.
  • A.6 Illustrative example responses of models: With Base Model + 10000 PFT examples, the response gives repetitive multistep steps, repeatedly states the boots and two pairs of heels total 38 + 0, and follows exact format.It loosely follows multistep reasoning while defining x as the cost of the more expensive pair of heels.
  • A.6 Illustrative example responses of models: Base Model + 1000 SFT + 9000 PFT examples gives the correct 104 answer through focused, formatted multistep reasoning.The response correctly computes 66, then 99, then adds 5 to obtain 104.
Loading 2502.11284v1…