PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus, Bryan Kian Hsiang Low

arXiv:2609.01573v1cs.CLcs.AIcs.LG

TL;DR

The paper asks how to divide a fixed annotation budget between SFT and RL, given limited allocation theory and evidence about cross-scale transfer. It defines near-optimal allocation regions and evaluates their scaling, transferability, methodological generality, and behavior under unequal annotation costs. The regions are broad even at 2–10% tolerances, generally widen with model scale, and transfer reliably from small proxy models to large targets.

  • Problem

    Fixed-budget SFT–RL allocation lacks a principled framework and evidence establishing whether allocation choices transfer across model scales.

  • Method

    The paper replaces exact-ratio optimization with near-optimal region analysis across scales, tasks, model families, RL methods, and annotation-cost ratios.

  • Results

    2–10% tolerances produce wide near-optimal regions that generally widen with model scale and transfer reliably from small proxy models to large targets.

  • Takeaways & Limitations

    Small proxy-model experiments can identify transferable near-optimal allocations without exhaustive large-scale search across the studied settings.

  • Takeaways & Limitations

    The analysis is restricted to two-stage SFT→RL pipelines, and its transfer results are studied in an in-distribution setting rather than OOD evaluation.

Abstract

from arXiv · show

How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.

1 Introduction

The paper frames SFT–RL annotation allocation as a near-optimality problem, showing that practical allocation regions are broad, scale with model size, and transfer from small proxies to larger models.

  • Motivation: Fixed-budget allocation must balance SFT initialization against RL data for policy improvement, but the appropriate ratio depends on scale, task, and method.Too little SFT weakens the initial policy, while excessive SFT leaves less budget for preference or reward-supervision data.
  • Research gap: Existing work reports broad SFT–RL trends but does not provide a principled allocation framework or establish cross-scale transfer of the optimal ratio.Exhaustive full-scale search is also expensive because allocation changes generally require retraining subsequent stages.
  • Near-optimality framework: The near-optimal region contains allocations within a fixed tolerance of the best observed performance, replacing a single exact optimum as the practical target.The analysis emphasizes regions that remain close to peak performance rather than one potentially unstable ratio.
  • Main findings: 2–10% tolerances still yield wide near-optimal regions, which generally widen with model scale and transfer reliably from small proxy models to large targets.This supports identifying useful allocations with small-model experiments instead of exhaustive large-scale grid search.
  • Generality: The widening and transfer patterns hold across tasks, model families, and both off-policy preference-based and on-policy RL methods.The study also examines how unequal SFT and RL annotation costs shift the allocation region.

2 Preliminaries

The paper formulates sequential SFT→RL training under a fixed annotation budget and studies near-optimal allocation regions using task-specific performance measurements and multiple RL paradigms.

  • Problem formulation: A pretrained model of size N receives allocation ratio r, assigning rB to SFT and (1−r)B to sequential RL training before evaluation with performance metric P.The baseline formulation measures B in annotated samples and assumes comparable SFT and RL annotation costs.
  • Problem formulation: The point optimum is ill-conditioned because post-training performance varies with task, model family, and training method, often producing a broad plateau near the maximum.This motivates optimizing the set of near-optimal ratios rather than a single r∗.
  • Problem formulation: An ε-near-optimal region contains ratios achieving at least (1−ε) of peak performance, with ε = 5% corresponding to retaining at least 95% of peak.As ε approaches zero, the region reduces to the point-optimum formulation.
  • Supervised fine-tuning: SFT adapts a pretrained language model using curated prompt–response demonstrations and a next-token prediction loss over the demonstration dataset.The policy likelihood πθ(y | x) represents the probability assigned to target response y for prompt x.
  • RL-based post-training: The study evaluates DPO and GRPO as RL-stage methods, representing off-policy preference optimization and on-policy reward-based policy optimization, respectively.DPO uses a static preference dataset and fixed reference model, whereas GRPO samples response groups from the current policy and applies a clipped objective.

3 Analysis

The analysis defines near-optimal allocation regions on discrete SFT–RL budget grids and evaluates their width, scale transfer, and robustness to training methods and annotation-cost asymmetry. Across tasks and model families, these regions broaden with tolerance and generally with model scale, making small-proxy guidance practical.

  • Experimental setup: The study evaluates four tasks across Llama 3, Qwen 2.5, and Qwen 3 families, using allocation and budget grids for sequential SFT followed by RL-stage training.The primary setup includes models up to 14B parameters and discrete allocation ratios.
  • Sensitivity of budget allocation: A 10% tolerance retains at least 90% of optimal performance while most tasks admit near-optimal ratios spanning 55%–75% of feasible allocation space.This supports a plateau-like performance structure rather than a single sharp optimum.
  • Scaling behavior of budget allocation: At fixed tolerance, near-optimal-region width generally increases with model scale, giving larger models more flexibility in choosing the SFT–RL ratio.The widening is a fixed-tolerance result and does not by itself establish reduced intrinsic allocation sensitivity.
  • Scaling behavior of budget allocation: Near-optimal regions transfer across model scales more reliably than point optima, and modest tolerances consistently improve transferability.A wider target region increases overlap with the source region, providing a structural explanation for improved transfer.
  • Extension across post-training methods: The same qualitative pattern holds for on-policy SFT–GRPO, although fluctuations are slightly stronger than with SFT–DPO, especially on Helpfulness.The conclusion about region-based transfer holds across both off-policy and on-policy post-training.
  • Extension to cost asymmetry: With heterogeneous annotation costs, larger tolerances broaden regions and strengthen transfer; increasing SFT cost generally widens the region at a fixed tolerance.When SFT becomes relatively expensive, the exact budget share matters less and logistics or sample reuse can guide allocation.
  • Allocation procedure: A three-step proxy procedure identifies target-region ratios satisfying the selected tolerance in 94.3% of cases at ε = 5% and 97.1% at ε = 10%.The procedure uses existing 1B–8B runs across tasks and Llama and Qwen 2.5 families, avoiding exhaustive target-scale tuning.

4 Related Work

Prior work studies SFT–preference-optimization trade-offs and broader training-stage budget decisions, but this paper addresses a distinct allocation question. It also complements work on when to stop SFT by varying the SFT budget share relative to RL.

  • Budget allocation between training stages: Raghavendra et al. report that SFT dominates in low-data regimes while preference optimization becomes more favorable at scale, without characterizing allocation regions or cross-scale transfer.Their work identifies a broad trend in the optimal SFT–DPO ratio as total budget changes.
  • Complementary SFT stopping work: AESL studies when to stop SFT at a fixed SFT data budget, whereas this paper studies the SFT budget share relative to RL.The two questions are complementary rather than interchangeable.
  • Related training-stage trade-offs: Other research examines budget trade-offs across pretraining, fine-tuning, distillation, and SFT–RL method interactions rather than this allocation-region problem.These lines of work address adjacent training-stage or method-comparison questions.

5 Conclusion

This work frames fixed-budget SFT–RL allocation through near-optimal regions rather than exact optima. These regions are wide, scale with model size, and transfer across models and post-training settings.

  • Near-optimal regions contain allocations within a fixed performance tolerance of the best observed allocation.
  • 2–10% tolerances still yield wide regions that widen with model scale and transfer from small proxies to larger targets.
  • The findings hold across tasks, model families, off-policy preference-based methods, on-policy RL, and asymmetric annotation costs.
  • Small proxy-model experiments can identify transferable allocation regions without exhaustive large-scale grid search.

Limitations

The study’s evidence is bounded by its in-distribution target-task setting, model and data scales, and publicly available task-specific corpora. Several larger-scale and out-of-distribution questions remain open.

  • Target-task training setting: The experiments use task-specific training data drawn from the same distribution as evaluation, so out-of-distribution transferability remains untested.The OOD regime is described as less reliably scaling, with larger generalization gaps and frequent nonmonotonicity.
  • Model and data scale ceiling: Model coverage reaches 14B parameters, leaving Llama 3 70B and Qwen 2.5 32B–72B for future verification.
  • Model and data scale ceiling: Budgets are generally capped at 15k examples per task, with several settings near the publicly available task-specific training-corpus ceiling.
  • Model and data scale ceiling: GSM8K provides ∼7.5K SFT examples and ∼16.6K synthetic preference pairs, constraining available data in several settings.

Ethical Considerations

The work is an empirical study using publicly released models and datasets, with no human-subjects research or newly released data and models. Its compute-conscious design uses parameter-efficient fine-tuning and caps most family sweeps at 8B parameters.

  • Experiments use publicly released Llama 3, Qwen 2.5, and Qwen 3 models and publicly available task-specific datasets under their original licenses.
  • The study conducts no human-subjects research and collects or releases no new data or models.
  • LoRA is used for parameter-efficient fine-tuning, while most model-family sweeps are capped at 8B parameters for tractable academic-compute requirements.
  • The authors report no specific dual-use risks beyond those associated with publicly released LLMs and standard post-training methods.

A Experimental Details

The experiments construct separate SFT and RL-stage datasets and apply the same sources, preprocessing rules, and evaluation protocols as prior work unless stated otherwise. DPO uses preference triples, while GRPO uses prompts with task-specific reward signals.

  • Dataset construction details: Each task has an SFT dataset of prompt-response pairs and an RL-stage dataset of DPO preference pairs or GRPO prompts with reward signals.
  • Dataset construction details: DPO formats preference data as triples (x, yw, yl), where yw and yl are preferred and rejected responses.
  • Dataset construction details: GRPO reuses DPO prompts x and uses task-specific evaluation metrics as rewards.
  • Hyperparameter settings: Experiments use LoRA with rank r = 32, scaling factor α = 32, dropout 0, no bias terms, and attention-module targets.
  • Hyperparameter settings: Main experiments run with seed 42 on L40s and H200s.

B Complete Performance Curves with Different Budgets

Performance curves are unstable at small budgets but stabilize for B ≥5k, where allocation ratios generally converge to comparable downstream performance. This pattern appears across model families, tasks, and both DPO and GRPO settings, with a task-specific non-monotonic SFT pattern for Qwen summarization.

  • Performance stability: B < 5k produces noisy, occasionally non-monotonic curves, especially for small models and selected instruction-following and math settings.The relative ordering of allocation ratios can fluctuate with budget in this regime.
  • Performance stability: B ≥5k yields stabilized curves where allocation ratios converge to comparable downstream performance across Llama 3 and Qwen 2.5 settings.The figures cover SFT–DPO and SFT–GRPO allocation ratios across tasks and model sizes.
  • Task-specific behavior: Qwen summarization shows persistent non-monotonic SFT performance across model scales, first decreasing before recovering at larger budgets.The paper hypothesizes that distribution mismatch may contribute to this warmup behavior.
  • Heterogeneous annotation costs: Increasing the SFT-to-DPO cost ratio ρ narrows the spread of performance curves in Llama 3 math experiments.Figure 10 organizes panels by model size and cost ratio, with x-axis budget measured in dollar-equivalent units.

D Analysis with Additional Model Family and Model Scale

Additional experiments extend the analysis to Qwen3, Qwen2.5 up to 14B, heterogeneous costs, denser allocation grids, and count-width estimation. Across these settings, near-optimal regions generally widen with tolerance and model scale, while transferability remains robust despite occasional local fluctuations.

  • Additional settings: Qwen3, Qwen2.5 up to 14B, SFT–GRPO, and heterogeneous-cost experiments extend the analysis beyond the main sweep.The additional settings cover multiple model families, scales, tasks, and RL paradigms.
  • Scale effects: Near-optimal regions generally widen with model scale, although occasional non-monotonic contractions occur at individual scales.The 14B Qwen2.5 region is far wider than the 2B proxy, while math width can dip slightly at 14B.
  • Scale effects: Transferability remains highly robust despite local width fluctuations across the additional model scales.The reported robustness further supports using near-optimal regions for cross-scale transfer.
  • Grid validation: The 9-point allocation grid agrees with the 5-point grid in trend direction and approximate magnitude across tested model sizes and tolerances.This supports the 5-point grid as a tractable protocol for capturing qualitative scaling and transferability behavior.
  • Width estimation: Count-width analysis replicates the range-width trends: near-optimal regions widen with tolerance and with model scale under fixed tolerance.Agreement between estimators supports treating the near-optimal set as approximately contiguous on the evaluated grid.

E.3 Ablation Study 3: Multi-Seed Validation

Multi-seed and low-budget ablations test whether the main qualitative findings depend on single-seed estimates or the stable-budget restriction. The trends generally persist, though lower budgets attenuate cross-scale effects and the tolerance definition can affect widening patterns.

  • Multi-seed validation: Mean performance curves are generally stable across three seeds, and mean near-optimal-region width curves are consistent in shape.The validation covers math and summarization in the Llama 3 family.
  • Multi-seed validation: At ε = 10%, cross-scale transfer hit rates show similar qualitative trends across all three seeds, including higher hit rates at larger tolerance.Per-seed matrices are reported because hit rates are integer-valued.
  • Low-budget ablation: Including budgets below 5k preserves widening with model scale and cross-scale transferability, although the effects become weaker.The attenuation is most noticeable for instruction-following, where low-budget noise can dominate the signal.
  • Tolerance definition: Absolute tolerance largely preserves Qwen widening but dampens Llama widening, with a negative absolute slope for Llama-Math.The contrast reflects how a fixed margin spans different portions of the performance range across scales.
  • Tolerance definition: Near-optimal regions transfer better than exact optima in 3 out of 4 absolute-tolerance settings.The exception is Llama-Math, where the reported exact-optimum comparison is degenerate.
  • Proxy versus fixed ratio: A proxy search generally outperforms or matches fixed r = 0.5, especially at 5% tolerance, where results are 0.90–0.95 versus 0.20–0.60.Whether r = 0.5 suffices depends on task, model family, and statistics unavailable in advance.
  • Proxy versus fixed ratio: The proxy strategy costs approximately 200 GPU-minutes versus approximately 500 GPU-minutes for a full 5-point 8B sweep, a 2.5× reduction.The proxy plus one target run also provides a >90% guarantee of hitting the target’s 5% near-optimal region on the first try.

F Additional Results for Cost Asymmetry Setting

The cost-asymmetry analysis extends beyond the main setting: heterogeneous costs shift the near-optimal allocation region, and the same qualitative behavior appears for summarization and GPU-time budgets.

  • Additional Results for Cost Asymmetry Setting: As SFT becomes relatively more expensive, the near-optimal region generally widens at the same tolerance, increasing allocation flexibility.This pattern is reported for the Summarization task across heterogeneous cost settings.
  • Additional Results for Cost Asymmetry Setting: The framework’s transferability and widening patterns are also observed for Summarization, indicating they are not specific to mathematical reasoning.The reported extension uses the same experimental protocol as the main analysis.
  • Additional Results for Cost Asymmetry Setting: At ρ = 0.5, where DPO requires roughly twice SFT’s GPU time, the near-optimal region is essentially a single ratio at small scale.The region begins widening at 8B in the Math task with Llama models.
  • Additional Results for Cost Asymmetry Setting: Under the concentrated ρ = 0.5 compute-time regime, cross-scale transfer is trivially perfect before the region widens.The cited case study treats GPU-time budgeting as a cost-asymmetry setting.
  • Additional Results for Cost Asymmetry Setting: LoRA is used instead of full-parameter fine-tuning because the Cartesian sweep requires hundreds or thousands of post-training runs.The paper identifies computational expense as the constraint motivating this choice.

Q.2: How robust are the region width and transferability statistics to training stochasticity?

The paper addresses stochasticity and grid-resolution concerns while restricting the main analysis to a more stable budget regime. It also cautions that scale-related widening under relative tolerance may partly reflect peak-performance scaling.

  • Robustness to Training Stochasticity: A 3-seed validation is conducted on math and summarization to assess whether the reported qualitative trends are robust to seed variance.The full sweep is not evaluated across multiple seeds because doing so would multiply its already substantial computational cost.
  • Robustness to Training Stochasticity: The main analysis excludes budgets B < 5k because performance curves can be noisy and allocation-ratio rankings can fluctuate across budgets.The paper highlights instruction-following on Llama and math on Qwen as prominent examples.
  • Robustness to Training Stochasticity: Under relative tolerance, widening with model scale can be partly driven by higher absolute peak performance rather than intrinsic changes in allocation sensitivity.The paper therefore interprets widening as a fixed-tolerance phenomenon and notes that the decomposition is task- and method-dependent.
Loading 2609.01573v1…