PaperlayerView exact revision on arXiv

Source-linked AI summary

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning

Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, Samet Oymak

arXiv:2506.17211v1cs.LG

TL;DR

Small language models can fail under SFT followed by RL when expert traces exceed their expressive capacity or successful initial rollouts are too unlikely. The paper introduces BREAD, which uses partial expert anchors and branched rollouts to densify rewards and progressively learn reasoning. BREAD matches or surpasses SFT plus vanilla GRPO with substantially less trace data and training compute, including on problems where those baselines fail.

  • Problem

    SFT plus RL can fail for SLMs when powerful-model traces are too difficult to imitate and sparse rewards prevent effective learning.

  • Method

    BREAD unifies SFT and RL through partial expert guidance, adaptive episode-anchor search, and branched rollouts that progressively become self-generated.

  • Results

    BREAD outperforms vanilla GRPO by more than 15% final accuracy and reaches the best-baseline accuracy in 25% of the training steps.

  • Takeaways & Limitations

    BREAD helps SLMs learn difficult reasoning problems that ordinary SFT and RL baselines fail to solve, while using a small fraction of expert traces.

  • Takeaways & Limitations

    The method focuses on SLMs and assumes access to a strong expert or successful traces; partial expert hints may still fail to produce reward.

Abstract

from arXiv · show

Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. The standard training approach combines a supervised fine-tuning (SFT) stage, often to distill capabilities of a larger model, followed by a reinforcement learning (RL)stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Under a suitable theoretical model, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization has exponentially small likelihood of success. To address these, we introduce BREAD: a GRPO variant that unifies the SFT and RL stages via partial expert guidance and branched rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3 times. Importantly, we demonstrate that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branched rollouts and expert guidance can substantially boost SLM reasoning.

1 Introduction

The paper identifies two weaknesses in standard SFT followed by RL for small-model reasoning: expert traces may be too difficult to imitate, and RL may begin from sparse-reward failures. It introduces BREAD to combine expert anchoring with branched rollouts, progressively shifting updates toward self-generated traces.

  • Motivation: SLMs remain challenged by long-context reasoning despite advances in frontier-model chain-of-thought capabilities.The paper contrasts frontier models with much smaller distilled models whose reasoning performance degrades substantially at 1.5B parameters.
  • Limitations of SFT+RL: SFT followed by RL can fail when expert traces are too difficult for the small model to express or when initialization rarely produces successful traces.The paper formalizes this failure mode in a mathematical setting and links it to difficult problems.
  • BREAD: BREAD integrates SFT and RL by anchoring optimization with expert traces while progressively increasing the share of self-generated reasoning.The method assumes access to an expert or another source of successful traces.
  • BREAD: BREAD uses branched rollouts to adapt guidance to problem difficulty, densify rewards, and build a step-by-step learning curriculum.Its workflow searches for suitable partial expert hints when ordinary rollouts fail, then updates the policy from the resulting rollouts.
  • Reported impact: BREAD matches or surpasses SFT plus vanilla GRPO while using less than 20% of correct-trace tokens and reducing training compute by about 75%.The reported contribution describes reductions in both rollout count and optimization steps.

2 Proposed Method: BREAD

BREAD combines regular rollouts with adaptive expert-anchor search, using partial traces to make downstream reasoning easier and progressively stitching successful subtrajectories into a complete solution. Theoretical analysis models why SFT+RL can fail under limited student expressivity and sparse rewards, while BREAD succeeds through shorter, denser learning steps.

  • Algorithm: BREAD first samples regular rollouts, then searches for a short expert hint when their success rate is too low.The hint is selected by adjusting its length until the resulting rollout success rate falls within a predefined range.
  • Compositionality: BREAD lowers a compositional task from completing T steps with success probability ϵ^T to completing only τ downstream steps with probability ϵ^τ.Increasing τ from 1 to T creates a curriculum that learns individual problem steps with Ω(1/ϵ) samples per step.
  • Navigation Task: In the Navigation Task, the student starts at State 0 and receives reward 1 only upon reaching State K.The Markov-chain model restricts the student to local transitions, while the expert may use larger jumps.
  • Why SFT+RL fails: SFT cannot learn an expert trace when its transitions exceed the student’s maximum jump distance, leaving RL dependent on an unlikely full successful trajectory.BREAD instead starts from an intermediate expert state, making the remaining completion task easier.
  • Formal failure result: Under the random-walk setting, SFT+RL has no impact with high probability unless the maximum trace length satisfies L ≥ Ω(K^2).The bound reflects the time required for the symmetric-random-walk initialization to reach State K.
  • Formal guarantee: BREAD succeeds in all T rounds with probability at least 1 − Te^−t while keeping trace length at most L, under the theorem’s stated length and rollout conditions.It incrementally learns successful subtrajectories and stitches them into a memorized full trace.

3 Empirical Insights into BREAD

Experiments show that vanilla GRPO and SFT can fail for small language models, while partial expert hints provide useful guidance and support adaptive curriculum learning.

  • RL-only evaluations: Vanilla GRPO learning stalls when the base model produces no high-quality traces, because reward signals are absent.Nearly half of evaluated queries lacked any high-quality trace.
  • SFT-only evaluations: SFT with difficult traces from stronger models can produce poor initialization for subsequent RL.Table 1 reports that small-model accuracy can decrease after SFT.
  • Motivation: These limitations motivate using expert traces directly during RL rather than first relying on SFT to learn them.
  • Expert hints: Partial expert hints improve accuracy, and longer appended hints provide more guidance to the model.The comparison uses Qwen2.5-3B-Instruct on the first 200 NuminaMath-CoT questions, with RL traces containing 40% hints.
  • Curriculum learning: BREAD’s Episode Anchor Search automatically selects branching points, creating a self-paced curriculum that adapts guidance to problem difficulty.Easier questions receive no or shorter hints, corresponding to smaller hint ratios.
  • Generalization: RL training with partial hints also improves accuracy on questions without hints, indicating generalization from short traces to full long traces.

4 Experiments

Across mathematical benchmarks, BREAD consistently outperforms the evaluated baselines, reaches baseline accuracy with substantially less compute, and remains effective on difficult questions using fewer expert traces.

  • Main results: BREAD outperforms all evaluated baselines in test accuracy on NuminaMath-CoT.It improves final accuracy by more than 15% over vanilla GRPO and remains better than SFT+GRPO.
  • Training efficiency: BREAD reaches the best-baseline accuracy in 25% of the training steps, corresponding to roughly 75% lower estimated compute FLOPs.The reported FLOP estimate uses forward-pass cost 2ND and backward-pass cost 4ND.
  • Hard questions: On a 500-question hard subset where baselines show little or no improvement, BREAD achieves clear upward performance with continued training.The subset was selected using pass@3 = 0, indicating no correct trace among three independent generations before training.
  • Training dynamics: Figure 8 examines test accuracy on very hard questions and the solve-none ratio, highlighting sparse-reward training stalls in vanilla GRPO and expert-trace injection in BREAD.The supplied figure text is truncated after describing EAS-based reward densification.
  • Sample efficiency: BREAD uses expert traces selectively: EAS requested traces for 36.7% of NuminaMath-CoT samples and 19.1% of Math samples in the reported training setting.Trace-budget-matched SFT baselines used the same selected subset or an equally sized random subset.
  • Cross-model results: In Table 2, BREAD reaches 0.788 on Qwen-1.5B-Instruct MATH and 0.647 on Qwen-3B-Instruct NuminaMath-CoT, exceeding the listed SFT and GRPO baselines.On Qwen-1.5B-Instruct MATH, BREAD approaches DeepSeek-R1-Distill at 0.818.

5 Related Work

Related work frames SFT and RL as complementary but insufficiently integrated for difficult SLM reasoning, where expert traces may be too complex and RL may fail to discover correct traces.

  • SFT and RL: Prior work debates whether reasoning gains arise primarily from SFT, RL, or capabilities already present in the base model.The cited literature includes claims that RL enhances reasoning, SFT mainly memorizes knowledge, and RLVR may barely increase correct-trace probability.
  • SFT and RL: The common reasoning-model pipeline uses SFT followed by RL, but hard questions can make expert solutions unsuitable for base models and leave RL unable to discover a correct trace.This motivates stronger integration of the two training stages.
  • Efficient RL and reasoning: Efficiency-oriented methods reduce training or inference costs, whereas BREAD selectively requests expert solutions and uses expert hints to increase RL training efficiency.The supplied passages contrast BREAD with prior efficient-reasoning work on supervised-signal usage and reward density.
  • Related RL methods: Unlike DAgger’s intermittent expert-action injection, BREAD inserts expert hints adaptively after failure and lets the model complete the remaining reasoning trace.The comparison also distinguishes BREAD’s branching from expert traces from Go-Explore’s branching from intermediate states.

6 Discussion and Limitations

BREAD improves GRPO by densifying rewards through branched rollouts, but its scope depends on strong expert traces and on the student benefiting from partial guidance.

  • BREAD assumes a strong expert or teacher model that provides high-quality traces for small-language-model training.The authors note that this assumption can be weakened by combining BREAD with standard RL when successful traces are available.
  • BREAD may fail when the student cannot obtain a reward signal even from partial teacher traces.In the extreme case, the student cannot complete the expert solution because it cannot follow the argument at all.
  • The authors suggest simpler, more digestible expert traces as a possible way to address this limitation.They leave training or prompting experts to produce such traces as future work.

The Supplementary Material is organized as follows:

The supplementary material provides implementation, algorithmic, theoretical, and computational details for BREAD and its expert-trace baseline.

  • The appendices cover FLOPs estimation, additional BREAD training and evaluation details, rollout reduction, expert-solution step distributions, and GRPO with Expert Trace.These topics are distributed across Appendices A–C.
  • BREAD algorithm: BREAD samples batches, searches for expert anchors, stores branched rollouts, and computes token-level advantages for policy updates.The full algorithm uses Episode Anchor Search and updates the policy from buffered question, anchor, and rollout tuples.
  • BREAD algorithm: Episode Anchor Search computes the probability that generated rollouts are correct and uses binary search to select expert-prefix boundaries when needed.The procedure returns rollouts with a selected expert prefix or recursively searches for a suitable boundary.
  • FLOPs estimation: The supplementary material estimates supervised-finetuning cost as 6ND and inference cost as 2ND, where N is parameter count and D is processed-token count.The estimate decomposes inference into a 2ND forward pass and supervised fine-tuning into one forward plus one backward phase.

A.2 Experiment Setting Details for BREAD

BREAD experiments use dataset-specific context limits, response limits, rollout settings, trace-splitting rules, and fixed hardware configurations.

  • Sequence and rollout settings: MATH experiments use 2048-token prompts and 4096-token responses, while NuminaMath-CoT uses 4096-token prompts and 8192-token responses.Both training and evaluation use the stated response limits for their respective datasets.
  • Sequence and rollout settings: Training uses eight rollouts, low-variance KL divergence, and a KL coefficient beginning at 0.00.The passage reports these as shared training settings for the experiments.
  • Expert-trace processing: Expert traces are split by sentences for MATH and paragraphs for NuminaMath-CoT, then aggregated into 10 evenly sized episodes.The splitting can also use a keyword list, and aggregation limits the number of episodes to control Episode Anchor Search cost.
  • Prompt template: Training prompts include the question and, when available, an expert hint, instructing the model to show reasoning in think tags and return the answer in boxed notation.Inference omits the expert hint according to the described template.
  • Hardware: Experiments run on eight L40S 40GB GPUs, except Qwen2.5-3B-Instruct training, which uses eight 80G H100 GPUs.

B.1 SFT training saturates and does not help after a while

The supplementary results examine SFT saturation, rollout efficiency, expert-trace length, and the relationship between GRPO with expert traces and SFT-style objectives.

  • SFT saturation: SFT training beyond the selected checkpoint does not further improve test accuracy, so SFT+GRPO starts from the 300-step checkpoint.The authors describe this checkpoint as already reaching the performance SFT can achieve.
  • Rollout efficiency: BREAD preserves accuracy with fewer rollouts, whereas vanilla GRPO loses accuracy when its rollout budget decreases from 8 to 5.On MATH with Qwen-2.5-3B-Instruct, BREAD reaches the same accuracy with five rollouts, reducing training FLOPs.
  • Expert-trace length: Most expert traces contain fewer than 20 steps, while a small number contain substantially more steps.The authors aggregate episodes because long traces without aggregation may slow training.
  • GRPO with Expert Trace: GRPO with Expert Trace reserves one of G rollouts for the expert trace while following the standard GRPO procedure for the remaining rollouts.
  • Objective comparison: GRPO with Expert Trace connects to GRPO plus an SFT loss, differing through the expert-trace coefficient and advantage-related terms.The supplementary discussion explicitly compares the objectives and lists three main differences, including these terms.
  • GRPO with Expert Trace: The expert-trace baseline assigns token-level credits using the expert trace’s advantage and the token probability ratio between old and current policies.The resulting objective can be interpreted as an entropy loss weighted by the expert-trace advantage.

D.1 Experimental Details

The experiments model pretrained small models as Markov chains with locally learnable transitions, then compare SFT, GRPO, and BREAD under controlled randomness. The BREAD procedure incrementally optimizes local transition dynamics using expert-anchored rollouts.

  • Model and assumptions: The pretrained small model is a Markov model whose learnable transitions are restricted by a maximum state jump d.Transitions are learnable exactly when the state distance is at most d.
  • Model and assumptions: Transition probabilities are concentrated on self-transitions, with Θ(ϵ) probability on other learnable transitions and zero probability on unlearnable transitions.The self-transition probability is 1 − Θ(dϵ).
  • BREAD procedure: BREAD learns incrementally by optimizing local transition dynamics from the right end of the expert trace.The procedure repeatedly samples trajectories from expert states, updates rewards, and updates the transition matrix until reaching a threshold.
  • Experimental setup: The experiments compare SFT, GRPO, and BREAD on fine-tuned Markov chains with randomly shifted high-probability transitions.For each state, one connected state is randomly assigned probability 1 − Θ(ϵ) rather than always favoring the self-transition.
  • Experimental setup: The experiments set T = K/3, vary ϵ or K across Figure 3 panels, and train for 10000 iterations with 1000 trajectories per iteration.Figure 3a fixes K = 30 and varies ϵ ∈ {0.01, 0.025, 0.05}; Figure 3b fixes ϵ = 0.05 and varies K ∈ {30, 50, 100}.

D.2.1 Proof of Lemma 1

The lemma’s proof constructs a setting where expert transitions exceed the student’s allowable jump size, making SFT ineffective and RL rewards exponentially unlikely. With sufficiently few RL samples, neither SFT nor GRPO changes the student model with high probability.

  • SFT failure: When the expert always jumps at least two states while the student can jump at most one, SFT cannot optimize the expert-trace likelihood.The supervised loss encounters zero student transition probabilities for the expert steps.
  • RL failure: A student trajectory follows a symmetric random walk, so reaching the required terminal state is governed by a random-walk hitting-time bound.The proof invokes concentration bounds for the maximum of a sum of independent Rademacher variables.
  • RL failure: With N ≤ e^(K^2/(4L)) samples, no student-generated trajectory receives reward.This follows by applying a union bound to the exponentially small probability that a trajectory reaches the target.

D.2.2 Proof of Theorem 1

The theorem proof establishes BREAD’s inductive success guarantee by solving one expert-anchored subtask at a time. Each round succeeds with exponentially improving probability as rollouts increase, yielding a high-probability successful trajectory across all rounds.

  • Inductive invariant: The proof inductively maintains that each round needs at most t traces for success probability at least 1 − e^(-t), while stitched subtasks remain length-bounded.The induction also controls the length of sub-traces generated from expert states.
  • Round construction: At round τ, BREAD starts from x_(T−τ) and uses a symmetric random walk to reach a state already contained in the constructed suffix.The new subtask’s length is bounded by the hitting time to x_(T−τ+1), and total length is bounded by summing these subtasks.
  • Failure control: The proof applies a random-walk hitting-time bound under L ≥ 5c^2K^2/T to control the probability of failure for each trace.The supplied derivation states the condition and the resulting per-trace failure bound.
  • Theorem conclusion: With t BREAD rollouts per round, the probability of success across T rounds is at least 1 − Te^(-t).This aggregates the per-round success probabilities using the stated induction.
Loading 2506.17211v1…