PaperlayerView exact revision on arXiv

Source-linked AI summary

Selecting Diverse SFT Traces Improves Post-RL Generalization

Dylan Zhang, Mingyuan Wu, Jinning Li

arXiv:2609.33780v1cs.LGcs.AI

TL;DR

Verified solutions can differ in how well they prepare models for RL, while common selection criteria do not directly characterize reasoning-route variety. The paper studies same-pool route selection with a rule-based fingerprint and finds that diverse selections improve post-RL coverage across puzzles and mathematics, including harder problems.

  • Problem

    Common reasoning-data selection criteria do not directly characterize whether retained solutions provide different reasoning routes, although SFT choices shape the starting policy for RL.

    Source: §1 · p. 1

  • Method

    The study compares route-diverse and route-similar selections from one pool at a fixed budget under matched training conditions, using a rule-based fingerprint of reasoning structure.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    Route-diverse selection improves post-RL coverage across puzzles and mathematics, including harder problems; on held-out RLVE environments, OLMo3-7B gains 16.9 percentage points of pass@8.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Route diversity is a practical criterion for selecting verified solutions to prepare models for RL, and the proposed selector makes it cheap to apply.

    Source: §6 · p. 11

    Fingerprints are domain-specific: OMEGA uses annotated strategy steps, while RLVE uses lexical cues and transitions.

    Source: H. Limitations and Future Work · p. 48

Abstract

from arXiv · show

Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL). We present a comprehensive study of route diversity, the variation in the sequences of reasoning steps in supervised fine-tuning (SFT) data, and propose a lightweight, rule-based fingerprint to select for it. From one pool at one budget, with matched training recipes and checkpoints, selecting diverse rather than similar routes improves post-RL problem coverage across puzzles and mathematics, including on problems harder than those seen in either training stage. In synthetic experiments, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 points on environments held out from SFT. In a single-model condition, where one model writes every candidate, diverse selection gains up to 6.2 points of mean pass@8 across 10 mathematics benchmarks. Pre-RL diagnostics suggest why: diverse SFT can produce both successful and failed attempts on more prompts despite slightly lower mean accuracy, giving group-relative RL more prompts with a learning signal. On 3 open-source corpora, our CPU-only selector, without model calls, outperforms more expensive alternatives in every comparison of mean post-RL performance. These results identify reasoning-route diversity as a practical criterion for selecting SFT data that better prepares models for RL.

1. Introduction

The section asks which verified solutions best prepare models for RL, noting that common selection criteria do not characterize route differences. It presents controlled route-selection studies and a scalable fingerprint, reporting stronger post-RL coverage and more prompts with mixed pre-RL outcomes.

1. Introduction

  • 16.9 percentage points: route-diverse OLMo3-7B SFT raises pass@8 on RLVE environments held out from SFT, with gains across seen and unseen environments and every difficulty band.The diverse model solves 1,133 questions the similar model misses, while the similar model solves 53 the diverse model misses.Source: Figure 1 · p. 2, §1 · p. 3
  • Route diversity means differing reasoning-step sequences, including for the same problem; the hypothesis is that varied practice puts correct attempts within sampling reach on more problems.Prior work reports benefits from distinct solutions, though adding them per problem can increase data volume together with variety.Source: §1 · p. 2
  • At a fixed budget, the study asks whether varied routes from one pool improve post-RL coverage, since common selection criteria do not characterize route differences.Prior studies caution that too much SFT can limit subsequent learning and generalization.Source: §1 · pp. 1–2
  • The study combines teacher-source sweeps with same-pool route-diverse versus route-similar selections, matching budgets, training recipes, evaluation, and checkpoints.Its fingerprint summarizes reasoning steps, their ordering, and path statistics to distinguish concentrated from spread-out route selections.Source: §1 · p. 3
  • 3.39 to 6.17 points: route-diverse selection improves mean pass@8 across 10 math benchmarks at three budgets when Qwen3-4B-Thinking-2507 writes every candidate.Source: §1 · p. 3
  • 54.7% versus 46.9%: the diverse OLMo3-7B checkpoint has mixed outcomes on more math prompts despite slightly lower mean accuracy, consistent with broader pre-RL learning opportunities.With binary rewards, all-success or all-failure groups have zero group-relative advantage, while mixed outcomes supply an outcome-based learning signal.Source: §1 · p. 3
  • On three released corpora, the CPU-only fingerprint selector beats random, topology, gradient-diversity, embedding, and lexical selection in every comparison of mean post-RL accuracy and pass@8.The rule-based, text-only method requires no model calls, additional generation, or gradients.Source: §1 · p. 3

2. Teacher Count as a Proxy for Route Diversity

At a fixed SFT trajectory budget, using more teacher sources improves post-RL coverage across puzzle and mathematics evaluations, including out-of-distribution problems.

Experiment setup

  • Comparisons hold the student, prompt pool, SFT trajectory budget, RL recipe, evaluation, and RL step constant while changing the SFT solutions.SFT solutions come from one or several teachers and use synthetic puzzles; RL is conducted on other puzzle or mathematics tasks.Source: §2 · p. 4

RL on a domain different from SFT

  • Twelve teachers lead in 54 of 56 benchmark, pool, and sampling-budget cells after RL on DAPO-Math-17k.For Qwen3-4B in the 16-environment pool, MATH-500 pass@1 rises from 34.14% to 65.08%.Source: §2 · p. 4

Out-of-distribution evaluation

  • On OMEGA’s out-of-distribution compositional split, every multi-teacher condition from two to five teachers exceeds one teacher at RL step 350.This holds when neither SFT nor RL trains on OMEGA.Source: §2 · p. 5

RL on a domain different from SFT

  • After RL on reasoning-gym, five teachers beat one on all four mathematics benchmarks in every pool comparison at pass@1 and pass@64.Neither SFT nor RL trains on these evaluation benchmarks.Source: §2 · p. 5

3. Selecting Verified Solutions for Route Diversity

The section selects verified solutions for route diversity using topology fingerprints, then tests whether diverse selections improve post-RL coverage and alter pre-RL reward signals.

3.1. Selecting verified routes

  • A route is a verified solution’s sequence of steps; its topology abstracts away wording, formatting, and teacher identity.The fixed-length fingerprint is computed from step annotations or trace text using fixed rules and a random projection, without model calls, new generation, or gradients.Source: §3.1 · p. 6, Figure 4 · p. 7
  • Diverse and similar datasets are selected from one candidate pool at the same target size, using farthest-point selection across clusters versus nearest-centroid selection in one dense region.RLVE environments provide verifiable answers, allowing evaluation on held-out environments and difficulties beyond those used for training.Source: §3.1 · pp. 6–7, §3.2 · p. 7

3.3. RLVE: generalization across difficulty and environments

  • At pass@8, route-diverse SFT leads by 16.9 points on environments held out from SFT, and the advantage also reaches harder-than-training problems.The diverse model solves 1,133 questions that Similar misses, while Similar uniquely solves 53; Qwen3 results also favor Diverse at both model sizes and budgets.Source: §3.3 · pp. 7–8

3.4. Analysis: Mixed Rewards Before RL

  • Before RL, OLMo3-7B has mixed rewards on 54.7% of 64 mathematics prompts with diverse SFT, versus 46.9% with similar SFT, despite slightly lower mean solve rate.On held-out RLVE questions, the diverse checkpoint also has more mixed-outcome and fewer all-fail prompts at both budgets.Source: §3.4 · p. 8
  • On RLVE, after the same RL, correct completions from diverse Qwen3 checkpoints vary more in wording than similar checkpoints, with mean bigram Jaccard distance 16.68% higher for Qwen3-4B and 15.13% higher for Qwen3-1.7B.Source: §3.4 · p. 8

3. Selecting Verified Solutions for Route Diversity

  • Selecting differing routes from an existing pool at a fixed budget needs no new generation and gives the same RL a better starting point.The diverse selection also leads on held-out OMEGA mathematics and Sokoban.Source: §3.4 · p. 8, §3.3 · p. 8

3.4. Analysis: Mixed Rewards Before RL

  • With one model generating every candidate, route-diverse SFT leads after the same GRPO at all three budgets by 3.39 to 6.17 points of mean pass@8 across ten mathematics benchmarks.The generator is Qwen3-4B-Thinking-2507, and both selections come from one pool.Source: §3.4 · p. 9
  • The post-RL coverage gap widens with sampling budget in the proposed account, but the illustrative model predicts it eventually narrows as both policies approach saturation.The observed peak gap varies by setting, occurring at intermediate difficulty for Qwen3-4B-Base and on the easiest band for OLMo3-7B.Source: §3.4 · p. 9

4. Route Selection on Released Reasoning Corpora

On three released reasoning corpora, route-diverse selection improves post-RL mathematics results and outperforms more expensive selection baselines.

Results

  • The selector beats random, topology, gradient-diversity, embedding, and lexical-feature baselines in every comparison, with relative mean-score gains of 1.2% to 10.8% at pass@1 and pass@8.Both selections use one released pool at one budget, followed by the same SFT and RL recipe.Source: §4 · pp. 9–10
  • Across all three corpora, diverse selection leads on every mathematics benchmark by 4.9 to 18.5 points of average accuracy.It also leads on all three OMEGA splits and GPQA-Diamond.Source: §4 · p. 10

Efficiency

  • On about 2.1 million solutions, selection takes about three hours on one CPU node and no GPU time, versus 64 to 232 GPU-hours for model-based baselines.The evaluated corpora are OpenThoughts3, INTELLECT-3, and Nemotron-Cascade 2.Source: §4 · pp. 9–10

5. Related Work

Related work studies how SFT prepares models for RL and how reasoning data are selected, with prior findings on divergent traces but differing methods and objectives.

Preparing a model for RL

  • Prior studies vary data timing, SFT loss, exploratory behaviors, reasoning primitives, or teachers when examining how SFT prepares models for RL.Other work compares SFT and RL in memorization, output diversity, and preservation of policy generality.Source: §5 · p. 10

Reasoning-data selection and structure

  • Data-selection work emphasizes instruction diversity, trace quality, model fit, task mixtures, or semantic breadth rather than directly measuring route differences.The section also situates this work alongside self-training, coverage selection, reasoning graphs, and path search.Source: §5 · p. 10
  • Distinct paths per problem have improved post-SFT accuracy, and divergent solutions selected at a fixed demonstration count have shown an advantage that persists after RL.The cited divergent-solution method uses a language model on each candidate, whereas route selection here reads trace text.Source: §5 · p. 10

6. Conclusion

At matched budgets, recipes, and checkpoints, route-diverse selection improves post-RL coverage; the CPU-only selector also beats alternatives across three released corpora.

6. Conclusion

  • Matched route-diverse selections improve post-RL coverage, including on problems harder than those used in either training stage.The comparisons use the same pool, demonstration budget, training recipes, and matched checkpoints.Source: §6 · p. 10
  • Across three released corpora, the CPU-only selector beats random, topology, gradient-diversity, embedding, and lexical selection in every mean post-RL comparison.Figure 8 compares relative mean-score gains at final RL checkpoint step 64.Source: §6 · p. 10, Figure 8 · p. 11
  • Before RL, the route-diverse checkpoint produces mixed rewards on more prompts despite slightly lower mean accuracy, suggesting more prompts provide a group-relative learning signal.The paper presents route diversity as a practical criterion for selecting verified solutions for RL preparation.Source: §6 · p. 11

AI Use Statement

The supplied passages describe AI use and detail the paired route-selection setup, fingerprint-based selection, evaluation, and measures of route spread.

AI Use Statement

  • Generative AI tools assisted with polishing the writing; the authors verified the content and take responsibility, and synthetic data came from open-source language models.The statement identifies writing assistance and synthetic-data sources.Source: AI Use Statement · p. 11

A. Method and Construction Details

  • Within each comparison, candidate eligibility, SFT trajectory budget, student initialization, training recipe, evaluation protocol, and checkpoint step are matched.Different testbeds may use different recipes; matching applies within each comparison.Source: A. Method and Construction Details · p. 20

A.1. Problem setup

  • The setup maps verified candidates to route fingerprints and distances, then forms fixed-budget diverse and similar sets that approximately maximize or minimize fingerprint spread.The two selections receive the same SFT and RL recipes, and post-RL differences are measured on the same held-out prompts with the same sample count.Source: §A.1 · pp. 20–21

A.2. Shared selection and training procedure

  • Fingerprints encode reasoning steps, transitions, trees, patterns, and other route features through dataset-specific feature blocks.Steps are segmented and labeled before route graphs and trees are built; feature vectors may include hashed route labels.Source: §A.2 · pp. 21–22
  • The selector allocates group quotas, clusters fingerprints, and uses greedy farthest-point sampling within clusters; the similar control keeps candidates nearest each group mean.The diverse set has the specified budget, and the two selections need not be disjoint.Source: §A.2 · pp. 22–23

A.3. Evaluation metrics and comparison units

  • Pass@k estimates coverage at a sampling budget, while matched question sets and checkpoints support paired comparisons; single-run results do not estimate training-seed variation.The paper cautions that an endpoint advantage alone does not establish a larger improvement during RL.Source: §A.3 · p. 24

A.4. Measuring topology spread

  • Within-prompt geometry measures separate the selected sets on distance and variance statistics, but pattern entropy and active-pattern count are nearly equal.These statistics characterize fingerprint geometry, not the number of semantically distinct algorithms.Source: §A.4 · p. 25

A.5. RLVE data construction

The appendix details how verified reasoning traces are fingerprinted and selected across several task settings, then reports teacher-source and task-pool sweeps. Across these comparisons, greater teacher-source variety generally improves post-RL coverage, with some benchmark and sampling-budget reversals.

A.5. RLVE data construction

  • RLVE fingerprints encode reasoning-span features and patterns; diverse selection clusters traces and chooses farthest points, while similar selection keeps centroid-nearest rows.The 50,000-row selections have about the same number and evenness of reasoning patterns, but diverse-selection solution distances are about 2.3 times larger.Source: §A.5 · pp. 25–26, Table 3 · p. 26

A.6. OMEGA data construction

  • OMEGA’s 323-dimensional topology fingerprint combines strategy-step transitions, trace position and path summaries, strategy labels, and lexical cues.After standardization and L2 normalization, diverse selection uses clustered farthest-point sampling and similar selection uses nearest-centroid sampling.Source: §A.6 · p. 27

A.7. Sokoban data construction

  • Sokoban fingerprints capture trace movement over the board, box-handling strategies, board-reconstruction behavior, and graph patterns in visited coordinates.Both selections retain 86,792 verified traces from one pool and receive the same 75-step GRPO training.Source: §A.7 · p. 28

A.8. Program-simulation data construction

  • Program-simulation candidates are checker-verified; the multi-model and single-model corpora each contain 10,000 trajectories and receive matched SFT and GRPO training.Evaluation uses 500 held-out prompts across seven difficulty levels, with 64 samples per prompt.Source: §A.8 · p. 28

B. Supporting Construction Sweeps

  • After RL on OMEGA or reasoning-gym tasks, multi-teacher SFT beats one teacher on OMEGA compositional coverage across reported budgets and pools.For RL on OMEGA, all 56 comparisons are positive; for RL on reasoning-gym tasks, the OOD evaluation lies outside both training stages.Source: B. Supporting Construction Sweeps · pp. 30–31, Figure 9 · p. 30, Figure 11 · p. 31
  • Before RL, Enigmata coverage generally rises as the SFT pool grows from 2 to 92 tasks, while twelve teachers reach 35.2% pass@64 versus 28.0% for one.The task-pool sweep has diminishing gains and local reversals; the teacher comparison uses twelve solutions per prompt in both conditions.Source: B. Supporting Construction Sweeps · p. 30, Figure 10 · p. 31

B.1. Teacher-source count after mathematics RL

  • At pass@1, all 32 multi-teacher mathematics comparisons beat one teacher; at pass@64, 27 are higher, one ties, and four are lower.The mean pass@64 gain is 4.1 points, while all comparisons remain positive through pass@8.Source: §B.1 · pp. 32–33, Figure 13 · p. 33

B.2. Qwen3-4B mathematics evaluation

The Qwen3-4B evaluation compares matched one- and twelve-teacher SFT conditions after mathematics RL. The section also specifies the pre-RL reward diagnostic and released-corpus training setup.

B.2. Qwen3-4B mathematics evaluation

  • Twelve teachers score higher than one in all 16 cells; the largest gain is 30.9 points of pass@1 on MATH-500 in pool 16, from 34.14% to 65.08%.Across seven sampling budgets, twelve teachers lead in 54 of 56 benchmark, pool, and budget cells.Source: §B.2 · p. 33, Table 5 · p. 34

C.1. Dolci-Think selection for reward diagnostics

  • The pre-RL diagnostic compares matched OLMo3-7B checkpoints on eight responses to each of 64 mathematics prompts, after selecting two 100,000-row datasets from one verified pool.The fingerprint combines 631 raw features, projected to 96 dimensions for cluster-based diverse selection or centroid-nearest similar selection.Source: §C.1 · p. 35

C.2. Released-corpus construction and training

  • Released-corpus training uses OLMo3-7B with one-epoch SFT followed by a shared 64-step GRPO recipe, with both conditions evaluated at the same RL checkpoint.OpenThoughts3 and INTELLECT-3 use math, code, and science quotas; Nemotron-Cascade 2 allocates its full budget to mathematics.Source: §C.2 · pp. 35–36

C.3. Released-corpus results by benchmark

Released-corpus comparisons show that Diverse and Similar do not perform uniformly across capabilities, while the appendix describes matched-budget selection baselines and how benchmark-level gains are calculated.

C.3. Released-corpus results by benchmark

  • INTELLECT-3 Enigmata favors Similar by 1.3 points in mean accuracy and 4.25 points in pass@4, while IFBench also declines on both OpenThoughts3 and INTELLECT-3.OpenThoughts3 IFEval ties; these comparisons use one training run per condition.Source: §C.3 · p. 36, Figure 15 · p. 37

C.4. Released-corpus selection baselines

  • Selection baselines draw the same 100,000-row budget from the same pool and quotas, then receive matched SFT and GRPO training.The baselines vary the selection criterion, including random, topology, embedding, lexical, and gradient-diversity approaches.Source: §C.4 · pp. 36, 38

C.5. Benchmark-level relative gains

  • Benchmark-level cells report relative percentage gains over a named baseline, whereas the aggregate is the relative change in the equally weighted mean benchmark score.The aggregate is neither question-weighted nor an average of the printed cell-level relative percentages.Source: §C.5 · p. 39

D. Extended Results

The extended results compare diverse and similar SFT across RLVE, OMEGA, Sokoban, and program simulation. Diverse selection leads across the reported RLVE difficulty levels and testbeds, including on problems harder than either training stage.

D.1. RLVE selection budgets (Qwen3)

  • The RLVE lexical-diversity analysis averages pairwise bigram Jaccard distance among eligible correct completions, weighting matched questions equally; longer-prefix gains are smaller and use different eligible-question sets.The measurement supports lexical variety, not a direct count of semantic reasoning routes.Source: §D.1 · pp. 41–42, Table 6 · p. 41
  • Figure 19 reports Diverse ahead of Similar on every metric for both Qwen3 sizes and both selection budgets after identical RL.Its right column gives the Similar and Diverse scores in percent; pass@1 is mean sampled accuracy, while pass@32 and pass@64 measure sampled coverage.Source: §D.1 · p. 41, Figure 19 · p. 42

D.2. RLVE per-difficulty (Qwen3-4B-Base)

  • Across all 15 RLVE difficulties, Diverse leads by 1.5 to 7.4 points in mean sampled accuracy and 3.4 to 10.4 points in coverage; difficulties 11 to 15 exceed both training stages.Coverage is reported as pass@32 for difficulties 1–10 and pass@64 for 11–15.Source: §D.2 · p. 42, Figure 20 · p. 43

D.3. OMEGA held-out mathematics

  • At pass@64, OLMo3-7B reaches 45.4% on held-out OMEGA OOD prompts with Diverse selection, versus 39.4% with Similar selection.Both conditions use 50,000 selected traces from one accepted pool and the same 75-step GRPO run.Source: §D.3 · p. 43, Figure 21 · p. 44
  • For Qwen3 students on held-out OMEGA prompts, the multi-model corpus leads at pass@64 for all three capacities; at 4B it solves 33.3% versus 21.7%.At 14B, the reported lead is 1.3 points, or four prompts.Source: §D.3 · p. 44, Figure 21 · p. 44

D.4. Diagnostics where the route is executable

  • At 64 samples, Diverse solves 53.6% of held-out Sokoban boards versus 31.4% with Similar, and 33.0% of program-simulation prompts versus 8.2%.In both testbeds, the gap grows with the number of samples.Source: §D.4 · p. 44, Figure 21 · p. 44

E. Per-Testbed Configuration

The testbed configurations specify candidate sources and verification, selection conditions, evaluation sets, and matched SFT-to-GRPO recipes across synthetic and mathematics settings.

Released corpora: selection configuration

  • For the released corpora, both conditions select 100,000 rows from a shared pool with identical domain quotas; Diverse uses cluster-proportional farthest-point selection, while Similar selects nearest-centroid rows.Dolci-Think uses 631 raw fingerprint features, and Nemotron-Cascade 2 uses 1,738; both feature maps project to 96 dimensions.Source: E. Per-Testbed Configuration · p. 46

Released corpora: evaluation and training

  • The released-corpus comparisons span 15 benchmarks with draw counts varying by benchmark group, and the Dolci-Think pre-RL diagnostic uses eight draws on 64 mathematics prompts.Released-corpus RL uses a 64-update schedule; baseline comparisons report final RL step 64, while Diverse and Similar results use step 8 or 16 depending on corpus.Source: E. Per-Testbed Configuration · p. 46

Single-model mathematics test

  • For the single-model mathematics test, both selections draw 10,000, 25,000, or 50,000 rows from one pool, then receive the same GRPO recipe and are evaluated at matched RL steps.The ten benchmarks use eight samples per problem and report pass@8.Source: E. Per-Testbed Configuration · p. 46

Sokoban

  • The Sokoban setup selects 86,792 rows per condition from replay-verified traces, evaluates 500 held-out boards with 64 samples each, and trains OLMo3-7B with a 75-step GRPO run.Fingerprints read each trace’s state-action structure.Source: E. Per-Testbed Configuration · p. 47

Program simulation

  • Program simulation uses executable-verified rewrite-system traces and compares equal-sized multi-model and single-model corpora on 500 held-out prompts, followed by the same 75-step GRPO setup.Evaluation reports pass@1, pass@4, and pass@64 from 64 samples per prompt.Source: E. Per-Testbed Configuration · p. 47

F. An Illustrative Coverage Model

The illustrative model links a policy’s share of useful reasoning moves to expected multi-sample coverage. It also relates per-prompt success probabilities to mixed-reward groups and distinguishes the paper’s pre-RL selection approach from methods that act inside RL.

F. An Illustrative Coverage Model

  • With per-rollout success approximated by βu(C), expected pass@k coverage rises with the policy’s share of useful moves, while the diverse–similar coverage gap eventually returns toward zero as both coverages approach one.The gap has a single maximum for a continuous sampling budget; for integer budgets, a maximum occurs at an adjacent integer.Source: F. An Illustrative Coverage Model · p. 47
  • For prompts solved less than half the time, raising useful-move coverage increases both expected coverage and mixed-reward probability while success remains below one half; across prompts, a lower mean solve rate can still yield mixed rewards on more prompts.The aggregate comparison allows success probabilities to cross across prompts.Source: F. An Illustrative Coverage Model · pp. 47–48

G. Additional Related Work

  • Related group-relative RL methods modify rewards, advantages, prompt or rollout selection, or policy updates inside RL, whereas this paper’s approach acts before RL.The cited work also reports analyses of RL’s parameter and output-distribution effects.Source: G. Additional Related Work · p. 48

H. Limitations and Future Work

The authors identify direct route counting, broader diagnostic measurements, and replicated comparisons as needed tests of their account.

H. Limitations and Future Work

  • A direct test would count distinct route fingerprints among sampled paths before and after RL, replicate comparisons across training seeds, and track mixed-reward prompts through training.The authors also propose matching conditions on the number of prompts with mixed rewards to test the connection causally.Source: H. Limitations and Future Work · p. 48
  • The fingerprints are domain-specific; validating RLVE fingerprints against route annotations and breaking down OMEGA results by category would clarify where gains concentrate.The account predicts concentration on prompts with intermediate success probabilities.Source: H. Limitations and Future Work · p. 48
Loading 2609.33780v1…