RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold

1. Introduction

• The paper studies synthetic-data effects on LLM math reasoning amid concerns about data scarcity and the risk that synthetic training amplifies biases or misinformation.Prior findings cited include improvements from scaling synthetic data [8] [29] and downward performance spirals [3] [50].Source: §1 · p. 1
• Self-generated positive responses can be as effective as 2× as much synthetic data from larger models, but may amplify spurious correlations and produce flat or inverse scaling.The introduction reports error scaling of approximately D^-0.05 to D^-0.15 with synthetic dataset size D under Zhang et al..Source: §1 · p. 1
• Per-step negative-data training improves performance over positive-only data, with gains comparable to scaling positive synthetic data by 8×.Negatives are selected to contrast good and bad choices at critical intermediate steps, helping avoid spurious steps amplified by positive-only training.Source: §1 · p. 2
• The authors model their negative-data construction as per-step advantage-weighted RL on mixed positive and negative data, contrasting it with positive-data imitation learning.They argue that this view explains generalization benefits through distribution-robust objectives and improved sample efficiency when advantages are estimated reliably.Source: §1 · p. 2
• The paper derives scaling laws for positive and negative synthetic data on GSM8K and MATH and studies how negative data can unlearn spurious correlations.Its contribution summary also presents a conceptual model inspired by RL to explain observed generalization benefits.Source: §1 · p. 2

2. Related Work

• Teacher-forcing on expert data maximizes next-token likelihood, but the paper points to math-reasoning failure modes that positive or negative synthetic data may address.The cited background describes conditioning each next-token prediction on all preceding tokens.Source: §2 · p. 3
• Prior synthetic-data work uses responses from stronger teachers or the learner itself, while this paper compares these sources and examines spurious intermediate steps.The related-work discussion connects learner-generated completions to ease of learning generalizable features and reduced undesirable memorization.Source: §2 · p. 3
• Randomly pairing correct and incorrect completions for preference optimization can perform poorly because of objective mismatch, and may require auxiliary losses.The related-work passage says that the best way to pose reasoning as preference optimization remains unclear.Source: §2 · p. 3
• The paper uses per-step credit assignment to connect preference optimization with advantage-weighted RL and explain how negative data can help unlearn spurious steps.It distinguishes this analysis from work that uses tree-based sampling to train a reward model on reasoning subsequences.Source: §2 · p. 3

3. Problem Setup and Synthetic Data Generation Pipeline

• The study focuses on GSM8K and MATH, using novel problems designed by capable models such as GPT-4 and Gemini 1.5 Pro.The synthetic problems are paired with responses obtained from those same models.Source: §3 · p. 3
• The generation pipeline prompts a capable model with a sampled real problem to create a similar solvable problem, then requests a step-by-step solution trace.Generated traces are assumed accurate and lightly filtered to remove duplicates, badly formatted answers, and model failures.Source: §3 · p. 4
• A trace is positive when its final answer is correct and negative when it is incorrect, as determined by a binary verifier using answer extraction and string matching.The original generated trace is defined as positive; the positive dataset contains problem-trace pairs sampled from the synthetic joint distribution.Source: §3 · p. 4
• A solution trace contains intermediate reasoning steps, so an incorrect final answer does not imply that every step is incorrect, nor does a correct answer ensure sound steps.The paper notes that even capable models' original synthetic answers may contain incorrect reasoning steps.Source: §3 · p. 4

4. Learning from Synthetic Data

• Rejection finetuning trains on positive traces sampled from the SFT policy for synthetic problems, applying the next-token prediction loss to those traces.The method samples multiple responses per problem because a single SFT-policy sample is not guaranteed to be positive.Source: §4 · p. 4
• Because negative data can encourage the model to produce errors under ordinary supervised fine-tuning, the paper uses contrastive DPO and examines two ways to construct negatives.The two variants are standard DPO and per-step DPO.Source: §4 · pp. 4–5
• The paper evaluates SFT on synthetic data and RFT on SFT-policy-generated positives as the amount of positive data scales on GSM8K and MATH.The figure caption notes fixed sets of 8k or 16k problems and reports that scaling model positives can amplify spurious correlations.Source: Figure 2 · p. 5
• Per-step DPO identifies the first pit in a sampled SFT-policy trace, where subsequent completions are expected to lead to incorrect answers, and uses that prefix in preference data.This differs from standard DPO, which rejection-samples a negative trace from the SFT policy and pairs it with the positive response.Source: §4 · p. 5

• Test error scales slowly with positive synthetic data: approximately D^-0.15 on GSM8K and D^-0.05 on MATH.The authors report that these trends are more underwhelming than pretraining scaling trends.Source: §5 · p. 5
• As self-generated positive data is scaled, test error can plateau or worsen on GSM8K, unlike scaling problems and solutions in SFT data.The verifier checks final answers, so incorrect or irrelevant intermediate steps can remain in accepted positive traces and create spurious associations during next-token training.Source: §5 · pp. 6–7
• Self-generated positive solutions from the finetuned policy yield about 2× the sample efficiency of the synthetic problems used to train it.The RFT datasets keep the questions fixed while adding verified positive solution traces sampled from the SFT policy.Source: §5 · pp. 6–7
• With one solution per question and equal-sized datasets, RFT data from the learner beats capable-model SFT data by >1%.The authors attribute a significant portion of the efficiency gain to learner-generated responses being easier to fit, not solely to extra response coverage.Source: §5 · p. 7, Figure 4 · p. 7
• Injecting spurious steps into otherwise positive RFT traces can make the trained model perform worse than the SFT policy and produce more such steps.The authors describe this as a failure mode later addressed by per-step credit assignment.Source: §5 · p. 7

6.1. Conceptual Model: Constructing Negatives to Enable Per-Step Credit Assignment

• Per-step credit uses Q-values to estimate success after a candidate step and defines its advantage relative to the preceding state; DPO on precise per-step preference pairs is equivalent to advantage-weighted RL.High-advantage steps are treated as critical, while low-advantage steps are treated as worse and should be unlearned.Source: §6.1 · p. 9

6.2. Scaling Results for Negative Data

• 8× the size of Dsyn: per-step DPO extends RFT’s 2× effective-data improvement on GSM8K and MATH, and performance continues to scale as data increases.Per-step DPO also improves pass@5 consistently, despite estimating advantages under a best-of-K policy with K=5.Source: Figure 7 · p. 10, §6.2 · pp. 10–11
• Standard DPO pairing of arbitrary positive and negative responses does not improve on SFT and can worsen scaling, while high-edit-distance pairing helps but underperforms per-step DPO.The conceptual model attributes the failure to an incorrect induced advantage that can reinforce spurious correlations.Source: Figure 7 · p. 10, §6.2 · p. 11

6.3. Why Does Credit Assignment from Negative Data Improve Model Generalization?

• Per-step DPO’s credit assignment identifies critical steps over irrelevant or incorrect ones, improving test performance and scaling trends over SFT and standard DPO with the same data.The section explains these generalization gains through the advantage-weighted RL objective.Source: §6.3 · p. 11

6.3.1. Advantage-Weighted RL De-Emphasizes Spurious Steps and Emphasizes Critical Steps

• Imperfectly cloned spurious steps receive low advantage when rollouts from their prefixes often end incorrectly, and only per-step DPO improves Q-values at every step compared with SFT.The authors report that per-step DPO also improves accuracy by >6% when trained on negative data from an RFT model overfit on distracting steps.Source: §6.3.1 · pp. 11–12, Figure 8 · p. 11

6.3.2. Why Does Generalization Improve?: Connecting Advantage-Weighted RL to DRO

• The authors interpret advantage weighting as similar to Group DRO: it upweights critical steps and downweights spurious ones, aiming to lower loss on critical states and improve test accuracy.They state that population-level benefits depend on accurate on-policy advantage estimates and sufficient policy regularization.Source: §6.3.2 · p. 12

6.3.3. But, Attaining Low Generalization Error Requires Low Advantage Estimation Error

• When spurious-step advantages are overestimated, training can reinforce them; BoK(πsft) with K=5 lowers estimator variance, while adding positive-trace pairs lowers test error across synthetic-data sizes.The fuller algorithm estimates Q-values on positive as well as negative traces, adding pairs at the first low-Q step in a positive trace.Source: §6.3.3 · p. 13, Figure 10 · p. 13

6.3.4. Validating Claims About Generalization: Controlled Analysis on a Didactic Problem

• The didactic star-graph analysis varies SFT initialization across under-trained, early-stopped, and over-trained checkpoints to study its role in per-step DPO’s generalization gains.The task probes a planning failure in which SFT memorizes a hard-to-predict adjacent node instead of identifying the critical token.Source: §6.3.4 · pp. 13–14
• Per-step DPO on negative data from under-trained and early-stopped SFT models sharply reduces training loss and improves test loss and performance, unlike SFT’s nearly flat test-loss landscape.SFT tends to memorize a critical token through non-generalizable features; per-step DPO can unlearn that spurious step and improve generalization.Source: §6.3.4 · p. 14
• From an over-trained SFT initialization, per-step DPO and SFT have identical test errors, and negative data yields no improvement over positive-only fine-tuning.The critical token already has a high advantage similar to other tokens, so per-step DPO does not reweight it; training cannot unlearn the model’s noisy-feature dependence.Source: §6.3.4 · pp. 14–15
• Advantage-weighted RL can unlearn spurious steps and improve generalization when advantage-estimation error is low and the under-trained base model assigns those steps low advantage.For the under-trained model, reweighting next-token loss by advantage estimates reduces the Q-value disparity and preferentially minimizes loss at the critical token.Source: §6.3.4 · pp. 14–15

• Sampling more positive traces from SFT models improves same-data sample efficiency up to 2×, but positive-only self-generated data can amplify spurious steps that hurt test performance.Source: §7 · p. 15
• 8× improvement in sample efficiency of the synthetic data is reported for training on an instance of the per-step objective.Source: §7 · p. 15
• Negative traces from the same SFT model can address positive-only failure modes by estimating per-step advantages and using them via RL.Source: §7 · p. 15
• For reasoning beyond evaluated datasets, capable-model synthetic data may contain errors, answer-referenced labels may reinforce spurious correlations, and transfer evaluation remains future work.Source: §7 · p. 15

• Per-step DPO and advantage-weighted RL have the same optimal policy when positive and negative continuations share an SFT-sampled prefix and pair labels use advantage estimates.Source: A. Proof of Theorem 6.1 · p. 21
• DPO is equivalent to optimizing a KL-penalized expected-reward objective with preferences defined by an induced Bradley–Terry model.Source: A. Proof of Theorem 6.1 · p. 21
• The optimal policy can be learned by running DPO on preference tuples sampled from the Bradley–Terry model induced by the reward function.Source: A. Proof of Theorem 6.1 · p. 21

B. Theory: Why Does Negative Data Improve Generalization?

• The analysis extends RL-versus-imitation guarantees to argue that negative-data training can improve over SFT by prioritizing high-advantage critical steps.Positive-only training is characterized as imitation learning, whereas the negative-data construction is equivalent to per-step advantage-weighted RL.Source: B. Theory: Why Does Negative Data… · p. 22
• The theorem lower-bounds the task-success difference between advantage-weighted-RL policy πneg and SFT policy πsft, with the bound accounting for prompt sampling and advantage-estimation error.The concentration term decreases as the synthetic prompt set grows; larger overestimation error ∆ can reduce the guaranteed improvement.Source: B. Theory: Why Does Negative Data… · pp. 22–23
• The guarantee is strongest when detected critical steps have high advantages and estimates are accurate; overestimating a spurious step can undermine improvement.The discussion notes that an overestimated spurious step may be labeled critical, so training on its negative data may not improve performance.Source: B. Theory: Why Does Negative Data… · pp. 23–24

C. Additional Related Work

• The related-work discussion identifies open-loop next-token errors and memorization of hard-to-learn tokens as SFT failure modes linked to poor generalization.The paper studies positive-data likelihood training and preference optimization on SFT-generated negatives as approaches to these failures.Source: C. Additional Related Work · p. 24

D. Synthetic Data Generation

• The synthetic-data prompt asks for a new, similar math question that is self-contained, reasonable, and answerable with a single number, without solving it.The prompt supplies an original question and requests output in a specified “CREATED QUESTION” format.Source: D. Synthetic Data Generation · p. 24
• For GSM8K, the authors change the prompt to request math word problems because this produced questions closer to GSM-style problems.The change replaces the general math-problem phrasing with “math word problems.”Source: D. Synthetic Data Generation · p. 24
• Generating the synthetic data used approximately $3,000 in OpenAI credits.Source: D. Synthetic Data Generation · p. 24

E. Negative Data Identifies Spurious Steps with Advantage Estimates

• The advantage probe assigns low values to steps that are incorrect or correct but irrelevant to solving the problem.The section presents this as a qualitative finding from examples of model-generated positive responses and identified spurious steps.Source: E. Negative Data Identifies Spurious… · pp. 24–25

F. Details on Star Graph Problem

• In the star-graph case study, the task tests whether advantage-weighted RL can generalize and unlearn a feature that causes SFT failure.Prior work is described as finding that SFT can minimize loss by memorizing a hard-to-predict node rather than learning to backtrack the path.Source: F. Details on Star Graph Problem · p. 27
• The star-graph dataset contains 30,000 random graphs with a center of out-degree two and paths of length four; test accuracy requires predicting every path node correctly.Evaluation averages 0/1 accuracy over 1,000 test graphs that differ from the training graphs.Source: F. Details on Star Graph Problem · p. 27
• The described SFT failure ranges from zero test accuracy when memorizing an adjacent token to 50% when choosing between the two paths without backtracking.The section says larger graphs or node vocabularies make overfitting on the hard-to-predict token easier, and uses a vocabulary size of 20.Source: F. Details on Star Graph Problem · p. 27
• For each SFT sample, the method estimates each step’s advantage with five conditional rollouts and pairs shared-prefix subsequences using the highest- and lowest-advantage next steps.It updates with one DPO gradient step per batch, then estimates advantages and constructs pairs on a fresh batch of star graphs.Source: F. Details on Star Graph Problem · p. 28

G. Implementation Details

• Experiments use GSM8K and MATH with pretrained DeepSeek-Math-7B and Llama2-7B models; each dataset has about 7,500 training examples.GSM8K has about 1,300 test examples and MATH about 5,000.Source: G. Implementation Details · p. 28
• Positive-data SFT runs for five epochs, while RFT data generation uses a two-epoch under-trained checkpoint and samples 100 responses per question.For the cited RFT setup, at most four diverse correct completions are retained; a separate scaling experiment samples until datasets reach 10,000–50,000 examples.Source: G. Implementation Details · p. 28
• Per-step DPO uses K=5 for best-of-K advantage estimation, with β=0.3 on GSM8K and β=0.1 on MATH, training for three epochs.The procedure uses learning rate 1e-6, batch size 64 prompts, RMSProp, and cosine learning-rate decay with 0.1 warmup ratio.Source: G. Implementation Details · pp. 28–29
• Replicating the experiments requires approximately 1,000 H100-cluster hours using eight GPUs.Source: G. Implementation Details · p. 29