MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

1 INTRODUCTION

• The paper characterizes RLVR dataset effectiveness as a function of the problems, target policy, and training rollout budget.Existing data efforts collect human-authored problems, build procedural environments, or synthesize surrogate tasks, but more data does not necessarily provide useful learning signal.Source: §1 · pp. 1–2
• About 88% of a 78K open-source math corpus falls into the zero-advantage trap for gpt-oss-120b at a 32K rollout budget, limiting the signal that corpus size alone provides.Filtering can select useful problems but discards hard problems at the target policy’s frontier.Source: §1 · p. 2
• Effective synthetic problems should require the reasoning operations and compositions needed for hard seeds, while allowing mixed rollout outcomes and at least one complete correct solution within budget.Directly prompting a strong teacher to vary hard seeds performs no better than training on the best existing corpus problems.Source: §1 · p. 2
• Across 8 math benchmarks, MBS raises gpt-oss-120b from 61.2 to 69.8 (+8.6 points) and DeepSeek v4 Flash from 72.4 to 77.9 (+5.4 points).The same approach improves Qwen3-4B by +6.9 pass@1 on SecQue and +2.4 on FinanceBench, using over 14,000 problems synthesized from 340 expert-curated seed questions.Source: §1 · p. 2
• MBS decomposes hard seeds top-down and recursively synthesizes new problems bottom-up, preserving the seeds’ reasoning blueprints while calibrating difficulty to the policy and budget.The construction grows families of problems from seed problems rather than directly varying the hard seeds.Source: §1 · p. 2, Figure 1 · p. 2

The zero advantage trap and the goldilock bands

• Roughly 88% of the 78K-problem corpus falls into the zero-advantage trap, with only 12.2% in the observed Goldilocks band at 32K.The corpus also includes 6.3% of problems with no correct responses in eight rollouts; the Goldilocks subset and a stricter in-budget-solvable subset outperform a full-distribution proxy despite using about an order of magnitude fewer problems.Source: §2 · p. 3, Figure 2 · p. 3, Figure 3 · p. 4

Difficulty is relative to the policy and training budget

• Changing the target policy reclassifies 32.3% of problems, while increasing the response limit from 16K to 64K lowers the hard-problem fraction from 33.1% to 10.6%.A response truncated at the budget does not establish that the policy lacks the required reasoning capability.Source: §2 · p. 3, Figure 2 · p. 3
• Filtering can retain useful existing problems but cannot create problems targeting capabilities absent from the filtered subset.Consequently, synthesis should target hard-seed reasoning operations while producing substantively new problems calibrated for mixed outcomes and at least one complete correct response within budget.Source: §2 · pp. 4–5

Hard problems are specifications for what data to synthesize

• Hard problems’ verified solutions specify reasoning operations and compositions for synthesis; a simpler worked subproblem enabled at least one target-policy success on 78/98 seeds and raised mean accuracy from 0% to 46.9%.Among 43 seeds with multiple verified blueprints from different teachers, 90.7% yielded the same blueprint or a compatible variant.Source: §2 · p. 4

Direct prompting a teacher to generate variations is not enough

• Direct seed synthesis often preserves techniques through shallow variants and yields fewer than one qualified problem per seed; filtering cannot repair its candidate pool.In the judged comparisons, MBS produces more qualified descendants per seed while more often preserving the blueprint in a new problem.Source: §2 · p. 5, Figure 4 · p. 5

3 MAGICBEANSTALK: EFFECTIVE DATA SYNTHESIS FROM HARD SEEDS

• MBS recursively decomposes each seed into easier subproblems, synthesizes new leaf problems, and recombines them into a tree preserving the seed’s reasoning blueprint.The method assumes access to teacher models and an ordered pool of student models that includes the target policy.Source: §3 · p. 5, §3.1 · p. 5

3.2 REJECTION-SAMPLE TEACHER TRACES

• For each problem, MBS accepts the first teacher trace whose final answer verifies and discards problems with no verified trace after the allowed attempts.A collection of teachers reduces the number of discarded problems due to solvability.Source: §3.2 · p. 6

3.3 TOP-DOWN PROBLEM DECOMPOSITION

• Teachers decompose verified problems into easier, self-contained subproblems; MBS retains a decomposition only when worked examples improve at least one evaluated student’s accuracy on the parent.The method retries ineffective decompositions and drops them if they remain ineffective.Source: §3.3 · p. 6

3.4 BOTTOM-UP SYNTHESIS OF CALIBRATED PROBLEMS

• Bottom-up synthesis generates harder variants at the leaves, then combines synthetic children into verified parent problems mirroring the original tree’s logical structure and difficulty.Candidates that exceed the weakest student’s accuracy threshold are discarded; retained trees are filtered for Goldilocks difficulty and in-budget solvability.Source: §3.4 · pp. 6–7
• For document-grounded problems, MBS retrieves unseen documents, extracts relevant snippets and entities, and re-grounds the synthetic tree while keeping each problem answerable from a document.The reported Table 1 evaluates Pass@1 across eight math benchmarks at a 16K output-token budget; AVG is the mean and Δ is the within-base-model improvement.Source: §3.4 · pp. 6–7, Table 1 · p. 7

4.1 FROM ZERO-ADVANTAGE SEEDS TO EFFECTIVE RLVR DATA

• MBS produced qualified descendants from 74.7% of hard seeds, versus 43.8% for seed synthesis, with candidate-level qualified rates of 42.8% versus 31.7%.A qualified descendant is in the Goldilocks band and has a normally terminated correct rollout within 16K.Source: §4.1 · p. 7, Table 5 · p. 9
• In math, 19.7% of MBS seed–candidate pairs scored at least 4 for both blueprint preservation and substantive novelty, versus 3.0% for seed synthesis.The corresponding finance rates were 86.0% and 17.0%.Source: §4.1 · p. 7
• Three frontier judges rejected 8.0% of sampled math answers and none of the finance answers; failure to verify may also reflect judge limitations on difficult problems.Source: §4.1 · p. 7, Table 4 · p. 9
• On the four-core-benchmark control run, MBS averaged 69.2, versus 65.3 for source-seed training and 63.9 for Goldilocks-filtered seed synthesis.This was a separate control run, not the full-budget results in Table 1.Source: §4.1 · p. 7
• On the original hard seeds, recovery rose from 9.5% for a fresh base sample to 25.0% after MBS training; MBS also produced 840 correct rollouts and mastered 14 seeds.Seed synthesis recovered 23.2%, produced 694 correct rollouts, and mastered 6 seeds.Source: §4.1 · p. 8

4.2 CONTINUAL RLVR ON MATH

• 61.2 to 69.8 (+8.6 points) for gpt-oss-120b and 72.4 to 77.9 (+5.4 points) for DeepSeek v4 Flash on the 8-benchmark average.The full source mixture reached 62.3 and Golden Goose 63.1; the in-budget Goldilocks source subset and seed synthesis reached 67.0 and 66.8.Source: §4.2 · p. 8
• At 24K and 32K inference, MBS remains 1.0 and 1.5 points above base, while source-seed training falls below base and seed synthesis converges to it.On MathArena Apex-Shortlist, MBS improves over base by 11.0, 13.3, and 10.9 points at 16K, 24K, and 32K, respectively.Source: Table 2 · p. 8, §4.2 · p. 9

4.3 MAGICBEANSTALK FOR FINANCE DATA

• MBS training improves pass@1 by 6.9 points on SecQue and 2.4 points on FinanceBench.Source: §4.3 · p. 9, Table 3 · p. 9

• Prior work builds hand-designed task environments (Lacombe et al., 2025; Stojanovski et al., 2025), adapts difficulty (Zeng et al., 2025; Xu et al., 2026), or mines Internet text (Lu et al., 2026).Source: §5 · p. 9
• Teacher-generated variations from seeds have been studied for SFT (Toshniwal et al., 2024; Zhao et al., 2025), VLMs (Liao et al., 2025; Acuna et al., 2025b; Jung et al., 2026; Acuna et al., 2025a), and RLVR (Yu et al., 2025a; Liang et al., 2025; 2026).Source: §5 · p. 10
• MBS analyzes what makes synthetic problems useful for RLVR, then uses recursive synthesis to turn hard seeds into new policy- and budget-calibrated problems.Source: §5 · p. 10
• Dynamic sampling, scaled rollouts, and oracle-guided exploration modify training algorithms, whereas this work focuses on data and is complementary to those methods.The cited approaches include dynamic sampling (Yu et al., 2025b), BroRL (Hu et al., 2025), and oracle prefixes, hints, or off-policy traces (Qu et al., 2026; Li et al., 2026).Source: §5 · p. 10

6 CONCLUSION

• MagicBeanStalk improves gpt-oss-120b by +8.6 and DeepSeek v4 Flash by +5.4 points across 8 math benchmarks; finance gains are +6.9 on SecQue and +2.4 on FinanceBench.In finance, over 14,000 verifiable problems are synthesized from 340 expert questions.Source: §6 · p. 10

AI USE STATEMENT

• The authors used generative AI for language polishing and LLMs to generate synthetic datasets, while retaining responsibility for the final work.They state that the tools did not contribute to the intellectual content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Exact reproduction of synthesis may be limited because proprietary teacher APIs can change, potentially changing generated solutions and problems.The paper says it will release the generated MagicBeanStalk datasets upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

6.1 ETHICS STATEMENT

• Synthetic training problems may contain errors, and the finance experiments do not establish readiness for training models used in financial decisions.The authors note that audits by three additional models may also contain mistakes and call for further testing before such reliance.Source: §6.1 · p. 10

B.1 A LARGE CORPUS CAN PROVIDE LITTLE LEARNING SIGNAL

• Only 12.2% of approximately 78,000 corpus problems fall in the observed Goldilocks band for gpt-oss-120b at a 32K rollout budget.The study sampled 8 responses per problem.Source: §B.1 · p. 14
• Roughly 88% of the corpus is in the zero-advantage trap: problems are solved in every rollout or in none.The passage reports that 6.3% were unsolved in any rollout; most were solved in all rollouts.Source: §B.1 · p. 14
• Goldilocks-only and in-budget-solvable subsets achieve higher average accuracy than the full-mixture proxy with about an order of magnitude fewer problems.Filtering concentrates useful signal, but retains only problems already present and cannot cover capabilities absent from the corpus.Source: §B.1 · p. 14

B.2 DIFFICULTY IS RELATIVE TO THE POLICY AND TRAINING BUDGET

• 32.3% of problems moved between zero-success, Goldilocks, and fully solved regions when the policy changed while the problem set and 32K budget stayed fixed.The comparison used 15,470 problems not solved in every GPT-5-nano rollout and evaluated gpt-oss-120b with 8 rollouts each.Source: §B.2 · p. 14
• Increasing gpt-oss-120b’s rollout budget from 16K to 64K reduced the hard-problem fraction from 33.1% to 10.6% and raised 8-benchmark average accuracy from 61.2% to 78.9%.A truncated response therefore does not establish that the policy lacks the required reasoning capability.Source: §B.2 · p. 14

B.3 VERIFIED HARD PROBLEMS SPECIFY FRONTIER CAPABILITIES

• A verified hard problem is solvable but has zero empirical success across normally terminated sampled responses, and its solution trace specifies a reasoning blueprint.The paper treats that blueprint as the essential reasoning operations and their composition that can reveal capabilities beyond the target policy’s competence.Source: §B.3 · p. 15
• After a simpler blueprint-based subproblem was given as an in-context example, the policy solved at least one rollout for 78/98 seeds and mean accuracy on the original hard problems rose from 0% to 46.9%.A judge checked that the subproblem did not explicitly reveal the original solution; the experiment used the first generated subproblem.Source: §B.3 · p. 15
• Among 43 seeds with at least two verified blueprints, 90.7% had the same or a compatible blueprint across frontier models, suggesting blueprints primarily reflect problem structure..Source: §B.3 · p. 15

C SEED SYNTHESIS: WHY DIRECT SEED SYNTHESIS IS NOT ENOUGH

• Seed-only training reached 65.3 average accuracy across four core math benchmarks, versus 63.9 for training on a similar number of Goldilocks-filtered synthetic problems.The direct seed-synthesis method produced no clear downstream improvement on unseen benchmarks.Source: C SEED SYNTHESIS: WHY DIRECT SEED… · p. 16
• In a blinded judge study, direct seed synthesis often retained recognizable reasoning techniques but frequently produced shallow variations rather than substantively new problems.The study assessed 300 generated math problems and 100 finance problems against seed blueprints.Source: C SEED SYNTHESIS: WHY DIRECT SEED… · p. 16
• Post-hoc filtering can calibrate difficulty and response length but cannot turn shallow variants into substantively new problems or produce candidates for seeds that yielded none.Seed synthesis returns fewer than one qualified problem per seed.Source: C SEED SYNTHESIS: WHY DIRECT SEED… · p. 16
• Under matched construction budgets, MBS yielded candidates meeting blueprint, novelty, and Goldilocks criteria for 28 seeds, versus 13 with direct synthesis.The control used 16.0M estimated construction tokens and MBS 14.3M; the table defines Goldilocks using 1–7 correct responses among 8 rollouts at 16K.Source: C SEED SYNTHESIS: WHY DIRECT SEED… · p. 16, Table 7 · p. 16
• The proposed target is substantively new problems that preserve hard-seed reasoning operations and composition, with mixed target-policy outcomes and at least one correct in-budget rollout.Hard problems specify what reasoning to target; the target policy and training rollout budget determine which generated problems are useful.Source: C SEED SYNTHESIS: WHY DIRECT SEED… · p. 16

D SUPPLEMENTARY REPRODUCIBILITY DETAILS

• The decomposition effectivity test checks whether conditioning on verified child problems and solutions raises at least one evaluated student's accuracy on the parent.It tests whether children provide useful scaffolds, not whether final problems qualify for the Goldilocks band.Source: D SUPPLEMENTARY REPRODUCIBILITY DETAILS · p. 16
• Both seed synthesis and MBS transfer to original hard seeds, but MBS masters more source seeds and generalizes better to math benchmarks.The evaluation uses approximately 1,250 source seeds that received 0/8 at 16K.Source: Table 9 · p. 17
• Final qualification requires 1–7 correct responses among 8 target-policy rollouts and at least one normally terminating correct response within the response budget.A positive decomposition-effectivity result from another student cannot substitute for this target-policy screen.Source: D SUPPLEMENTARY REPRODUCIBILITY DETAILS · p. 17

E CORE PROMPT TEMPLATES

• Verified solutions are teacher-side construction context, not RLVR training targets, and generated problems must be self-contained with a correct worked solution and one final answer.The seed-variation template also requires novelty rather than a simple rewrite.Source: E CORE PROMPT TEMPLATES · pp. 17–18
• Other templates decompose parents into easier subproblems, test scaffolded solving, and generate harder leaf variants that require more reasoning steps.For document-grounded finance problems, every resulting node must be answerable from a new supporting passage alone, without importing seed-document facts.Source: E CORE PROMPT TEMPLATES · pp. 17–18, D SUPPLEMENTARY REPRODUCIBILITY DETAILS · p. 17
• Core prompts use verified seed reasoning to generate novel problems testing the same skills, then combine synthetic children while preserving their original dependency pattern.The composition prompt asks for a parent matching the original difficulty and abstraction level rather than simplifying it.Source: E CORE PROMPT TEMPLATES · p. 18

F.1 BLUEPRINT PRESERVATION AND PROBLEM NOVELTY

• The study pairs one MBS candidate with one direct seed-synthesis candidate per seed, matching candidates on target-policy correct-rollout count within the observed Goldilocks band.The samples contain 300 math pairs and 100 finance pairs; GPT-5.6 scoring is symmetric and method-label blinded.Source: §F.1 · p. 18
• Reasoning preservation scores whether generated problems retain the seed's core operations and dependencies, while novelty scores whether they exceed paraphrase or superficial substitution.The scales run from 1 to 5, and the scores are assigned independently.Source: §F.1 · p. 18
• The joint criterion requires both reasoning preservation and problem novelty to score at least 4 for the same candidate.Novelty without blueprint preservation does not count as successful transfer.Source: §F.1 · p. 18
• The component-analysis table covers teacher coverage, leaf variants, and recursive composition; its qualified-problem scope uses 8 gpt-oss-120b rollouts and a complete correct response within 16K.Teacher coverage uses 100 hard seeds under an equal four-call budget, and the leaf comparison uses 10 fixed seeds.Source: Table 10 · p. 19

G MBS ADDITIONAL ABLATION

• With four teacher calls per seed, heterogeneous teachers cover more seeds; effectivity removes 16/22 decompositions, harder leaves raise qualified coverage, and recursion adds seeds missed by one-level composition.The recursive-composition gain includes qualified problems from 116 source seeds not covered by one-level composition.Source: G MBS ADDITIONAL ABLATION · p. 19