Source-linked AI summary
Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text
Ximing Lu, David Acuna, Jaehun Jung, Jian Hu, Di Zhang, Shizhe Diao, Yunheng Zou, Shaokun Zhang, Brandon Cui, Mingjie Liu, Hyunwoo Kim, Prithviraj Ammanabrolu, Jan Kautz, Yi Dong, Yejin Choi
TL;DR
RLVR training increasingly saturates because existing verifiable data are limited, while many reasoning-rich texts are difficult to verify automatically. Golden Goose converts such text into multiple-choice fill-in-the-middle tasks with generated distractors, producing scalable RLVR data. GooseReason revives saturated models, achieves state-of-the-art results across 15 benchmarks, and extends to cybersecurity with a new state of the art.
Problem
RLVR scaling is limited by finite verifiable data, while reasoning-rich open-ended texts are often excluded because their solutions are difficult to verify automatically.
Method
Golden Goose masks crucial reasoning steps in source text and generates plausible incorrect distractors, converting unverifiable text into verifiable multiple-choice fill-in-the-middle tasks.
Results
GooseReason revives saturated models, sustains RL gains, and achieves new state-of-the-art performance for 1.5B and 4B-Instruct models across 15 benchmarks; its cybersecurity data yields a 4.44% gain across three benchmarks.
Takeaways & Limitations
Reasoning-rich unverifiable internet text can be automatically reused as RLVR data for continued training and specialized-domain improvement.
Takeaways & Limitations
The pipeline may inherit biases or toxic content from its reasoning-rich internet-text sources, and cybersecurity capabilities have dual-use risks.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.
1. Introduction
Golden Goose addresses RLVR data saturation by converting reasoning-rich, unverifiable text into verifiable multiple-choice tasks. GooseReason-0.7M revives saturated models, sustains RL gains across domains, and extends the approach to cybersecurity.
- Motivation: RLVR scaling is constrained by finite verifiable data, causing improvements to saturate increasingly quickly, especially for stronger language models.Open-ended reasoning tasks are often excluded because their solutions are difficult to verify automatically.
- Method: Golden Goose masks crucial reasoning steps in source text and generates plausible but incorrect distractors to create verifiable MCQ fill-in-the-middle tasks.The pipeline can first extract educationally valuable passages from noisy web scrapes and filters out easy problems.
- Dataset: Over 0.7 million GooseReason tasks span mathematics, programming, and general scientific domains, complementing existing RLVR data.The dataset is designed to remain pluggable into existing RL recipes and provide fresh learning signals after prior data become stale.
- Results: 2.27% absolute improvement replaces a prior 0.79% degradation when GooseReason continues RL training on a saturated Qwen-4B-Instruct model.GooseReason-4B-Instruct achieves state-of-the-art performance among 4B-Instruct models across 15 diverse benchmarks.
- Cybersecurity deployment: 4.44% absolute gain across three cybersecurity benchmarks is achieved from 180K GooseReason-Cyber examples after 100 RL steps.This establishes a new cybersecurity state of the art and surpasses the prior model’s 1.44% average gain despite its extensive domain-specific training.
- Results: 13× more effective examples than ProRL are provided by GooseReason across math, code, and STEM, with over 450,000 additional effective examples.An effective example has both successful and failed model rollouts, producing a meaningful learning signal for RL.
oose
Golden Goose converts reasoning-rich but unverifiable text into verifiable multiple-choice RLVR tasks by masking key spans and generating plausible distractors. The pipeline also handles noisy web data through passage extraction and supports corpora previously excluded from RLVR curation.
- Data Synthesis Pipeline: Golden Goose masks a contiguous span of important reasoning steps, treats it as the answer, and generates plausible but incorrect distractors.The resulting multiple-choice task asks the student model to select the masked span from a randomized candidate set, with verification by exact option matching.
- Data Synthesis Pipeline: For noisy sources, the pipeline first extracts or summarizes a coherent educational passage before constructing the masked context and distractors.If no suitable passage exists, the synthesis process returns an empty string.
- Source Corpora: The pipeline leverages reasoning-rich unverifiable corpora that prior RLVR curation typically excluded, including incomplete forum solutions, coding problems without test cases, and open-ended scientific material.These sources include AoPS, rStar-Coder data lacking test cases, and university-level scientific textbooks with difficult-to-validate answers.
- Design Choices: Open-ended infill was rejected as an alternative because RL-tuned reasoning models often solve the problem from scratch instead of generating the required masked content.Multiple-choice formulation also avoids the computational overhead of using a powerful LLM judge during RL training.
- Design Choices: Increasing the distractor count raises task difficulty by reducing elimination-based strategies; with 9 options, over 70% of GooseReason-Math problems were medium-difficulty.With only 3 options, many problems became overly easy for ProRL-1.5B-v2.
3. Experiment
The experiments test GooseReason-0.7M for continued RL after data saturation, compute-constrained RL from scratch, and cybersecurity specialization. Across these settings, adding GooseReason improves scaling and yields strong benchmark results.
- Scaling Beyond Data Saturation: 2.71% versus 0.63% in math, 2.12% versus 0.95% in coding, and 3.48% versus 0.13% in STEM show GooseReason's gains over continued ProRL training.The largest margin occurs in STEM, where general-science RLVR data is scarcer; the evaluation primarily uses non-MCQ benchmarks.
- Scaling Beyond Data Saturation: After 300 steps, Qwen-4B-Instruct lost 1.29% in math and 1.52% in STEM under continued ProRL training, whereas GooseReason produced gains of 2.18%, 2.24%, and 2.40%.The reported GooseReason gains correspond to math, coding, and STEM, respectively.
- Benchmark Results: GooseReason-4B-Instruct achieved new state-of-the-art results among 4B-Instruct models across 15 diverse benchmarks and comparable or better performance than Qwen3-30B-Instruct.The evaluation covers mathematics, code generation, STEM, instruction following, and logical reasoning.
- Compute-Constrained Scaling: At the same number of steps under a fixed compute budget, joint training with GooseReason-0.7M consistently outperformed ProRL-only training from scratch.This comparison used Qwen-4B-Instruct and 200 RL steps.
- Cybersecurity Evaluation: 4.44% absolute gain across 3 cybersecurity benchmarks established new cybersecurity state-of-the-art performance after 100 RL steps with GooseReason-Cyber.The evaluation used CTI-Bench, CyberMetricc, and SecEval.
4. Related works
Prior RLVR scaling relies on longer training, larger rollout budgets, human-curated data, or handcrafted verifiable environments. Golden Goose instead converts reasoning-rich unverifiable internet text into scalable multiple-choice RLVR tasks.
- Scaling RLVR: ProRL and BroRL scale RLVR algorithmically through longer training and more rollouts per example, respectively.These approaches target effective scaling axes for avoiding RLVR saturation.
- Large-Scale Human Annotation: Human-authored and verified RLVR datasets provide large-scale math and code data but require specialized expert curation.The paper positions Golden Goose as complementary to these datasets.
- Automated Data Synthesis: Automated RLVR synthesis commonly relies on manually designed environments for games, puzzles, and formal domains, limiting expansion beyond hundreds of environments.Such procedural generators are difficult to adapt to knowledge-intensive domains such as medicine, economics, and cybersecurity.
- Golden Goose: Golden Goose converts reasoning-rich unverifiable internet text into verifiable multiple-choice tasks, including free-form textbooks and coding problems lacking unit tests.This broadens RLVR data sources beyond handcrafted environments and programming problems.
5. Conclusion
The paper presents Golden Goose as a scalable way to convert reasoning-rich unverifiable text into RLVR data. GooseReason-0.7M improves RL scaling across general domains and GooseReason-Cyber extends the approach to cybersecurity.
- Conclusion: Golden Goose is a simple, scalable pipeline that converts reasoning-rich unverifiable internet text into verifiable multiple-choice tasks.The approach is presented as a way to unlock text that conventional RLVR datasets typically exclude.
- Conclusion: Over 0.7 million GooseReason-0.7M tasks span mathematics, programming, and general scientific domains.The dataset is designed to complement existing RLVR datasets and support further RL scaling.
- Conclusion: GooseReason revives saturated models, sustains gains across math, coding, and STEM, and achieves new state-of-the-art results for 1.5B and 4B-Instruct models across 15 benchmarks.The paper also reports new cybersecurity state-of-the-art performance from tasks synthesized from raw web scrapes, surpassing a 7B domain-specialized model.
6. Impact Statements
Golden Goose is presented as a way to accelerate reasoning-model progress in domains where verifiable RLVR data is scarce, with cybersecurity as a demonstrated application. The cybersecurity deployment establishes new state-of-the-art results while raising dual-use and source-content concerns.
- Golden Goose targets reasoning-intensive domains where verifiable RLVR data is scarce, including STEM, math theorem proving, and open-ended domains.
- The paper demonstrates Golden Goose in cybersecurity and reports new state-of-the-art results in that domain.
- The cybersecurity application has dual-use potential, supporting automated defense and vulnerability analysis while theoretically enabling offensive misuse.
- Because the pipeline relies on reasoning-rich internet text, biases or toxic content in source corpora may be inherited.
A. Details of Data Synthesis
The data-synthesis prompts convert source material in cybersecurity, math and STEM, and code into fill-in-the-middle multiple-choice tasks. They require masking important consecutive content and generating plausible but incorrect distractors, with cybersecurity adding passage extraction and a no-suitable-passage fallback.
- Cybersecurity Domain: Cybersecurity synthesis first extracts a coherent, educationally valuable passage of around 100–600 words before constructing the task.
- Cybersecurity Domain: The cybersecurity prompt replaces a consecutive multi-sentence span containing important reasoning steps with [MASK] and uses the removed text as the ground-truth answer.
- Cybersecurity Domain: Cybersecurity synthesis returns an empty answer when no passage is suitable for educational cybersecurity learning.
- Math and STEM Domain: Math and STEM synthesis masks several consecutive important reference-solution steps and generates at least ten plausible, stylistically similar distractors that are incorrect.
- Code Domain: Code synthesis masks several consecutive important solution lines and generates at least ten plausible distractors similar in style and length but incorrect.
B. Details of Experiments
The experiments break down comparisons of continued or from-scratch RL across math and coding benchmarks, including original data, GooseReason-0.7M, RLVE, and fixed-compute joint training. Additional breakdowns cover IFEval and GPQA Diamond for continued training on Qwen-4B-Instruct.
- Continued RL on Qwen-4B-Instruct: Continued RL on Qwen-4B-Instruct after data saturation is broken down across six math benchmarks, comparing original ProRL data with added GooseReason-0.7M.
- Continued RL on Qwen-4B-Instruct: Continued RL on Qwen-4B-Instruct after data saturation is also broken down across four coding benchmarks, comparing the original ProRL data with added GooseReason-0.7M.
- Continued RL on Qwen-4B-Instruct: Additional Qwen-4B-Instruct results compare continued RL on IFEval and GPQA Diamond.
- Continued RL on ProRL-1.5B-v2: Continued RL on ProRL-1.5B-v2 is broken down across six math benchmarks for original ProRL data, added GooseReason-0.7M, and RLVE.
- Continued RL on ProRL-1.5B-v2: The corresponding ProRL-1.5B-v2 breakdown covers four coding benchmarks under the compared training-data settings.
- RL from scratch on Qwen-4B-Instruct: From-scratch RL on Qwen-4B-Instruct under a fixed compute budget compares ProRL data alone with joint training using GooseReason-0.7M across six math and four coding benchmarks.