PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

Shobhita Sundaram, John Quan, Ariel Kwiatkowski, Kartik Ahuja, Yann Ollivier, Julia Kempe

arXiv:2601.18778v3cs.LGcs.CL

TL;DR

RL fine-tuning can stall when hard problems provide sparse or nonexistent rewards, raising whether pretrained LLMs can generate curricula for problems they cannot solve. SOAR uses asymmetric self-play with a teacher rewarded by measured student progress on hard problems, and finds that grounded meta-RL can unlock learning plateaus while avoiding intrinsic-reward collapse. The paper argues that useful stepping stones can be generated without preexisting ability to solve the target problems.

  • Problem

    RL methods stall on problems with sparse rewards, motivating whether pretrained LLMs can leverage latent knowledge to generate curricula for problems they cannot solve.

  • Method

    SOAR uses asymmetric self-play and bilevel meta-RL to train a teacher that generates question-answer pairs, rewarding it with student improvement on hard problems.

  • Results

    Grounded rewards enable learning under sparse, binary rewards, outperform intrinsic self-play rewards, and show that question structure matters more than solution correctness for learning progress.

  • Takeaways & Limitations

    Useful stepping-stone questions do not require the teacher to solve the hard problems and can expand the model’s learning frontier beyond direct RLVR.

  • Takeaways & Limitations

    Bilevel RL is computationally expensive because stable training requires parallel students, and scaling beyond 3–8B models remains future work.

Abstract

from arXiv · show

RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? We explore this with SOAR: An asymmetric self-play framework that uses meta-RL to surface these pedagogical signals. A teacher model proposes synthetic problems for a student model, and is rewarded with its improvement on a subset of hard problems, thus grounding the curriculum in real student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of math benchmarks (0/128 success) reveals three core findings. First, it is possible to realize bilevel meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful problems. Second, grounded rewards outperform intrinsic learnability rewards used in prior LLM self-play, reliably avoiding typical instability and diversity collapse modes. Third, the structure and well-posedness of questions are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data

1 Introduction

RLVR stalls when problems are too difficult to solve, motivating SOAR, which uses asymmetric self-play and grounded meta-RL to generate stepping-stone curricula from latent model knowledge. The study finds that teacher-generated questions can unlock hard-problem learning without exposing the teacher to those problems.

  • RLVR provides little or no learning signal when the model cannot already solve difficult problems to some extent.
  • SOAR asks whether a pretrained LLM can generate stepping-stone curricula for problems it cannot solve.The framework is motivated by latent pedagogical knowledge acquired during pretraining.
  • SOAR trains a teacher to propose question-answer pairs and rewards it according to student improvement on a difficult subset.The hard dataset serves as a black-box grounding signal rather than an intrinsic self-play reward.
  • The teacher learns to generate useful questions that get the student unstuck on hard problems without seeing those problems.
  • A model’s ability to generate effective stepping stones is distinct from its ability to solve the corresponding hard problems.Meta-RL sharpens the pretrained model’s capacity to propose useful questions into a reliable learning signal.
  • Grounded meta-RL can escape genuine learning plateaus by helping models discover data that expands their learning frontier.

2 Related Work

Prior curriculum and self-play methods often rely on curated data or intrinsic proxies, which can be brittle in sparse-reward mathematical settings. This work instead grounds curriculum generation in measured student progress on a fixed target set through a bilevel meta-RL loop.

  • Mathematical self-play faces sparse, brittle rewards because correctness is essentially binary and provides no gradient toward partial solutions.
  • Recent fully data-free LLM self-play systems use intrinsic or proxy rewards but risk degenerate tasks, reward hacking, and weak progress guarantees.
  • Automated curricula commonly reorder existing data or select curated prompts, while generated environments can become degenerate under unconstrained objectives.
  • SOAR differs by grounding rewards in student progress on a fixed target set rather than intrinsic difficulty or curiosity proxies.
  • The bilevel objective is implemented as a nested meta-RL loop, with RLOO in the outer loop avoiding inner-loop unrolling and backpropagation through time.

3 Method

SOAR frames curriculum generation as a bilevel teacher-student meta-RL problem: a teacher proposes synthetic question-answer pairs, while student improvement on hard problems grounds teacher rewards. The method uses nested outer and inner RL loops, promotion, and black-box progress signals to generate useful stepping stones without automatic verification.

  • 3.1 Overview: SOAR asks whether latent knowledge can generate stepping-stone question-answer pairs for problems the pretrained model cannot solve.The framework targets difficult datasets with 0/128 successful generations, where direct RL may fail because positive rewards are sparse.
  • 3.1 Overview: The teacher and student are initialized from the same target model, with the teacher generating synthetic problems and the student training on them.The teacher is intended to provide gradient signal that moves the student beyond a performance plateau.
  • 3.2 Outer Loop: Teacher Training: The outer RL loop trains the teacher with RLOO, while the inner RL loop trains the student and returns its subsequent Dtrain improvement as the teacher’s black-box reward.The nested formulation avoids explicitly unrolling the inner optimization and backpropagating through student updates.
  • 3.1 Overview: SOAR replaces intrinsic curriculum rewards with measured student improvement on difficult ground-truth problems.Synthetic question utility is treated as emergent: pairs are rewarded when student training improves performance on Dtrain, implicitly penalizing degenerate problems and reward hacking.
  • 3.2 Outer Loop: Teacher Training: Each generated dataset is evaluated by training the student for 10 steps and comparing its greedy success on sampled reward questions with a baseline student.Rewards are averaged over parallel student trainings, normalized by subtracting initial accuracy, and assigned to the corresponding teacher rollout group.
  • 3.3 Inner Loop: Student Training: When the moving-average reward exceeds a threshold, SOAR promotes the best trained student baseline and accumulates its successful datasets as Promotion Questions.The promoted student is evaluated directly, while PQ trains a fresh base student to isolate the value of the synthetic questions.
  • 3.3 Inner Loop: Student Training: SOAR-generated questions outperform Hard-Only and Intrinsic-T baselines on MATH/HARP fail@128, with larger gains at higher k, and transfer to held-out OlympiadBench.The evaluation compares synthetic-question training and promoted-student inference against direct hard-set training and intrinsic-reward teachers.

4 Experiment Setup

The experiments study SOAR on stringent fail@128 math subsets using primarily Llama-3.2-3B-Instruct, with held-out evaluation and transfer tests. They compare promoted students and fresh students trained on synthetic questions against direct training, intrinsic self-play, contemporary self-play, and a curated-data upper bound.

  • 4.1 Models and Datasets: The study primarily uses Llama-3.2-3B-Instruct and evaluates MATH, HARP, and OlympiadBench fail@128 subsets.Each subset retains problems with 0/128 sampled successes and is split 50-50 into training and test sets.
  • 4.1 Models and Datasets: SOAR is trained on MATH and HARP while OlympiadBench is held out for transfer evaluation.Each outer iteration samples 64 teacher problems and 64 reward questions; student promotion uses a three-step reward moving average above τ = 0.01.
  • 4.3 Evaluation: Teacher evaluations use the checkpoints where training rewards stabilize: step 200 for MATH and step 170 for HARP.The study evaluates generated-problem and promoted-student performance after training completion.
  • 4.3 Evaluation: The promoted student is selected by best validation performance, and observed runs contain one of {128, 192, 256} synthetic questions.At most four promotions were observed in practice.
  • 4.3 Evaluation: Promotion Questions are tested by training a fresh base student on synthetic questions combined with the fail@128 training set.This isolates the value of the synthetic questions from the promoted student’s particular training trajectory.
  • 4.3 Evaluation: Baselines include direct Hard-Only training, an intrinsic learnability teacher, SeRL self-play, and a full-data upper bound using 6750 MATH training problems.The Hard-Only comparison also includes a MATH run with group size 128 to separate SOAR from additional compute.
  • 4.3 Evaluation: Performance is reported as pass@k for k ∈{1, 4, 8, 16, 32}, using 32 samples per problem and mean ± standard deviation over 6-12 seeds.Fresh-student evaluations use early stopping based on training reward convergence.

5 Results

SOAR uses synthetic questions and grounded meta-RL to improve learning on hard math subsets where direct training stalls. The results show gains over hard-only and intrinsic-reward baselines, stable and diverse teacher policies, and useful learning signals from coherent questions even when solutions are incorrect.

  • Learning on hard subsets: Synthetic questions substantially outperform Hard-Only and Intrinsic-T baselines, with larger gains at higher pass@k.Hard-Only training plateaus despite additional sampling and extended training.
  • Learning on hard subsets: +9.3% pass@32 on fail@128-MATH and +4.2% on fail@128-HARP are achieved by PQ over Hard-Only.PS achieves +8.5% on MATH and +3.6% on HARP over Hard-Only.
  • Generalization: Synthetic curricula transfer to OlympiadBench despite no out-of-distribution optimization, suggesting generalizable reasoning pathways.The transfer is reported for PQ-MATH, PQ-HARP, and Intrinsic-T.
  • Comparison with curated data: Synthetic PQ-MATH questions recover 75% of the performance gains from full-MATH training, while PQ-HARP recovers 50%.HARP-PQ outperforms 128 real HARP questions and matches 128 real MATH questions.
  • Teacher policies: Grounded-T questions produce competitive student performance with lower variance than Base-T and more stable outcomes than Intrinsic-T.Grounded-T primarily sharpens in-domain teacher behavior; on OlympiadBench, Grounded-T and Base-T trajectories overlap more.
  • Teacher policies: Grounded rewards preserve semantic diversity, whereas Intrinsic-T collapses into a narrow conceptual space associated with reward hacking and instability.Grounded-T matches Base-T diversity, while Intrinsic-T has V S = 10.82.
  • Question structure over answer correctness: Only 32.8% of PQ problems contain fully correct solutions, yet 63% are mathematically well-posed, indicating structure and conceptual content can matter more than answer correctness.Intrinsic-T questions have higher correctness at 55% but perform worse, while meta-RL reduces ambiguity errors relative to Base-T.

6 Discussion and Conclusions

The paper presents grounded bilevel meta-RL as a way to escape sparse-reward plateaus by generating stepping-stone questions, while grounding teacher rewards in measured student progress. It argues that this approach can expand learnability beyond direct RL and avoid self-play collapse, but remains computationally expensive and scoped as a proof of concept.

  • Breaking the sparse-reward plateau in RL fine-tuning: Grounded meta-RL can kickstart RL fine-tuning when initial success rates are too low to collect useful RLVR signal.The setup uses generated question-answer pairs and a bilevel self-play loop to address hard problems that direct training cannot learn from.
  • Breaking the sparse-reward plateau in RL fine-tuning: Generating effective stepping stones is distinct from solving the target hard problems, allowing the teacher to help the student without seeing those problems.The paper frames this as exploiting a latent ability sharpened by meta-RL.
  • Grounded rewards mitigate self-play collapse: Grounding teacher rewards in measured student progress prevents teacher degeneration and preserves diversity, unlike intrinsic objectives prone to reward hacking, diversity collapse, and seed instability.The contrast is with learnability and self-consistency rewards that are decoupled from task performance.
  • Expanding the learning frontier: The approach is positioned as expanding the envelope of learnability beyond direct RLVR, within the paper’s broader debate over latent abilities and learning-frontier expansion.The authors describe the results as evidence that grounded meta-RL can elicit capabilities inaccessible through repeated sampling alone.
  • Limitations: The primary limitation is the computational cost of bilevel RL, because stable training requires parallel students despite relatively cheap 10-20-step inner loops.The authors also characterize the work as a proof of concept and identify scaling beyond 3-8B models as future work.

B.5 Datasets

The datasets are constructed by filtering math problems for complete failure under 128 sampled solutions, then creating evaluation splits for three benchmarks. The filtering and split procedures differ according to each benchmark’s source and prior test-set status.

  • Fail@128 Filtering: 0/128 success defines the fail@128 filter: retain problems for which 128 sampled solutions all fail.Solutions are sampled with Llama-3.2-3B-Instruct using a 1024-token budget and temperature 1.0.
  • OlympiadBench: OlympiadBench uses English, text-only, automatically verifiable questions from a 674-question subset, followed by a random train/test split.
  • HARP: HARP draws fail@128 problems from the full dataset and creates a random train/test split because HARP was originally designed as a test set.
  • MATH: MATH draws the initial pool from its 5000-question official test split to reduce confounding from possible exposure to training questions.The motivation is a measured zero-shot accuracy gap between the official training and test splits.
  • Dataset sizes: Table 2 reports the original problem-pool sizes and the resulting train/test split sizes.

B.6 Evaluation

The evaluation compares curriculum and mixed synthetic-real training across hard math benchmarks, with checkpointing, seeding, and reporting procedures designed to account for convergence and variance. Mixed training is more stable on HARP and OlympiadBench, while curriculum training is selected for MATH.

  • Mixed synthetic-real training: Synthetic questions are evaluated by training fresh students on combinations of generated questions and real fail@128 training data.The study compares curriculum training, which uses a 64-step synthetic warm-start, with mixed training throughout.
  • Training-strategy selection: On HARP and OlympiadBench, mixed training yields significantly more stable learning dynamics, whereas curriculum training shows an early performance spike followed by a crash.
  • Training-strategy selection: On MATH, Base-T performs better with curriculum training, so curriculum is adopted for the MATH experiments.
  • Student checkpoint selection: Student checkpoints are selected using a smoothed, normalized reward slope, with the earliest point below 15% of the maximum slope marking the start of a plateau.Test performance is averaged over the following 200-step window, and varying the threshold from 10% to 20% has negligible effects.
  • Statistical evaluation: The evaluation uses nested seeding, multiple teacher and student runs, and aggregated means and standard deviations to capture training variation.Teacher-sampling experiments use 2-3 students per teacher seed, yielding at least 8 seeds per reported metric.
  • Compute: Each SOAR training run uses four GPU nodes for approximately 48-60 hours, while each fresh-student RLOO evaluation takes approximately 12 hours on one node.

C.1 Full Student Training Curves

The full student curves compare PQ, Hard-Only, and the full MATH upper bound across MATH, HARP, and OlympiadBench, including teacher-sampled variants. Extending Hard-Only training on HARP from 1500 to 6500 steps does not improve performance.

  • Full student training curves: Figure 9 compares PQ, Hard-Only, and the full MATH upper bound across MATH, HARP, and OlympiadBench student-training curves.The curves report different pass@k values and include the PS inference performance as a horizontal reference line.
  • Teacher variants: Figures 11-13 show student curves for Grounded-T, Base-T, Intrinsic-T, and Grounded-T without promotion, with means and standard deviations over seeds.
  • Extended Hard-Only training: 6500 steps of Hard-Only HARP training does not improve performance over 1500 steps.Figure 10 reports shading as ± 1 SD over 3 seeds.

C.2 Full Evaluations on fail@128 MATH, HARP, and OlympiadBench.

SOAR-generated curricula outperform direct and intrinsic-reward baselines on fail@128 MATH and HARP, while also transferring to OlympiadBench. Grounded-T improves performance, stability, and cross-seed consistency, and its learned questions capture useful pedagogical signals.

  • PQ and PS consistently outperform inference-only, Hard-Only, and intrinsic baselines across all MATH inference budgets.They recover most of the gain from training with real curated problems.
  • PQ and PS consistently outperform inference-only, Hard-Only, and intrinsic baselines on HARP across all inference budgets.SOAR questions also outperform similarly sized curated MATH/HARP subsets.
  • Improved teaching ability is decoupled from improved solving ability: trained Grounded-T teachers show no inference improvement over the base model on fail@128 problems.Direct Hard-Only training provides very little improvement, while extra compute does not recover PQ gains.
  • PQ and PS transfer from MATH and HARP reward signals to improving OlympiadBench performance.They match or outperform 128 questions sampled from the curated HARP training set.
  • Grounded-T and PQ perform better than Intrinsic-T despite containing fewer correct solutions, providing evidence that learned pedagogical signals reside in the teacher distribution.Grounded-T and PQ sampled questions have similar performance to accumulated PQ datasets.
  • Grounded-T outperforms all comparison teachers, particularly at higher inference budgets, with lower variance and greater stability across student and teacher seeds.Grounded-T without student promotion performs worse, validating the importance of promotion.

C.4 Categorizing Correctness of Synthetic Questions.

Synthetic-question analysis finds that well-posedness matters more than solution correctness for student progress. Well-posed questions with incorrect answers improve performance, while formatting is not the main driver.

  • Synthetic questions are categorized by well-posedness, correctness, and error type, including arithmetic, logical-fallacy, ill-posed, and ambiguous cases.Correctness is evaluated only when a question is well posed.
  • Well-posedness matters more than solution correctness for learning progress.Teacher training improves both well-posedness and correctness, but the best-performing datasets contain fewer correct solutions than Intrinsic-T.
  • Adding well-posed questions with incorrect answers improves student performance on HARP.The controlled comparison uses Correct-only and Well-posed synthetic subsets alongside real fail@128 questions.
  • Student format matching reaches 100% within approximately 50 steps across Hard-Only, Correct-only, and Well-posed conditions.This rules out formatting as the main explanation for the performance gains.

D.1 Sampled dataset size

Larger sampled Grounded-T datasets generally improve student performance and reduce error, with 128 questions offering similar mean performance to 64 but significantly greater consistency. The default n=64, τ=0.01 configuration performs best, while multi-turn sampling underperforms single-turn sampling.

  • The teacher is trained with n=64 while evaluation samples n ∈{32, 64, 128} questions for fresh students.This compares datasets smaller than, equal to, and larger than the teacher’s training partition size.
  • Sampling 128 Grounded-T questions yields similar mean performance to 64 questions but significantly smaller error.Performance improves with increasing dataset size, indicating benefits from sampling beyond the teacher’s training size.
  • The default configuration n=64, τ=0.01 performs best across the evaluated settings.n=64 has modest advantages over n=32 at larger evaluation dataset sizes.
  • Multi-turn teacher sampling performs worse than default single-turn sampling across all pass@k values and sampled dataset sizes.The multi-turn strategy avoids output filtering but does not exceed one student promotion.
  • Grounded-T outperforms Hard-Only across all inference budgets with Llama-3.1-8B-Instruct.The result transfers to the larger model despite using the smaller model’s default hyperparameters and observing no promotions.

E Teacher Training Dynamics

SOAR teacher training alternates between search and exploitation as student promotions change the grounding baseline. Grounded rewards preserve completion diversity, unlike intrinsic rewards that converge while diversity declines.

  • SOAR follows a cyclical search-exploitation pattern during teacher training.Reward rises during exploitation, triggers student promotion, then drops as the improved baseline makes prior curricula less useful.
  • Intrinsic-T reward rises smoothly while teacher-completion diversity falls; Grounded-T preserves diversity throughout training.Diversity is measured as average pairwise cosine distance of completion embeddings.
  • Student promotions occur when the 3-step moving average of teacher rewards exceeds τ=0.01.After promotion, the teacher adapts to the improved student and reward later recovers.
Loading 2601.18778v3…