PaperlayerView exact revision on arXiv

Source-linked AI summary

Frontier Learning: Training LLM Reasoners at the Edge of Capability

Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic, Aurelien Lucchi

arXiv:2609.35426v1cs.LGcs.AIcs.CL

TL;DR

Fixed-pool RL post-training can lose its learning signal as models improve and exhaust useful problems. Frontier learning uses procedural generators and regret-guided exploration to keep training near the model's evolving capability frontier. It outperforms standard baselines across reasoning tasks, with a reported DICE result of 71.8 versus 33.3.

  • Problem

    Fixed-pool GRPO can lose its optimization signal when all rollouts for a problem receive the same reward, and its useful coverage can be exhausted as the policy improves.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    Frontier learning uses procedural generators online and a regret-guided level buffer, mutation, and exploration to adapt training to the policy's evolving capabilities.

    Source: §1 · p. 2

  • Results

    Across five puzzle, math, and graph reasoning tasks and three model families, frontier learning outperforms baselines; on DICE it scores 71.8 versus 33.3 for the strongest standard baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The training distribution can be adapted online by generating problems near the model's evolving capability frontier rather than relying only on a fixed pool.

    Source: §2 · p. 3, §6 · p. 10

    The approach requires a procedural generator with continuous or discrete task-specific parameters; extending it to tasks without procedural generation remains future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

1 INTRODUCTION

Fixed-pool training can lose its signal as a model improves, motivating online generation of problems near its evolving capability frontier. Frontier learning combines procedural generation with regret-guided level selection and reports gains over fixed-pool and other baselines.

1 INTRODUCTION

  • Fixed-pool methods become ineffective once the policy reliably solves every problem, because GRPO supplies no signal when rollouts receive the same reward.Filtering or reweighting can focus training on informative items but remains bounded by the pool's coverage.Source: §1 · p. 1
  • Frontier learning generates problems online from task-specific attributes, selecting levels that are informative for the policy's current capabilities.Generator attributes can interact in complex ways and need not provide a monotonic difficulty measure.Source: §1 · pp. 1–2
  • The related-work distinction is online exploration of task-specific difficulty levels, rather than only filtering fixed pools or generating problems offline.The paper positions its focus as LLM post-training on reasoning tasks, unlike prior procedural-generation work on navigation and control.Source: §1 · p. 2
  • On DICE, frontier learning reaches 71.8 versus 33.3 for the strongest standard baseline over 500 steps.The contribution summary reports that gains are largest over long training horizons, when fixed-pool baselines plateau.Source: §1 · p. 2
  • A dynamic level buffer, regret-based priorities, and state-dependent mutation with exploration maintain and expand the set of frontier levels.The approach uses rollouts already collected for GRPO to prioritize levels; regret-guided mutation drives expansion.Source: §1 · p. 2

2 BACKGROUND

RLVR post-training optimizes a policy for verifiable-answer problems, while GRPO derives response advantages from within-problem reward comparisons. Fixed datasets can become uninformative as the policy improves, and MAD normalization is used to preserve frontier-level update magnitude.

2 BACKGROUND

  • RLVR optimizes expected reward over problems with verifiable correct answers, using a reward function that evaluates response correctness.The objective is the expected reward of policy responses on problems sampled from the dataset.Source: §2 · p. 3
  • GRPO constructs response advantages from within-group reward comparisons, avoiding a separate value network.For each problem, the policy generates multiple responses whose rewards are compared.Source: §2 · p. 3
  • MAD normalization keeps a problem's total absolute advantage constant, avoiding reduced update magnitude for frontier cases with all but one rollout wrong.The section contrasts this with standard-deviation normalization.Source: §2 · p. 3
  • Fixed-data GRPO can expose the policy to only zero-gradient problems when all rollouts are correct or all are wrong, and this can persist after the policy masters the dataset.Filtering or resampling assumes informative problems exist in the pool; the section argues the training data must adapt online to the model's evolving capability.Source: §2 · p. 3

3 FRONTIER LEARNING

Frontier learning continually adapts training data generated from procedural task levels to track a policy’s changing capability. It focuses on the moving band of informative levels using regret-guided sampling, exploration, and state-dependent mutation.

3.1 PROCEDURAL DATA GENERATION

  • A level is a combination of generator attributes that defines a structured difficulty class from which arbitrarily many problems can be sampled.Prior offline use fixes levels and problems before training, so the training distribution may not adapt to the policy’s current capability.Source: §3.1 · p. 4

3.2 OBJECTIVE FOR OPEN-ENDED POST-TRAINING

  • The capability frontier is a thin, moving band of levels with informative problems, while most levels are too easy or too difficult for the current policy.The objective is expected performance on instances drawn uniformly from levels, estimated in practice using held-out levels unseen during training.Source: §3.2 · p. 4

3.3 COMPONENTS OF FRONTIER LEARNING

  • Frontier learning uses a dynamic level buffer and combines regret-based prioritization, exploration, and mutation to track and extend the capability frontier.Each iteration turns the selected levels into generated problems for a GRPO update, then uses rewards and regret estimates for future selection.Source: §3.3 · p. 4, Figure 2 · p. 5

3.3.1 REGRET AND PRIORITY

  • Problem-wise regret is positive when a problem is sometimes solved but not reliably solved, and zero when it is impossible or fully mastered.Level-wise regret aggregates problem-wise regret over problems drawn from that level; a high value indicates many problems in the informative regime.Source: §3.3.1 · p. 5
  • A level’s priority adds a staleness correction to its regret, while moving-window estimates let priority shift toward harder levels entering the informative band.The staleness term periodically revisits buffer levels and helps detect changes in policy capability on dormant levels.Source: §3.3.1 · p. 5

3.3.2 REGRET-GUIDED SAMPLING WITH EXPLORATION

  • Batch assembly samples buffer levels by priority, then replaces a fraction in expectation with uniformly sampled levels to explore beyond the buffer.Zipfian sampling concentrates training on the frontier while preserving diversity across the buffer; the expected exploratory fraction is ϕ.Source: §3.3.2 · p. 6

3.3.3 STATE-DEPENDENT MUTATION

  • State-dependent mutation perturbs nearby levels, focusing most mutation effort on informative levels while allowing easy or impossible levels to seed variants.This lets informative levels seed incrementally harder variants as the policy improves, while limiting the production of impossible levels.Source: §3.3.3 · p. 6

4 ALGORITHM

The algorithm initializes a level buffer, assembles batches through priority sampling with exploration and mutation, then generates problems and updates the policy with GRPO. Returns from trained levels refresh regret and priority for subsequent batches.

4 ALGORITHM

  • Grid seeding initializes the buffer with levels sampled across cells of the attribute space to provide broad difficulty coverage from the start.The stated purpose is to prevent random seeding from placing all initial levels in one difficulty band.Source: §4 · p. 6
  • Each batch draws buffer levels by Zipfian priority sampling, may replace them with novel exploratory or mutated levels, and evicts the lowest-priority level when the buffer is full.Exploration samples uniformly from the level space; mutation perturbs a single attribute, and only novel candidates enter the buffer.Source: §4 · pp. 6–7
  • The generator produces problems for each active level, policy rollouts supply rewards for a GRPO update, and trained-level returns refresh regret and priority.As regret falls for levels that have become easy, priority shifts toward higher-regret frontier levels in the next batch.Source: §4 · pp. 6–7

5 EXPERIMENTS

Frontier learning leads the standard-budget results on all three reported tasks and continues improving during longer DICE training, including on harder levels where baselines plateau. The experiments span five procedurally generated tasks and compare online frontier learning with fixed-buffer and fresh-level baselines.

5 EXPERIMENTS

  • Experiments cover five procedural tasks across puzzle, math, and graph domains, using Qwen3 variants plus Llama-3.2-3B-Instruct and Olmo3-7B-Instruct.Tasks are COUNTDOWN, SOKOBAN, DICE, DECIMAL ARITHMETIC, and LARGEST ISLAND.Source: §5 · p. 7
  • Baselines include fixed-buffer Uniform, SEC, and PLR and fresh-level Domain Randomization; evaluation uses unseen configurations and averages greedy accuracy over four seeds.Each evaluation level has 200 unique problems.Source: §5 · pp. 7–8

5.1 RESULTS

  • 50.9 ± 0.5% accuracy on COUNTDOWN versus 44.7 ± 1.7 for the strongest baseline; frontier learning also has the highest final performance on DECIMAL ARITHMETIC and SOKOBAN.Overall gains are smaller on the latter two tasks.Source: Table 1 · p. 8, §5.1 · p. 8
  • 71.8 ± 6.2% for the full DICE method; PLR+Explore reaches 33.5 ± 1.8, PLR+Mut 62.5 ± 4.5, and Unif+FlatMut 47.1 ± 10.8.The ablation findings identify mutation-driven expansion and regret-guided prioritization as important, with random exploration adding further gains, especially on Hard and Extra-hard levels.Source: Table 2 · p. 8, §5.1 · p. 9
  • 71.8 ± 6.2% accuracy on DICE after 500 steps, versus 33.3 ± 1.2 for SEC, 33.2 ± 2.1 for ACE-GRPO, and 25.4 ± 5.9 for rollout-matched DAPO.Standard baselines plateau after roughly 100–200 steps while frontier learning continues improving.Source: §5.1 · pp. 8–9
  • 72.6% accuracy on Hard and 20.7% on Extra-hard DICE levels, while standard baselines remain under 5.5% on those categories.Frontier learning reaches near-perfect performance on Easy and Medium configurations; its continued gains extend to harder regions.Source: §5.1 · p. 9

6 CONCLUSIONS

The conclusion presents frontier learning as online curriculum adaptation through a procedural generator, using buffer growth, regret-based prioritization, mutation, and exploration. The related work distinguishes its search of generator configuration spaces from fixed-pool curricula, ordered-difficulty generation, environment design, and LLM-based task generation.

6 CONCLUSIONS

  • Frontier learning combines a dynamic level buffer, regret-based priority, and mutation with exploration to adapt training distributions to evolving capability; experiments report outperformance across puzzle, math, and graph tasks.The approach is instantiated within GRPO.Source: §6 · p. 10
  • The approach is limited to procedural generators with continuous or discrete task-specific parameters; non-procedurally generated instances remain a future direction.Source: §6 · p. 10

A ADDITIONAL RELATED WORK

  • Unlike fixed-pool sampling and allocation methods, frontier learning searches a procedural generator’s configuration space and uses configuration-level regret to guide generation.The related-work discussion contrasts it with approaches that reallocate or prioritize problems in a fixed pool.Source: A ADDITIONAL RELATED WORK · p. 17
  • Unlike SCALER’s predefined difficulty ordering, frontier learning considers multidimensional generator configurations that need not have a known or monotonic difficulty order.Source: A ADDITIONAL RELATED WORK · p. 17
  • Related to UED replay-and-editing methods, frontier learning aggregates binary rollout regret by procedural configuration and uses it for replay, state-dependent mutation, and exploration.Its regret targets the outer edge of the productive region and guides where the configuration space expands.Source: A ADDITIONAL RELATED WORK · p. 17
  • Unlike LLM-based generation methods using proposers, teachers, or environment designers, frontier learning searches an existing generator’s explicit configuration space without an LLM proposal or rewriting stage.The paper frames this as trading generation flexibility for inexpensive, interpretable exploration and mutation.Source: A ADDITIONAL RELATED WORK · p. 17

B PROCEDURAL GENERATOR

The procedural generator varies Sokoban difficulty through grid size, box count, and maximum solution depth. Anchor levels specify task-attribute combinations, and the task spaces need not follow a monotone difficulty ordering.

B PROCEDURAL GENERATOR

  • Figure 4 contrasts Sokoban anchor levels L1 and L7, from a 6×6 grid with 1 box and depth ≤12 to a 10×10 grid with 6 boxes and depth ≤80.The figure also identifies walls, floor, boxes, goals, and boxes on goals by color.Source: Figure 4 · p. 18

C ANCHOR LEVELS DEFINITION

  • Sokoban levels vary grid size, number of boxes, and maximum solution depth; a smaller grid with more boxes can be harder than a larger grid with fewer.The generator scales these attributes, but their combinations do not enforce a strict monotone ordering on any single axis.Source: C ANCHOR LEVELS DEFINITION · p. 18, B PROCEDURAL GENERATOR · p. 18
  • The anchor-level table groups task-specific configurations into difficulty bins, including Extra Hard for Dice.It specifies Countdown, Sokoban, Decimal Arithmetic, and Largest Island attributes and their abbreviations.Source: Table 4 · p. 19

D IMPLEMENTATION DETAILS AND HYPERPARAMETERS

The implementation specifies shared training settings and additional stabilization for long runs. The learning-curve evaluation tracks accuracy over 200 GRPO steps on three tasks.

D IMPLEMENTATION DETAILS AND HYPERPARAMETERS

  • All methods use four random seeds, with runs assigned H200 or GH200 GPUs by task family.Qwen3-4B-Base and LLaMA3.2 use a 2048-token response limit; SOKOBAN uses Qwen3-4B thinking mode with 4096 tokens.Source: D IMPLEMENTATION DETAILS AND… · p. 20
  • The 500-step runs add a small KL loss and asymmetric clip-higher PPO clipping to prevent entropy collapse over extended training.These stabilization changes apply across all methods in the 500-step runs only.Source: D IMPLEMENTATION DETAILS AND… · p. 20

E LEARNING CURVES

  • Figure 5 tracks accuracy on fixed anchor evaluation sets over 200 GRPO steps for COUNTDOWN, SOKOBAN, and DECIMAL ARITHMETIC.The figure presents the standard-budget learning curves for these three tasks.Source: Figure 5 · p. 21, E LEARNING CURVES · p. 21

F PER-DIFFICULTY ACCURACY BREAKDOWN

Frontier Learning’s per-difficulty gains vary by task and difficulty, while extended training continues to improve validation accuracy on harder configurations. As training progresses, the learnable configurations expand toward harder regions.

F PER-DIFFICULTY ACCURACY BREAKDOWN

  • On COUNTDOWN and DECIMAL ARITHMETIC, gains concentrate primarily on Easy and Medium bins while Hard-bin performance remains largely comparable across methods.Figures 6 and 7 show the per-bin learning curves and final validation performance.Source: F PER-DIFFICULTY ACCURACY BREAKDOWN · p. 21, Figure 6 · p. 22, Figure 7 · p. 23
  • On SOKOBAN, Frontier Learning’s largest improvements occur on easier configurations, with smaller gains as difficulty increases.This pattern is reported for the per-difficulty breakdown.Source: F PER-DIFFICULTY ACCURACY BREAKDOWN · p. 21
  • Across the 500-step horizon, Frontier Learning continues to improve validation accuracy and reaches substantially higher accuracy on harder tasks.The extended-training result is illustrated in Figure 9, whose validation curves cover four difficulty bins for DICE.Source: F PER-DIFFICULTY ACCURACY BREAKDOWN · p. 21, Figure 9 · p. 24
  • Frontier Learning prioritizes levels near the current learning frontier rather than spending substantial compute on levels already mastered or too difficult to solve reliably.The frontier shifts as capabilities improve, with learnable configurations expanding toward harder regions.Source: F PER-DIFFICULTY ACCURACY BREAKDOWN · p. 21

G.1 FRONTIER LEARNING

The COUNTDOWN ablation tests sensitivity to the staleness coefficient and informative success-rate interval. Results vary only modestly around the default configuration.

G.1 FRONTIER LEARNING

  • Changing the staleness coefficient or informative success-rate interval produces only modest performance changes around the defaults.The sensitivity study uses Llama-3.2-3B-Instruct on COUNTDOWN; Table 7 reports the tested configurations.Source: §G.1 · p. 24, Table 7 · p. 25

G.2 STEP SIZE SENSITIVITY OF THE BASELINES

The DICE learning-rate sweep finds that neither baseline exceeds approximately 35% final accuracy at any tested rate. The paper’s 1 × 10−6 rate is best or joint-best for both baselines, while larger rates are worse and increasingly unstable.

G.2 STEP SIZE SENSITIVITY OF THE BASELINES

  • At rates ≥2 × 10−6, both baselines perform uniformly worse and become increasingly unstable, collapsing to near-zero accuracy at 2 × 10−6–5 × 10−5.The sweep tested DICE for 500 steps with seed 42 and held other hyperparameters fixed.Source: §G.2 · p. 25
  • The sweep tested 5 × 10−7, 9 × 10−7, 1 × 10−6, 2 × 10−6, 5 × 10−6, and 5 × 10−5 on DICE.Uniform is the non-regret-based baseline; SEC is the best-performing baseline in Table 2.Source: §G.2 · p. 25
  • ≈35%: no tested learning rate lifts either baseline’s final accuracy above this level, and 1 × 10−6 is best or joint-best for both.The baselines are Uniform and SEC.Source: Figure 11 · p. 25, §G.2 · p. 25

H COMPUTE ACCOUNTING.

Compute accounting finds comparable wall-clock cost on the 200-step tasks, but higher runtime and longer responses for frontier learning on DICE. Matched-time and Largest Island comparisons indicate that greater compute or longer generations alone do not account for its performance advantage.

200-step experiments

  • Across SOKOBAN, COUNTDOWN, and DECIMAL ARITHMETIC, frontier learning has comparable wall-clock cost to baselines and does not systematically generate longer responses.These are 200-step experiments; methods use the same number of optimizer updates, prompts, rollouts per prompt, and maximum response length.Source: H COMPUTE ACCOUNTING · p. 25, Table 8 · p. 25, Table 9 · p. 26

Long-horizon compute on DICE

  • 27.8±0.6 hours and 1383±26 tokens per response for frontier learning, versus 18.7±1.1 hours and 877 ± 65 for PLR on DICE.This is the 500-step DICE experiment.Source: H COMPUTE ACCOUNTING · p. 26, Table 11 · p. 27
  • After 500 steps, frontier learning reaches 72.6% on Hard and 20.7% on Extra-hard, while no standard baseline exceeds 5.5% on those categories.The passage describes the additional runtime as partly a consequence of capability expansion, rather than simply a larger nominal training budget.Source: H COMPUTE ACCOUNTING · p. 26
  • The longer DICE responses are consistent with movement toward harder configurations, and the added runtime is partly attributed to capability expansion.The comparison is against simply using a larger nominal training budget.Source: H COMPUTE ACCOUNTING · p. 26
  • 68.0 ± 5.5% accuracy at PLR’s 18.7-hour run time; PLR+Mut uses 28.5 ± 0.3 hours and longer responses (1426 ± 14 tokens) yet finishes lower.The matched-time comparison is on DICE.Source: H COMPUTE ACCOUNTING · p. 26, Table 12 · p. 27

Long horizon compute on LARGEST ISLAND

  • On LARGEST ISLAND, frontier learning’s runtime and response length sit in the middle of the baseline range, so its accuracy advantage is not attributed to greater time or longer outputs.The accounting covers the 500-step Olmo-3-7B-Instruct experiment.Source: H COMPUTE ACCOUNTING · p. 26, Table 13 · p. 27
Loading 2609.35426v1…