PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier

Lorenz Wolf, Connor Watts, Roger Creus Castanyer, Geoffrey Bradway, Maxwill Lin, Augustine N. Mavor-Parker, Matthew Daborn-Sargent

arXiv:2606.18284v1cs.LGcs.AIcs.CL

TL;DR

Training agents increasingly requires a supply of valid tasks at the current capability frontier, but evaluating such tasks with repeated solver rollouts is especially expensive for software engineering. PROPEL amortizes this cost by training an activation probe on offline solver-labeled tasks and using it as a frozen-reference reward during generator RL. Across math, code, and software engineering, the method shifts generation toward the learnable frontier, while experiments also expose transfer and diversity limitations.

  • Problem

    Fixed task distributions saturate, while synthetic tasks can be trivial, impossible, or ill-posed; direct generator optimization also requires expensive repeated solver rollouts.

    Source: §1 · pp. 1–2

  • Method

    PROPEL trains a probe once on solver-labeled tasks and frozen-reference activations, then uses its predicted utility as the generator’s RL reward.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    Across math, code induction, and software engineering, PROPEL increases generation near the learnable frontier, including code-induction rates from 10.1% → 20.0% and 5.3% → 12.6%.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Activation probes can provide a near-free surrogate for solver utility, making generator RL tractable in settings such as agentic SWE.

    Source: §1 · p. 2

    The probe is fixed after training, can degrade under policy drift, and still requires solver rollouts to construct its labeled corpus.

    Source: §7 · p. 10

Abstract

from arXiv · show

The limiting resource for training agents via reinforcement learning (RL) is increasingly frontier task supply: valid, solvable tasks just difficult enough to train the current model. As reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation yields tasks that are trivial, impossible, or ill-posed. Training a task generator with RL to optimize validity and learnability can address this bottleneck, but direct optimization requires repeated solver rollouts per candidate. For software-engineering (SWE) tasks, a single rollout can take tens of minutes; solver-in-the-loop generator training is intractable. We introduce PROPEL, a solver-amortized framework for training task generators at the targeted solve rate. PROPEL trains a lightweight activation probe on a one-time labeled corpus of generated tasks and solver outcomes. The probe predicts target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate: for coding, tasks generated at the learnable frontier increase from $10.1\% \rightarrow 20.0\%$ for a Qwen2.5-3B-Instruct solver and from $5.3\% \rightarrow 12.6\%$ for a Qwen2.5-7B-Instruct solver. For SWE, PROPEL increases the share of generations at the targeted solve rate from $9.8\% \rightarrow 19.6\%$ for Qwen3.5-27B on repositories not seen during training of probe and generator.

1 INTRODUCTION

PROPEL addresses the expensive solver-in-the-loop bottleneck by using activation probes as rewards for task-generator RL. Across math, code, and software engineering, it increases generation near the target solver’s learnable frontier while introducing diversity-mitigation methods.

1 INTRODUCTION

  • Solver-in-the-loop generator training is costly because SWE rollouts can take tens of minutes, while reliable solve-rate estimates require many trials per candidate.The same bottleneck appears more mildly in math and code generation.Source: §1 · pp. 1–2
  • PROPEL replaces repeated solver rollouts with a probe trained on offline solver-labeled tasks and evaluated through frozen-reference activations during RL.The pipeline uses a one-time labeled pool, probe training, probe-based rewards, and held-out solver evaluation.Source: §1 · p. 2, Figure 1 · p. 2, CONTRIBUTIONS · p. 3

CONTRIBUTIONS

  • 10.1% → 20.0% and 5.3% → 12.6%: PROPEL roughly doubles code-induction tasks at the learnable frontier for Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct solvers.The reported relative gains are +98% and +138%, respectively.Source: §1 · p. 2, CONTRIBUTIONS · p. 3
  • Fixed-probe optimization causes mode collapse to a semantic topic, while worst-case optimization mitigates it and maintains a +86% relative frontier-rate gain over base.Source: CONTRIBUTIONS · p. 3
  • Across math, code induction, and SWE, PROPEL yields significant utility gains over the baseline, including a 1.7× math lift and frontier-rate gains at larger SWE scale.The SWE experiments use a Qwen3.5-27B generator and solver, while the math result is measured on the oracle-validated output distribution.Source: CONTRIBUTIONS · p. 3
  • A probe trained on Qwen3.5-4B activations transfers across generator families to Mistral-7B-Instruct-v0.3 and Phi-3.5-mini-instruct without per-family retuning.This provides evidence that the encoded utility signal generalizes across model families.Source: CONTRIBUTIONS · p. 3

2 RELATED WORK

Prior work trains task generators through self-play, uses probes and internal states for prediction or reward, and studies learned-reward overoptimization. PROPEL combines these directions by replacing solver rollouts with a probe-based task-utility reward and extending feature-reward training to multi-turn SWE trajectories.

Synthetic task generation and self-play

  • Prior task-generation systems target the solver’s learnable frontier through proposer–solver self-play in deduction, abduction, induction, and software-engineering bug repair.Absolute Zero motivates induction-task design, while Self-play SWE-RL motivates rewards peaking at intermediate solver pass rates.Source: §2 · p. 3

Probes and internal-state rewards

  • Probes have been used for reasoning correctness, candidate scoring, judge calibration, and internal-state supervision before PROPEL applies them to task-generation utility.PROPEL extends RLFR by predicting task training utility from generator internal features and supports multi-turn SWE trajectories.Source: §2 · p. 4

Reward overoptimization and mode collapse

  • Learned reward optimization is associated with Goodhart effects and mode collapse, motivating probe co-evolution and diversity-oriented mitigation in PROPEL.Probe-high but solver-failed outputs become negatives for the next probe in the adversarial co-evolution procedure.Source: §2 · p. 4

3 PROBE REWARDS FOR OPTIMIZING PROBLEMS AT THE EDGE OF LEARNING

PROPEL defines task-generation utility around validity and target-solver learnability, then replaces expensive solver-in-the-loop rewards with activation-probe rewards from a frozen reference model.

3.1 PROBLEM SETTING

  • A useful generated task is valid and lies near the target solver’s learnable frontier, where the solver sometimes succeeds and sometimes fails.Validity is checked cheaply, while utility is computed from solver outcomes.Source: §3.1 · p. 4

3.2 UTILITY SIGNAL FOR TASK GENERATORS

  • The utility label marks tasks whose mean solve rate falls inside a target band: 1/8–3/8 for K=8, or 1/3–2/3 for SWE with K=3.Tasks below the band are too hard, while tasks above it are already too easy; SWE uses a broader band because solver trials are limited.Source: §3.2 · p. 4

3.3 PROBE-REWARD RL

  • PROPEL trains an activation probe offline on expensive solver labels, then predicts task utility from hidden states of a frozen reference generator during RL.The pipeline collects an offline task pool, computes utility, extracts reference activations, trains and selects probes, and optimizes the generator with probe rewards.Source: §3.3 · p. 5
  • The reward gates on validity and uses the probe’s raw logit to rank valid tasks, while invalid generations receive a fixed negative penalty.A validity predicate determines whether a task can be used; the probe ranks valid tasks by estimated frontier utility.Source: §3.3 · p. 5
  • Probe quality is assessed beyond classification accuracy because useful RL rewards must vary across tasks sampled from the base policy.Reward variance under policy, together with calibration and balanced accuracy, is used to diagnose whether a probe supplies actionable optimization signal.Source: §3.3 · p. 5
  • Generator optimization uses GRPO for math and code and DAPO-style advantage calculations for SWE, all with the probe-based reward.The expensive utility signal is therefore kept outside the generator’s RL loop.Source: §3.3 · pp. 4–5

4 EVALUATION

The evaluation measures realized target-band utility on fresh generations across math, code induction, and software engineering, using domain-specific validity and solver procedures.

4.1 METRICS

  • Fresh generations are evaluated by running the solver K times and reporting the share whose realized solve count falls inside the target utility band.The probe score is not used for evaluation; every result recomputes utility from solver rollouts.Source: §4.1 · pp. 5–6

4.2 DOMAINS

  • The study covers competition-style math, executable function-induction puzzles, and bug-writing tasks on real Python repositories.Math and AZR use Qwen3.5-4B generators with Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct target solvers; SWE uses Qwen3.5-27B.Source: §4.2 · p. 6
  • Math tasks are validated through symbolic answer equivalence and a strict two-oracle reference policy requiring agreement across four attempts.The generator draws competition-style prompts from GSM8K, MATH-500, or AIME-style pools.Source: §4.2 · p. 6
  • AZR tasks provide five input-output examples and a hint, requiring the solver to reproduce a seeded Python function on five hidden inputs.Executable ground truth eliminates the need for an oracle filter.Source: §4.2 · p. 6
  • SWE generation introduces behavior-changing bugs while preserving tests, and verification requires a clean patch, runnable suite, and at least one newly failing previously passing test.Each valid bug receives K=3 solver trials in fresh sandboxes under the OpenCode harness.Source: §4.2 · p. 6
  • SWE probe inputs aggregate activations across full multi-turn agent trajectories, with training using verifier scores and probe rewards while solver labels remain offline.Training uses repository sandboxes and up to 35 tool-using turns per trajectory.Source: §4.2 · p. 6

5 PROBE TRAINING DATA AND SELECTION

PROPEL trains one probe per domain and target solver from offline labeled data, selecting probes with held-out performance, calibration, and reward variance diagnostics.

5.1 PROBE-TRAINING DATA

  • SWE retains all 1,757 Probe-Train rows and uses class-reweighted loss because positive frontier tasks are sparse.Math and AZR instead use balanced probe-training datasets, whose sizes are summarized in Table 1.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 PROBE SELECTION

  • Table 2 reports one RL reward probe per domain–solver pair along with validation balanced accuracy, ECE, and base-policy RVP.SWE additionally applies post-hoc temperature calibration during RL scoring, while its RVP is omitted because of computational cost.Source: Table 2 · p. 7
  • Probe selection sweeps reference-model layers, pooling, probe heads, and— for SWE—trajectory pooling and class-imbalance handling.Only the selected probe for each domain–solver pair is used as the RL reward.Source: §5.2 · p. 7
  • Selection uses held-out balanced accuracy and calibration, with RVP as the final Math/AZR tiebreaker on 512 base-policy completions.SWE selection uses cross-repository folds so probes must transfer beyond repositories seen during probe training.Source: §5.2 · p. 7, Table 2 · p. 7
  • RVP ranges from 0.008 to 0.145, and downstream RL gains track it more closely than validation balanced accuracy.AZR has the highest RVP and largest gain, while math has the lowest RVP and slightly smaller gains.Source: §5.2 · p. 7

6 RESULTS

Across math, code induction, and software engineering, PROPEL improves utility and transfers beyond both probe and RL training distributions, while fixed-probe optimization can cause semantic mode collapse.

6 RESULTS

  • 53,664 versus 22,592 solver trials: PROPEL uses fewer trials offline and no solver calls during generator RL, yet achieves more than twice SIL’s utility improvement.The comparison is on AZR, where solver trials are much cheaper than in SWE; PROPEL replaces online solver feedback with a frozen-reference forward pass and probe head.Source: §6 · p. 7, Table 3 · p. 8
  • Across code induction, PROPEL roughly doubles utility U_S across the output distribution at both solver sizes, yielding significant gains over the baseline generator.Table 4 evaluates AZR, math, and SWE, defining utility as U_S(·) from Equation 1 and reporting validity, diversity, and top-topic metrics.Source: §6 · p. 7, Table 4 · p. 8
  • 2.0×: the same SWE checkpoint improves utility on 550 tasks from 11 held-out repositories never seen during RL.PROPEL also improves utility on repositories out of distribution for the probe, indicating gains beyond both training distributions.Source: §6 · p. 8, Table 4 · p. 8
  • 74%: single-probe optimization concentrates AZR generations on one semantic topic, while worst-case optimization reduces concentration from 0.74 to 0.69 for Qwen2.5-3B-Instruct.For Qwen2.5-7B-Instruct, worst-case optimization reduces top-topic concentration from 0.67 to 0.54 while retaining utility gains.Source: §6 · p. 8, Table 4 · p. 8
  • 39.6%→67.4% validity and 5.2%→9.6% utility: cold transfer to Mistral-7B-Instruct-v0.3 improves AZR without per-family tuning.On math, utility rises from 7.4% to 26.4% for Mistral-7B-Instruct-v0.3, although diversity metrics degrade; transfer to Phi-3.5-mini-instruct is more modest.Source: §6 · p. 9, Figure 3 · p. 9

Self-BLEU #

  • Increasing KL regularization trades lower utility and topic collapse for higher diversity, while hard-gate rewards provide the best joint operating point in the reported ablation.The math run fully collapsed at KL 0.02; the selected coefficients were β=0.1 for math and 0.05 for AZR.Source: Self-BLEU # · p. 9

7 CONCLUSION

PROPEL uses activation probes on frozen reference-model representations as a single-forward-pass surrogate for solve rate, shifting generators toward the learnable frontier across several domains. The conclusion emphasizes transfer potential while noting limited coverage, surrogate drift, mode collapse, and persistent offline labeling cost.

7 CONCLUSION

  • Activation probes trained on a frozen reference model provide a single-forward-pass surrogate for solve rate during task-generator RL.Optimizing this surrogate shifts coding-task generation toward the target solver’s learnable frontier.Source: §7 · pp. 9–10
  • Optimizing activation-probe rewards shifts generator outputs toward the target solver’s learnable frontier, roughly doubling the rate at which AZR generators produce frontier-band coding tasks.Source: §7 · p. 9
  • The activation signal reflects properties of the task itself rather than any particular generator.Source: §7 · p. 10
  • The evidence is limited to three domains, and broader families, scales, and task types are needed before treating the results as general.The probe can degrade under sustained policy drift, while current mode-collapse mitigations are partial and trade efficacy for diversity.Source: §7 · p. 10
  • The probe’s labeled corpus still requires solver rollouts, so PROPEL amortizes rather than eliminates the cost of constructing supervision.Future work also considers adapting the generator as the solver improves, but stabilizing the resulting inner-outer loop remains open.Source: §7 · p. 10

A SWE RL TRAINING RESULTS

The SWE generator-RL run raises the probe-based reward, while PROPEL’s out-of-distribution gain comes from shifting bugs away from trivial solves toward the targeted difficulty band.

A SWE RL TRAINING RESULTS

B WHERE THE SWE OOD GAIN COMES FROM

  • +9.8 percentage points: PROPEL’s OOD frontier-rate gain is roughly twice its in-distribution gain.The OOD base frontier rate is 9.8%, compared with 21.8% in-distribution.Source: B WHERE THE SWE OOD GAIN COMES FROM · p. 15
  • 19.6% OOD: PROPEL reduces trivial-solve rates to 19.6% while concentrating bugs in the target band of some-but-not-all solver trials.The corresponding in-distribution trivial-solve rate is 18.4%; the comparison is restricted to bugs with at least two valid solver trials.Source: B WHERE THE SWE OOD GAIN COMES FROM · p. 15, Table 5 · p. 15
  • 13.5 versus 14.3 lines: base OOD and in-distribution trivial-solve bugs have similar diff sizes, so the OOD gain reflects edit location rather than edit shape.The passage attributes the difference to shallow edits landing on less-critical control flow in unfamiliar repositories.Source: B WHERE THE SWE OOD GAIN COMES FROM · p. 15

C ADDITIONAL ANALYSIS ON SWE

Additional SWE analyses show that RL produces bugs with broader verifier failure surfaces and more efficient generator trajectories, while solve-count mass moves from saturated outcomes into the targeted band.

C ADDITIONAL ANALYSIS ON SWE

  • 5 failing tests: the median failure count rises from 1 to 5 on RL-Train and from 2 to 5 on Held-Out Eval after RL.The share failing at least 10 tests rises from 11.9% to 28.9% on RL-Train and from 18.0% to 30.4% on Held-Out Eval.Source: C ADDITIONAL ANALYSIS ON SWE · p. 16, Table 6 · p. 17
  • 26.8 → 15.1 turns: generator trajectories shorten while patch validity rises from 0.58 to 0.97, bug validity from 0.32 to 0.79, and shaped reward from 0.28 to 0.68.Clean stopping increases from 151 to 247 of 256 episodes, while max-turn and length terminations fall from 61 and 44 to 2 and 7.Source: C ADDITIONAL ANALYSIS ON SWE · p. 16, Table 7 · p. 17

D SOLVE-COUNT DISTRIBUTIONS

  • The main solve-count mechanism is redistribution from the saturated 8/8 bin into the 1–3@8 target band used to train the probe.Figure 6 compares PROPEL and base-policy solve-count distributions and includes PROPEL WCO for AZR.Source: D SOLVE-COUNT DISTRIBUTIONS · p. 16, Figure 6 · p. 16

E AZR 3B ADVERSARIAL COEVOLUTION

AZR adversarial coevolution constrains proxy optimization with an auxiliary probe, while the broader pipeline filters repositories, constructs disjoint splits, and selects probes using cross-repository validation.

E AZR 3B ADVERSARIAL COEVOLUTION

  • False positives are valid tasks scored as useful by the probe but outside the target solver solve-count band; they become negatives for an auxiliary probe and constraints during later RL.The original probe continues supplying utility, while valid-task reward uses the minimum of the two probe scores.Source: E AZR 3B ADVERSARIAL COEVOLUTION · p. 17
  • 99.7% validity and 88.6% all-8 solves: using the auxiliary probe alone avoids one collapsed family but shifts generation toward easy string-manipulation tasks.The standalone run achieves only 1.8% Utility, so the auxiliary probe is retained as a constraint rather than replacing the utility objective.Source: E AZR 3B ADVERSARIAL COEVOLUTION · p. 17
  • 0.1009 → 0.1533 Utility: the false-positive variant improves Utility and Valid from 0.6654 to 0.8398 while keeping diversity metrics close to base.It recovers most of the unconstrained single-probe Utility of 0.1995 without its semantic concentration.Source: E AZR 3B ADVERSARIAL COEVOLUTION · p. 17, Table 8 · p. 17

F.2 SPLITS

  • Repositories never overlap across Probe-Train, RL-Train, and Held-Out Eval, whose roles are probe training, generator-RL evaluation/training, and strict post-RL generalization.Probe training and held-out scoring use the ≥2-valid-trial filter, retaining 73.2%, 78.0%, and 84.5% of the respective rows or usable rows.Source: §F.2 · p. 18, Table 9 · p. 18

F.1 REPOSITORY SOURCING

  • 29.8%, 15.5%, and 8.9%: dataset-wide repository yields for Probe-Train, RL-Train, and full Held-Out Eval measure episodes passing bug validity and producing at least two valid solver trials.The pipeline excludes flaky or failed repositories, ranks survivors by tests-per-entity, and drops repositories with no usable labeled trajectories.Source: §F.1 · p. 18, §F.3 · p. 19

G PROBE TRAINING AND SELECTION DETAILS

  • The probe grid uses pooled frozen-reference activations, covering 48 Math/AZR cells and 270 SWE cells with trajectory pooling and class-imbalance variants.Math and AZR use one pooled activation per rendered task; SWE pools one activation per generated assistant turn across multi-turn trajectories.Source: G PROBE TRAINING AND SELECTION DETAILS · p. 19, Table 10 · p. 19

G.1 PROBE-SWEEP RESULTS AZR/MATH

G.2 PROBE SWEEP RESULTS SWE

SWE probes are selected through a large cross-repository sweep, with reward variance serving as a complementary diagnostic rather than a universal selector.

G.2 PROBE SWEEP RESULTS SWE

  • 270 cells: the SWE sweep varies layer, trajectory pooling, probe architecture, and class-imbalance strategy, selecting epochs by five-fold cross-repository balanced accuracy.Each sweep cell is trained three times with seeds 0, 1, and 2 and averaged.Source: §G.2 · p. 22
  • The selected SWE configuration is the weighted-loss probe with the best cross-fold balanced accuracy, averaged over three random seeds.The held-out Spearman correlation is computed on a cross-repository held-out fold, though the supplied text does not provide its value.Source: §G.2 · p. 22, Table 11 · p. 22

H PROBE SELECTION: RVP VALIDATION PANEL

  • 0.735 Pearson r: reward variance under policy correlates positively with realized short-RL reward gain across the 12-probe panel.Within Math the correlation is 0.95, whereas AZR candidates cluster in a narrow high-variance band with correlation −0.20, so RVP is used as a coarse sanity check.Source: H PROBE SELECTION: RVP VALIDATION PANEL · p. 22

I RL HYPERPARAMETERS

  • SWE RL uses the SkyRL stack across 3 nodes and 24 H100s, while Math and AZR use a shared recipe on 4 GPUs per run.KL coefficient, group size, and checkpoint are screened using validity, diversity, and collapse alerts.Source: I RL HYPERPARAMETERS · p. 22

J QUALITATIVE EXAMPLES: BASE VS RL ON AZR

The qualitative examples contrast base-generator failures with PROPEL tasks in the learnable-frontier band, while the domain prompts specify validity and task-construction constraints for math, induction, and SWE generation.

J QUALITATIVE EXAMPLES: BASE VS RL ON AZR

  • 2 of 8 solved places a PROPEL-generated AZR task in the targeted learnable-frontier 1–3@8 band, contrasting with base outputs at 0 of 8 and 8 of 8.The examples are drawn from confirmatory multi-seed runs underlying Table 4; the base cases illustrate unsolvable or mis-specified and trivial or saturated tasks.Source: J QUALITATIVE EXAMPLES: BASE VS RL ON… · pp. 22, 24, 25

K.1 MATH DOMAIN

  • Math prompts require one self-contained problem with a definite answer from a different mathematical area, using only the specified problem-output format.The template also requests a creative twist or unexpected connection and forbids solutions, hints, teaching notes, and extra commentary.Source: §K.1 · p. 26

K.2 AZR FUNCTION INDUCTION DOMAIN

  • The AZR generator requires a deterministic Python function with at least one argument, bounded execution, exactly specified input blocks, and a natural-language message describing the function.The message is intended to help a test subject recover the function from input-output pairs, without including the code snippet.Source: §K.2 · p. 27

K.3 SWE DOMAIN

  • The SWE interaction format constrains the agent to a thought section followed by one bash code block containing one command or a connected command sequence.The shell is located at /workspace, and the response format is enforced by rejection rules.Source: §K.3 · pp. 27–28
  • SWE prompts target subtle bugs that fail existing tests, have exactly one correct fix, and are difficult to diagnose from test errors alone.The instructions favor changes in data flow or edge-case behavior, require codebase browsing and test inspection, and reject simple condition flips as too easy.Source: §K.3 · p. 28

Related papers

Loading 2606.18284v1…