Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence

Motivation

• Scaling beyond human supervision addresses how LRMs can continue improving when human judgment and design cannot cover increasingly large and difficult learning experience.Deterministic verification supports mathematics and code, but open-ended tasks require proxy rewards and increasingly costly human evaluation.Source: §1 · p. 3

Framework

• The framework analyzes this transition through connected reward and experience axes: how behavior is evaluated and how learning trajectories are produced.Human intent remains essential, while recurring supervision during learning increasingly shifts toward model-generated feedback and experience.Source: §1 · p. 3

Reward axis

• Scalable rewards broaden reinforcement learning beyond exactly verifiable tasks, but reduced independent grounding increases vulnerability to miscalibration, correlated errors, and reward exploitation.Candidate signals include learned evaluators, rubrics, language-model judges, internal certainty, response agreement, references, and interaction outcomes.Source: §1 · p. 3

Experience axis

• Experience generation progresses from fixed human-written tasks toward self-generated curricula, executable environments, live-interface tasks, and reusable information, provided experience remains diverse, solvable, relevant, and dynamically valid.The central concern shifts from trajectory quantity to the quality and sustainability of generated learning processes.Source: §1 · pp. 3–4

Co-evolution

• Rewards and experience must co-evolve: stronger policies expose weaknesses that guide new tasks and environments, while reliable evaluation converts resulting trajectories into further updates.This coupling can also produce failure when policies, generators, and evaluators adapt together while independent performance stagnates or declines.Source: §1 · p. 4

Receding-supervision ladder

• The five-level L0–L4 receding-supervision ladder tracks which learning components still require continued human provision, from per-instance judgments to co-evolving policies, rewards, tasks, and environments.The ladder describes operational responsibility during learning, not capability, reliability, alignment, or the amount of human knowledge embedded in the infrastructure.Source: §1 · p. 4, Figure 1 · p. 4

Evaluation framework

• The paper evaluates autonomous learning systems through three complementary objects: policy capability, feedback fidelity, and experience quality.These distinctions organize discussion of reliable rewards, sustainable curricula, valid environments, efficient learning, and safe autonomous improvement.Source: §1 · pp. 4–5

2. Preliminaries

• The preliminaries define notation and a common learning framework linking reasoning, rewards, policy optimization, uncertainty, and the five-level supervision ladder.Table 1 summarizes the paper’s notation.Source: §2 · p. 5, Table 1 · p. 5

2.1. From Single-Turn MDP to Sequential POMDP

• MDPs model reasoning actions, transitions, scalar rewards, and discounting, while POMDPs extend them to partially observed sequential settings using interaction histories.Single-turn RLVR is the T=1 special case; longer horizons introduce partial observability and long-horizon credit assignment.Source: §2.1 · pp. 5–6

2.2. The Reward Function as the Object of Scaling

• Reward scaling separates feedback provenance from experience generation, progressing from per-instance human judgments to reusable evaluators, model signals, references, and environment outcomes.Automated rewards may still depend on human-created references, rules, or environments.Source: §2.2 · pp. 6–7

2.3. Policy Optimization

• Policy optimization determines how rewards update the model: PPO uses a learned value baseline, whereas GRPO uses group-relative advantages, clipped updates, and KL regularization without a value network.PPO’s value model adds substantial memory and computation costs in common LRM implementations.Source: §2.3 · pp. 7–8

2.4. Uncertainty Measures

• Sequence surprisal and predictive token entropy provide internal uncertainty signals, but neither establishes correctness without additional calibration or grounding.Lower surprisal indicates higher policy probability, while lower token entropy indicates a more concentrated next-token distribution.Source: §2.4 · p. 8

2.5. A Five-Level Ladder of Receding Supervision

• The five-level ladder tracks receding supervision across two axes: feedback moves beyond direct human evaluation, while experience shifts from fixed human design toward generated or discovered settings.L0 uses per-instance human supervision; L1 reuses human-grounded evaluators; L2 uses nonhuman evaluation with externally provided tasks and environments.Source: §2.5 · p. 8

L3 Experience beyond Human Design.

• L3 generates or reorganizes experience under broad human goals, while L4 co-evolves feedback, experience, and policy with human oversight concentrated on intent, safety, and independent evaluation.The ladder classifies which learning component becomes less human-dependent, and increasing autonomy risks shared model mistakes, unintended behavior, invalid curricula, and narrow self-play experience.Source: L3 Experience beyond Human Design · pp. 8–9

• Reward beyond human evaluation derives learning signals from model statistics, stored references, or observable environment outcomes.These sources extend beyond per-instance human-provided answers, preferences, or acceptance criteria.Source: §3 · p. 9, Figure 3 · p. 10
• Human-grounded evaluators reuse human criteria to score many outputs, while self-trained evaluators form a boundary toward less human-dependent reward evidence.Source: §3 · p. 9, Figure 3 · p. 10
• Whether a policy is optimized online or offline is independent of the reward-evidence regime.Source: §3 · p. 10

3.1. Per-Instance Human Supervision

• Per-instance supervision (L0) assigns each example a human- or expert-supplied comparison, quality judgment, reference answer, specification, or test suite, after which reward computation can be automatic.Automatic scoring does not eliminate per-instance supervision because extending training to new tasks still requires new targets (Ouyang et al., 2022; Guo et al., 2025a).Source: §3.1 · p. 10
• Rule-based rewards substantially advance mathematical reasoning and code generation when answers can be matched or programs compiled and tested (Shao et al., 2024; Guo et al., 2025a; Yu et al., 2025c; Lin et al., 2025b; Mu et al., 2025; Liu et al., 2026c).Mathematical outputs are checked against ground-truth answers, while generated programs are executed against predefined tests.Source: §3.1 · pp. 10–11
• Scaling this regime becomes difficult because every new task requires an answer, test, preference, or comparable criterion, with expert and open-ended domains especially costly.Open-ended tasks may admit multiple valid responses under context-dependent standards.Source: §3.1 · p. 11
• Moving beyond per-instance supervision requires reusing human judgments across many instances or sourcing evaluation differently, corresponding to L1 and L2.This need arises as the amount and difficulty of generated experience increase.Source: §3.1 · p. 11

3.2. Human-Grounded Evaluation

• At L1, reusable reward models and LLM judges apply human preferences, annotations, or criteria across queries, enabling graded and reference-free evaluation while remaining human-grounded.DeepSeekMath V2 uses a generative verifier to assess theorem-proving correctness, completeness, and rigor, including identified flaws (Shao et al., 2025b).Source: §3.2 · p. 11

3.2.1. Outcome and Process Reward Models

• Learned reward models provide partial-credit or process-level feedback, while automatically generated step labels can lag behind LLM or human judgments, with consensus filtering improving reliability.Math-Shepherd automatically annotates process steps by sampling continuations from a reasoning prefix and assessing whether they reach the reference answer; PRIME derives implicit process rewards from outcome labels and policy rollouts (Cui et al., 2025a).Source: §3.2.1 · p. 11

3.2.2. LLM Judges and Rubric-Based Evaluators

• LLM judges and rubric-based evaluators scale multidimensional assessment across open-ended tasks, using references, evidence, checklists, adaptive criteria, and group-wise comparisons to improve reliability.Specialized judges can add domain evidence and criteria, but their rewards remain human-grounded when records, conversations, and rubric standards determine what they value (Team et al., 2025b).Source: §3.2.2 · p. 13

3.2.3. Self-Trained Evaluators and Boundary Cases

• Self-trained evaluators reduce labeled-preference requirements by generating, judging, and filtering their own comparisons, yet their ladder position depends on remaining supervised initialization, prompts, formats, or human criteria.Offline preference optimization changes how feedback is consumed rather than where its evaluative standard originates, and fixed data can lose coverage as policies change (Rafailov et al., 2023; Casper et al., 2023; Xiong et al., 2024).Source: §3.2.3 · p. 14

3.3. Reward Sources beyond Human Evaluation

• L2 rewards comprise model-derived, reference-anchored, and environment-grounded signals, distinguished by the evidence used to score rollouts rather than by each artifact’s historical origin.Reusable references and fixed environment dynamics can evaluate many new trajectories, while retrieval lies between these cases because terminal rewards may still use known answers or task rules.Source: §3.3 · p. 14

3.3.1. Model-Derived Signals

• Model-derived rewards use confidence, entropy, consensus, or trajectory consistency without external correctness guarantees, so optimization can elicit existing capabilities but also produce overconfidence, confirmation bias, or collapse.Entropy minimization requires useful initial probability mass and confidence–correctness correlation; consensus follows the model’s own answer distribution and may become wrong or self-reinforcing.Source: §3.3.1 · pp. 15–16

3.3.2. Reference-Anchored Signals

• Reference-anchored rewards compare sampled reasoning with an available target through likelihood, variational, perplexity, reconstruction, or process-based signals, providing external evidence while inheriting target coverage and supervision limits.These methods can shape reasoning without a separately trained evaluator for every rollout, but a single reference may cover only part of the acceptable answer space.Source: §3.3.2 · p. 17

3.3.3. Environment-Grounded Signals

• Environment-grounded rewards score executable consequences, formal acceptance, game outcomes, or observable states, separating policy evidence more strongly than confidence or consensus but remaining vulnerable to incomplete tests, inaccurate simulators, exploitable rules, and specification mismatch.Tool use or interaction alone does not establish environment-grounded reward; the terminal evaluation must be tied to the resulting state or outcome.Source: §3.3.3 · pp. 18–20

3.4. Limitations and Failure Modes

• Automatic rewards remain trustworthy only when their supporting evidence is independent, valid, and aligned with the intended success criterion.Human-grounded, model-derived, reference-anchored, and environment-grounded signals can each misrepresent quality or reinforce policy errors.Source: §3.4 · p. 20

3.4.1. Failures of the Reward Signal

• Reward signals fail through judge bias, consensus collapse, incomplete references, verifier noise, specification exploitation, and coupled policy-evaluator errors.False positives are generally more damaging than false negatives, while recursive self-evaluation requires evidence independent of both actor and judge.Source: §3.4.1 · pp. 20–21

3.4.2. Capability Expansion and Transfer

• RLVR can improve sampling efficiency without expanding capability, although prolonged training sometimes increases rare high-reward actions and expands boundaries on difficult tasks (Yao et al., 2025; Liu et al., 2025f).Exact support cannot exceed initially possible outputs under standard online optimization, and finite sampling can shrink observed support.Source: §3.4.2 · p. 21
• Transfer is task- and base-model-dependent: RL can preserve nonmathematical abilities more reliably than SFT, but superficial heuristics and limited generalization remain possible (Huan et al., 2026; Alam & Rastogi, 2025; Li et al., 2026b).Reward cost, independence, and validity are distinct properties, and reaching L3 additionally requires generating or discovering the experience itself.Source: §3.4.2 · p. 21

4. Experience beyond Human Design

• The experience axis progresses from fixed human-designed experience through task generation, environment construction, integrated pipelines, and autonomous co-evolution, as summarized in Figure 4.The framework tracks which of the task distribution Π, environment dynamics (S, P), and reward R remain externally supplied or adapt within learning.Source: Figure 4 · p. 22, §4 · p. 22

4.1. Fixed Human-Designed Experience

• Fixed human-designed tasks and environments isolate reward comparisons but constrain coverage, diversity, and horizon because new tasks and dynamics remain outside the learning loop.Changing reward sources cannot reveal domains absent from ΠH or repair missing tools, states, causal mechanisms, or poorly calibrated curricula in EH.Source: Table 3 · p. 23, §4.1 · pp. 24–25

4.2.1. Task Generation in Fixed Environments

• Task generation adapts the task distribution while retaining a fixed environment, with learnability objectives targeting problems near the solver’s competence frontier.Self-generated traces, instructions, agentic tasks, proposer-solver self-play, corpus-grounded self-play, and reward-free curricula reduce human task curation but do not independently change both experience sources.Source: §4.2.1 · pp. 25–26

4.2.2. Environment Construction for Fixed or Seeded Tasks

• Environment construction adapts (S, P) around fixed or seeded tasks through adversarial design, executable code, synthesized tool worlds, learned dynamics, live applications, and persistent memory.Regret and related difficulty signals favor environments that challenge but do not overwhelm the current policy, while validation and correction address the brittleness of interactive worlds.Source: §4.2.2 · pp. 26–27

4.3. Integrated Experience Generation

• Integrated experience generation begins at L3, where limited external seeds can support ongoing training trajectories by jointly supplying or adapting tasks, environments, and feedback.Agent World Model, ScaleEnv, EvoEnv, and related systems generate both interaction settings and tasks, whereas ZeroGUI and CuES represent boundary cases with some externally retained components.Source: §4.3 · p. 28

4.4. Autonomous Co-evolution

• Autonomous co-evolution corresponds to L4: reward acquisition, task generation, environment construction, and policy improvement form a persistent adaptive loop in which each component reshapes the others.Existing systems couple subsets of this loop through generator-verifier feedback, proposer-solver self-play, adaptive opponents, learned environments, difficulty-aligned curricula, or self-modifying agents; the full regime requires joint improvement without an externally maintained curriculum.Source: §4.4 · pp. 28–29

4.5. Failure Modes across the Experience Axis

• Poor difficulty calibration can make group rewards nearly constant and eliminate GRPO advantages, while noisy solve-rate estimates weaken methods designed to preserve informative difficulty ranges.The learnability signal, regret objective, and frontier-targeting challenger address this failure mode, including in R-Zero (Huang et al., 2025a).Source: §4.5 · pp. 29–30
• Coupled policy, verifier, generator, and environment loops can produce reward hacking, internally consistent but externally invalid conventions, and unreliable self-verification.Unit tests, code verification, and game outcomes constrain these behaviors, but systematic bias remains; reliable verification may require independent audits and its own objective.Source: §4.5 · p. 30
• Self-generated curricula can collapse toward familiar regions, reducing task diversity; multidimensional filtering, adversarial objectives, and interestingness models provide pressure to keep curricula expanding.These mechanisms are used in SeRL, LSP, and open-ended learning to counter distributional narrowing and mode collapse.Source: §4.5 · p. 30
• Changing environments or opponents increase gradient variance and compound long-horizon credit assignment when scalar outcomes must be distributed across self-generated trajectories.Role-conditioned or group-relative baselines can improve stability in multi-agent training.Source: §4.5 · p. 30
• Constructed environments inherit model errors and biases, can make verification costly, and may plateau below oracle supervision, motivating independent anchors in autonomous pipelines.Vision-language reward estimation and world-state modeling can propagate inaccuracies through every collected trajectory.Source: §4.5 · p. 30
• Autonomous experience must remain grounded through consensus, execution feedback, corpus grounding, or occasional audits, with evaluation covering capability, feedback fidelity, experience quality, and reproducibility.The dominant constraint varies by domain: executable checkers help mathematics and code, environment fidelity dominates tool use and control, and self-play risks proposer-judge collusion in open-ended settings.Source: §4.5 · p. 30, Figure 5 · p. 31

5. Evaluation, Datasets, and Benchmarks

• Evaluation must separately assess policy capability, feedback fidelity, and experience quality rather than treating held-out performance as sufficient evidence.The three objects become jointly important as reward acquisition and experience generation enter the learning loop.Source: §5 · p. 31
• Across ladder levels, evaluation expands from human-defined held-out standards to audits of reusable rewards, task quality, and long-term learning-loop stability.At L0, humans define the standard; L1–L2 require evaluator and reward audits, L3 adds task and environment quality, and L4 requires longitudinal evidence.Source: §5 · pp. 31–32

5.1. Capability Evaluation

• Capability tests measure whether π_θ solves intended problems under a held-out standard, independently of how its training reward was produced.Model-reward-trained policies can be tested on human-authored mathematics, while agents trained in evolving environments can be evaluated in fixed sandboxes.Source: §5.1 · p. 32

5.1.1. Hard-verifiable reasoning

• Hard-verifiable tasks provide scalable, comparatively reliable external anchors through exact answers, unit tests, game outcomes, or constraint satisfaction.Representative coverage includes mathematics, code, repository repair, logic puzzles, games, and multiple-choice reasoning, with disjoint tests needed to limit leakage for executable-reward methods.Source: §5.1.1 · pp. 32–33

5.1.2. Semi-verifiable and expert-graded tasks

• Semi-verifiable and expert-graded benchmarks extend capability evaluation to broad knowledge, multi-step reasoning, and specialized domains while retaining residual expert grounding.Their expensive labels make them useful transfer anchors, but they assess transfer under expert grounding rather than independence of the training reward.Source: §5.1.2 · pp. 33–34

5.1.3. Open-ended generation and interactive preference

• Open-ended evaluation relies on preferences, rubrics, or live comparison, creating evaluator entanglement and sensitivity to judge choice, verbosity, response order, and stylistic bias.Fixed judge panels, independent human agreement checks, and external safety suites help address these risks, but judge bias can still confuse fluency with substantive quality.Source: §5.1.3 · pp. 34–35
• Capability benchmarks cannot establish faithful training feedback or informative experience: gains may instead reflect miscalibrated rewards, narrowed curricula, or exploitation of evaluator preferences.The paper therefore treats capability evidence as necessary but insufficient before directly evaluating feedback and experience.Source: §5.1.3 · p. 35

5.2. Feedback Fidelity

• Feedback fidelity evaluates the reward mechanism itself—outcome or process models, LLM judges, and model-derived signals—not only the resulting policy.The central question is whether the signal represents the intended criterion rather than a convenient correlate.Source: §5.2 · p. 35, §5.2.1 · p. 35

5.2.1. Outcome reward models

• Outcome-reward evaluations span pairwise preference accuracy, unseen prompts, best-of-four selection, downstream utility, long-form sensitivity, personalization, robustness, overoptimization, and multimodal or multi-attribute criteria.RewardBench (Lambert et al., 2024), RewardBench 2 (Malik et al., 2025), Long-form RewardBench (Huang et al., 2026c), and Personalized RewardBench (Ma et al., 2026) broaden coverage beyond simple preference accuracy; RM-Bench (Liu et al., 2024e), PPE (Frick et al., 2025), RewardMATH (Kim et al., 2024a), VL-RewardBench (Li et al., 2024c), HelpSteer2 (Wang et al., 2024d), and RewardAnything (Yu et al., 2025e) target specific failure modes and conditioning.Source: §5.2.1 · p. 35

5.2.2. Process reward models

• Process reward models must be evaluated on step-level error detection as well as final-answer discrimination, using measures such as step-level F1 or PRM-Score.ProcessBench (Zheng et al., 2025a), PRMBench (Song et al., 2025b), Socratic-PRMBench (Li et al., 2025g), and VILBENCH (Tu et al., 2025) assess process-level credit assignment, including vision-language reasoning.Source: §5.2.2 · p. 35

5.2.3. LLM judges and verifier utility

• LLM judges require tests of human agreement, factual discrimination, utility, modality and context robustness, instruction adherence, pipeline quality, and position bias, while judge reuse for final evaluation creates circularity.A disjoint judge, held-out human subset, or verifiable proxy provides a more credible improvement measure than reusing the training judge family.Source: §5.2.3 · p. 36

5.2.4. Intrinsic certainty and consensus signals

• Intrinsic rewards based on uncertainty, self-consistency, likelihood, or references require calibration, sampling and distribution-shift tests, reference-sensitivity checks, and external audits.The audit should match the training mechanism: frozen reward models need RewardBench-style or downstream-utility tests, judge-trained policies need independent evaluation, process-supervised policies need step and outcome measures, and intrinsic-reward policies need calibration and external-checker audits.Source: §5.2.4 · p. 36

5.3. Experience Quality

• Experience evaluation must examine the task and environment stream because internal reward can rise despite curriculum narrowing, inaccurate dynamics, or verifier co-adaptation.The relevant objects are generated or adapted task distributions Π and environment dynamics (S, P), alongside the trained policy.Source: §5.3 · p. 36

5.3.1. Agentic and interactive environments

• Interactive benchmarks cover web, desktop, mobile, multimodal, workplace, scientific, game, and tool-use settings, while synthesized environments require validation, reproducibility, criterion correctness, frozen-policy solve rate, and public-suite transfer.These measures can separate policy performance from execution conditions by varying agent scaffolds and environmental volatility, and can test whether generated worlds reproduce established benchmark transitions.Source: §5.3.1 · pp. 36–37

5.3.2. Task quality under self-generated curricula

• Self-generated curricula should pair solver accuracy with learnability, diversity, novelty, rejection, retention, and filtering statistics to detect recycling, mastered-skill repetition, invalid proposals, and corrupted interactive trajectories.The learnability score p̄(1 − p̄) peaks at intermediate solve rates and vanishes when all sampled solutions succeed or fail, but it does not measure novelty or semantic diversity.Source: §5.3.2 · p. 37

5.3.3. Closed-loop evaluation

• Autonomous co-evolution requires frozen external tests, curriculum-diversity trajectories, training–held-out-verifier agreement, component-freezing ablations, and transfer from synthetic environments to public suites.PAST-Bench tests retained experience through matched task sequences, while AI4AI-Bench reruns modified procedures from scratch with a fixed evaluator; success on either component is not evidence of a fully autonomous loop. Generated-world success may reflect generator-specific regularities if it fails to transfer to WebArena or OSWorld (Zhou et al., 2024; Xie et al., 2024b).Source: §5.3.3 · pp. 37–38

Dataset roles

• Dataset reuse across training, evaluation, and evaluator construction can invalidate apparent independence, so roles must be stated explicitly.The key distinction is whether the same data, evaluator, or environment enters both learning and final assessment.Source: §5.4 · p. 38

Verifiable training and evaluation sets

• Verifiable suites enable inexpensive objective training and evaluation, but saturation, contamination, leakage, and protocol bias require complementary static and refreshed benchmarks.Fresh data, perturbations, and closed-test construction reduce particular leakage risks without eliminating protocol bias.Source: §5.4 · p. 38

Preference and evaluator-development sets

• Preference and process-reward datasets support evaluator development, but evaluators require disjoint tests to measure general fidelity rather than familiarity with annotation schemes.RewardBench-style data (Lambert et al., 2024; Malik et al., 2025), HelpSteer2 (Wang et al., 2024d), ProcessBench (Zheng et al., 2025a), PRMBench (Song et al., 2025b), and Socratic-PRMBench (Li et al., 2025g) instantiate differing judgment standards.Source: §5.4 · p. 38

Interaction logs and environments

• Long-horizon evaluation needs interaction logs, tool-use corpora, and frozen synthesized-environment splits with generator prompts and validation scripts to make environment fidelity auditable.WebArena-style environments (Zhou et al., 2024; Xie et al., 2024b; Chezelles et al., 2025) and tool-use corpora (Qin et al., 2024) supply trajectories for this evaluation.Source: §5.4 · p. 38

Sampling, decoding, and comparability

• Complete reporting must control decoding, sampling, budgets, tools, observations, resets, seeds, and environment versions because protocol differences can dominate attributed learning-signal effects.Greedy and sampled results answer different questions and require separate reporting; agent evaluations additionally require action budgets and environment specifications.Source: §5.4 · p. 38

Cross-domain transfer

• Cross-domain transfer should report training-domain performance, target-domain performance, and their gap, since in-domain gains without transfer may reflect overfitting, reward exploitation, or curriculum narrowing.Improvements across science QA, tool use, or open-ended interaction provide stronger evidence of generalization, though further analysis is needed (Wei et al., 2024; Wang et al., 2023b; Mialon et al., 2024).Source: §5.4 · pp. 38–39

Outcome and stability metrics

• Exact match, numerical equivalence, unit-test Pass@k, mean sample accuracy, G-Pass@k, and CoT-Pass@k measure correctness and increasingly reliable reasoning across repeated samples.CoT-Pass@k additionally requires a valid reasoning trace, distinguishing occasional success from reliable reasoning.Source: §5.5 · p. 39

Preference and judge metrics

• Preference and open-ended evaluations use pairwise accuracy, best-of-four selection, downstream-RLHF-linked metrics, win rates, rubric scores, structured G-Eval, and process-fidelity measures such as step-level F1 or PRM-Score.Judge-based results require diagnostics for position, length, and style bias.Source: §5.5 · p. 39

Experience-quality metrics

• Self-generated tasks and synthesized environments require solve-rate, learnability, diversity, novelty, filtering, executability, validation, transition-fidelity, and transfer metrics, while closed loops track internal reward against held-out capability.Persistent divergence between internal and external signals warns of reward exploitation or collapse but does not identify a unique cause.Source: §5.5 · p. 39

Contamination, bias, and tooling

• Static benchmark leakage requires contamination analyses and refreshed or procedurally varied suites, while LLM evaluators require explicit position- and style-bias audits.ConTAM (Singh et al., 2024), ConStat (Dekoninck et al., 2024), and related analyses illustrate vulnerability to leakage.Source: §5.5 · p. 39
• Community harnesses standardize prompting, decoding, and aggregation, but reproducible reports must specify decoding seeds, judge prompts, environment settings, and exact harness revisions.These choices can materially change reported comparisons.Source: §5.5 · p. 40

Recommended minimum scorecard

• The recommended scorecard combines frozen in-domain and transfer tests, feedback-source audits, experience-quality measurements, contamination analysis, compute-matched baselines, and longitudinal external evaluation for methods approaching L4.L4-oriented methods should compare internal reward trajectories with frozen external capability evaluations.Source: §5.5 · p. 40

• Evaluation practice is stronger for policy capability than feedback fidelity or experience quality, allowing internal rewards to rise without unchanged evidence of general capability.Public leaderboards rarely test agreement between training judges and independent evaluators, while curriculum diversity, environment fidelity, and closed-loop stability metrics remain fragmented.Source: §5.6 · p. 40
• Contamination controls, matched-budget comparisons, reproducible agent benchmarks, and multimodal or long-context evaluations remain inconsistently developed.Live websites and graphical interfaces change over time, making some agent evaluations expensive and partly non-reproducible; multimodal evaluation remains weakly connected to reduced-supervision learning.Source: §5.6 · p. 40
• The evaluation framework measures capability on diverse suites, audits feedback signals against training methods, and assesses experience through curriculum quality, environment validity, and closed-loop stability.Refreshed suites reduce contamination, while expert checks and frozen judge panels provide sparse external anchors for feedback and experience.Source: §5.6 · p. 40
• No single leaderboard can establish progress toward self-sustaining learning, because conclusions depend on which components were externally evaluated versus kept inside the training loop.Missing evidence does not invalidate a method, but it limits the conclusions supported by reported gains.Source: §5.6 · p. 40

6. Challenges and Future Directions

• Autonomous rewards and experience weaken auditability, allowing proxy exploitation, curriculum collapse, generator-verifier collusion, and training–evaluation overlap to hide behind rising internal reward.Learning must therefore continue with less intervention while preserving evidence independent of the optimization loop.Source: §6 · pp. 40–41

6.1. Stabilizing Rewards beyond Per-Instance Human Evaluation

• Reward scaling must preserve independent evidence because intrinsic and consensus signals can reinforce shared errors, while external checks remain incomplete or exploitable.Independent executable checks, frozen evaluators, corpus-grounded tests, and periodic audits can calibrate model-derived rewards without restoring dense supervision.Source: §6.1 · p. 41

6.2. Scaling Experience without Curriculum Collapse

• Experience generation must optimize longitudinal learnability, semantic diversity, environment fidelity, and transfer rather than merely producing more trajectories.Frontier-targeting curricula, preserved unsolved tasks, staged executability checks, differential testing, and held-out real interactions address collapse and synthetic-environment errors.Source: §6.2 · pp. 41–42

6.3. Efficiency of Experience Sampling and Token Utility

• Experience generation is a resource-allocation problem: frontier-focused sampling, low-information rejection, shorter feedback, replay, and difficulty-aware test-time compute can increase useful learning per budget.Credible reports should pair capability gains with the generation, verification, and environment-interaction costs required to obtain them.Source: §6.3 · pp. 42–43

6.4. Evaluation under Receding Supervision

• Evaluation must jointly measure capability, feedback fidelity, and experience quality using independent anchors, matched budgets, live benchmarks, and audits of reward robustness, diversity, validity, and transfer.Fresh evaluation distributions help, but cannot alone exclude contamination or protocol bias.Source: §6.4 · p. 43

6.5. Trust, Safety, and Value Preservation under Autonomy

• As supervision recedes, human values migrate into data, tools, objectives, environments, and evaluation, where co-adapting feedback can amplify sycophancy, unsafe confidence, and omitted safety cases.Safety audits should test policies, evaluators, and curricula, retaining independent human or constitutional vetoes for drift and unreliable self-generated evidence.Source: §6.5 · p. 43

6.6. Toward Broader Training Paradigms and Superintelligence

• Sparse anchors, generated curricula, and closed loops may eventually support pretraining or continual mid-training, but safe open-ended capability growth remains undemonstrated.Progress toward superintelligence depends on stable rewards, expanding semantic coverage, drift-sensitive evaluation, and independent execution, interaction, audits, and value constraints.Source: §6.6 · p. 43

Conclusion

• The dual-axis framework tracks reward evidence and learning experience, while a five-level ladder identifies which components still require continued human provision.The reward axis concerns rollout evaluation; the experience axis concerns tasks and environments.Source: §7 · p. 44
• Reusable and model-derived rewards plus adaptive task and environment generation reduce direct human judgment but risk invalid evidence, miscalibrated difficulty, curriculum narrowing, errors, and collusion.Continued improvement requires feedback that remains faithful and experience streams that remain informative under adaptation.Source: §7 · p. 44
• The evaluation framework separates policy capability on frozen tasks, feedback fidelity under optimization, and experience quality across diversity, validity, and transferability.Capability gains alone are insufficient when feedback and experience are generated within the same learning loop.Source: §7 · p. 44
• Human intent persists through evaluators, objectives, task seeds, environments, and sparse audits; supervisory autonomy neither constitutes general intelligence nor guarantees reliable capability growth.Such learning loops may contribute to progress toward superintelligence, but the paper does not equate autonomy with superintelligence.Source: §7 · p. 44

Limitations

• The ladder is analytical rather than mutually exclusive, comparisons are constrained by methodological differences, coverage is selective, and movement along it does not imply superintelligent behavior.It tracks operational supervision during learning rather than historical human knowledge embedded in models, references, corpora, tools, or environments; superintelligence remains conceptual.Source: §7 · p. 44