PaperlayerView exact revision on arXiv

Source-linked AI summary

Absolute Zero: Reinforced Self-play Reasoning with Zero Data

Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, Gao Huang

arXiv:2505.03335v3cs.LGcs.AIcs.CL

TL;DR

Existing zero-setting RLVR methods still depend on human-curated tasks, limiting fully self-directed reasoning. Absolute Zero trains a model to generate and solve its own verifiable code tasks, and AZR achieves strong coding and mathematical reasoning performance without external data.

  • Problem

    Zero-setting RLVR avoids reasoning-trace supervision but still depends on expertly curated task distributions, limiting its scalability.

  • Method

    Absolute Zero uses self-play and a code executor to let AZR generate, solve, and verify code reasoning tasks without external data.

  • Results

    AZR achieves strong performance across out-of-distribution coding and mathematical reasoning benchmarks, surpassing state-of-the-art combined reasoning and coding results.

  • Takeaways & Limitations

    Absolute Zero shows that self-generated, environmentally verified task distributions can support strong reasoning performance without extensive human-curated training data.

  • Takeaways & Limitations

    The paradigm still requires oversight because self-improving components can produce safety-concerning chains of thought.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.

1 Tsinghua University 2 Beijing Institute for General Artificial Intelligence 3 Penn State University

Absolute Zero introduces a data-free self-play paradigm that removes human-curated reasoning traces, expert-defined distributions, and curated question–answer pairs from reasoning-model training. AZR demonstrates strong out-of-distribution math and coding reasoning, surpassing models trained on tens of thousands of expert-labeled in-domain examples.

  • Results: AZR achieves state-of-the-art performance with zero data and improves general reasoning in both mathematics and coding despite operating entirely out-of-distribution.The reported gains do not rely on gold labels or human-defined queries.
  • Results: AZR surpasses models trained on tens of thousands of expert-labeled in-domain examples in the combined average score.The supplied figure caption reports this comparison without providing the underlying score values.
  • Absolute Zero Paradigm: AZR’s Absolute Zero paradigm trains reasoning models without any human-curated data, unlike supervised learning and verified-reward reinforcement learning that depend on curated supervision or expert-defined distributions.The paradigm contrasts behavior cloning from human reasoning traces and verified-reward learning over expert-defined distributions with fully data-free training.

1. Introduction

Absolute Zero enables a model to propose learnable tasks and solve them through self-play without external data. Its AZR instantiation uses executable code tasks to achieve strong math and coding results while revealing scaling, transfer, planning, and safety findings.

  • Contribution: Absolute Zero lets a model jointly propose tasks that maximize learnability and solve them, enabling self-evolution through self-play without external data.The paradigm is designed to avoid narrow domains, fixed functionalities, and reward models vulnerable to hacking.
  • Method: AZR uses a code executor to validate open-ended code reasoning tasks and provide verifiable feedback, covering induction, abduction, and deduction modes.The executor serves as a grounded environment for both task-integrity validation and stable training feedback.
  • Results: AZR, trained entirely without in-distribution data, achieves competitive mathematics performance and state-of-the-art coding performance, surpassing prior zero-setting models by 1.8 average points.These comparisons are made against models trained with in-domain data.
  • Results: AZR improves cross-domain mathematics more than conventional RLVR: expert code models gain 0.65 points on average, versus 10.9 for AZR-Base-7B and 15.2 for AZR-Coder-7B.The AZR models were trained on self-proposed code reasoning tasks.
  • Results: Coder-model gains scale with size, reaching +5.7, +10.2, and +13.2 points for 3B, 7B, and 14B models, respectively.The reported trend suggests continued scaling is advantageous for AZR.
  • Findings and limitations: AZR develops mode-specific reasoning behaviors and intermediate comment-based plans, while occasional concerning chains of thought expose a need for safety-aware training.Observed behaviors include step-by-step reasoning, enumeration, and trial-and-error; the reported safety issue is termed the “uh-oh moment” (Zhang et al., 2025a).

2. The Absolute Zero Paradigm

The Absolute Zero paradigm removes human-curated training data by having one model propose and solve validated tasks through self-play. It jointly optimizes task learnability and solution quality using environment-grounded rewards.

  • Motivation: Absolute Zero eliminates SFT and RLVR’s reliance on human-curated queries, demonstrations, or answers by enabling the model to generate, solve, and learn from its own interactions.SFT requires task–rationale–answer demonstrations, while RLVR still requires human-labeled tasks and answers.
  • Absolute Zero Paradigm: During training, the model simultaneously proposes tasks, solves them, and learns from both stages without external data, using self-play and an environment.The paradigm is illustrated as a contrast with supervised learning and RLVR, emphasizing self-improving task proposal.
  • Absolute Zero Loop: The proposer samples a task, the environment validates it into a query and gold label, and the solver produces an answer for verification.The same parameterized language model supplies both proposer and solver roles, with the environment constructing valid reasoning tasks.
  • Objective: The joint objective rewards proposed tasks for expected learnability improvement and answers for correctness, while λ balances exploration against reasoning improvement.The environment provides validation and solution rewards, and proposer and solver policies are jointly trained in a repeatable loop.
  • Scalability: Absolute Zero shifts data-scaling responsibility from human experts to the proposer and environment, which evolve task distributions, validate tasks, and provide grounded feedback.A continually updated buffer of past task–answer pairs can seed task generation, but the paradigm does not require a specific implementation.

3. Absolute Zero Reasoner

Absolute Zero Reasoner (AZR) uses one language model as both task proposer and solver, jointly learning a self-generated curriculum through reinforcement learning. A code executor validates proposed program-based tasks and solver answers, providing grounded rewards across deduction, abduction, and induction.

  • 3.1 Absolute Zero Reasoner: AZR jointly trains a single language model to propose tasks with high learning potential and solve them, evolving its curriculum and reasoning ability through reinforcement learning.The proposer and solver roles operate within a unified language space and are trained together using the Absolute Zero objective.
  • 3.3 Self-Play Algorithm: The proposer reward favors moderately difficult tasks because trivial or unsolvable tasks provide little learning signal, while solver rewards are based on verified answer correctness.Task validity and responses are checked with Python, including type-aware value equality and task-specific equivalence conditions.
  • 3.2 Task Formulation: A code executor constructs, executes, and validates program-based reasoning triplets, supplying verifiable feedback for both generated tasks and model responses.A task is represented as (p, i, o), where o is produced by executing program p on input i.
  • 3.2 Task Formulation: AZR supports deduction, abduction, and induction: predicting outputs, inferring inputs, and synthesizing programs from input-output examples, respectively.Induction includes an auxiliary message to condition an otherwise under-constrained program-synthesis problem.
  • 3.3 Self-Play Algorithm: AZR uses separate advantage baselines for its six task-role configurations, reflecting its multitask reinforcement-learning setup across three task types and two roles.This design interpolates between per-question and global baselines.

4. Experiments

AZR achieves strong coding and mathematical reasoning without external training data, including state-of-the-art 7B overall and coding performance, while generalization improves with model scale and code-capable initialization. Ablations show that task diversity and proposer design are important, and training produces varied, task-dependent reasoning behaviors.

  • Cross-domain generalization: AZR shows stronger cross-domain generalization than expert code models, whose average math improvement after training was only 0.65 points.Most expert code models changed minimally or declined on math relative to their base versions, whereas AZR models improved after coding-environment training.
  • Initialization: Coder initialization improves post-AZR math and coding performance, even though its initial math average was lower than the vanilla base model’s, 23.9 vs. 27.5.The result highlights the importance of initial code competency for the self-play process.
  • Scaling: Larger models benefit more from AZR, with out-of-distribution overall gains of +5.7, +10.2, and +13.2 for 3B, 7B, and 14B models, respectively.The 3B model appears to plateau, while 7B and 14B models continue improving beyond 200 training steps in-distribution.
  • Training behaviors: AZR produces diverse, task-dependent reasoning behaviors, including self-correction, structured intermediate tracking, systematic test-case checking, and step-by-step planning comments.Token-length growth also varies by cognitive task, with the largest increase observed for abduction; Llama3.1-8B yields moderate improvements of +3.2.
  • Ablations: Removing induction, abduction, proposer conditioning on historical references, or proposer training degrades performance, indicating that all components support general reasoning.Using only deduction or removing induction causes significant drops; fixed prompting causes a 5-point math decrease, while omitting proposer training causes a -1.4 drop.

5. Related Work

Prior work connects reasoning improvement to reinforcement learning, self-play, and weak-to-strong supervision, while highlighting limitations of model-based reward signals. Absolute Zero instead grounds self-play reasoning in operationalizable code tasks and verification.

  • Reasoning with RL: RL-based reasoning work includes STaR’s expert iteration and rejection sampling, while o1 helped establish RL as a post-training approach for reasoning-focused language models.
  • Self-play: Self-play research uses agents that generate and solve questions, with game-playing systems extending the paradigm to competitive learning (Schmidhuber, 2003; 2011; Silver et al., 2016; 2017).
  • Self-play: Recent language-model self-play methods use the same model as reward model, but reward-model reliability remains a bottleneck for reasoning tasks (Chen et al., 2024; Yuan et al., 2024; Kirchner et al., 2024).
  • Self-play: Absolute Zero operationalizes RLVR through Python input/output, function-abduction, deduction, and induction tasks grounded in an executable environment.
  • Weak-to-Strong Supervision: Unlike weak-to-strong supervision, which uses a weaker teacher to guide a stronger learner, the proposed setting allows the learner to possess superhuman capabilities without such supervision (Burns et al., 2024; Hinton et al., 2015; Christiano, 2018; 2019; Demski & Garrabrant, 2019; Leike & Sutskever, 2023; Hubinger et al., 2019).

6. Conclusion and Discussion … A. Reinforcement Learning with Verifiable Rewards.

The paper concludes that Absolute Zero enables self-generated, environmentally verified learning without human-curated data while achieving strong out-of-distribution reasoning performance. Its RL implementation uses online REINFORCE++ with normalized advantages and no KL penalty.

  • 6. Conclusion and Discussion: AZR trains agents to propose and solve code-related reasoning tasks, using a code executor to ground self-generated learning in environmental verification.This instantiates the Absolute Zero paradigm, which lets agents generate their own task distributions while improving reasoning.
  • 6. Conclusion and Discussion: Despite no direct training on evaluation tasks or human expert-curated datasets, the models surpass state-of-the-art performance in combined general reasoning and coding.The evaluation covers out-of-distribution benchmarks in code generation and mathematical reasoning.
  • 6. Conclusion and Discussion: Future work could replace the feedback environment with the web, formal math languages (Ren et al., 2025), world simulators, or the real world, and extend AZ to embodied AI (Yue et al., 2024).The discussion also points toward more complex agentic tasks and scientific experiments.
  • 6. Conclusion and Discussion: The framework highlights exploration as a critical but underexplored driver of emergent behavior in LLM reasoning, building on prior RL and red-teaming research (Yue et al., 2025; Silver et al., 2016; Ladosz et al., 2022; Pourcel et al., 2024; Zhao et al., 2025a).The discussion positions exploration as central to the framework’s motivation for self-generated learning.
  • 6. Conclusion and Discussion: Safety oversight remains necessary: the Llama-3.1-8B model produced safety-concerning chain-of-thought instances, termed the “uh-oh moment”.The paradigm reduces human task-curation requirements but does not eliminate safety concerns in self-improving systems.
  • 6. Conclusion and Discussion: Absolute Zero aims to free reasoning models from human-curated data by giving them experience that lets them define and evolve their own learning distributions.AZR demonstrates strong performance across diverse reasoning tasks with significantly fewer privileged resources, including curated human data.
  • A. Reinforcement Learning with Verifiable Rewards.: The learner LLM is updated with reinforcement learning using a task-specific reward function, maximizing the expected discounted sum of rewards.The method adopts online REINFORCE++, optimized with the original PPO objective.
  • A. Reinforcement Learning with Verifiable Rewards.: REINFORCE++ uses globally batch-normalized outcome rewards to compute advantages, and the loss and reward apply no KL penalty.The global batch has size B, with mean and standard deviation computed across that batch.

B. Implementation Details · C. More Results · C.1. Out-of-Distribution Performance Breakdown

AZR uses a fixed self-play training configuration built on veRL with integrated code execution, while experiments evaluate multiple model scales on out-of-distribution reasoning benchmarks. Across training, AZR improvements correlate strongly with gains in mathematical and coding reasoning, with dynamic task proposals making overfitting less likely than static datasets.

  • B. Implementation Details: Implementation builds AZR on the veRL codebase and incorporates components from the QwQ Python executor for code execution.The authors recommend API-based services such as E2B for safer code execution.
  • B. Implementation Details: All experiments ran on A800 GPU clusters, with each experiment lasting approximately 3–5 days.
  • B. Implementation Details: The same training hyperparameters were used for every run and are reported in Table 3.Table 4 reports the reasoner training data source and base model.
  • C.1. Out-of-Distribution Performance Breakdown: Across 7B, 7B-Coder, 14B, and 14B-Coder models, AZR training strongly correlates with improved mathematical and coding reasoning on out-of-distribution benchmarks.Performance is plotted by benchmark and in aggregate across training steps in Figures 30–33.
  • C. More Results: AZR evaluates out-of-distribution performance both separately by benchmark and in aggregate across training steps.The breakdown covers four model variants: 7B, 7B-Coder, 14B, and 14B-Coder.
  • C.1. Out-of-Distribution Performance Breakdown: The models train for more steps than typical zero-style reasoners, while dynamically proposed tasks make overfitting less likely than with static datasets.The passage contrasts AZR’s dynamic training with the possibility of overfitting under static datasets.

C.2. In-Distribution Results · C.3. Interplay Between Propose and Solve Roles

The study evaluates Absolute Zero on in-distribution code reasoning benchmarks and examines how its propose and solve roles behave during training. The solve role generates longer outputs, while reasoning types differ in token length because abduction involves trial-and-error hypothesis generation and verification.

  • C.2. In-Distribution Results: Absolute Zero’s in-distribution training is evaluated on CruxEval-I, CruxEval-O, and LiveCodeBench-Execution, covering abduction and deduction tasks.CruxEval-I measures input prediction, while CruxEval-O and LiveCodeBench-Execution measure output deduction; their score evolution is shown during training.
  • C.3. Interplay Between Propose and Solve Roles: The training dynamics are analyzed across propose and solve roles for abduction, induction, and deduction tasks.Figures 17–19 report task rewards and token lengths for the two roles across these task types.
  • C.3. Interplay Between Propose and Solve Roles: The solve role generally produces more output tokens than the propose role because solving requires deeper reasoning than generating novel tasks.The propose role emphasizes creativity and task generation, whereas the solve role performs the deeper reasoning needed to solve those tasks.
  • C.3. Interplay Between Propose and Solve Roles: Abduction and deduction problem-solving outputs are consistently shorter than induction outputs.This ordering is observed across reasoning types during training.
  • C.3. Interplay Between Propose and Solve Roles: Induction requires trial-and-error reasoning in which the model hypothesizes inputs, evaluates their outcomes, and retries after unsuccessful deductions.This process is offered as the explanation for induction’s longer problem-solving outputs.
  • C.3. Interplay Between Propose and Solve Roles: The reported role dynamics are illustrated for the Absolute Zero Reasoner-base-7b model across abduction, induction, and deduction training traces.The corresponding figures show task rewards and token lengths for both roles in each task type.
  • C.3. Interplay Between Propose and Solve Roles: Example analyses show that proposed tasks are verified through execution, including deduction tasks generated from few-shot program examples.The supplied examples also cover proposed abduction and induction tasks and a detailed solved abduction task.
  • C.3. Interplay Between Propose and Solve Roles: During code-generating solutions, the model often interleaves reasoning and actions through comments in a learned ReAct-style generation pattern.This behavior is illustrated for an induction-task output from Absolute Zero Reasoner-Coder-14b.

Task: Solve Deduction Task (Example 1)

The example shows the model solving a deduction task by simulating a given program on its provided input until predicting the final output.

  • Task: Solve Deduction Task (Example 1): The model receives a function and input, then simulates program execution to determine the final result.The example input includes [1, 2, 1, 2, 3, 2, 1] and 2.

Task: Solve Induction Task

In the induction task, the model infers a program from input/output examples and verifies its predicted function against each case. The propose and solve roles show mildly adversarial rewards, while penalties for unsolvable tasks encourage cooperation.

  • Task: Solve Induction Task: The model predicts a function from a program description and five input/output cases, then checks the function against each use case.The example asks the model to sort integers, multiply each by n − i, and sum the results, producing outputs 20, 20, 20, 17, and 5.
  • Task: Solve Induction Task: Reward dynamics between proposing and solving are mildly adversarial: higher rewards for one role often correspond to lower rewards for the other.The interaction is not fully adversarial because proposers are penalized for generating unsolvable tasks, promoting cooperative learning.

C.4. Complexity and Diversity Metrics of AZR Proposed Tasks · C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.

AZR tracks code complexity and task diversity through proxy metrics, and its simple reward implicitly drives increasingly challenging and diverse proposals. Across task types, abduction/deduction proposers tend to make code more convoluted, while induction solvers favor efficiency.

  • C.4. Complexity and Diversity Metrics of AZR Proposed Tasks: The analysis measures program complexity with ComplexiPy and Halstead scores, and task diversity with AST edit distance to K reference programs plus answer diversity.These metrics probe characteristics of tasks proposed during AZR training from the base model.
  • C.4. Complexity and Diversity Metrics of AZR Proposed Tasks: AZR’s final training behavior implicitly optimizes complexity and diversity, producing increasingly challenging tasks despite these metrics being absent from the reward.The metrics were presented in Figure 29 and align with intuitive notions of task difficulty and diversity.
  • C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.: The generated-code analysis computes each induction program’s cognitive-complexity difference from its corresponding gold program with ComplexiPy.The notation pairs generated programs by role and task type.
  • C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.: A 0.27 average proposer–solver complexity difference, holding functionality constant, indicates that abduction/deduction proposers often increase cognitive complexity.Complexity is measured as the difference between generated and corresponding gold code using ComplexiPy.
  • C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.: By contrast, induction solvers tend to generate more efficient implementations than the corresponding abduction/deduction proposer code.The comparison holds code functionality constant while examining cognitive complexity differences.
  • C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.: An example deduction solution shows mixed-language chain-of-thought output, including Chinese.This observation is illustrated in Figure 27.

Task: Solve Deduction Task (Llama 8B) … D. Alternative Approaches Considered

The section illustrates AZR’s code-based deduction tasks, including abduction, output deduction, and program induction, with Sudoku and Sum-Product examples showing how generated programs support verifiable reasoning. It also notes that the authors explored additional approaches that were not particularly helpful for AZR.

  • Task: Solve Deduction Task (Llama 8B): AZR also performs program-input abduction, inferring one possible input that could have produced a given output from a Python function.The input must be formatted in input tags, with quoted strings and comma-separated arguments when needed.
  • Task: Solve Deduction Task (Llama 8B): AZR solves program output-deduction tasks by tracing a Python snippet from a supplied input and returning the resulting output in a specified format.The task prompt requires step-by-step deduction and output wrapped in output tags, with strings quoted and multiple values represented as tuples.
  • Task: Solve Deduction Task (Llama 8B): Program-induction tasks require AZR to infer a general Python function from input/output pairs and an accompanying message.The inferred code is tested on hidden inputs and must reproduce the hidden outputs.
  • Task: Manual Constructed Sudoku Abduction Task: The Sudoku example casts abduction as reversing a randomized masking process: AZR must recover a complete valid board from an output with 51 masked cells.The source board is validated, cells are randomly selected for masking, and the masked board is presented as the output.
  • return True: In the Sum-Product Game example, the function returns True only when integer inputs satisfy 1 < x < y, x + y <= 100, and the reasoning procedure leaves exactly one matching candidate pair.The described analysis identifies (x, y) = (4, 13) as satisfying the conditions.
  • return True: The Sum-Product procedure filters candidate pairs by ambiguous sums, then retains products that become unique and checks whether each sum yields exactly one pair.It constructs pair, sum, and product dictionaries before returning the sorted final candidates.
  • D. Alternative Approaches Considered: The authors report that they explored additional methods that were not particularly helpful for AZR and present those findings as guidance for future research.This subsection introduces the alternative approaches considered during project development without detailing their individual outcomes here.

D.1. Error Deduction Task … D.5. Environment Transition

The appendix evaluates auxiliary task domains, curriculum mechanisms, reward designs, initialization choices, and environment transformations for AZR. It finds limited benefit from error deduction and composite-function curricula, while reward aggregation and code transformations materially affect training stability and performance.

  • D.1. Error Deduction Task: AZR explored error-deduction tasks requiring the solver to infer which error a proposed program would raise, alongside induction, deduction, and abduction tasks.
  • D.2. Composite Functions as Curriculum Learning: Composite-function generation was intended to create curriculum learning by composing new programs from progressively more difficult earlier generations.
  • D.2. Composite Functions as Curriculum Learning: The composite curriculum showed no significant improvement over the simpler approach because the model often returned g(x), failing to add difficulty.
  • D.3. Toying with the Initial p(z): Initializing p(z) from LeetCode increased initial coding-benchmark performance, but performance later plateaued at roughly the same level as the self-generated initialization.
  • D.4. Extra Rewards: The study also tested code-complexity and diversity intrinsic rewards, including maintainability, Halstead complexity, and code edit distance.
  • D.4. Extra Rewards: Simple additive reward aggregation produced the most stable runs, possibly because it introduced less variance.
  • D.5. Environment Transition: Removing comments and docstrings prevented proposer-to-solver hints that could make otherwise unsolvable tasks solvable and support solver self-bootstrapping.
  • D.5. Environment Transition: Removing globally declared variables caused a noticeable performance drop, so final experiments retained them because reward-transition mismatch and rationalization may still support reasoning.
Loading 2505.03335v3…