PaperlayerView exact revision on arXiv

Source-linked AI summary

Absolute Zero: Reinforced Self-play Reasoning with Zero Data

Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, Gao Huang

arXiv:2505.03335v3cs.LGcs.AIcs.CL

TL;DR

Existing zero-setting RLVR methods still depend on human-curated tasks, limiting scalability. Absolute Zero lets a model generate and solve its own code-based tasks with executable verification, achieving state-of-the-art performance in coding and combined reasoning evaluations without human-curated training data.

  • Problem

    Existing zero-setting RLVR methods still depend on expertly curated task distributions, creating a scalability gap for human-produced training data.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    Absolute Zero trains a model to generate and solve its own reasoning tasks through self-play, using a code executor as an environmental source of verifiable reward.

    Source: §1 · p. 2, §3.2 · p. 6, §6 · p. 14

  • Results

    AZR achieves state-of-the-art performance in coding and combined general reasoning evaluations despite training without human-curated or in-distribution data, and scales across model sizes and classes.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The results support Absolute Zero as a promising approach for developing strong reasoning models without extensive domain-specific training data.

    Source: §6 · p. 14

    The self-improving system still requires oversight because experiments revealed safety-concerning chains of thought.

    Source: §6 · p. 14

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.

1 Tsinghua University 2 Beijing Institute for General Artificial Intelligence 3 Penn State University

Absolute Zero introduces a self-play paradigm in which a model autonomously proposes learnable tasks and solves them using verifiable environmental feedback, without human-curated data. AZR achieves state-of-the-art combined math-and-coding performance, surpassing models trained on tens of thousands of expert-labeled examples.

Empirical result

  • AZR surpasses models trained on tens of thousands of expert-labeled in-domain examples in the combined average score across mathematics and coding, despite using zero data.The system relies on neither gold labels nor human-defined queries and operates entirely out-of-distribution.Source: Figure 1 · p. 1

Absolute Zero paradigm

  • Absolute Zero enables a model to autonomously propose tasks optimized for learnability and learn to solve them through verifiable feedback, eliminating human-curated training data.Unlike supervised learning and verified-reward reinforcement learning, the paradigm avoids human-curated reasoning traces, expert-defined distributions, and curated question-answer pairs.Source: Figure 2 · p. 2

1. Introduction

The introduction argues that zero-setting RLVR still depends on human-curated tasks, limiting scalability and autonomous learning, and proposes Absolute Zero as self-evolving, environment-grounded self-play without external data. It presents AZR as a code-executor-based implementation achieving strong math and coding results while revealing scaling, transfer, planning, behavioral, and safety findings.

Motivation and paradigm

  • Absolute Zero enables models to propose learnable tasks and solve them through open-ended, environment-grounded self-play without human supervision or external data.Verifiable environmental feedback replaces learned reward models, helping avoid reward hacking while supporting autonomous learning (Silver et al., 2017; Chen et al., 2025; 2024; Hughes et al., 2024).Source: §1 · p. 2

Method

  • AZR uses a code executor to validate and reward self-proposed tasks across induction, abduction, and deduction, with end-to-end multitask reinforcement learning.The executor validates task integrity and answers, while a new advantage estimator is tailored to the multitask setting.Source: §1 · p. 2

Results

  • AZR achieves competitive mathematical reasoning and new state-of-the-art coding performance without in-distribution training data, outperforming prior zero-setting models by 1.8 points.It surpasses models trained with code datasets using RLVR and exceeds previous zero-setting models trained with in-domain data by an average of 1.8 absolute points.Source: §1 · pp. 2–3

Empirical findings

  • AZR shows stronger cross-domain transfer than conventional RLVR: expert code models gain only 0.65 math-accuracy points on average after RLVR, versus 10.9 for AZR-Base-7B and 15.2 for AZR-Coder-7B.For AZR coder models, gains rise from +5.7 to +10.2 to +13.2 points at 3B, 7B, and 14B, respectively; coding capabilities may potentially amplify reasoning improvements after AZR training.Source: §1 · p. 3

Emergent behaviors

  • AZR naturally develops intermediate comment-based plans and mode-specific reasoning behaviors, with abduction showing the largest token-length growth through trial-and-error.Induction often interleaves step-by-step plans with code, while deduction and induction grow more modestly in token use (Yao et al., 2023; Ren et al., 2025).Source: §1 · p. 3

2. The Absolute Zero Paradigm

The Absolute Zero paradigm removes human-curated task and answer data by having one model propose, solve, and learn from validated tasks through self-play. Its proposer and solver roles are jointly trained using learnability and correctness rewards supplied by an environment.

2.1. Preliminaries

  • Absolute Zero eliminates SFT and RLVR’s reliance on human-curated queries, demonstrations, or answers by learning through self-play interactions with an environment.SFT imitates human- or AI-provided task-rationale-answer demonstrations, while RLVR removes rationale labels but still requires human-labeled tasks and answers.Source: §2.1 · p. 3

2.2. Absolute Zero

  • Absolute Zero jointly trains a model to propose tasks, solve them, and learn from both stages without external data.The paradigm shifts data scaling from human experts to a proposer policy and environment that evolve the learning task distribution and provide grounded feedback.Source: §2.2 · pp. 3–4
  • The proposer samples a task, the environment validates it into a query-answer pair, and the solver generates an answer for verification.The same parameterized model plays proposer and solver roles, with the environment transforming proposed tasks into validated reasoning problems.Source: §2.2 · p. 4
  • Learnability rewards encourage challenging but learnable task proposals, while solver rewards evaluate answer correctness and jointly guide continuous self-improvement.A nonnegative coefficient λ balances exploration of new learnable tasks against improving reasoning and problem-solving abilities.Source: §2.2 · p. 4

3. Absolute Zero Reasoner

AZR is a self-play system in which one language model proposes verifiable coding tasks and solves them to improve its reasoning without external data. It combines three reasoning modes, code-executor verification, replay buffers, and task-relative reinforcement learning.

3.1. Two Roles in One: Proposer and Solver

  • AZR jointly trains a single model as proposer and solver, rewarding tasks with moderate difficulty and solutions verified by Python execution.The proposer estimates learnability through solver rollouts, while the solver receives correctness-based rewards and format penalties.Source: §3.1 · pp. 4–6

3.2. Learning Different Modes of Reasoning: Deduction, Induction, and Abduction

  • AZR represents coding tasks as program-input-output triplets and supports deduction, abduction, and induction, with held-out examples encouraging generalized program synthesis.Deduction predicts outputs, abduction infers inputs by output equivalence, and induction synthesizes programs from partial input-output examples plus a message.Source: §3.2 · p. 6

3.3. Absolute Zero Reasoner Learning Algorithm

  • The self-play algorithm initializes task buffers, proposes and validates tasks, samples solver queries, computes both role rewards, and updates the model with TRR++.Algorithm 1 organizes the repeated propose-solve-update loop across all three task types.Source: §3.3 · p. 6, §3.3.1 · p. 7

3.3.1. Buffer Initialization

  • AZR bootstraps its buffers from valid seed triplets, optionally beginning from a zero identity triplet, without updating model parameters during seeding.The seed buffer is expanded into deduction, abduction, and induction initialization data before self-play begins.Source: Figure 5 · p. 6, §3.3.1 · pp. 6–7

3.3.2. Task Proposal Inputs and Buffer Management

  • During self-play, proposers condition on sampled prior triplets to promote diversity, while previously validated tasks fill batches when proposals are invalid.Valid proposed triplets are added to the corresponding buffers regardless of whether they receive task reward.Source: §3.3.2 · p. 7

3.3.3. Constructing Valid Tasks

  • Proposed tasks are accepted only after execution, safety, determinism, and formatting checks, with AZR restricting generated programs to deterministic functions.Determinism is approximated by two independent executions, and valid solver queries expose the appropriate parts of each task representation.Source: §3.3.3 · pp. 7–8

3.3.4. Answer Verification

  • Answer verification uses task-specific value equivalence: output equality for deduction, program-output equivalence for abduction, and held-out mapping correctness for induction.Abduction avoids exact input matching because programs need not be bijective.Source: §3.3.4 · p. 8

3.3.5. Task-Relative REINFORCE++

  • TRR++ computes separate normalized baselines for the six combinations of three task types and two roles, providing structured variance reduction for multitask reinforcement learning.This interpolates between per-question and global baselines used in related policy-optimization approaches.Source: §3.3.5 · p. 8

4. Experiments

Across benchmark evaluations, AZR achieves leading coding and overall reasoning performance despite training only on self-proposed tasks, with stronger gains at larger model scales. Ablations and behavioral analyses show that task diversity, proposer design, and task-specific reasoning patterns materially shape performance.

4.1. Experiment Setup

  • Experiments use ID and OOD coding and mathematical benchmarks, greedy decoding, Qwen2.5 and Llama model variants, and batch size 64 × 6 with learning rate 1e−6.OOD evaluation includes HumanEval+, MBPP+, LiveCodeBench, AIME’24, AIME’25, AMC’23, MATH500, Minerva, and OlympiadBench.Source: §4.1 · pp. 8–9

4.2. Results

  • 1.8 absolute percentages: Absolute Zero Reasoner-Coder-7B surpasses the previous best 7B overall average and exceeds expert-data models by 0.3 absolute percentages in coding.This result is achieved despite no direct access to human-curated data or downstream math and coding tasks.Source: Table 1 · p. 9, §4.2 · p. 9
  • AZR gains 10.9 and 15.2 points in math for base and coder models, while improving out-of-distribution code performance by 3.2 and 5.0 points.Expert code models average only a 0.65-point math increase, whereas math models average a 2.0-point coding increase.Source: §4.2 · pp. 9–10
  • +5.7, +10.2, and +13.2 overall gains occur for 3B, 7B, and 14B models, with larger models continuing to improve beyond 200 training steps.Llama3.1-8B also improves by +3.2, though its gains are more limited than those of stronger Qwen2.5 models.Source: §4.2 · p. 10
  • AZR proposes diverse programs and solves them through verification, trial-and-error, state tracking, structured intermediate results, and systematic test-case checking.Code responses often interleave final outputs with immediate step-by-step plans, while abduction produces the largest task-dependent token-length increase.Source: §4.2 · pp. 10–11, Figure 7 · p. 11
  • The Llama-trained AZR model exhibits state tracking but also produces an unusual potentially concerning chain of thought described as an “uh-oh moment.”The authors encourage future investigation of this behavior’s implications.Source: §4.2 · p. 11
  • Removing both induction and abduction, or removing induction alone, significantly degrades math performance; removing proposer conditioning causes a 5-point math and 1-point code drop.Not training the proposer produces a moderate -1.4 performance drop.Source: §4.2 · pp. 11–12
  • At k=256/512, AZR matches or outperforms its base model across high-pass@k coding and math evaluations except AIME24 at k=512, while also leading MMLU-Pro averages.These results indicate broad answer diversity and reasoning coverage compatible with further test-time scaling.Source: §4.2 · p. 12, Figure 8 · p. 12, Figure 9 · p. 13

5. Related Work

The paper extends zero-setting RLVR into Absolute Zero, where the learner self-generates tasks and answers without external data, drawing on self-play and replacing external supervision with verifiable rewards.

Reasoning with RL

  • Absolute Zero extends zero-setting RLVR by training directly from a base LLM without SFT, external prompts, or answers, using only self-proposed data refined through RLVR.This approach aims to surpass existing zero-setting models rather than merely match them.Source: §5 · p. 13

Self-play

  • The work builds on self-play traditions that generate evolving learning tasks, from proposal–prediction agents and Go systems to automatic goal-generation paradigms.Prior examples include AlphaGo and AlphaZero (Silver et al., 2016; 2017), asymmetric self-play, unsupervised environment design, and autotelic agents.Source: §5 · p. 13
  • Unlike prior self-play language-model methods relying on reward models or fixed human-defined tasks, this work uses self-play to elicit long CoT for reasoning.It frames tasks as Python input/output, function-abduction, deduction, and induction problems grounded in an operationalizable environment for RLVR.Source: §5 · pp. 13–14

Weak-to-Strong Supervision

6. Conclusion and Discussion … A. Reinforcement Learning with Verifiable Rewards.

Absolute Zero trains reasoning agents to generate and solve their own code-grounded tasks, achieving strong out-of-distribution coding and mathematical reasoning without human-curated training data. The framework also exposes open questions about exploration, safety, environments, and reinforcement-learning design.

6. Conclusion and Discussion

  • Absolute Zero has agents generate their own learning-task distributions and improve reasoning through environmental guidance, instantiated by AZR’s code-task proposal and solving loop.The code executor grounds both task validation and answer verification.Source: §6 · p. 14
  • AZR surpasses state-of-the-art combined reasoning and coding performance on out-of-distribution benchmarks without direct task training or human expert-curated datasets.The models also scale across model sizes and enhance other model classes.Source: §6 · p. 14
  • Absolute Zero may extend to web, formal mathematics, simulators, real-world, embodied, scientific, multimodal, and more complex agentic environments, with adaptive distributions, learned rewards, diversity incentives, and improved progress estimation.The discussion specifically cites MAGELLAN as work on estimating learning progress.Source: §6 · p. 14
  • The framework elevates exploration from solving tasks to discovering and refining the learning-task space itself, expanding the boundaries of problem spaces beyond fixed datasets.The authors identify this meta-level exploration as an important direction for future research.Source: §6 · p. 14
  • Self-improving systems still require oversight because safety-concerning chains of thought, termed “uh-oh moment” instances, appeared in Llama-3.1-8B and were not addressed by this work.Managing the safety of self-improving components remains a critical research direction.Source: §6 · p. 14
  • AZR suggests a shift toward reasoning models that possess experience by defining and evolving their own learning distributions, potentially freeing them from constraints imposed by human-curated data.The authors frame this as the beginning of an “era of experience” (Silver & Sutton, 2025; Zhao et al., 2024).Source: §6 · p. 14

A. Reinforcement Learning with Verifiable Rewards.

  • The reinforcement-learning appendix updates the learner LLM with online REINFORCE++ using the original PPO objective, normalized outcome advantages, and no KL penalty in the loss or reward.Rewards are task-specific, with mean and standard deviation computed across the global batch.Source: A. Reinforcement Learning with… · p. 23

B. Implementation Details · C. More Results · C.1. Out-of-Distribution Performance Breakdown

AZR is implemented with established reinforcement-learning and code-execution infrastructure, while its dynamically proposed tasks support sustained out-of-distribution gains in mathematical and coding reasoning across model scales. The reported implementation also emphasizes fixed training settings and safer execution practices.

B. Implementation Details

  • AZR was built on the veRL codebase and integrates components from the QwQ Python executor for code execution.The implementation recommends API-based services such as E2B for safer execution.Source: B. Implementation Details · p. 23
  • AZR experiments were conducted on A800 GPU clusters, with each experiment lasting approximately 3–5 days.Source: B. Implementation Details · p. 23
  • The training configuration remains fixed across runs, with the hyperparameters documented in Table 3.Table 3 specifies the hyperparameters used during AZR self-play training.Source: B. Implementation Details · p. 23, Table 3 · p. 23
  • The implementation documents the training-data source and base model used for reasoner training in Table 4.The supplied passage identifies Table 4’s scope but does not provide its underlying entries.Source: Table 4 · p. 24
  • The task-generation safety setup forbids specified Python modules from appearing in proposed task programs.The forbidden-module list is presented in Figure 10.Source: Figure 10 · p. 24

C.1. Out-of-Distribution Performance Breakdown

  • Training with AZR strongly correlates with improving mathematical and coding reasoning across out-of-distribution benchmarks and aggregate performance for four model variants.The breakdown covers 7B, 7B-Coder, 14B, and 14B-Coder models across training steps in Figures 30–33.Source: §C.1 · p. 24
  • Compared with typical zero-style reasoners, AZR trains for more steps, while dynamically proposed tasks may reduce the likelihood of overfitting associated with static datasets.Source: §C.1 · p. 24

C.2. In-Distribution Results · C.3. Interplay Between Propose and Solve Roles

Absolute Zero Reasoner shows steadily improving in-distribution performance on CruxEval-I, CruxEval-O, and LiveCodeBench-Execution during training. Its propose and solve roles exhibit distinct reasoning dynamics: solving produces longer outputs, while induction is more token-intensive than abduction or deduction.

C.2. In-Distribution Results

  • CruxEval-I evaluates input prediction, whereas CruxEval-O and LiveCodeBench-Execution evaluate output-prediction deduction tasks.The three benchmarks track capabilities in the task domains used by the training curriculum.Source: §C.2 · p. 24
  • The benchmark curves do not perfectly correlate with broader coding or mathematical reasoning, limiting their interpretation to task-specific progress indicators.The passage explicitly contrasts Figure 16 with Figure 30 when qualifying this proxy relationship.Source: §C.2 · p. 24

C.3. Interplay Between Propose and Solve Roles

  • The solve role generally produces more output tokens than the propose role during training.The proposed explanation is that proposing emphasizes creative task generation, whereas solving requires deeper reasoning and therefore longer outputs.Source: §C.3 · p. 24
  • During problem solving, abduction and deduction tasks consistently use fewer tokens than induction tasks.This ordering is attributed to induction involving repeated hypothesis generation, outcome evaluation, reflection, and retries after failed deductions.Source: §C.3 · p. 24

C.2. In-Distribution Results

  • Training consistently improves Absolute Zero Reasoner-base-7b’s in-distribution scores on CruxEval-I, CruxEval-O, and LiveCodeBench-Execution.These benchmarks evaluate abduction through input prediction and deduction through output prediction, but serve only as task-specific proxies for broader coding and mathematical reasoning.Source: §C.2 · p. 24, Figure 16 · p. 25

C.3. Interplay Between Propose and Solve Roles

  • The training analysis compares task rewards and token lengths across the propose and solve roles for abduction, induction, and deduction.Figures 17–19 provide the corresponding task-specific dynamics for the three reasoning types.Source: Figure 17 · p. 26, Figure 18 · p. 26, Figure 19 · p. 27
  • The system validates proposed deduction, abduction, and induction tasks by executing generated programs and checking their correctness.The examples also show program-based verification for proposed tasks and solution outputs across the three task types.Source: Figure 20 · p. 27, Figure 22 · p. 29, Figure 23 · p. 30
  • Absolute Zero Reasoner learns a ReAct-style pattern in which comments interleave reasoning and acting during code generation.This behavior is observed even after chain-of-thought output when answering requires code, including an induction-task example from Absolute Zero Reasoner-Coder-14b.Source: Figure 21 · p. 28

Task: Solve Deduction Task (Example 1)

The deduction task asks the model to predict a function’s output by simulating program execution on a provided input.

Task setup

Execution

  • To solve the deduction task, the model simulates execution of the given program.Source: Figure 25 · p. 32

Output prediction

Task: Solve Induction Task

The solve-induction task asks the model to infer a code function from input/output examples and verify its prediction against every provided case. Training exhibits mildly adversarial proposer–solver rewards while penalizing unsolvable tasks to preserve cooperation.

Solve Induction Task

  • The model infers a function that sorts integers, weights each element by n − i, sums the results, and matches the five supplied outputs.For the examples, outputs are 20, 20, 20, 17, and 5.Source: Task: Solve Induction Task · p. 33
  • After proposing the function, the model tests it on each use case and confirms whether all five cases pass.The example explicitly shows sequential validation against the given input/output pairs.Source: Figure 26 · p. 33
  • Proposer and solver rewards are mildly adversarial, but penalizing unsolvable tasks encourages cooperative learning.Higher reward for one role often coincides with lower reward for the other, without making the interaction entirely adversarial.Source: Task: Solve Induction Task · p. 34

C.4. Complexity and Diversity Metrics of AZR Proposed Tasks · C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.

AZR’s proposed tasks are evaluated through program-complexity and diversity proxies, which the policy appears to improve implicitly despite their exclusion from the reward. Complexity comparisons further indicate that abduction/deduction proposers make code more convoluted, while induction solvers favor efficiency.

C.4. Complexity and Diversity Metrics of AZR Proposed Tasks

  • AZR task analysis measures program complexity with ComplexiPy and Halstead scores.These serve as proxy measures for the complexity of proposed programs.Source: §C.4 · p. 34
  • Task diversity is assessed using average AST edit distance from K reference programs and answer diversity over historical input-output pairs.Answer diversity is defined as 1 − p(answer), where p(answer) is the empirical probability of a specific answer.Source: §C.4 · p. 34
  • Without explicit reward terms for these metrics, AZR increasingly proposes tasks with higher apparent difficulty and diversity.The observed trend aligns with intuitive notions of task difficulty and diversity under the simple reward function in Equation (4).Source: §C.4 · p. 34

C.5. Generated Code Complexity Dynamics Between Abd/Ded and Ind.

  • Generated induction programs are compared with corresponding gold code using ComplexiPy cognitive-complexity differences.The comparison is performed for each generated-program pair while holding code functionality constant.Source: §C.5 · p. 34
  • 0.27 is the average proposer–solver complexity difference under fixed functionality, indicating that abduction/deduction proposers often make code more convoluted.The analysis uses ComplexiPy to quantify the cognitive complexity difference between generated programs and corresponding gold code.Source: §C.5 · p. 34
  • Induction solvers tend to generate more efficient implementations than the abduction/deduction proposers’ more convoluted code.This contrast is observed while holding the code’s functionality constant.Source: §C.5 · p. 34

Task: Solve Deduction Task (Llama 8B) … D. Alternative Approaches Considered

The paper illustrates AZR’s deduction-task formats and reasoning behavior through code-output inference, Sudoku abduction, and Sum-Product Game verification, while noting that several alternative approaches were not particularly helpful.

Task: Solve Deduction Task (Llama 8B)

  • Training examples suggest that AZR-Llama3.1-8b strengthened state-tracking behavior, while proposed-task complexity and diversity increased over training.The metrics cover program complexity and diversity across programs and answers, with upward trends in all reported measures.Source: Figure 28 · p. 36, Figure 29 · p. 36
  • The deduction-task suite also includes generating diverse inputs, inferring possible inputs from code and outputs, and reconstructing code from input/output pairs.These prompts frame the tasks as program generation, abduction, and induction problems evaluated against hidden examples.Source: Task: Solve Deduction Task (Llama 8B) · pp. 44–45
  • AZR’s output-deduction task asks the model to infer a Python snippet’s output from given code and input, formatting the result in output tags.The deduction prompt explicitly requires step-by-step reasoning and tuple or quoted-string formatting when applicable.Source: Task: Solve Deduction Task (Llama 8B) · p. 45

Task: Manual Constructed Sudoku Abduction Task

  • Sudoku abduction reverses solving: a validated complete board is randomly masked at 51 positions, and the model infers the original input from the masked output.The function rejects incomplete or invalid boards, copies the valid board, and masks randomly selected cells; AZR-Coder-14b verifies an initial guess before answering.Source: return True · pp. 47–48, Figure 42 · p. 48

return True

  • The Sum-Product Game function returns True only when integer inputs satisfy 1 < x < y, x + y <= 100, and uniquely meet the game’s candidate constraints.The described filtering uses ambiguous products by sum, then retains pairs whose products become unique and whose sum yields exactly one final candidate.Source: return True · p. 50
  • (4, 13) is identified as the pair satisfying the Sum-Product Game conditions, including 1 < 4 < 13 and 4 + 13 = 17 <= 100.The accompanying figure describes AZR-Coder-14b analyzing the constraints, identifying a candidate, and verifying it.Source: return True · p. 50, Figure 43 · p. 50

D. Alternative Approaches Considered

  • The authors report that the additional approaches explored in this section were not particularly helpful for Absolute Zero Reasoner, but may guide future research.They present these unsuccessful methods as findings worth sharing with the community.Source: D. Alternative Approaches Considered · p. 51

D.1. Error Deduction Task … D.5. Environment Transition

The authors explore auxiliary task designs, curricula, reward formulations, initialization strategies, and environment transformations for AZR, finding that several more complex interventions offer little benefit or reduce performance. The final setup therefore favors simpler reward aggregation and retains information that supports solver learning.

D.1. Error Deduction Task

  • Error-deduction tasks produced no noticeable downstream improvement and required more computation, so they were excluded from the final AZR version.The task required proposing code that raises an error and deducing the resulting error type.Source: §D.1 · p. 51

D.2. Composite Functions as Curriculum Learning

  • Composing generated programs was intended to create curriculum learning, with newer programs built from earlier ones becoming progressively more difficult for the solver.The implementation sampled one to three non-composite programs and filtered outputs that failed to use all selected components.Source: §D.2 · p. 51
  • The composite-function curriculum showed no significant advantage over the simpler approach because models often returned trivial identities such as f(g(x)) = g(x).The authors suggest stricter execution-based rewards as future work and deprioritized this direction.Source: §D.2 · p. 51

D.3. Toying with the Initial p(z)

  • Using a LeetCode-sourced initial p(z) improved early coding performance but plateaued similarly, while math performance was lower than with the official self-generated setup.Recency-based reference sampling also showed potential collapse, so uniform sampling was retained.Source: §D.3 · p. 51

D.4. Extra Rewards

  • Simple additive reward aggregation produced the most stable runs, possibly because it introduced less variance than alternative combinations of extrinsic and intrinsic rewards.The experiments included complexity and diversity rewards as intrinsic signals, but diversity strategies did not significantly change performance.Source: §D.4 · pp. 51–52

D.5. Environment Transition

  • Removing comments and docstrings caused a significant performance drop, suggesting proposer–solver communication through code annotations helps the solver bootstrap from otherwise unsolvable tasks.The authors therefore treat these messages as potentially useful hints rather than removing them.Source: §D.5 · p. 52
  • Removing globally declared variables caused a noticeable drop in self-play performance, so the final experiments retained them despite possible answer leakage.The authors attribute this partly to a mismatch between generation and post-transition reward assignment, while noting that exposed answers may still support nontrivial reasoning, consistent with rationalization in STaR (Zelikman et al., 2022).Source: §D.5 · p. 52

Related papers

Loading 2505.03335v3…