PaperlayerView exact revision on arXiv

Source-linked AI summary

Maximum Likelihood Reinforcement Learning

Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, Andrea Zanette

arXiv:2602.02710v3cs.LG

TL;DR

Binary terminal-reward tasks induce a likelihood over correct rollouts, but standard reinforcement learning only approximates it to first order. MaxRL uses compute-indexed objectives to bridge expected-reward RL and maximum likelihood, Pareto-dominating tested methods and achieving up to 20× test-time scaling efficiency gains over GRPO.

  • Problem

    Binary success observed after stochastic generation defines an implicit likelihood, but non-differentiability makes directly optimizing this principled objective difficult.

    Source: §1 · pp. 1–2

  • Method

    MaxRL uses a compute-indexed family of objectives based on pass@k events to interpolate between expected-reward RL and exact maximum likelihood.

    Source: §1 · p. 2

  • Results

    MaxRL Pareto-dominates standard RL objectives across all tested settings and achieves up to 20× test-time scaling efficiency gains over GRPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MaxRL scales more effectively than traditional RL when additional compute and training data are available in correctness-based domains.

    Source: §1 · p. 2, §8 · p. 12

    The finite-sample estimator can assign zero gradient to inputs with no successes, losing unbiasedness for the full maximum-likelihood series.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to $20\times$ gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.

1 Introduction

Correctness-based tasks induce an implicit likelihood over successful rollouts, but non-differentiable stochastic generation makes direct likelihood optimization difficult, so reinforcement learning is used as a workaround. MaxRL addresses this mismatch with a compute-indexed objective that approaches maximum likelihood as sampling increases and improves scaling efficiency over standard RL methods.

Motivation

  • In correctness-based tasks, stochastic generation followed by external binary verification induces a probability of success for each input, defining an implicit likelihood over correctness.Directly maximizing this likelihood is principled but difficult because intermediate sampling is non-differentiable and success probabilities may be small.Source: §1 · pp. 1–2

Method

  • MaxRL interpolates between expected-reward reinforcement learning and exact maximum likelihood by incorporating higher-order correctness information as sampling compute increases.Its likelihood approximation is defined through a Maclaurin expansion in pass@k events and recovers likelihood optimization in the infinite-compute limit.Source: §1 · p. 2

Objective mismatch

  • Standard reinforcement learning optimizes only the first-order approximation of the maximum-likelihood objective, whereas maximum likelihood reweights harder, low-success inputs more strongly.These different weighting schemes produce distinct optimization dynamics.Source: §1 · p. 2

Estimator

  • An on-policy estimator has an expected gradient that exactly matches MaxRL’s compute-indexed likelihood approximation, so additional sampling improves the optimized objective rather than only reducing variance.This provides a sample-based implementation requiring a one-line change to standard reinforcement-learning objectives.Source: §1 · p. 2

Empirical results

  • MaxRL Pareto-dominates RLOO (Ahmadian et al., 2024) and GRPO (Shao et al., 2024) across all tested settings, with up to 20× test-time scaling efficiency gains over GRPO.It also scales more favorably when additional compute and data are available, particularly on mathematical reasoning tasks with a perfect verifier.Source: §1 · p. 2

2 Preliminaries

This section formalizes correctness-based tasks, latent generation and decoding, pass-rate metrics, and the distinction between maximum-likelihood and reinforcement-learning objectives. Correctness is binary at the decoded-output level, while ML optimizes log pass rate and RL optimizes pass rate.

Task and model setup

  • Correctness-based tasks assign each input a binary success or failure outcome, with semantic correctness defined by a task-dependent output equivalence relation.Inputs x are drawn from ρ, correct answers are y*(x), and the model induces p_θ(· | x).Source: §2 · p. 2

Latent generation models

  • Latent generation models sample trajectories z from m_θ(z | x), then deterministically decode them into outputs whose correctness is evaluated after decoding.Expectations over model outputs therefore mean expectations over latent samples followed by deterministic decoding.Source: §2 · pp. 2–3

Pass rate

  • Pass rate is the probability of a correct answer for one input, while pass@k is the probability that at least one of k i.i.d. samples is correct.These metrics are defined under the model’s conditional output distribution p_θ(· | x).Source: §2 · p. 3

Maximum likelihood (ML)

  • Maximum likelihood maximizes the log-likelihood of correctness, p_θ(y*(x) | x), analogous to cross-entropy training in differentiable supervised learning.The objective treats correctness probability as the observed-data likelihood.Source: §2 · p. 3

Reinforcement learning (RL)

  • Reinforcement learning uses the binary reward r(x, y) = I{y = y*(x)} and maximizes expected pass rate rather than its logarithm.Thus, RL and ML differ in whether they optimize the expectation of pass rate or the expectation of its logarithm.Source: §2 · p. 3

3 Maximum Likelihood Reinforcement Learning (MaxRL)

This section shows that expected-reward reinforcement learning optimizes only the first-order term of maximum likelihood, motivating MaxRL, a compute-indexed family that approaches maximum likelihood as rollout compute increases.

3.1 Maclaurin Expansion of Maximum Likelihood

  • Maximum likelihood combines pass@k gradients through an infinite harmonic mixture, whereas classical reinforcement learning retains only expected pass@1, a first-order approximation.Higher-order terms capture rare successes, which are especially important when the pass rate p is small.Source: §3 · p. 3, §3.1 · p. 3

3.2 MaxRL Objective Function

  • MaxRL targets maximum likelihood under fixed sampling and non-differentiable generation, addressing the increasing sample cost of estimating large-k pass@k gradients when p is small.This finite-sample difficulty motivates the method’s design.Source: §3.2 · p. 4
  • MaxRL truncates the maximum-likelihood expansion at order T, with T = 1 recovering reinforcement learning and T →∞ recovering maximum likelihood.Intermediate T values interpolate between the two objectives by incorporating higher-order correctness events.Source: §3.2 · p. 4
  • MaxRL provides a principled tradeoff between additional rollout compute and higher-fidelity approximations to maximum likelihood.The truncation level T controls which orders of correctness events contribute to learning.Source: §3.2 · p. 4

4 Gradient Estimators for MaxRL

MaxRL’s maximum-likelihood gradient is the expectation of score functions under successful trajectories, enabling estimators that improve their approximation as rollout compute increases. The resulting implementation uses successful-sample normalization, a variance-reducing control variate, and a one-line advantage modification.

4 Gradient Estimators for MaxRL

  • The maximum-likelihood gradient equals the conditional expectation of score functions given success, and finite-series terms can alternatively be estimated using pass@k gradient estimators.This conditional form motivates averaging gradients from successful trajectories; improvements to pass@k estimators transfer to the truncated maximum-likelihood objective.Source: §4 · p. 4

4.1 Empirical Gradient Estimator

  • MaxRL samples from the unconditional policy and averages score functions only over successful trajectories, avoiding the computational demands of directly sampling the success-conditioned distribution.When no successes occur, some inputs receive zero gradient, so the estimator is not unbiased for the infinite maximum-likelihood series but is unbiased for the truncated objective.Source: §4.1 · pp. 4–5
  • Increasing rollout count N raises MaxRL’s approximation order to maximum likelihood, whereas REINFORCE only reduces variance for a fixed pass@1 objective.MaxRL normalizes successful score functions by K, while REINFORCE normalizes by total samples N; the estimator is unbiased for the truncated MaxRL objective with T = N.Source: §4.1 · p. 5

4.2 Variance Reduction via Control Variates

  • A zero-mean unconditional average-score control variate preserves unbiasedness while reducing the estimator’s variance in practice.Standard baseline arguments do not directly apply because the successful-sample normalization K is random and correlated with the rollouts.Source: §4.2 · pp. 5–6

4.3 On-Policy Implementation

  • The on-policy implementation changes REINFORCE-style training by one advantage-calculation line, normalizing by per-task mean reward and dropping terms when K = 0.The practical estimator is unbiased for MaxRL order T = N −1 rather than T = N because baseline subtraction is absent when no positive rollouts occur.Source: §4.3 · p. 6

5 A Unifying Weight-Function View

The paper unifies maximum likelihood, MaxRL, classical RL, and GRPO through population-level pass-rate weight functions that determine how learning signals are allocated across inputs. MaxRL approaches maximum-likelihood weighting for hard inputs as sampling compute increases, whereas GRPO remains fundamentally distinct and can emphasize easy inputs.

5 A Unifying Weight-Function View

  • Population-level weight functions w(p) fully characterize how maximum likelihood, MaxRL, classical RL, and GRPO allocate learning signal across input difficulty.Figure 1 plots these functions, with derivations provided in Appendix E.Source: §5 · p. 6, Figure 1 · p. 7
  • As T increases, MaxRL uniquely approaches maximum-likelihood weighting in the low-pass-rate regime, while truncated objectives interpolate between REINFORCE and maximum likelihood.The key distinction is how strongly objectives emphasize hard, low-pass-rate inputs.Source: §5 · p. 6, Figure 1 · p. 7
  • GRPO’s empirical standard-deviation normalization induces a population-level objective fundamentally different from REINFORCE, consistent with recent analyses (Davis and Recht, 2025; Liu et al., 2025b; Xiong et al., 2025b; Yang et al., 2026a; Thrampoulidis et al., 2026).This provides a reinterpretation of GRPO beyond its heuristic Z-normalization motivation.Source: §5 · pp. 6–7
  • For small p, GRPO upweights low-pass-rate inputs approximately as 1/√p, but additional sampling does not make it a better maximum-likelihood approximation.At sufficiently large pass rates, GRPO’s weighting increases as p →1, assigning more weight to very easy inputs unlike likelihood-based objectives.Source: §5 · p. 7, Figure 1 · p. 7

6 Experiments

Across controlled, data-rich, data-scarce, and large-model settings, MaxRL consistently improves compute scaling, performance, and output diversity over standard RL baselines. Its optimization dynamics also resemble maximum likelihood by emphasizing harder prompts and sustaining broader pass-rate distributions.

6 Experiments

  • The experiments evaluate on-policy MaxRL against RLOO and GRPO across exact-ML, infinite-data, data-scarce, and billion-parameter reasoning settings.The study covers ImageNet classification, procedurally generated mazes, GSM8K, and four mathematical reasoning benchmarks.Source: §6 · p. 7

6.1 Comparisons with Exact Maximum Likelihood

  • With sufficient rollouts, MaxRL closely tracks exact maximum likelihood, whereas REINFORCE fails from low initial pass rates and struggles even with very high sampling budgets.MaxRL uses the same sparse successful trajectories as REINFORCE but extracts more learning signal through likelihood-inspired reweighting, improving as rollout counts increase.Source: Figure 2 · p. 8, §6.1 · p. 8

6.2 Infinite Data Regime

  • In the infinite-data maze setting, MaxRL scales more effectively than RLOO and GRPO: 4-rollout MaxRL matches or exceeds 128-rollout RLOO, while 16-rollout MaxRL exceeds 128-rollout GRPO across pass@k metrics.GRPO and RLOO show almost no gap between pass@k and pass@1, whereas MaxRL retains a substantial gap suggesting further improvement with compute.Source: Figure 3 · p. 8, §6.2 · p. 9

6.3 Data-Scarce Regime

  • In the fixed-data GSM8K regime, MaxRL overtakes RLOO and GRPO at about 30 epochs, continues improving through 50 epochs, and preserves substantially healthier pass@k performance.RLOO and GRPO peak in pass@1 around 10 epochs but suffer massive pass@k degradation, while MaxRL reaches a higher peak and is more resistant to overfitting.Source: §6.3 · p. 9, Figure 4 · p. 9

6.4 Large Reasoning Model Training

  • MaxRL Pareto-dominates GRPO across two Qwen model sizes and four math benchmarks, improving pass@1 and pass@k while achieving up to 20× test-time scaling efficiency.MaxRL improves pass@k over both the base model and GRPO checkpoint in 7 of 8 settings, with a perfect verifier filtering incorrect solutions.Source: Figure 5 · p. 10, §6.4 · p. 10

6.5 Behavior Analysis of MaxRL

  • MaxRL produces larger gradient norms on harder prompts and yields a larger fraction of training prompts with at least one correct rollout than GRPO and RLOO.It maintains a broad pass-rate distribution, while GRPO and RLOO push per-prompt rates toward 0 and 1; MaxRL also becomes more stable as rollout budget increases.Source: §6.5 · pp. 10–11, Figure 6 · p. 11, Figure 7 · p. 11

7 Related Work

The paper distinguishes purely interactive RL from supervised or hybrid training and positions MaxRL against related finite- and asymptotic-objective analyses. Its key distinction is deriving an exact finite-sample estimator–objective equivalence without adaptive sampling.

Supervised training vs reinforcement learning

  • Unlike supervised or hybrid training, this work assumes no high-quality demonstrations or stronger model and uses purely interactive RL to optimize a cross-entropy-like objective.Supervised learning is stable and sample-efficient but cannot directly optimize non-differentiable objectives; RL can use reward feedback, while on-policy learning appears important for optimal performance (Tajwar et al., 2024; Xu et al., 2024).Source: §7 · p. 11

Closely related work

  • Xiong et al. (2025b) studies adaptive rollout allocation and sampling strategies, whereas this work uses no adaptive sampling and derives an estimator approaching maximum likelihood as rollout compute increases.Both works address expected-reward underweighting of low-pass-rate prompts through log-likelihood-like weighting, but their empirical focuses differ: adaptive-sampling comparisons versus data and compute scaling.Source: §7 · p. 12
  • Davis and Recht (2025) characterize asymptotic population objectives induced by binary-reward RL, while this work studies how finite sampling explicitly defines objectives.The paper establishes an exact estimator–objective equivalence at each finite rollout count and evaluates interpolation with increased compute in controlled settings and large-scale LLM post-training.Source: §7 · p. 12

8 Conclusion … B Extended Related Works

MaxRL is presented as a compute-indexed likelihood framework for binary-reward reinforcement learning that becomes maximum-likelihood-like with increased compute while outperforming traditional RL in scaling with data and compute. The appendices document reproducibility and situate MaxRL among supervised learning, reinforcement learning, RLVR, exploration, and generalization research.

8 Conclusion

  • MaxRL approaches maximum likelihood as compute increases, improves scaling with additional compute and data, and is currently limited to binary rewards.The authors identify continuous rewards, multi-turn RL, and off-policy PPO-style training as future directions.Source: §8 · p. 12

A Public Code Release

  • The authors release code, scripts, datasets, and implementation details needed to reproduce the paper’s results.The assets are available through the project website and open-source repository.Source: A Public Code Release · p. 30

B Extended Related Works

  • Supervised learning uses fixed target distributions, whereas reinforcement learning addresses policy-induced, nonstationary data and delayed evaluative feedback but suffers from sequential credit-assignment challenges.Behavior cloning can incur compounding errors from state drift, while RL uses reward-based gradients and includes methods such as Q-learning, policy gradients, TRPO, DDPG, and SAC.Source: B Extended Related Works · pp. 30–31
  • Hybrid methods combine supervised and reinforcement learning through interactive correction, inverse or adversarial imitation, demonstrations, trajectory optimization, and preference-based alignment such as InstructGPT and DPO.MaxRL is related to forward-KL methods including RWR and AWR, while finite-compute MaxRL defines a sampling-indexed objective rather than merely approximating a fixed EM objective.Source: B Extended Related Works · p. 31
  • In LLM reasoning, RLVR methods optimize expected reward or pass rate, whereas MaxRL recovers a cross-entropy-based classification objective within an RL training pipeline.The related work distinguishes MaxRL from PPO, GRPO, RLOO, and other algorithms that primarily vary advantage or off-policy updates.Source: B Extended Related Works · pp. 31–32
  • RLVR can sharpen distributions and reduce pass@k because expected-reward optimization marginalizes learning signal from harder prompts, motivating exploration methods targeting diversity and new knowledge.The literature includes intrinsic bonuses, entropy methods, curricula, adaptive sampling, additional rollouts, hints, and risk-sensitive approaches, while reporting mixed evidence on pass@k degradation.Source: B Extended Related Works · pp. 32–33
  • Related reinforcement-learning research studies generalization beyond training environments using procedural benchmarks, regularization, invariances, task-shift methods, robust RL, exploration, and theoretical PAC-MDP frameworks.In LLM alignment, analogous work examines transfer to unseen prompts, behaviors, and user distributions.Source: B Extended Related Works · p. 33

C Theoretical Results … H.3 Equivalence of Validation Top-1 Accuracy and Majority Voting Accuracy

The supplementary sections prove the theoretical identities underlying MaxRL, characterize practical estimator bias, unify RL objectives through weighting functions, and specify implementation and ImageNet evaluation procedures. They also establish that validation top-1 accuracy equals majority-voting accuracy.

C Theoretical Results

  • The maximum-likelihood gradient has a conditional-expectation representation, and the order-T=N MaxRL gradient admits an unbiased sample estimator.The proofs derive the likelihood-gradient identity and show unbiasedness for the MaxRL estimator using successful rollouts and pass@N.Source: C Theoretical Results · p. 34

D Analysis of the Practical Gradient Estimator from Algorithm 1

E More on Unifying Weight-Function View on RL Objectives

  • The population gradients of classical RL, GRPO, maximum likelihood, and MaxRL can all be expressed as a common score-gradient form with objective-specific weighting functions.For maximum likelihood, the weighting function is 1/ppassθ(x), while classical RL uses wRL=1; MaxRL is also rewritten in this framework.Source: E More on Unifying Weight-Function… · pp. 37–38

F RL Implementation Details for LLM Experiments

  • In the LLM implementation, GRPO, RLOO, and MaxRL differ primarily through their advantage functions, while dynamic per-sample baselines avoid a separate value network.The experiments are fully on-policy, so the importance ratio is one and clipping has no effect; all three methods estimate expected return online.Source: F RL Implementation Details for LLM… · p. 39

G Pass@k Calculation

  • For ImageNet, pass@k is computed analytically from πθ(y|x), whereas other generative settings use sampled rollouts and a bootstrap-based low-variance unbiased estimator.The closed-form calculation averages analytically computed pass@k values over validation examples; the sampling-based procedure generates n≥k samples and counts correct outcomes.Source: §G.1 · p. 40, §G.2 · p. 40

H.1 Training Procedure

  • ImageNet training uses binary-reward RL objectives whose per-example losses are averaged over batches before gradient descent updates.GRPO, REINFORCE, and MaxRL share the sampled binary-reward setup and differ in advantage calculation.Source: §H.1 · p. 41

H.2 Training Hyperparameters

  • ResNet-50 models are trained on ImageNet for 20 epochs with SGD momentum, cosine decay, one-epoch warmup, varied rollout counts, and single-GPU runs.The reported standard learning rate is 0.1 after sweeping the specified learning-rate grid; training uses augmentation while evaluation does not.Source: §H.2 · p. 41

H.3 Equivalence of Validation Top-1 Accuracy and Majority Voting Accuracy

  • Validation top-1 accuracy is equivalent to majority-voting accuracy after averaging the per-example metric over validation examples.This equivalence connects the ImageNet metric to majority voting used in traditional LLM chain-of-thought reasoning tasks (Wang et al., 2023a).Source: §H.3 · pp. 41–42

H.4 Gradient Norm Analysis … I.3 Training Setups

Across ImageNet and maze experiments, MaxRL produces gradients aligned with maximum likelihood, outperforms GRPO as compute increases, and achieves better rollout coverage. The maze studies use a compact Transformer, procedurally generated symbolic mazes, and two RL-training regimes designed to compare baselines under different compute budgets.

H.4 Gradient Norm Analysis

  • Cross-entropy and MaxRL assign larger gradient norms to hard inputs, whereas GRPO emphasizes medium-difficulty inputs and REINFORCE produces negligible gradients.After 1500 steps from scratch, REINFORCE’s pass rate remains below 0.003 because its low gradient norm prevents sufficient model improvement; on pretrained models it produces gradients but remains weak on hard inputs.Source: §H.4 · p. 42, Figure 8 · p. 42

H.5 Comparison with GRPO and Additional Metrics

  • MaxRL outperforms GRPO and scales better with additional compute, while GRPO and REINFORCE have worse coverage than MaxRL.Figure 9 evaluates pass@1, pass@128, majority voting accuracy, and checkpoint coverage; GRPO improves with more compute but remains suboptimal to MaxRL and cross-entropy.Source: Figure 9 · p. 42, §H.5 · p. 43

I.1 Model Architecture

  • The maze experiments use a 3M-parameter, four-layer Qwen2-style decoder-only Transformer with 256 hidden units, 1024 intermediate units, four attention heads, and 512-token maximum length.The model uses grouped-query attention, RMSNorm, SiLU activations, RoPE, tied embeddings, bfloat16 precision, and a 32-token vocabulary.Source: §I.1 · p. 44

I.2 Task Description

  • Mazes are procedurally generated with Prim’s algorithm, represented as symbolic grid tokens, and solved by autoregressively generating directional actions followed by DONE.Grid cells encode WALL, PATH, START, or GOAL; rows use NEWLINE, boundaries use GRID_START and GRID_END, and actions come from a fixed vocabulary.Source: §I.2 · p. 44

I.3 Training Setups

  • The maze dataset contains 1 million distinct 17 × 17 training mazes and 256 non-overlapping test mazes, with 1,500-step SFT pretraining before RL.SFT uses AdamW, learning rate 5 × 10^-4, and batch size 32 to initialize the required trajectory-output format.Source: §I.3 · p. 45
  • RL training uses learning rate 1 × 10^-4 with one on-policy parameter update per step, evaluated in two compute settings.The settings are batch size 256 for 9,000 steps with varying training rollouts, and batch size 32 for 20,000 steps with 128 rollouts per task.Source: §I.3 · p. 45

I Maze Experiments

  • The larger-batch regime limits comparisons to MaxRL, GRPO, and RLOO, while the cheaper 20,000-step regime supports broader baseline comparisons and final Pass@k summaries.The 256-batch setting is used for Figures 3 and 11; the 32-batch setting is used for Figure 12 and Table 3 because training all baselines at larger batch size exceeds the compute budget.Source: §I.3 · p. 45

I.3 Training Setups

  • Each training run uses four RTX 4090 GPUs, with a modified verl rollout engine to increase sampling parallelism and accelerate training.The model’s small size removes the memory bottleneck, motivating replacement of the default vLLM engine.Source: §I.3 · p. 45

I.4 Training Dynamics in Our Highest Compute Budget Setting … J.2 Hyperparameters

Across maze training and broader baseline comparisons, MaxRL improves both average accuracy and sampling coverage, while the GSM8K experiments specify a data-scarce evaluation setup and detailed training configuration. The reported settings include high-compute maze training, SmolLM2-360M on GSM8K, and fixed hyperparameters for baseline comparisons.

I.4 Training Dynamics in Our Highest Compute Budget Setting

  • MaxRL outperforms GRPO and RLOO across pass@1, pass@32, pass@128, and pass@256 during 9,000-step, high-compute maze training.The setting uses a 3M model, 17x17 maze puzzles, batch size 256, and 128 training rollouts per task.Source: Figure 11 · p. 45, §I.4 · p. 45

I.5 Comparison Table with Additional Baselines

  • In the broader baseline comparison, MaxRL significantly outperforms entropy bonus, PKPO (Walder and Karkhanis, 2025), Differential Smoothing (Gai et al., 2026), and SELF (Nguyen et al., 2025b) across metrics.MaxRL is the only method described as maintaining both strong pass@1 and pass@k coverage.Source: Figure 12 · p. 46, §I.5 · p. 46
  • 84.4 pass@1 and 94.3 pass@256 let MaxRL outperform standard GRPO by more than 44 pass@1 points while exceeding broader baselines in maze accuracy and coverage.PKPO reaches 74.5 pass@1 with a nearly flat pass@k curve, while SELF reaches 87.5 pass@256 but only 46.1 pass@1.Source: §I.5 · p. 46, Table 3 · p. 46

J GSM8K Experiments

  • The GSM8K experiments study data-scarce, compute-rich reinforcement fine-tuning using SmolLM2-360M-Instruct, 1,500 training steps, GSM8K, and the non-overlapping GSM8K-Platinum test set.Each prompt requests step-by-step reasoning and a final answer inside boxed notation.Source: §J.1 · p. 47

J.1 Prompt template

  • The GSM8K prompt template appends a step-by-step instruction requiring the final answer within \boxed{} before tokenizer chat-template processing.The supplied example asks a hiking-time problem and produces the final answer \boxed{5}.Source: §J.1 · p. 47

J.2 Hyperparameters

  • The GSM8K setup uses 256 prompts per batch, 128 generations per prompt, learning rate 1 × 10−5, 1,500 steps, zero KL and entropy coefficients, and eight Nvidia GH200 devices.Baseline settings use T = 16 for PKPO and γ = 0.05 for differential smoothing.Source: §J.2 · p. 47, Table 4 · p. 47

J.3 Effect of Additional Compute … K.2 Hyperparameters

On GSM8K, MaxRL improves with additional rollout compute, outperforms all compared baselines, and preserves intermediate-difficulty prompts that provide learning signal. The experiments also specify the Qwen Math prompt format and default on-policy training and evaluation settings.

J.3 Effect of Additional Compute

  • Almost 30%: T = 128 outperforms T = 1 (REINFORCE) at pass@2048, with larger gains at higher pass@k as training rollouts increase.For MaxRL, increasing rollouts directly increases the approximation order T of the finite-sample maximum-likelihood objective.Source: §J.3 · pp. 47–48, Figure 13 · p. 48

J.4 Comparison with Additional Baselines

  • MaxRL achieves the best performance on every GSM8K metric, reaching 33.2 pass@1 and 83.4 pass@1024, including a 7.5-point advantage over PKPO at pass@1024.At pass@1024, MaxRL exceeds GRPO, RLOO, GRPO with entropy bonus, and Differential Smoothing by more than 30 points.Source: Table 5 · p. 48, §J.4 · p. 48

J.5 Per-Prompt Pass-Rate Distribution During Training

  • MaxRL retains a broad per-prompt pass-rate distribution, whereas GRPO and RLOO concentrate prompts toward pass rates of 0 or 1.This preserves non-zero learning signal on intermediate-difficulty prompts and supports extracting more signal from a fixed training dataset.Source: §J.5 · p. 48, Figure 14 · p. 49

K.1 Prompt template

  • The Qwen Math prompt template appends step-by-step reasoning instructions and requires the final answer within \boxed{{}}, then applies the Qwen3 tokenizer chat-template.The displayed example formats an integer digit-sum problem with these instructions.Source: §K.1 · p. 50

K.2 Hyperparameters

  • The default training setup is fully on-policy, with no importance ratios, clipping, KL penalty, or entropy bonus.A GRPO-plus-entropy-bonus baseline is also evaluated on GSM8K; MaxRL outperforms it, although the bonus slightly mitigates GRPO’s issues.Source: §K.2 · p. 50
  • Training rollouts use temperature 1.0 without special or adaptive sampling, while evaluation uses temperature 0.6 and top-p 0.95 without top-k or min-p sampling.The setup also omits fixes for inference-training logit mismatch.Source: §K.2 · p. 50
  • Table 6 provides the default training hyperparameters for Qwen3-1.7B-Base and Qwen3-4B-Base training.The table defines the default settings used for these Qwen3 experiments.Source: §K.2 · p. 50, Table 6 · p. 50

K.3 Additional Results · L Comparison with Additional Baselines · L.1 PKPO

Additional Qwen3 results show that MaxRL matches or exceeds base and GRPO performance across extended benchmarks and majority voting, while retaining favorable training dynamics and validation accuracy. Against PKPO, MaxRL outperforms direct pass@k optimization across small and large-scale settings.

K.3 Additional Results

  • Up to 20.5× speedup over GRPO accompanies similar or better pass@1 on four additional reasoning benchmarks, with MaxRL matching or outperforming base and GRPO.The benchmarks are AIME 2024, HMMT Feb 2025, HMMT Nov 2025, and JEEBENCH (Arora et al., 2023; MathArena, 2025a; MathArena, 2025b).Source: Figure 15 · p. 51, §K.3 · p. 51
  • MaxRL outperforms both the pretrained base model and GRPO on majority-voting accuracy across all five reported benchmarks.Majority voting draws N independent rollouts and selects the most frequent final answer (Wang et al., 2023a).Source: §K.3 · p. 51, Table 7 · p. 51
  • Across both Qwen3 model sizes, MaxRL generates longer chains-of-thought, retains higher actor entropy, and produces larger gradient norms than GRPO.These training-dynamics trends are reported for Qwen3-4B-Base and Qwen3-1.7B-Base.Source: §K.3 · p. 51, Figure 16 · p. 52, Figure 17 · p. 52
  • During training, MaxRL consistently beats GRPO in pass@1 for Qwen3-4B-Base, while Qwen3-1.7B-Base catches up after initially trailing and ultimately converges higher.Pass@1 is estimated using 32 samples on intermediate evaluation datasets.Source: §K.3 · p. 51, Figure 18 · p. 52, Figure 19 · p. 52

L.1 PKPO

  • MaxRL outperforms PKPO at every evaluated pass@k, including pass@16 despite PKPO explicitly optimizing pass@16.The comparison uses PKPO with T = 16 on maze and GSM8K, holding other hyperparameters at their defaults; REINFORCE similarly underperforms MaxRL at pass@1.Source: §L.1 · p. 53, Table 8 · p. 53, Table 9 · p. 53
  • On POLARIS-53K Qwen3-4B-Base training, MaxRL matches or exceeds PKPO across four evaluation datasets and has larger gaps on BeyondAIME and AIME 2025.PKPO is tested with T values 4, 8, and 16; MaxRL wins at every evaluated pass@k on the harder datasets.Source: §L.1 · p. 53, Figure 20 · p. 53

L.2 ZPD … M.1 Non-Zero Gradient Signal Analysis

Across difficulty-aware sampling, nonlinear pass-rate weighting, and gradient-signal analyses, MaxRL consistently outperforms the compared alternatives while retaining a broader source of useful training gradients. These comparisons span ImageNet, maze, GSM8K, and large-scale POLARIS-53K training settings under matched budgets or recipes.

L.2 ZPD

  • On ImageNet, ZPD performed similarly to REINFORCE and failed to improve significantly beyond the randomly initialized model.ZPD reweights prompts using p(1−p), implemented here through a pass-rate-based advantage without additional curriculum sampling.Source: Figure 21 · p. 54, §L.2 · p. 54
  • MaxRL significantly outperformed ZPD in pass@k for every evaluated k on maze and GSM8K, including a ~30% pass@2048 improvement on GSM8K.The methods used identical training settings and compute budgets.Source: Figure 22 · p. 54, §L.2 · p. 54
  • 12.6× to 29.0× gains in test-time scaling efficiency over ZPD were achieved by MaxRL on all evaluation datasets under identical Qwen3-4B-Base training conditions.The comparison used POLARIS-53K, 1000 RL-training steps, and a perfect verifier.Source: §L.2 · p. 54, Figure 23 · p. 55

L.3 Power Function

  • On ImageNet’s supervised classification setting, MaxRL outperformed the Power Function at both pass@1 and pass@128 despite using 128x less rollout compute, with the gap larger at pass@128.Source: §L.3 · p. 55
  • On maze and GSM8K, MaxRL significantly outperformed the Power Function in pass@k for every evaluated k, including a ~20% pass@2048 gain on GSM8K.The comparison used identical training hyperparameters and compute budgets.Source: Figure 25 · p. 56, §L.3 · p. 56
  • MaxRL achieved up to 12.4× test-time scaling efficiency gains over the Power Function on all four evaluation datasets, while matching or outperforming it.This Qwen3-4B-Base and POLARIS-53K comparison used identical training setup, 1000 RL-training steps, and a perfect verifier.Source: Figure 26 · p. 56, §L.3 · p. 56

M.1 Non-Zero Gradient Signal Analysis

  • MaxRL consistently produced non-zero pass rates for a larger fraction of maze and GSM8K training tasks than GRPO and RLOO, preserving useful gradient signal as training progressed.Tasks with zero correct rollouts contribute no gradient; MaxRL’s harder-prompt upweighting and larger gradient norms therefore keep more of the dataset contributing after easier prompts saturate.Source: §M.1 · p. 57, Figure 27 · p. 57

M.2 Policy Gradient Variance Analysis · M.3 On the Distribution Sharpening Narrative · M.4 Non-Binary Reward Settings

The analyses show that MaxRL’s baseline reduces gradient variance, its stability improves with more rollouts, and it can solve problems unavailable to base and GRPO models. A natural non-binary-reward extension also outperforms GRPO, although broader evaluation remains future work.

M.2 Policy Gradient Variance Analysis

  • MaxRL’s baseline consistently lowers gradient-estimator variance across rollout counts and tasks, with the largest effect at low rollout counts and convergence at higher counts.This result is reported for GSM8K, Polaris-53K, and Maze experiments using sampled minibatches.Source: §M.2 · p. 57, Figure 28 · p. 58
  • MaxRL’s normalized gradient variance decreases as rollout count increases, following the same qualitative trend as GRPO and RLOO.The analysis compares inverse signal-to-noise ratio because the methods optimize different objectives.Source: Figure 29 · p. 58, §M.2 · p. 58
  • Together, the variance results indicate that MaxRL becomes more stable with larger rollout budgets despite stronger amplification of very-low-success-rate problems.The conclusion concerns both baseline-induced variance reduction and rollout-budget scaling.Source: §M.2 · p. 58

M.3 On the Distribution Sharpening Narrative

  • With 4096 samples per AIME 2025 question, MaxRL solves a strict superset of the problems solved by the base and GRPO-trained models, including three exclusive problems.The three additional problems are 10, 24, and 27.Source: §M.3 · pp. 58–59, Table 10 · p. 59
  • Correct MaxRL trajectories for the three exclusive AIME problems have high perplexity under both the base and GRPO-trained models, suggesting capabilities beyond the base model.An equal number of incorrect base-model rollouts had much lower perplexity; broader study is left for future work.Source: Figure 30 · p. 59, §M.3 · p. 59

M.4 Non-Binary Reward Settings

  • For non-binary rewards, MaxRL extends naturally under non-negative rewards with non-zero policy expectation, using a Monte-Carlo estimate of expected reward.The extension is evaluated on a modified Maze task with continuous rewards in [0, 1], where shorter trajectories receive higher rewards.Source: §M.4 · pp. 59–60
  • In the continuous-reward Maze setting, MaxRL outperforms GRPO by a large margin on both Mean@32 and Best@32 validation reward, while GRPO’s Best@32 collapses.Both Mean and Best reward continue increasing for MaxRL at the end of training, but broader theoretical and empirical evaluation remains future work.Source: Figure 31 · p. 60, §M.4 · p. 60

Related papers

Loading 2602.02710v3…