PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning over mathematical objects: on-policy reward modeling and test time aggregation

Pranjal Aggarwal, Marjan Ghazvininejad, Seungone Kim, Ilia Kulikov, Jack Lanchantin, Xian Li, Tianjian Li, Bo Liu, Graham Neubig, Anaelia Ovalle, Swarnadeep Saha, Sainbayar Sukhbaatar, Sean Welleck, Jason Weston, Chenxi Whitehouse, Adina Williams, Jing Xu, Ping Yu, Weizhe Yuan, Jingyu Zhang, Wenting Zhao

arXiv:2603.18886v1cs.AIcs.CL

TL;DR

Deriving structured mathematical objects matters for STEM, but targeted data and evaluations for this ability are limited. This paper introduces Principia benchmarks and on-policy training and aggregation approaches, reporting gains on mathematical-object derivation and conventional numerical and MCQA tasks.

  • Problem

    Research has limited targeted data for deriving complex mathematical objects, and checking such answers is difficult because equivalent expressions can take many forms.

    Source: §1.1 · p. 3, §1.3.1 · p. 7

  • Method

    The paper introduces PrincipiaBench, an RL training collection, and VerifyBench, alongside on-policy LLM reward modeling and test-time aggregation.

    Source: §1.7 · p. 19, §2.3 · p. 23, §2.3.2 · p. 24

  • Results

    7.22-18.35% average improvements over each base LM on PrincipiaBench accompany enhanced reasoning on numerical and MCQA tasks.

    Source: Introduction · p. 2, §1.7 · p. 20

  • Takeaways & Limitations

    Accurate rewards when training on the Principia Collection require a model-based verifier.

    Source: §1.7 · p. 20

    PrincipiaBench excludes problems needing missing context or essential definitions, so it evaluates derivation from self-contained statements rather than incomplete inputs.

    Source: §1.3.1 · p. 7

Abstract

from arXiv · show

The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.

Introduction

The introduction argues that evaluations limited to numerical or multiple-choice answers risk overstating broad reasoning competence, motivating methods for deriving precise mathematical objects. It previews the Principia suite, on-policy reward modeling, and parallel aggregation as approaches to stronger reasoning.

Introduction

  • Deriving mathematical objects such as matrices and piecewise functions is important for precise scientific expression, but evaluations often use numerical or multiple-choice answers for easier scoring.The introduction cautions that inferring broad reasoning competence from these restricted formats risks overstating model capabilities.Source: Introduction · p. 1
  • PrincipiaBench contains 2,558 carefully filtered mathematical-object problems, while the Principia Collection provides 248K synthetic problems requiring six types of mathematical-object answers.The benchmark draws on four sources and was used to evaluate 27 LMs; the paper reports that it offers a more challenging testbed for frontier-LM reasoning.Source: Introduction · pp. 1–2
  • 7.22-18.35% average improvements over each base LM on PrincipiaBench followed RL training on the Principia Collection with a strong LLM verifier.The RL-trained models also improve 7.5-17.5% on AIME 2024 and 12.31-25.47% on GPQA-Diamond, and outperform competitive baselines sharing the same base LM.Source: Introduction · p. 2
  • RLLM trains an LM as a reasoning Reward Model, then uses its k-wise comparative judgments to reward a policy LM across verifiable and non-verifiable domains.RLLM outperforms RLHF with scalar reward models and RLVR with rule-based verifiers on math, physics, and instruction-following tasks; on-policy-trained reward models also outperform prompted and off-policy alternatives.Source: Introduction · p. 2
  • ParaGator trains parallel generation and aggregation online end-to-end, using pass@k optimization to encourage complementary solutions and pass@1 optimization to select or synthesize a final answer.It addresses redundant or insufficiently diverse generations and the mismatch between off-policy training and inference; experiments report strong results against existing parallel reasoning methods.Source: Introduction · p. 2

1 Principia: Training Language Models to Reason over Mathematical Objects

RL training on the Principia Collection improves mathematical-object reasoning and also benefits numerical and MCQA performance. A strong model-based verifier and excluding MCQA data yield the best performance.

1 Principia: Training Language Models to Reason over Mathematical Objects

  • +18.23%, +10.23%, and +15.16% average gains on PrincipiaBench were achieved by Principia-trained Qwen3-4B-Base, Qwen2.5-7B-Base, and OctoThinker-8B-Long-Base, respectively.PrincipiaBench requires deriving mathematical objects.Source: Figure 1 · p. 3
  • Principia Collection training also improves performance on the numerical AIME-2025 and MCQA SuperGPQA benchmarks, demonstrating cross-format reasoning gains.Source: Figure 1 · p. 3
  • Using the GPT-OSS-120B model-based verifier and excluding MCQA data provides the best performance.Source: Figure 1 · p. 3

1.1 Overview

The paper addresses the scarcity of data requiring graduate-level STEM reasoning to derive complex mathematical objects by introducing benchmarks and a training dataset. It reports improved reasoning from RL training and evaluates tools for judging mathematical-object equivalence.

1.1 Overview

  • On PrincipiaBench, Qwen3-235B scores 55.58 and o3 scores 62.90, below their AIME-2024 scores of 83.33 and 85.63 and GPQA-Diamond scores of 74.34 and 82.29.PrincipiaBench contains 2,558 carefully filtered problems from four sources, requires complex mathematical-object answers, and was used to evaluate 27 LMs.Source: §1.1 · p. 3
  • The Principia Collection contains 248K problem statements grounded in mathematics and physics taxonomies and requiring six types of mathematical-object answers.The answer types are equations, inequalities, intervals, sets, matrices, and piecewise functions.Source: §1.1 · p. 3
  • The authors introduce Principia VerifyBench to assess verifier judgments of mathematical-object equivalence, finding o3 correct in 94.05% of 168 cases where it disagreed with math-verify.They report that rule-based verifiers such as math-verify are surprisingly fragile for complex mathematical-object answers and use o3 as the evaluation judge.Source: §1.1 · pp. 3–4
  • RL training on the Principia Collection improves average PrincipiaBench performance by 7.22–18.35% over each base LM and transfers gains to AIME 2024 and GPQA-Diamond.The reported improvements are 7.5–17.5% on AIME 2024 and 12.31–25.47% on GPQA-Diamond; trained models also outperform competitive baselines sharing the same base LM.Source: §1.1 · p. 4

1.2 Motivation: Need for a benchmark and training data for mathematical objects

Existing benchmarks and post-training data emphasize easy-to-evaluate short answers or multiple-choice options, while mathematical and scientific work often requires de novo derivation of mathematical objects. Removing options reveals substantial performance drops, motivating benchmarks and training data focused on object derivation.

Motivation: Need for a benchmark and training data for mathematical objects

  • The field lacks benchmarks and training data for deriving complex mathematical objects, partly because simpler formats are easier to evaluate and accurate reward signals are difficult.The passages contrast multiple-choice selection with researchers’ everyday work of building new knowledge through de novo derivations.Source: §1.2 · pp. 4–5
  • Removing options on SuperGPQA mathematical-object instances drops performance 10–20%; o3 falls 69.10 →62.90 and Qwen3-235B 69.33 →55.58, suggesting MCQA overestimates derivation ability.The reported declines are for the mathematics and engineering subset filtered to instances requiring mathematical-object answers.Source: Figure 3 · p. 5, §1.2 · p. 5
  • With ten answer choices, Qwen3-235B uses backward chaining from options, but without them makes false assumptions, misses valid cases, and constructs illogical arguments.The passage presents this as an example of option-driven reasoning that changes fundamentally when choices are removed.Source: §1.2 · p. 5
  • Most existing RL post-training answers are numerical, multiple-choice, or short text spanning ten to a few hundred tokens, and prompts rarely resemble structured graduate-level STEM specifications.Existing problem statements mostly fall within a few hundred tokens, while longer answers below roughly 100–200 tokens are still not complex mathematical objects.Source: §1.2 · pp. 5–6
  • The Principia Collection addresses this gap with exclusively complex mathematical-object answers and graduate-level STEM problem statements encoding concepts and multiple conditions.The passages note that token length itself is not a measure of difficulty; the longer answers consist of mathematical objects and the longer prompts encode graduate-level concepts.Source: §1.2 · p. 6

1.3 The Principia Suite

The Principia Suite introduces a benchmark for directly deriving mathematical objects, a meta-evaluation benchmark for answer-equivalence verifiers, and a large synthetic post-training collection. Together, these resources target object-form answers and the challenges of validating and generating them.

1.3.1 PrincipiaBench

  • PrincipiaBench assesses direct derivation of equations, inequalities, intervals, sets, matrices, or piecewise functions, using 2,158 final problems filtered for single-question, self-contained statements.The authors filtered candidate problems for undesirable traits, manually inspected them, and removed 128 cases with highly variable judgments.Source: §1.3.1 · pp. 6–7
  • Equivalence verification is difficult because mathematical objects admit many valid forms, and rule-based methods such as Sympy can fail on equivalent expressions.The authors use o3 as a judge that classifies reference-answer and prediction pairs as equivalent or not equivalent.Source: §1.3.1 · pp. 7–8

1.3.2 Principia VerifyBench

  • Principia VerifyBench contains 168 unanimously labeled disagreements between o3 and math-verify: 75 equivalent and 93 not equivalent.The initial sample comprised 200 disagreements, split evenly by which verifier judged the answers equivalent; instances were retained only when two annotators agreed.Source: §1.3.2 · p. 9

1.3.3 Principia Collection

  • The Principia Collection contributes 248,748 graduate-level problem–answer pairs whose answers are mathematical objects.The final collection retains 248,748 instances after filtering.Source: §1.3.3 · pp. 9, 12
  • The collection mines broad-to-specialized mathematics and physics topics to promote problem diversity, then refines generated statements to require deeper reasoning.Problem proposals are revised when they remain straightforward, as illustrated by adding an interpolating Hamiltonian, free-energy convexity, and a pair-correlation function.Source: §1.3.3 · p. 10
  • For collection labels, eight GPT-OSS-120B answers are grouped through pairwise equivalence and transitivity checks; a prediction equivalent to at least five becomes the reference.A union–find algorithm keeps groups consistent when equivalence judgments violate transitivity.Source: §1.3.3 · pp. 10–12

1.4 Experimental Results

Principia training improves mathematical-object derivation and transfers to numerical and MCQA tasks, while PrincipiaBench remains challenging for frontier models.

1.4.1 Experimental Setup

  • Experiments train 4B–8B base models with GRPO and evaluate mathematical objects, MCQA, and numerical tasks using PrincipiaBench, GPQA/SuperGPQA, and AIME.Training is capped at 1K steps with checkpoint selection by validation scores; PrincipiaBench reports mean@8, GPQA-Diamond mean@16, SuperGPQA mean@8, and AIME mean@32.Source: §1.4.1 · p. 13

1.4.2 Main Results

  • Across four base models, Principia training consistently improves PrincipiaBench performance, and Principia models achieve the highest scores in every setting.The reported gains are Qwen2.5-7B-Base +10.12% (12.75 →22.87), OctoThinker-8B-Long-Base +15.30% (3.75 →19.05), Qwen3-4B-Base +18.35% (11.31 →29.66), and Qwen3-4B +7.22% (41.23 →48.45).Source: Table 2 · p. 14, §1.4.2 · p. 14
  • Principia-4B outperforms Polaris-4B on SuperGPQA, GPQA-Diamond, and AIME, and Principia models also lead on average across object, MCQA, and numerical benchmarks.The authors attribute this transfer to learning to construct complex mathematical forms, despite the Principia Collection containing only mathematical-object answers.Source: §1.4.2 · p. 14, Table 2 · p. 14
  • Principia Collection training yields more consistent validation improvements than DeepScaleR or WebInstruct-Verified and a more stable increase in response length.DeepScaleR primarily uses numerical-answer problems, while WebInstruct-Verified uses simple mathematical objects and numerical answers; Figure 10 measures response length on the validation set.Source: Figure 9 · p. 15, §1.4.2 · p. 15, Figure 10 · p. 15
  • o3 scores 62.90 and Qwen3-235B 55.58 on PrincipiaBench, where most evaluated LMs perform substantially worse than on established reasoning benchmarks.The authors characterize PrincipiaBench as a challenging testbed for evaluating frontier-model reasoning about mathematical objects.Source: §1.4.2 · p. 15

1.5 Analysis

The analysis finds that complex mathematical-object answers benefit from strong model-based verification, while mathematical-object training transfers across answer formats and weight merging outperforms joint training. MCQA-only training performs poorly, and adding MCQA data to mixed-format training reduces performance.

1.5.1 Benefits of model-based verifiers over rule-based verifiers

  • 95.24% agreement for GPT-OSS-120B and 94.05% for o3, versus 5.95% for math-verify on Principia VerifyBench.The benchmark deliberately samples cases where o3 and math-verify disagree, and the paper concludes that o3 is more likely to be correct in those cases; fine-tuned verifiers have under 90% agreement, while Qwen3-4B reaches 92.26%.Source: §1.5.1 · p. 16
  • 13.31 →31.80 on the Principia Collection with GPT-OSS-120B verification, versus 13.31 →16.30 with math-verify and 13.31 →9.54 with general-verifier.Verifier type makes little difference on DeepScaleR, whose total averages range from 28.67 to 30.45; the results instead vary substantially on complex mathematical-object data.Source: §1.5.1 · pp. 16–17, Table 4 · p. 17

1.5.2 Unifying different answer formats

  • 33.23 vs. 29.92 total average for weight merging versus joint training on math-object and numerical data; with all three formats, the scores are 31.95 vs. 28.39.Weight merging also produces the strongest PrincipiaBench scores in these settings: 22.93 on Physics, 45.21 on ARB, and 34.38 on SuperGPQA*; incorporating MCQA reduces performance versus math-object and numerical data alone.Source: Table 5 · p. 18, §1.5.2 · p. 18
  • 25.04 total average for MCQA-only training, with 24.96 on PrincipiaBench Avg., 17.71 on AIME-24, and 19.79 on AIME-25, indicates weak transfer across formats.The authors suggest that option-anchored heuristics, such as reasoning backward from choices, can be reinforced by MCQA training and impede deeper derivational reasoning.Source: Table 5 · p. 18, §1.5.2 · p. 18
  • 45.53 on SuperGPQA, 43.56 on GPQA-Diamond, 22.71 on AIME-24, and 23.96 on AIME-25 for math-only training show transfer to MCQA and numerical benchmarks.The synthetic datasets use the same subject entities while requiring mathematical-object, numerical-value, or multiple-choice answers, enabling comparison of answer formats while controlling for problem diversity.Source: §1.5.2 · pp. 17–18, Figure 11 · p. 17

1.6 Related Work

This section situates Principia against standard numerical-answer and multiple-choice reasoning evaluations and synthetic-data approaches, arguing that both underserve reasoning over complex mathematical objects.

Related Work

  • LM reasoning is commonly evaluated with numerical-answer math and Olympiad exams (Cobbe et al., 2021; Hendrycks et al., 2021; Luong et al., 2025a) or science multiple-choice questions (Wang et al., 2024b; Rein et al., 2024; Du et al., 2025).Source: §1.6 · p. 19
  • Unlike Ma et al. (2025a), which evaluates General-Reasoner only on multiple-choice and numerical-answer Olympiad tasks, this work introduces PrincipiaBench for deriving mathematical objects.Source: §1.6 · p. 19
  • Deriving or reasoning over mathematical objects is a harder, broader test of reasoning and a critical skill for experiments and scientific writing, yet remains overlooked between standardized exams and scientific discovery.Source: §1.6 · p. 19
  • Prior reasoning-data work augments existing problem–answer pairs with chain-of-thoughts to ensure correctness, but this limits problem-source diversity (Zelikman et al., 2022; Kim et al., 2023a; Mukherjee et al., 2023).Common post-training datasets draw on sources such as AMC, AIME, IMO, IMC, and Chinese K-12 exams, but contain few problems requiring complex mathematical objects.Source: §1.6 · p. 19
  • Forum- and document-based synthesis likewise inherits source limitations; Principia instead generates problems from graduate-level subjects using MSC2020 and PhySH entities, with mathematical objects as answers.Source: §1.6 · p. 19
  • Training on the Principia Collection yields strong performance on PrincipiaBench and other reasoning benchmarks.Source: §1.6 · p. 19

1.7 Conclusion

The Principia Collection improves mathematical-object derivation and generalizes to MCQA and numerical tasks, while the paper introduces a benchmark, training dataset, and verifier-evaluation suite for this capability.

1.7 Conclusion

  • The paper introduces PrincipiaBench for mathematical-object derivation, the Principia Collection for RL post-training, and Principia VerifyBench for evaluating verifiers.Existing benchmarks relying on multiple-choice formats fail to capture mathematical-object derivation.Source: §1.7 · p. 19
  • Principia Collection-trained LMs improve substantially on PrincipiaBench and show enhanced reasoning on MCQA and numerical tasks.Experiments also show that a model-based verifier is necessary to provide accurate rewards when training on the Collection.Source: §1.7 · p. 19
  • The authors propose PrincipiaBench for tracking reasoning improvements, the Collection and its pipeline for synthetic-data research, and VerifyBench for training and evaluating model-based verifiers.Source: §1.7 · p. 20

2 RLLM: Unified Post-Training via On-Policy-Trained Language Model as a Reward Model · 2.1 Overview · 2.2 Preliminaries

RLLM uses an on-policy-trained, reasoning language model as a reward model to unify post-training across verifiable, hard-to-verify, and non-verifiable tasks. The section contrasts this approach with RLHF’s scalar rewards and RLVR’s rule-based verifiers, and outlines each paradigm’s scope.

2 RLLM: Unified Post-Training via On-Policy-Trained Language Model as a Reward Model

  • RLLM uses an LM’s reasoning to produce explicit judgments and its instruction-following ability to flexibly design rewards, covering verifiable, hard-to-verify, and non-verifiable tasks.It is presented as a unified alternative to scalar reward models and rule-based verifiers.Source: Figure 12 · p. 21, §2.1 · pp. 21, 23

2.2 Preliminaries

  • Scalar RLHF reward models provide a single score without intermediate reasoning, limiting reasoning improvements and leaving them vulnerable to reward hacking and poor OOD generalization.Classical RLHF trains a scalar reward model on human preference pairs and uses it to optimize a policy via RL.Source: §2.1 · p. 21, §2.2 · p. 23
  • RLVR is effective when outcomes are verifiable, but its need for high-quality reference answers limits it to tasks such as easy-to-verify math and coding, rather than hard-to-verify or non-verifiable tasks.Its verifiable-reward approach has shown success in mathematical problem-solving and verifiable instruction following (Shao et al., 2024; Lambert et al., 2024).Source: §2.1 · p. 21, §2.2 · p. 23

2.1 Overview

  • RLLM outperforms RLHF and RLVR across verifiable and non-verifiable tasks, with especially large gains on hard-to-verify math and physics problems.For Qwen3-1.7B, Figure 13 reports consistently higher accuracy and win rates across four post-training data settings, particularly on hard-to-verify problems.Source: Figure 13 · p. 22, §2.1 · pp. 22–23
  • RLLM trains a thinking reward model on policy-sampled responses using synthetic teacher or verifier labels, then uses its reasoning-backed generative rewards to optimize the policy with RL.The reward model produces a reasoning trace before assigning a score.Source: §2.1 · p. 22
  • Effective policy improvement requires both on-policy reward-model training and a strong generator-verifier capability gap; off-policy reward models struggle to provide accurate downstream signals.The gap is created by using a stronger reward model to train a weaker policy.Source: §2.1 · pp. 22–23

2.2 Preliminaries

  • RLVR replaces RLHF’s reward model with a verifier that rewards responses only when they match a reference answer (Lambert et al., 2024).Verifiers can be rule-based, such as math checks or coding unit tests, or model-based.Source: §2.2 · p. 23

2.3 RLLM: Reinforcement Learning with Language Models as Reward Models

RLLM uses a generative, reasoning language model as a reward model to unify post-training, with reward judgments adapted to task type and reference availability.

2.3 RLLM: Reinforcement Learning with Language Models as Reward Models

  • Unlike scalar RLHF rewards or deterministic RLVR verifiers, RLLM uses generative LM judgments and applies RL to train the reward model and optimize a policy.The LM-as-RM generates reasoning traces and scores, but policy optimization uses only the scores.Source: §2.3 · pp. 23–24

2.3.1 Task Settings and LM-as-RM Variants

  • For verifiable and subjective tasks, LM-as-RM can score responses pointwise with or without references, or jointly score candidate sets pairwise or listwise.Reference-based judgments can act as generative verification for problems requiring substantial reasoning.Source: §2.3.1 · p. 24

2.3.2 LM-as-RM Training via RLVR

  • LM-as-RM training uses on-policy responses labeled by a strong teacher, then trains the model with RLVR to predict the teacher’s score.Math labels are typically binary, non-verifiable-task labels continuous, and the synthetic dataset is balanced; J1 formats the judgments for RLVR, while GRPO rewards matching the teacher score (Whitehouse et al., 2025b; Shao et al., 2024).Source: §2.3.2 · pp. 24–25

2.3.3 RLLM Policy Training with LM-as-RM

  • Policy optimization samples rollouts and uses LM-as-RM scores; pointwise scoring is most efficient, while pairwise and listwise judgments compare candidates.Exhaustive pairwise comparisons have O(n^2) judgment complexity, whereas pivot pairwise uses O(n) evaluations; the policy and reward model may share an initialization, yielding self-rewarding RLLM (Yuan et al., 2025a).Source: §2.3.3 · p. 25

2.4 Experimental Setup

The setup trains Qwen3-1.7B policies with J1-trained reward models or verifiers and evaluates them across verifiable reasoning and non-verifiable instruction following against RLHF, RLVR, and prompted-LM baselines.

2.4.1 LM-as-RM Models

  • J1-Qwen3-32B reward models and verifiers are trained on 18,774 synthetically annotated, balanced responses to hard-to-verify prompts.The recipe draws prompts from OpenMathReasoning and Principia Collection, samples 16 Qwen3-1.7B responses per prompt, and uses GPT-OSS-120B to label correctness; reference-free and reference-based variants are called J1-Qwen3-32B-RM and J1-Qwen3-32B-Verifier.Source: §2.4.1 · p. 25

2.4.2 RLLM Policy Models

  • RLLM trains Qwen3-1.7B Instruct policies with J1 reward models across hard-to-verify problems with or without references and easy-to-verify problems with or without references.The reference-free hard-to-verify setting uses J1-Qwen3-32B-RM, while the reference-based setting uses J1-Qwen3-32B-Verifier; easy-to-verify experiments test whether an LM-as-RM can substitute for rule-based verification. The main hard-to-verify setting samples 3,000 problems from OpenMathReasoning (Moshkov et al., 2025a).Source: §2.4.2 · p. 26

2.4.3 Evaluation Benchmarks and Metrics

  • Evaluation covers easy-to-verify math, hard-to-verify physics, and non-verifiable instruction-following benchmarks, using mean@16, mean@8, and evaluator-rated win rates, respectively.Math benchmarks are AIME24, AIME25, BRUMO25, HMMT24, and HMMT25; physics uses the Physics subset of PrincipiaBench, with o3 evaluating final answers; instruction following uses AlpacaEval 2.0 and ArenaHard 2.0, evaluated by GPT-4o and GPT-4.1, respectively.Source: §2.4.3 · p. 27

2.4.4 Baselines

  • RLLM is compared with RLHF scalar-reward models, RLVR verifiers, and prompted LMs-as-reward-models.The RLHF baselines include Skywork-Reward-V2-Llama-3.1-8B (Liu et al., 2025b), Athene-RM-8B, and AceMath-7B-RM (Liu et al., 2024); RLVR includes math-verify and general-verifier (Ma et al., 2025b). Prompted reward-model baselines include Llama-3.1-8B-Instruct, Qwen-1.7B, Qwen-32B, and GPT-OSS-120B.Source: §2.4.4 · p. 27

2.5 Results

RLLM with strong LM-as-RMs improves hard-to-verify reasoning performance over scalar-RM and verifier-based alternatives, while also extending to non-verifiable instruction following. The analyses show that gains depend on the reward model’s capability and training-data alignment, and can increase with additional evaluation compute.

2.5.1 Main Results

  • 8% absolute average improvement on competition math over Qwen3-1.7B Instruct comes from RLLM’s best setting: hard prompts and the strongest on-policy-trained LM-as-RM.The strongest RLLM model also outperforms GPT-OSS-120B as the RM and generalizes to hard-to-verify Physics; RLLM also outperforms RLHF and RLVR in non-thinking mode and with other model backbones.Source: Table 6 · p. 28, §2.5.1 · pp. 28–30
  • All RLLM models except the one trained with weaker Llama-3.1-8B-Instruct outperform RLHF, and self-rewarding RLLM improves over its base policy.The self-rewarding Qwen3-1.7B policy/RM setting improves from 35.20 →38.84; RLLM also scales with LM-as-RM size and capability.Source: Table 6 · p. 28, §2.5.1 · p. 28
  • 2% improvement in math results (41.16 →43.24) follows on-policy J1 training of Qwen3-32B-Verifier, while RLLM thinking-verifiers outperform RLVR with rule-based verifiers.RLLM also generalizes better to hard-to-verify Physics in this reference-based setting.Source: Table 7 · p. 28, §2.5.1 · p. 29
  • 39.63 →43.41: reference-free RLLM on hard-to-verify data outperforms reference-based RLVR on easy-to-verify data.RLVR performs better on easy than hard data (35.67 →39.63), while RLLM performs better on hard than easy data (41.18 →43.41).Source: Table 8 · p. 29, §2.5.1 · p. 29
  • RLLM matches or outperforms RLHF on non-verifiable instruction-following tasks and obtains better win rates on harder prompts.For these tasks, the study uses a pairwise LM-as-RM because responses lack a strict correctness criterion; Table 9 compares win rates and length-controlled win rates.Source: Table 9 · p. 30, §2.5.1 · p. 30

2.5.2 Analyses and Ablations

  • 10% improvement in judgment accuracy averaged across 8 seeds occurs with J1 training of Qwen3-32B, whereas training Qwen3-1.7B on its own responses brings little downstream benefit.Off-policy J1-trained reward models improve in-distribution judgment accuracy but do not transfer to downstream policy gains, unlike the on-policy model.Source: §2.5.2 · pp. 30–31, Table 10 · p. 31, Table 11 · p. 31
  • 16% better judgment accuracy for Qwen3-32B-Verifier over Qwen3-32B-RM with reference answers rises to 92% after J1 training.The J1 recipe improves LM-as-RM performance in both reference-based and reference-free settings.Source: §2.5.2 · p. 31
  • More LM-as-RM judgments improve win rates on hard prompts, while results for other prompt categories mostly saturate at pairwise comparisons.The tested reward assignments are pointwise, pairwise, and listwise, including a pairwise setup that uses a randomly selected pivot rollout.Source: §2.5.2 · p. 31

2.6 Conclusion

The conclusion presents RLLM—RL with RL-trained language models as reward models—as a unified post-training recipe, reporting gains over RLHF and RLVR, especially on hard-to-verify tasks. It also identifies on-policy training and the generator-verifier gap as important to successful RLLM training.

2.6 Conclusion

  • RLLM is presented as a single unified post-training recipe across easy-to-verify, hard-to-verify, and non-verifiable tasks.Source: §2.6 · p. 31
  • RLLM outperforms RLHF (with scalar RMs) and RLVR (with rule-based rewards), with particularly large gains on hard-to-verify tasks.Source: §2.6 · pp. 31–32
  • J1 training on Qwen3-32B yields a 10% absolute improvement, whereas J1 training on Qwen3-1.7B yields no further improvement; reference-based J1-Qwen3-32B-Verifier is 10% better than J1-Qwen3-32B-RM.Source: Figure 15 · p. 32, Figure 17 · p. 33
  • On-policy LM-as-RM training and the generator-verifier gap are identified as important components of successful RLLM training.Off-policy-trained RMs improve on their in-distribution validation data but do not yield downstream task improvements, potentially due to limited OOD generalization.Source: §2.6 · p. 32, Figure 16 · p. 33

3 Learning to Aggregate through Online Reinforcement Learning

This section presents ParaGator, an online reinforcement-learning approach that jointly optimizes diverse candidate-solution generation and final-answer aggregation, extending aggregation across multiple inference rounds. It evaluates the approach on competition-math and scientific-reasoning benchmarks, reporting improvements over offline self-aggregation, particularly on AIME and Principia, with results varying by backbone and benchmark.

3.1 Overview

  • Prior methods either train only the aggregator on off-policy candidates or risk diversity collapse; ParaGator instead trains the generator for diverse candidates and the aggregator for a correct final answer.The framework optimizes both stages online and end-to-end, aligning generated candidates with those encountered by the aggregator.Source: §3.1 · pp. 34–35, §3.3 · p. 35

3.3.1 Candidate Solution Generation and Pool Selection

  • Pass@k candidate optimization rewards a pool when at least one sample solves the problem, encouraging complementary solution modes rather than a single dominant trajectory.All generated candidates remain visible to the aggregator and contribute to the candidate-level RL signal; no heuristic filtering or reranking is applied.Source: §3.3.1 · p. 36

3.3.2 Self-Aggregation

  • The same model aggregates its current-policy candidate pool on-policy, with pass@1 reward for a correct final answer, and both stages are optimized jointly using RLVR.Training uses GRPO with a KL-regularized RLVR loss, so candidate generation and aggregation rollouts come from the same policy.Source: §3.3.2 · p. 36, Pass@1 Aggregation Optimization · p. 36, §3.3.3 · p. 36

3.3.4 Inference Details

  • At inference, the model repeatedly aggregates evolving candidate pools for T iterations, extending the single-round training scaffold to sequential test-time scaling.Each round’s aggregation rollouts form the next pool, preserving the generator–aggregator interface used in training.Source: §3.3.4 · p. 37

3.4.1 Self-aggregation improves frontier models

  • Parallel generation plus aggregation improves performance over standard generation and majority voting across four competition-math benchmarks on three strong open models.The models are Kimi-K2-Thinking, Qwen3-4B-Thinking-2507, and Qwen3-4B-Instruct-2507; the benchmarks are AIME, Brumo, HMMT, and IMO-Answerbench.Source: Figure 20 · p. 37, §3.4.1 · p. 37

3.4.2 The role of candidate diversity (pass@k) in self-aggregation

  • Repeated aggregation never exceeds initial pass@4, while higher initial sampling temperature improves pass@k and aggregation pass@1 despite similar initial pass@1.The temperature comparison fixes aggregation sampling temperature at 1.0.Source: §3.4.2 · pp. 37–38, Figure 21 · p. 38, Table 13 · p. 38

3.5.1 Competition Math

  • ParaGator achieves the highest reward in both competition-math training rounds and the best average post-aggregation performance, with strongest results on most benchmarks.Dr.GRPO favors initial-round reward while offline aggregation training favors aggregation reward; joint online multitask training partly mitigates this mismatch but underperforms ParaGator.Source: Figure 22 · p. 39, §3.5.1 · p. 39, Table 15 · p. 40

3.5.2 Scientific Reasoning

  • 33.77 PrincipiaBench average and 23.72 competition-math average are ParaGator’s best Qwen3-4B-Base post-aggregation results; for Instruct, it leads competition math at 61.38 while online multitask leads PrincipiaBench at 58.68.ParaGator also achieves the best Base AIME score (27.96) and the best Instruct AIME score (71.20); its Instruct PrincipiaBench average is 58.07.Source: §3.5.2 · p. 40, Table 16 · p. 41

3.6 Related Work

3.7 Conclusion

  • Traditional parallel decoding and self-aggregation are bottlenecked by off-policy generations and mode collapse; ParaGator aligns online candidate generation with downstream aggregation.Source: §3.7 · p. 42
  • ParaGator uses pass@k optimization to explore diverse, complementary candidates and pass@1 optimization to train the aggregator to synthesize them into a final answer.Source: §3.7 · p. 42
  • ParaGator consistently improves offline self-aggregation for Qwen3-4B-Base and Qwen3-4B-Instruct-2507 across competition math and scientific reasoning, with especially pronounced gains on AIME and Principia.Source: §3.7 · p. 42

4 Appendix

The appendix documents Principia’s contributors, implementation, prompts, and procedures for constructing and validating questions, as well as grouping equivalent predictions and determining whether a majority answer exists.

4.1 Contributions

  • The appendix lists contributors separately for Chapters 1–3 and notes that all authors have Meta AI affiliation except contributors marked † (ULCA) and †† (CMU).Source: §4.1 · p. 53

4.2 Hyperparameters for RL training

  • For verifier-based RL, policy and verifier models run on separate nodes; inference uses vLLM 0.8.4, policy training uses verl 0.4.0 with vLLM rollouts, and communication uses litellm 1.72.2.Source: §4.2 · p. 53

4.3.1 Prompts for Training and Evaluating Principia Models

  • The default Principia prompt requires responses to place step-by-step reasoning in <reason> tags and a concise final prediction in <answer> tags.It specifies that the tags appear only once and that no extra text should appear outside them.Source: §4.3.1 · p. 53

4.3.2 Prompts for Principia Collection Construction

  • The collection-construction prompts first elicit eight distinct, specific, actionable, nontrivial capabilities, then generate reasoning-intensive questions tied to a topic, capability, and required answer format.The capability prompt also asks for self-contained, relevant skills; the question prompt cautions against concept-checking questions.Source: §4.3.2 · pp. 53–54
  • Good questions should require deep, multi-step reasoning while yielding one decisive answer, with clear necessary conditions and no hints or image dependence.They should be grounded in the topic, capability, and requested answer format rather than testing concepts or routine calculations.Source: §4.3.2 · p. 58
  • To refine candidate questions, the prompts recommend solving them to assess difficulty and ambiguity, checking uniqueness and non-triviality, and revising or filtering before finalizing.They also advise balancing depth against excessive complexity and checking edge cases when pedagogically useful.Source: §4.3.2 · pp. 58–59
  • Harder variants should add non-obvious constraints, contextual interpretation, or edge cases while avoiding textbook imitation and remaining clear, fair, and reasoning-intensive.The prompt frames this as raising a straightforward question’s difficulty while preserving its topic, required capability, and answer format.Source: §4.3.2 · pp. 61–62
  • Question validation checks answer-type relevance, rejects statements without a requested answer, flags compositional questions asking for more than two answers, and detects included answers or overly prescriptive solution guidance.Source: §4.3.2 · p. 62
  • The validation prompt requires a decision enclosed in <decision> tags, using “This is problematic” when any issue exists and “Looks good to me” otherwise.Source: §4.3.2 · p. 62
  • The voting procedure checks equivalence between unique predictions, removes weak links with agreement below τ = 0.6 when refining transitivity, then groups predictions using disjoint-set union.For groups of at least three, it selects as representative the prediction whose length is closest to the median.Source: §4.3.2 · p. 63
  • A majority answer is returned only when the largest equivalence group meets the threshold of at least 5 of 8 votes; otherwise, the procedure returns no majority or no valid predictions.The procedure reports the winning group's count and refined equivalence map alongside a majority answer.Source: §4.3.2 · p. 63

5 Prompt Templates

The section illustrates prompts and training examples for language-model reward models, including scoring non-verifiable tasks, using reference answers as context, and generating policy-response examples. It also contrasts math problems with expression and integer reference answers.

5 Prompt Templates

  • For non-verifiable tasks, the pairwise LM-as-reward-model prompt asks the model to score fine-grained response quality from 0 to 10.The prompt differs from those for verifiable tasks by requesting a score rather than a correctness judgment.Source: Figure 25 · p. 64
  • The pointwise reference-based prompt adds a reference answer as context but still asks the reward model to evaluate the entire response.The reference answer is additional context, not a replacement for evaluating the response.Source: Figure 26 · p. 66
  • Synthetic correct and incorrect LM-as-reward-model examples pair a question and reference answer with a policy-sampled response checked against the reference using GPT-OSS-120B.The figures show examples in which the sampled response matches or does not match the reference answer.Source: Figure 27 · p. 67, Figure 28 · p. 68
  • The policy-training examples contrast a hard-to-verify math problem with an expression reference answer and an easy-to-verify problem with an integer reference answer.Source: Figure 29 · p. 69, Figure 30 · p. 69

Related papers

Loading 2603.18886v1…