PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning over mathematical objects: on-policy reward modeling and test time aggregation

Pranjal Aggarwal, Marjan Ghazvininejad, Seungone Kim, Ilia Kulikov, Jack Lanchantin, Xian Li, Tianjian Li, Bo Liu, Graham Neubig, Anaelia Ovalle, Swarnadeep Saha, Sainbayar Sukhbaatar, Sean Welleck, Jason Weston, Chenxi Whitehouse, Adina Williams, Jing Xu, Ping Yu, Weizhe Yuan, Jingyu Zhang, Wenting Zhao

arXiv:2603.18886v1cs.AIcs.CL

TL;DR

Existing evaluations often use numerical or multiple-choice answers, leaving precise mathematical-object derivation under-tested. The paper introduces the Principia suite and on-policy verifier and aggregation recipes. These methods improve mathematical-object reasoning and also transfer to numerical and MCQA tasks.

  • Problem

    Existing benchmarks and post-training datasets largely use numerical values or multiple-choice options, while graduate-level STEM problems requiring complex mathematical-object derivation remain scarce.

  • Method

    The paper builds PrincipiaBench, Principia Collection, and Principia VerifyBench, then trains language models with strong model-based judges and on-policy aggregation.

  • Results

    RL training improves PrincipiaBench performance by 7.22–18.35% on average and also improves AIME 2024 and GPQA-Diamond performance.

  • Takeaways & Limitations

    Training on mathematical-object data improves reasoning across mathematical-object, numerical, and multiple-choice answer formats.

  • Takeaways & Limitations

    Some Physics benchmark instances are not self-contained because they require figures that are not provided.

Abstract

from arXiv · show

The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.

Introduction

The paper targets a gap in reasoning evaluations that favor numerical and multiple-choice answers over formally deriving mathematical objects. It introduces data, benchmarks, and on-policy methods to improve this capability and extend test-time aggregation.

  • Deriving mathematical objects such as matrices and piecewise functions is foundational for articulating scientific phenomena and solving graduate-level STEM problems.
  • Existing evaluations largely use numerical values or multiple-choice options for convenient automated scoring, risking overstated reasoning capabilities.
  • PrincipiaBench and the Principia Collection target reasoning that requires complex mathematical-object outputs rather than restricted answer formats.
  • RLLM trains a policy model using comparative judgments from an on-policy language-model reward model instead of relying only on scalar rewards or rule-based verification.
  • On-policy training of the LM-as-RM outperforms prompted and off-policy alternatives across mathematical, physics, and instruction-following tasks.
  • Parallel aggregation is extended through on-policy training to address redundant or insufficiently diverse parallel solutions during test-time reasoning.

1 Principia: Training Language Models to Reason over Mathematical Objects

Principia training improves reasoning on mathematical-object derivation and also transfers to numerical and multiple-choice benchmarks. Strong model-based verification and excluding MCQA data produce the best reported configuration.

  • Principia Collection training improves PrincipiaBench performance for Qwen3-4B-Base, Qwen2.5-7B-Base, and OctoThinker-8B-Long-Base.Average gains are +18.23%, +10.23%, and +15.16%, respectively.
  • Training also improves performance on AIME-2025 and SuperGPQA, indicating reasoning gains across mathematical answer formats.
  • Using GPT-OSS-120B as a model-based verifier and excluding MCQA data yields the best performance.

1.1 Overview

The Principia suite supplies a mathematical-object benchmark, a large training collection, and verifier evaluation, with model-based verification supporting substantial cross-format improvements.

  • PrincipiaBench contains 2,558 carefully filtered problems requiring complex mathematical-object answers from four source collections.
  • The Principia Collection contains 248K graduate-level STEM problem statements requiring equations, inequalities, intervals, sets, matrices, or piecewise functions.
  • Principia VerifyBench evaluates whether outputs with different expressions are mathematically equivalent using human-labeled judgments.
  • o3 judgments were correct in 94.05% of 168 cases where math-verify and o3 disagreed.
  • RL training across four language models improved average PrincipiaBench performance by 7.22–18.35% over each base model.
  • The paper contributes PrincipiaBench, Principia Collection, and Principia VerifyBench for evaluating and improving mathematical-object reasoning.

1.2 Motivation: Need for a benchmark and training data for mathematical objects

The paper argues that current benchmarks and post-training datasets inadequately test de novo mathematical-object derivation. Removing answer options exposes this weakness, motivating Principia’s benchmark and training data.

  • The field lacks benchmarks and training data requiring complex mathematical-object derivation beyond numerical values and multiple-choice options.
  • Options can provide anchors for backward chaining, allowing models to select answers without fully deriving the underlying object.
  • Frontier LM performance on mathematical-object questions in SuperGPQA drops 10–20% when answer options are removed.For o3, performance changes from 69.10 to 62.90; for Qwen3-235B, from 69.33 to 55.58.
  • Widely used RL datasets mostly contain short numerical, multiple-choice, or text-span answers rather than complex mathematical objects.
  • Principia Collection addresses this gap with complex-object answers and graduate-level STEM problem statements containing multiple conditions.

1.3 The Principia Suite

The Principia suite targets precise derivation of mathematical objects by combining a benchmark, verifier evaluation, and a large training collection. It filters problems for answer-type, single-question, self-contained requirements and addresses equivalence-checking difficulties in non-canonical mathematical representations.

  • PrincipiaBench: PrincipiaBench assesses LM ability to derive mathematical objects directly, without relying on multiple-choice formats.Its filtering excludes numerical, proof, text-span, boolean, and multiple-choice answers.
  • PrincipiaBench: Problems are filtered to require one question and self-contained statements, excluding multi-part tasks and cases needing missing figures or definitions.The filtering rationale is to evaluate object derivation rather than partial-credit assignment, clarification requests, or reasoning from incomplete information.
  • Verifier evaluation: Complex mathematical objects create verification challenges because equivalent answers can differ in symbolic order, notation, LaTeX commands, variables, and surrounding text.The paper reports failures involving Sympy’s recognition of reordered terms, alternate notation, and text spans.
  • Verifier evaluation: Principia VerifyBench uses human annotations to evaluate answer-equivalence judgments for verifier reliability.The annotation process retained 168 instances after requiring agreement between two annotators: 75 equivalent and 93 not equivalent.
  • Principia Collection: The Principia Collection contains 248,748 graduate-level problem statements and corresponding answers focused exclusively on mathematical objects.Examples pair subject entities with problem statements and answers, while the collection construction also uses a model-based verifier and majority voting over equivalent forms.

1.4 Experimental Results

The experiments evaluate Principia-trained models on mathematical-object, MCQA, and numerical tasks using held-out validation and test benchmarks. Training on the Principia Collection consistently improves mathematical-object derivation and transfers to other answer formats, while frontier models remain challenged by PrincipiaBench.

  • Training setup: The study trains four base LMs with GRPO on the Principia Collection across 4B–8B model scales and multiple model families.The training setup includes Qwen-based and Llama-based models, zero-RL and continual-training configurations.
  • Evaluation setup: PrincipiaBench contains 2,558 problems requiring complex mathematical-object answers, while evaluation also covers MCQA and numerical benchmarks.The test groups include PrincipiaBench, GPQA-Diamond and SuperGPQA, and AIME-2024 and AIME-2025.
  • Main results: +10.12%, +15.30%, +18.35%, and +7.22% are the PrincipiaBench improvements for Qwen2.5-7B-Base, OctoThinker-8B-Long-Base, Qwen3-4B-Base, and Qwen3-4B, respectively.The corresponding score changes are 12.75 →22.87, 3.75 →19.05, 11.31 →29.66, and 41.23 →48.45.
  • Cross-format transfer: Training on mathematical-object data improves MCQA and numerical performance, including results on SuperGPQA, GPQA-Diamond, and AIME.Principia-4B surpasses Polaris-4B and Qwen3-4B (thinking) on the cited benchmarks, while several Principia models achieve the best average performances.
  • Benchmark difficulty: o3 scores 62.90 and Qwen3-235B scores 55.58 on PrincipiaBench, yet the passage reports that most LMs perform substantially lower there than on common reasoning benchmarks.The comparison includes SuperGPQA, GPQA-Diamond, and AIME.
  • Data alignment: Training on DeepScaleR or WebInstruct-Verified is insufficient for precisely deriving complex mathematical objects, whereas Principia training produces more consistent improvements.The passage also reports a more stable response-length increase for Principia training than for the baselines.

1.5 Analysis

The analysis examines verifier quality, answer-format mixtures, and training strategies. Strong general-purpose model verifiers are especially valuable for complex mathematical objects, and weight merging outperforms joint training when combining formats.

  • Verifier analysis: o3 and GPT-OSS-120B achieve 94.05% and 95.24% agreement with Principia VerifyBench ground-truth labels, respectively.The results support using strong general-purpose LMs as verifiers for evaluation or reward provision.
  • Verifier analysis: General-purpose Qwen3-4B reaches 92.26% verifier agreement, while specially fine-tuned verifiers remain below 90%.The passage attributes the lower fine-tuned-verifier agreement primarily to low recall.
  • Verifier ablation: On Principia Collection training, math-verify raises performance from 13.31 to 16.30, general-verifier lowers it to 9.54, and GPT-OSS-120B raises it to 31.80.Verifier choice matters much less on DeepScaleR, whose total averages range from 28.67 to 30.45.
  • Format mixtures: The answer-format mixture experiment controls problem diversity by using shared subject entities while varying mathematical-object, numerical, and MCQA answer formats.The generated numerical and MCQA datasets contain 300K and 350K questions, respectively.
  • Format mixtures: The MCQA-only model reaches 25.04 total average performance, with 24.96 on PrincipiaBench, 17.71 on AIME-24, and 19.79 on AIME-25.The passage characterizes its performance as weak across nearly all benchmarks.
  • Format mixtures: The math-only model achieves 45.53 on SuperGPQA, 43.56 on GPQA-Diamond, 22.71 on AIME-24, and 23.96 on AIME-25.These results are reported as transfer from mathematical-object supervision to MCQA and numerical benchmarks.
  • Format mixtures: Weight merging beats joint training with math-object and numerical data, achieving 33.23 versus 29.92 total average performance.The same pattern holds when all three formats are combined: 31.95 for weight merging versus 28.39 for joint training.

1.6 Related Work

Prior reasoning evaluations commonly use numerical or multiple-choice answers, leaving mathematical-object manipulation underexamined. The paper positions Principia as targeted data and benchmarking for this gap and reports strong performance across both PrincipiaBench and other reasoning benchmarks.

  • Existing evaluations: Reasoning benchmarks commonly use grade-school or Olympiad problems with numerical answers and broader disciplinary tasks with multiple-choice answers.The cited examples span mathematics, physics, chemistry, and biology.
  • Motivation: Mathematical-object problems require producing or reasoning over structured expressions and are presented as a more challenging test bed than multiple-choice or numerical exams.The passage connects this capability to experiments and scientific writing.
  • Motivation: The paper identifies a gap between ambitions for reasoning-intensive scientific applications and standardized tests that rely on simplified answer formats.The stated applications include autonomous scientific discovery and assisting scientists’ workflows.
  • Synthetic reasoning data: The paper reports that Principia Collection training yields strong performance on both PrincipiaBench and other reasoning benchmarks.This is stated as the paper’s result following its discussion of synthetic reasoning data.

1.7 Conclusion

The paper introduces the Principia suite to address mathematical-object derivation, then reports improvements on both object-focused and conventional reasoning tasks. It also argues that model-based verification is necessary for accurate rewards and presents the artifacts as resources for future research.

  • Contributions: The Principia suite comprises PrincipiaBench, the Principia Collection, and Principia VerifyBench for benchmarking, training, and verifier meta-evaluation.The three artifacts target mathematical-object derivation and verifier assessment in RL training and benchmarking.
  • Findings: Principia-trained LMs improve substantially on PrincipiaBench and also improve on MCQA and numerical tasks.The conclusion describes this as enhanced reasoning across answer formats.
  • Findings: The paper reports that model-based verifiers are necessary for providing accurate rewards when training on the Principia Collection.This conclusion follows the paper’s verifier experiments.
  • Implications: PrincipiaBench is proposed as a benchmark for tracking progress in RL algorithms and post-training datasets, while the Collection supports future synthetic-data work.Principia VerifyBench is presented as a way to train and evaluate model-based verifiers.

2 RLLM: Unified Post-Training via On-Policy-Trained Language Model as a Reward Model

RLLM uses an on-policy-trained language model as a reasoning reward model, addressing limitations of scalar RLHF rewards and rule-based RLVR verifiers. Experiments report gains across verifiable and non-verifiable tasks, with on-policy training and a strong generator–verifier gap identified as important.

  • Comparison with RLHF and RLVR: RLHF uses scalar reward models, while RLVR uses verifiers that require reference answers and are limited by task verifiability.RLHF reward models provide single scores; RLVR depends on rule-based or model-based verification of outcomes.
  • RLLM overview: RLLM trains a language model as a thinking reward model and uses its judgments to optimize a policy.The framework is presented as a unified alternative to scalar reward models and rule-based verifiers.
  • Empirical comparison: RLLM outperforms RLHF and RLVR across easy-to-verify and hard-to-verify math and physics benchmarks, as well as non-verifiable instruction-following tasks.The comparison includes verifiable and non-verifiable settings across the PrincipiaBench evaluation suite.
  • Reward-model training: On-policy training of the LM-as-RM outperforms prompted and off-policy alternatives, including a larger GPT-OSS-120B model.The result supports training the reward model on outputs sampled from the policy it will evaluate.

2.3 RLLM: Reinforcement Learning with Language Models as Reward Models

RLLM uses generative, reasoning-based rewards from language models across reference-free, reference-based, pointwise, pairwise, and listwise settings. Its training samples policy outputs for LM-as-RM learning, then uses the resulting scores to optimize the policy.

  • Framework: RLLM is a special case of RLAIF that uses a language model as a thinking reward model across diverse task types.The LM-as-RM produces generative rewards rather than relying on scalar rewards or deterministic verification functions.
  • Reward formulations: Reference-free LM-as-RM evaluation scores a response independently, while reference-based evaluation grounds judgment in a provided reference answer.Reference-based evaluation is intended for easy-to-verify and hard-to-verify problems requiring substantial reasoning.
  • Reward formulations: Pointwise, pairwise, and listwise variants produce scores for one response, comparative response pairs, or sets of K responses.Comparative variants jointly reason about multiple candidate responses before assigning scores.
  • LM-as-RM training: The LM-as-RM is trained on policy-sampled responses labeled by a stronger teacher, then optimized with RL to predict teacher scores.Mathematical responses typically receive binary correctness labels, while non-verifiable tasks use continuous quality scores.
  • Policy training: Policy training samples rollouts and uses only the LM-as-RM’s scores, not its reasoning traces, to compute optimization rewards.The reward computation depends on the selected pointwise, pairwise, or listwise configuration and reference availability.
  • Pairwise variants: Pivot pairwise evaluation reduces exhaustive comparison cost from O(n^2) to O(n) LM-as-RM evaluations while using a shared reference rollout.Both ordered comparisons are evaluated to reduce positional bias.

2.4 Experimental Setup

The experiments train Qwen3-1.7B policies with larger J1-trained Qwen3-32B reward models and evaluate RLLM across verifiable reasoning and non-verifiable instruction-following tasks. The setup varies references, verification difficulty, model combinations, and baselines.

  • Training recipe: J1 reward models are trained on-policy using 16 responses per prompt generated by the Qwen3-1.7B policy.The training uses hard-to-verify prompts from OpenMathReasoning and the Principia Collection.
  • Training recipe: The primary experiments optimize a Qwen3-1.7B policy with J1-Qwen3-32B reward models, creating a substantial generator–verifier gap.The paper identifies this gap as essential for strong downstream policy improvements.
  • Training settings: The study compares reference-free and reference-based training on hard-to-verify samples, plus easy-to-verify samples with or without references.Reference-free hard-to-verify training uses J1-Qwen3-32B-RM, while reference-based training uses J1-Qwen3-32B-Verifier.
  • Evaluation: Evaluation covers five competition-math benchmarks, Physics subset problems requiring mathematical-object reasoning, and AlpacaEval and ArenaHard instruction-following tasks.Math uses mean@16, Physics uses mean@8, and instruction-following uses evaluator-based win rates.
  • Baselines: Baselines include scalar-reward RLHF, rule-based or model-based RLVR, and prompted language-model reward models spanning multiple families and sizes.Prompted LM-as-RM comparisons include Llama-3.1-8B-Instruct, Qwen models, and GPT-OSS-120B.

2.5 Results

RLLM consistently improves post-training across verifiable and non-verifiable reasoning settings, with the strongest results from capable, on-policy-trained LM-as-RMs. Its benefits extend to hard-to-verify and out-of-domain tasks, while off-policy or weak self-reward models show limited gains.

  • 2.5.1 Main Results: J1-Qwen3-32B-RM improves AIME24 by 12% over the Qwen3-1.7B Instruct model in the reference-free setting.Scaling the LM-as-RM improves RLLM results across the evaluated benchmarks.
  • 2.5.1 Main Results: All RLLM variants outperform all RLVR variants in the reference-based setting.Reference-based thinking-verifiers outperform rule-based RLVR alternatives, including math-verify, which can fail on complex mathematical objects.
  • 2.5.1 Main Results: All RLLM models except the weaker Llama-3.1-8B-Instruct variant outperform RLHF models, with results improving from 36.12 to 43.41.This comparison shows that strong LM-as-RMs can outperform scalar reward models during online RL, even when scalar RMs score highly on static benchmarks.
  • 2.5.2 Analyses and Ablations: On-policy J1 training improves a Qwen3-32B verifier by 2% over prompted Qwen3-32B, whereas off-policy-trained RMs do not transfer to downstream policy improvements.The generator-verifier gap matters: J1 training provides little benefit for a weak Qwen3-1.7B self-evaluator, but strengthens the 32B evaluator.
  • 2.5.1 Main Results: 39.63 →43.41: reference-free RLLM trained on hard-to-verify data outperforms reference-based RLVR trained on easy-to-verify data.The comparison supports using strong LM-as-RMs to extend reinforcement-learning post-training beyond easily verifiable domains.
  • 2.5.1 Main Results: 41.18 →43.41: RLLM training on hard-to-verify data yields better downstream performance than training on easy-to-verify data.The best-performing configuration uses hard prompts, the strongest on-policy-trained LM-as-RM, and either reference-free or reference-based rewards.
  • 2.5.2 Analyses and Ablations: Reference-based judgments are more accurate than reference-free judgments, and Qwen3-32B-Verifier reaches 92% after J1 training.Reference answers improve judgment accuracy, while on-policy training further improves the reference-based verifier.

2.6 Conclusion

The paper presents RLLM as a unified post-training recipe spanning easy-to-verify, hard-to-verify, and non-verifiable tasks. Its analyses identify strong on-policy training and an adequate generator-verifier gap as important components, while test-time aggregation remains a separate scaling direction.

  • 2.6 Conclusion: RLLM is presented as a unified post-training recipe across easy-to-verify, hard-to-verify, and non-verifiable tasks.The conclusion positions RLLM as an alternative to both scalar and rule-based reward approaches.
  • 2.6 Conclusion: Strong LM-as-RMs produce particularly large gains when training on hard-to-verify tasks.The paper also identifies on-policy LM-as-RM training and the generator-verifier gap as important components of successful RLLM training.
  • 2.6 Conclusion: On-policy J1 training improves judgment accuracy by 10% for Qwen3-32B but provides almost no improvement for Qwen3-1.7B.The generator-verifier analysis links stronger verifier training gains to downstream policy improvements, unlike weak self-evaluation.
  • 2.6 Conclusion: Off-policy-trained reward models improve in-distribution validation rewards but do not produce downstream task improvements.The result suggests that reward-model validation gains alone do not ensure useful out-of-distribution transfer.
  • 2.6 Conclusion: Reference-based LM-as-RMs are more accurate than reference-free ones, with J1-Qwen3-32B-Verifier 10% better than J1-Qwen3-32B-RM.The comparison isolates the benefit of supplying reference answers to the verifier.

3 Learning to Aggregate through Online Reinforcement Learning

ParaGator jointly trains parallel candidate generation and self-aggregation online, using distinct objectives to align diverse candidates with downstream synthesis. Across math and scientific reasoning benchmarks, this approach improves aggregation performance, especially on complex tasks.

  • Method analysis: Self-aggregation is upper bounded by initial pass@k performance, and incorrect candidate solutions cannot be synthesized into correct solutions.This motivates improving the quality of the initial candidate pool rather than optimizing aggregation alone.
  • Online training: ParaGator trains one model end-to-end for diverse candidate generation and final-answer aggregation using online reinforcement learning.Candidate generation and aggregation are optimized jointly rather than as separate pipeline stages.
  • On-policy aggregation: On-policy aggregation trains the model on candidate pools sampled from its current generator, eliminating the off-policy mismatch targeted by prior methods.The alignment also teaches the generator to produce candidates suited for downstream aggregation.
  • Reward analysis: Across both backbones, pass@k training achieves the strongest reward in both initial and aggregation rounds, avoiding the specialization trade-offs of Dr.GRPO and offline aggregation training.Online multitask training partially mitigates the mismatch by optimizing both stages jointly.
  • Results: ParaGator reaches 33.77 on PrincipiaBench and 23.72 on competition math for Qwen3-4B-Base, while Qwen3-4B-Instruct-2507 reaches 61.38 on competition math.The reported gains extend to AIME, where the corresponding scores are 27.96 and 71.20.
  • Distinct objectives: The generator uses pass@k optimization to encourage diverse, complementary solutions, while the aggregator uses pass@1 optimization to produce one correct answer.These objectives assign different roles to the two stages while preserving end-to-end training.

4 Appendix

The appendix documents contributors, implementation details, prompting formats, and procedures for constructing mathematical problem capabilities. It also specifies structured generation requirements and example domain-specific reasoning skills.

  • Contributors: The appendix lists separate contributors for Chapters 1, 2, and 3, with most authors affiliated with Meta AI.ULCA and CMU affiliations are marked separately.
  • Implementation: Policy and verifier models run on separate nodes using vLLM, verl, and litellm for inference, training rollouts, and inter-node communication.The verifier determines answer equivalence while verl trains the policy model.
  • Prompting: The default Principia training prompt requires step-by-step reasoning inside <reason> tags and a concise final prediction inside <answer> tags.Each tag is required to appear once, with no text outside the reasoning and answer regions.
  • Capability generation: Problem-statement construction prompts request eight specific, actionable, relevant, non-trivial, self-contained, and non-overlapping capabilities.The format requires numbered items enclosed between single <start> and <end> markers.
  • Domain examples: The appendix supplies mathematical capability examples spanning multiplicative number theory, algebraic-geometry coverings, recurrences, representations, and curves over finite or local fields.Examples include Chebotarev density, monodromy, characteristic polynomials, symmetric-group characters, and Hasse-Weil bounds.

5 Prompt Templates

The prompt-template appendix specifies judge formats for verifiable and non-verifiable tasks and provides examples of correct, incorrect, and difficult-to-verify mathematical responses. It also includes comparative evaluation traces and scientific problem examples.

  • LM-as-RM templates: Pairwise judges compare two responses impartially, while pointwise reference-based judges evaluate each response against a reference answer.The pairwise template uses qualitative comparison, whereas pointwise judging assigns correctness scores from 1 or 0.
  • Non-verifiable judging: For non-verifiable tasks, pairwise LM-as-RM prompts assign scores from 0 to 10 to capture fine-grained response quality.This differs from verifiable-task evaluation, which can use correctness-oriented judgments.
  • Comparative evaluations: The appendix reports that RLLM outperforms RLHF and RLVR in Qwen3-1.7B non-thinking evaluations and matches or exceeds RLHF on instruction-following win rates.For ArenaHard 2.0, RLLM obtains the best win rates on hard prompts.
  • Reference-based judging: Reference-based pointwise judging provides the reference answer as additional context while still evaluating the entire response.The response is assessed independently rather than only by matching a specific final span.
  • Training examples: Synthetic LM-as-RM examples are created by sampling a policy response and checking whether it matches the reference answer with GPT-OSS-120B.The appendix shows both correct and incorrect training examples, plus easy- and hard-to-verify mathematical references.
  • Judge traces: A qualitative comparison shows J1-Qwen3-32B correctly identifies an incorrect response and states the reference-matching answer, unlike the weaker J1-Qwen3-1.7B trace.The example illustrates the role of stronger reasoning traces in LM-as-RM judgments.
Loading 2603.18886v1…