PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math

Guijin Son, Donghun Yang, Hitesh Laxmichand Patel, Hyunwoo Ko, Amit Agarwal, Sunghee Ahn, Kyong-Ha Lee, Youngjae Yu

arXiv:2602.06291v1cs.CL

TL;DR

Verifying research-level mathematical solutions requires extensive review, while existing evaluators may struggle when correctness is unavailable. The paper proposes Consequence-Based Utility, which tests candidates through downstream performance on related verifiable questions and consistently outperforms oracle-free baselines.

  • Problem

    Evaluating research-level mathematical solutions remains dependent on extensive review and validation.

  • Method

    Consequence-Based Utility evaluates candidate solutions by measuring solver accuracy on related neighborhood questions whose correctness is verifiable.

  • Results

    CBU consistently outperforms oracle-free baselines; it achieves 71.38 AUC versus 69.48 for the corresponding LLM judge.

  • Takeaways & Limitations

    CBU maintains stronger correct-wrong separation than LLM judges as difficulty increases, supporting evaluation of research-level questions.

  • Takeaways & Limitations

    CBU requires additional effort to construct neighborhood questions, and automated variants depend on reliable generation without human oversight.

Abstract

from arXiv · show

Recent progress in reasoning models suggests that generating plausible attempts for research-level mathematics may be within reach, but verification remains a bottleneck, consuming scarce expert time. We hypothesize that a meaningful solution should contain enough method-level information that, when applied to a neighborhood of related questions, it should yield better downstream performance than incorrect solutions. Building on this idea, we propose \textbf{Consequence-Based Utility}, an oracle-free evaluator that scores each candidate by testing its value as an in-context exemplar in solving related yet verifiable questions. Our approach is evaluated on an original set of research-level math problems, each paired with one expert-written solution and nine LLM-generated solutions. Notably, Consequence-Based Utility consistently outperforms reward models, generative reward models, and LLM judges on ranking quality. Specifically, for GPT-OSS-120B, it improves Acc@1 from 67.2 to 76.3 and AUC from 71.4 to 79.6, with similarly large AUC gains on GPT-OSS-20B (69.0 to 79.2). Furthermore, compared to LLM-Judges, it also exhibits a larger solver-evaluator gap, maintaining a stronger correct-wrong separation even on instances where the underlying solver often fails to solve.

1. Introduction

The paper addresses the validation bottleneck in LLM-assisted research-level mathematics by introducing Consequence-Based Utility, an oracle-free evaluator based on downstream performance on related neighborhood questions. It reports stronger ranking and correct–wrong separation than existing oracle-free evaluators, and releases resources and guidance for applying the method.

  • Motivation: Rigorous validation remains comparatively underemphasized in LLM-assisted science, despite hypothesis generation and experimental planning receiving substantial attention.Validation currently depends heavily on costly human experts or unreliable and biased LLM judges.
  • Method: Consequence-Based Utility scores each candidate solution by the accuracy a solver achieves on closely related neighborhood questions when the candidate is used as an in-context exemplar.Higher induced accuracy indicates greater utility and does not require access to ground-truth answers.
  • Results: Compared with LLM judges, Consequence-Based Utility preserves stronger correct–wrong separation even when the underlying solver fails to solve the evaluated question.Error analysis attributes the gains to more reliable downranking of incorrect reasoning, unjustified compression, and unjustified interpretation.
  • Resources: The work releases EXPERTMATH, comprising expert-written research-level math problems with author solutions alongside LLM-generated problems.It also provides practical guidance on constructing neighborhood questions and choosing the number of rollouts needed for stable utility estimates.
  • Results: Consequence-Based Utility consistently outperforms LLM judges, reward models, and generative reward models in ranking candidate solutions.The method evaluates solutions through downstream neighborhood-question performance rather than direct oracle-based verification.

2. Preliminary and Related Works

Recent human–LLM mathematics collaborations motivate scalable oracle-free validation because experts must filter hallucinated or incomplete proof attempts. The paper formalizes candidate solutions and compares validation paradigms including reward models, generative reward models, and LLM judges, while excluding majority voting for research problems.

  • Motivation: Human–LLM collaborations on open research problems still require experts to reject hallucinations, repair missing steps, and make arguments checkable.These experiences motivate validation that does not require a domain expert for every candidate.
  • Validation framework: Oracle-free validation replaces an unavailable correctness oracle with a validator that outputs scores for candidate selection or ranking.The oracle returns 1 only for fully correct candidates, whereas the validator provides a usable ranking signal.
  • Related validators: The paper excludes majority voting because research-solution validity often cannot be reduced to a single discrete final answer.Majority voting is more naturally suited to contest-style or short-answer mathematics.
  • Related validators: Reward models approximate solution quality with a scalar score, commonly learned from pairwise preferences using a Bradley–Terry model.The study defaults to generative reward models because process reward models may be less stable than outcome-level scoring.
  • Related validators: LLM judges evaluate candidates by generating a critique and an integer rating on a 1–10 scale, which the paper uses directly as a numeric validator.The judge is modeled as Jψ(Q, C(i)) producing both natural-language feedback and a discrete score.

3. Consequence-Based Utility

Consequence-Based Utility evaluates difficult-to-verify solutions by the breadth and coherence of the verifiable consequences they enable. It hypothesizes that correct or near-correct candidates transfer method-level information to related questions, producing stronger downstream performance.

  • Motivation and hypothesis: The approach replaces direct scoring with “support by consequences,” assessing a candidate through the breadth and coherence of what it enables.This perspective is motivated by settings where reference answers are unavailable or costly, or solutions are long and subtle.
  • Motivation and hypothesis: Correct or near-correct candidates are hypothesized to contain method-level information that transfers to a neighborhood of related questions.Transferred information should yield consistently higher downstream performance than incorrect candidates.
  • Implementation in the LLM setting: For each candidate, the evaluator conditions the model on the original problem and candidate, then tests solutions on related questions with verifiable correctness.The neighborhood N(Q) contains previously solved or otherwise easier instances, and a verifier assigns binary correctness scores to each rollout.
  • Implementation in the LLM setting: Candidate usefulness is estimated by averaging scores from T independent rollouts for each related question.This converts downstream in-context performance into an empirical utility estimate.
  • In-context learnability as a correctness signal: The method treats in-context learnability as a correctness signal, extending prior uses of in-context performance for valuing examples and demonstrations.The section also relates context conditioning to training signals such as distillation from privileged traces.

4. Experiment Setup

The experiments use EXPERTMATH, a human-validated research-level mathematics dataset with author-created neighborhood variants and paired expert solutions. Candidates are ranked by Consequence-Based Utility against LLM judges, reward models, and generative reward models using solver performance on related questions and ranking-based metrics.

  • Dataset: The dataset starts from 70 faculty-authored questions spanning representation theory, algebraic combinatorics, geometry, and homotopy theory.
  • Dataset: Neighborhood questions preserve each original problem’s core idea while perturbing the statement and are designed to become straightforward once the original is understood.Authors reuse key lemmas or reductions and make variants slightly easier when feasible.
  • Dataset: The final dataset contains 192 original research-level problems—70 originals and 122 variants—with expert-written solutions and 630 human-validated LLM-generated solutions.Each problem includes four correct and five incorrect LLM-generated solutions, manually checked against the ground truth and for derivational validity.
  • Evaluation: Consequence-Based Utility is compared with LLM judges, reward models, and generative reward models using GPT-OSS-20B/120B and Qwen3-30B-A3B/235B-A22B on neighborhood questions conditioned on each target and candidate.Reward-model baselines include AceMath-RM-72B and Qwen2.5-Math-RM-72B.
  • Evaluation: Evaluation focuses on ranking and separating binary-labeled correct solutions above incorrect ones using Acc@1, Recall@5, and AUC among five higher-is-better metrics.Score calibration is not evaluated because the dataset provides binary rather than graded quality labels.

5. Main Results

Consequence-Based Utility (CBU) outperforms reward models and LLM judges across the evaluated results, including on difficult questions that the underlying solver cannot solve. Its scores also provide a more predictive and linearly separable signal of correctness than LLM-judge scores.

  • Overall comparison: CBU outperforms all baselines, with reward models weakest and LLM judges substantially stronger but consistently below same-backbone CBU.For Qwen3-235B-A22B, CBU achieves 71.38 AUC versus 69.48 for the corresponding LLM judge and 67.85 for Qwen3-235B-GenRM.
  • Overall comparison: +34.29 HumanWin points is GPT-OSS-120B’s largest reported CBU gain over LLM-judge scoring, increasing from 48.57 to 82.86.Across GPT-OSS-120B metrics, gains range from +6.13 on Recall@5 to +34.29 on HumanWin.
  • High-difficulty evaluation: CBU remains robust as question difficulty increases, while LLM-judge separability drops sharply in the high-difficulty regime.Both evaluators retain nonzero correct-wrong separation even when 1 −avg@64 ≈1, but CBU is better suited to the research-level difficulty tail.
  • Correctness prediction: 18.25 points is the largest gain from using CBU rather than LLM-judge scores for logistic-regression correctness prediction, observed on Qwen3-30B-A3B.Across all four backbones, the CBU score outperforms the LLM-judge score, with gains ranging from 6.02 points to 18.25 points.
  • Correctness prediction: 79.65 accuracy is achieved by combining CBU and LLM-judge scores on Qwen3-235B-A22B, improving over 72.79 from the LLM-judge score alone.Combining both scores further improves accuracy; for GPT-OSS-20B, the corresponding values are 73.09 to 73.90.

6. Additional Analysis

Consequence-Based Utility reduces overconfidence in wrong solutions and better preserves human-written correctness signals. Its advantage appears linked to penalizing polished but non-reconstructable reasoning, especially unjustified compression and invalid steps that LLM judges may overlook.

  • CBU reduces overconfidence on wrong solutions while better preserving human-written correctness signals.Figure 5 measures the probability that each validator assigns a solution an above-average score relative to its mean for that instance.
  • In 112 analyzed cases, CBU scored below the mean while the LLM judge scored above it.GPT-5-Pro supplied initial error labels, which were confirmed by a mathematics PhD student.
  • 71.4% of cases involved unjustified compression, and 68.8% involved incorrect reasoning.These were the two most concentrated failure modes among the 112 cases.
  • 31.3% of cases involved external references, consistent with LLM judges responding to authority-like cues.The analysis used four non-exclusive error types, including incorrect reasoning, unjustified compression, unjustified interpretation, and external references.
  • CBU may downrank wrong or underspecified solutions because they provide little transferable information for solving neighborhood variants.This is presented as a plausible explanation for CBU’s stronger validation performance.

7. A Practitioner’s Guide to Consequence-Based Utility

Consequence-Based Utility requires multiple rollouts, but matched-budget experiments show that small rollout counts capture most of its signal. In practice, its advantage depends on obtaining sufficiently difficult neighborhood questions: it outperforms LLM judges on RealMath variants but not on easier Daft-Math.

  • Inference budget: 64 rollouts are used for both CBU and LLM-Judge, with comparable average token consumption, preventing a systematic inference-budget advantage.CBU requires multiple downstream-solving rollouts, whereas an LLM judge can score in one pass.
  • Rollout efficiency: Error decreases monotonically with rollout count, and CBU converges at a similar or faster rate than LLM judges.CBU often converges faster for GPT-OSS-20B and Qwen3-30B-A3B, while GPT-OSS-120B is nearly identical.
  • Rollout efficiency: n ≥8 rollouts keeps mean normalized error below 0.05 across all tested backbones, indicating that few rollouts capture most of the signal.Estimates are compared with a 64-rollout reference using bootstrap resampling over sampled rollout subsets.
  • Neighborhood construction: CBU substantially outperforms LLM-judge scoring on both RealMath neighborhood variants for GPT-OSS-20B.RealMath automatically generates graduate-level problems by transforming theorems from mathematics papers, offering a practical alternative to faculty-written variants.
  • Neighborhood construction: On Daft-Math, LLM-judge scoring is stronger, with Acc@1 93.51 vs. 85.58 for CBU.The contrast is consistent with CBU performing better on higher-difficulty questions, whereas Daft-Math contains easier competition-level variants.

8. Discussions and Future Work · A. Additional Analyis

The paper proposes Consequence-Based Utility as an oracle-free method for estimating solution correctness from downstream performance without ground truth. Across research-level mathematics, it outperforms LLM judges and reward models, while its applicability may be more limited than theirs.

  • 8. Discussions and Future Work: CBU estimates solution correctness from downstream performance when ground truth is unavailable.It is presented as an oracle-free evaluation method.
  • 8. Discussions and Future Work: Across research-level mathematics, CBU consistently outperforms LLM judges and reward models.The comparison is reported across the paper’s evaluation setting.
  • 8. Discussions and Future Work: CBU remains effective with both expert-written and LLM-generated neighborhoods.Its effectiveness is reported across both neighborhood construction sources.
  • 8. Discussions and Future Work: Applicability is identified as a key limitation of CBU.The discussion contrasts this potential limitation with the broader applicability of LLM judges.
  • 8. Discussions and Future Work: LLM judges are described as broadly applicable despite exhibiting systematic biases.This comparison motivates concern about CBU’s applicability relative to LLM judges.
  • 8. Discussions and Future Work: The discussion frames CBU’s applicability as a tradeoff against the systematic biases of more broadly applicable LLM judges.The passage contrasts the proposed method with LLM judges rather than claiming that either approach is universally preferable.

A.1. Output Score Distribution of LLM-Judges · A.2. Prompt Sensitivity of LLM-Judges

LLM-judge backbones differ substantially in score distributions: GPT-OSS judges provide usable dynamic range, while Qwen judges show ceiling effects that reduce candidate discrimination. Prompt sensitivity was tested by rerunning GPT-OSS-20B and GPT-OSS-120B scoring with ProofGrader and UQ templates.

  • A.1. Output Score Distribution of LLM-Judges: Figure 7 compares density histograms of judge scores on a 1–10 scale across candidate solutions and backbones.The figure summarizes output-score distributions for the evaluated LLM judges.
  • A.1. Output Score Distribution of LLM-Judges: GPT-OSS-20B and GPT-OSS-120B spread scores across much of the 1–10 scale, providing usable dynamic range for ranking.Their scores assign nontrivial mass across the range.
  • A.1. Output Score Distribution of LLM-Judges: Qwen3-30B-A3B and especially Qwen3-235B-A22B concentrate scores near 10, producing a strong ceiling effect.The concentration indicates saturation relative to the broader GPT-OSS distributions.
  • A.1. Output Score Distribution of LLM-Judges: The Qwen judges’ saturation suggests overconfident scoring and reduces score-based discrimination among candidates.This follows from scores being heavily concentrated near the top of the scale.
  • A.2. Prompt Sensitivity of LLM-Judges: The experimental LLM-judge prompt was adapted from prior evaluation prompts used by Zhang et al. (2025b) and Phan et al. (2025).The study examined whether findings depended on this specific prompt.
  • A.2. Prompt Sensitivity of LLM-Judges: The study reran GPT-OSS-20B and GPT-OSS-120B scoring with two alternative templates: the 0–7 ProofGrader prompt and the binary-correctness UQ prompt.ProofGrader follows Ma et al. (2025), while UQ comes from Nie et al. (2025).

A.3. Token Count: CBU VS. LLM-Judges

CBU has inference costs comparable to LLM-judges across backbones, while the comparison also evaluates sampling diversity through pairwise cosine similarity of embedded generations.

  • Inference cost: CBU’s average token usage per generation stays within ±15% of the LLM-judge across backbones.Reported differences include +1.3% on Qwen3-235B, +15.0% on Qwen3-30B, +9.5% on GPT-OSS-120B, and -7.4% on GPT-OSS-20B.
  • Inference cost: +1.3% is CBU’s token-usage difference versus the judge on Qwen3-235B.The passage reports this as one example of comparable average token usage across backbones.
  • Inference cost: +15.0% is CBU’s token-usage difference versus the judge on Qwen3-30B.This is the largest positive difference among the listed backbone examples.
  • Sampling diversity: Sampling diversity is quantified by embedding repeated generations with Gemini Embedding 001 and computing mean pairwise cosine similarity.Table 5 reports token counts and pairwise cosine similarity statistics as mean ± std [min, max].

B. Evaluation Metrics

This section formally defines the evaluation metrics and introduces the notation used for candidate rankings, correctness labels, scorer outputs, and the human-written solution.

  • B. Evaluation Metrics: Table 6 provides formal definitions of the evaluation metrics used in the study.The definitions rely on ranked candidate indices, correctness labels, candidate sets, scorer outputs, and the human-written solution.
  • B. Evaluation Metrics: π_k denotes the index of the k-th ranked candidate, whose correctness label is y(π_k) ∈ {0, 1}.The label indicates whether the ranked candidate is correct.
  • B. Evaluation Metrics: C and W denote the sets of correct and wrong candidates, while s(·) denotes the scorer and H the human-written solution.These symbols define the candidate groups and scoring reference used in the metric definitions.

D. Details on EXPERTMATH . … 4. External References

The appendix documents EXPERTMATH’s composition, representative expert-written and RealMath problems, the prompts used for evaluation and generation, and strict procedures for error analysis and external references. It also specifies output constraints that emphasize conservative, evidence-based diagnosis rather than external validation or direct judgment of final answers.

  • D. Details on EXPERTMATH .: EXPERTMATH contains 192 expert-written mathematics problems and 425 LLM-generated problems derived from RealMath.The generated problems are planned for release on Hugging Face, while the expert-written set remains under embargo until after July 2026.
  • D. Details on EXPERTMATH .: 35 is obtained for the Grassmannian trace by counting symmetric plane partitions or invariant 3 × 2 semistandard Young tableaux.The second method uses the reflected complement action on tableau columns.
  • D. Details on EXPERTMATH .: The appendix also presents expert-written questions on PGL2 Deligne–Lusztig theory, Auslander–Reiten translates, and the Varchenko–Gelfand ideal.For the PGL2 case, the solution identifies r + 1 double cosets; for the Varchenko–Gelfand problem, degree-two generators suffice.
  • E. Prompts.: The prompts include Consequence-Based Utility evaluation, default and specialized LLM judges, RealMath practice-problem generation, and error analysis.The utility prompt asks the model to solve a question using a supplied question-solution set, whereas the judge prompts require independent reasoning and comparison.
  • E. Prompts.: RealMath generation requests five standalone, difficult practice problems targeting distinct sub-skills and collectively covering the original problem’s key skills.The prompt forbids trivial decompositions or plug-in variants and requires integer answers bigger than 1000.
  • 2. Unjustified Compression: Error analysis is conservative: it tags only clear, high-confidence reasoning failures and permits zero, one, or multiple categories.Each tagged category requires 1–3 exact solution quotes with explanations of the claim, the problem, and what would fix it.
Loading 2602.06291v1…