Source-linked AI summary
Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification
Fateme Mazdarani, Carlos Toxtli
TL;DR
Most mathematical evaluations emphasize final answers, leaving unclear whether LLMs can verify valid non-canonical traces. The paper introduces a controlled linear-equation benchmark for final-answer, step-level, and first-error evaluation, finding that base models perform well on canonical traces but degrade sharply on perturbed ones, especially for localization.
Problem
Most mathematical benchmarks emphasize final-answer accuracy, providing limited evidence about whether evaluators accept valid non-canonical solution traces.
Method
The paper generates controlled linear-equation traces with exact labels and evaluates final answers, trace correctness, and first-error localization across canonical and perturbed solutions.
Results
Base models perform well on canonical traces but degrade sharply under perturbation, with valid perturbed T1 false-rejection rates of 75.56% for GPT-OSS 20B, 83.89% for Qwen3-14B, and 85.28% for Phi-4-Reasoning.
Takeaways & Limitations
Evaluator robustness is distinct from final-answer accuracy, and adaptation can reduce the gap although benefits are model dependent and majority-vote test-time compute can amplify systematic mistakes.
Takeaways & Limitations
The study is limited to single-variable linear equations with unique rational solutions and does not establish that the behavior generalizes to richer mathematical or scientific tasks.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final-answer accuracy. This can obscure whether a model can verify a non-canonical but valid solution trace. We introduce a controlled linear-equation benchmark for evaluating LLMs in the evaluator role. Each instance asks the model to judge final-answer correctness, step-level trace correctness, and the first incorrect step. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Across GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, base models perform well on canonical traces but degrade substantially on perturbed traces, especially for error localization. On valid perturbed traces, base-model false-rejection rates reach 75.6-85.3%, showing strong sensitivity to canonical solution form. Supervised fine-tuning, distillation, and test-time compute improve robustness in some settings, but gains are model dependent and can trade off against canonical performance. The results show that reliable process-level verification remains challenging, and evaluator robustness should be measured separately from solver accuracy, even in a simple algebraic domain with exact ground truth.
I. INTRODUCTION
The paper frames evaluator robustness as the ability to distinguish valid non-canonical solution paths from genuine errors, not merely check final answers. It introduces a controlled benchmark and three-part protocol to measure this capability in linear equations.
- Correct final answers can still contain invalid steps, while valid solutions may follow non-standard paths that should not be penalized.
- Template sensitivity describes reliance on familiar solution forms rather than step-level validity.
- The benchmark uses fixed linear equations and perturbed traces with exact labels for correct, incorrect, and tricky solutions.
- The protocol separately evaluates final-answer correctness, step-level correctness, and first-error localization.
- The study evaluates GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, then tests supervised fine-tuning, distillation, and test-time compute.
II. RELATED WORK
Prior mathematical LLM benchmarks primarily measure solving performance, while verifier research studies process scoring and intermediate reasoning. This work focuses specifically on grading solution traces under controlled variability.
- Most mathematical benchmarks assess LLMs as problem solvers rather than evaluators.
- Existing verifier research includes process-based and outcome reward models, step-level verifier training, and formally verified process labels.
- Prior perturbation studies mainly alter problem statements, context, symbolic forms, or values in the solver setting.
- This benchmark holds the original equation fixed while perturbing the supplied solution trace to isolate evaluator robustness.
C. Evaluating Worked Solutions
The benchmark generates worked solutions for fixed linear equations across three trace types, separating final-answer checking from process checking and error localization.
- The generated equations have the form ax + b = cx + d with unique rational solutions.
- The taxonomy combines canonical or perturbed traces with three evaluation types for the same equation.
- T1 contains mathematically equivalent steps and a correct final answer, measuring false rejection of valid reasoning.
- T2 injects one arithmetic or algebraic error and propagates it to an incorrect final answer.
- T3 contains an invalid intermediate step but restores the correct final answer, separating answer checking from process checking.
B. Reproducibility Details
The benchmark uses exact, checker-based semantics to validate algebraic traces and distinguish equivalence-preserving transformations from candidate-generating steps. Its figures emphasize that some transformations require explicit verification obligations.
- Reproducibility Details: Equation instances are generated from integer coefficients with a ≠ c, while train and test equations remain instance-disjoint.
- Reproducibility Details: Step indexing is 0-based, and Q3 identifies the first invalid step or uses −1 when no invalid step exists.
- Trace Semantics and Validation: The checker does not treat every familiar manipulation as a bidirectional equivalence over the real line.
- Trace Semantics and Validation: A step is accepted as strictly equivalent when its solution set is logically equivalent to the next step over the active domain.
- Trace Semantics and Validation: Candidate-generating transformations may add extra roots but are accepted only after every candidate is checked against the original equation.
D. Perturbation Families
The benchmark tests whether evaluators can handle valid but unconventional solution paths and distinguish them from genuine errors. It uses controlled perturbations and three separate judgments: final-answer correctness, full-trace validity, and first-error localization.
- Perturbation design: Six perturbation families generate valid but unconventional traces while holding the original equation fixed.Four preserve equivalence directly; multiplying by an expression involving x and squaring both sides generate obligations checked by substitution.
- Evaluation outputs: The model receives the original equation and complete worked trace, then returns Q1, Q2, and Q3.Q1 checks the final answer, Q2 checks the full trace, and Q3 identifies the first invalid step or returns −1.
- Metrics: Q2 false-rejection rate measures rejection of valid T1 traces, while Q2 error rates on T2 and T3 indicate false acceptance of invalid processes.Localization is strict and requires exact agreement with the first-error index.
F. Model Selection and Adaptation
The study evaluates three open-weight reasoning models and tests parameter-efficient fine-tuning, explanation-based distillation, and test-time voting as robustness adaptations. These interventions target acceptance of valid non-canonical traces, rejection and localization of invalid ones, or correction of isolated response errors.
- Model selection: The evaluation covers GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning as open-weight reasoning-oriented models.The models were screened to reduce confounding from inability to follow mathematical notation.
- Fine-tuning: LoRA fine-tuning teaches acceptance of valid non-canonical traces and rejection or localization of invalid transitions.Training uses T1 samples for acceptance and T2/T3 samples for rejection and localization.
- Distillation: Gemini-enhanced distillation retains 4,000 samples whose explanations preserve ground-truth Q1, Q2, and Q3 labels.The intervention tests whether richer natural-language rationales improve verifier behavior beyond label-only supervision.
- Test-time compute: Test-time compute aggregates seven sampled responses by majority vote for each output field.This tests whether additional samples correct isolated mistakes or amplify systematic template sensitivity.
G. Data Composition and Partitioning
The benchmark uses balanced canonical and perturbed evaluation sets across three trace types, enabling comparisons between standard and altered solution forms. Results show strong canonical performance but sharp degradation under perturbation, particularly for valid T1 traces.
- Data composition: The training set contains 27,000 samples, including 9,000 canonical and 18,000 perturbed traces.Canonical traces are evenly split across T1/T2/T3, while perturbed traces are balanced across trace type and perturbation family.
- Partitioning: Each held-out evaluation set contains 1,080 samples, with 360 samples per trace type.The perturbed evaluation set is balanced across perturbation families.
- Perturbed-trace results: 75.56% of valid perturbed T1 traces are falsely rejected by GPT-OSS 20B, compared with 83.89% for Qwen3-14B and 85.28% for Phi-4-Reasoning.These Q2 false-rejection rates quantify the models’ sensitivity to non-canonical solution forms.
- Canonical-versus-perturbed comparison: GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning achieve 98.33%, 93.89%, and 92.22% Q2 accuracy on baseline T1 traces, but only 24.44%, 16.11%, and 14.72% on perturbed T1 traces.The models often treat the first unfamiliar transformation as an error despite deterministic acceptance of every transition.
2) Outcome and Process Are Entangled:
Perturbations entangle outcome and process judgments: models can misjudge correct final answers or valid reasoning when derivations look unfamiliar, while adaptation can improve robustness with model-dependent trade-offs.
- Outcome and Process Are Entangled:: On T3 traces, base-model Q1 accuracy drops from 96.67% to 67.78% for GPT-OSS 20B, from 71.39% to 63.06% for Qwen3-14B, and from 99.44% to 56.94% for Phi-4-Reasoning.The final answer remains correct by construction, so unfamiliar derivations can contaminate outcome judgment.
- Outcome and Process Are Entangled:: GED-FT improves GPT-OSS 20B perturbed Q2 from 54.06% to 76.94% and Q3 from 24.07% to 58.98%.On valid T1 perturbations, Q2 rises from 24.44% to 94.72%, indicating improved acceptance of valid non-canonical derivations.
- Outcome and Process Are Entangled:: Qwen3-14B’s best perturbed result comes from FT+TTC, raising localization from 14.35% to 81.67%, whereas GED-FT underperforms.The usefulness of distilled explanations depends on the student model and explanation style.
- Outcome and Process Are Entangled:: GED-FT is most effective for Phi-4-Reasoning, reaching 96.39% on perturbed Q1, 96.76% on Q2, and 93.70% on Q3.Standard FT can over-accept traces, so valid and invalid examples must remain balanced.
C. Test-Time Compute by Majority Voting
Majority-vote test-time compute helps when errors are noisy but can stabilize systematic mistakes; after adaptation, it can reduce residual uncertainty.
- Test-Time Compute by Majority Voting: Majority-vote TTC aggregates k = 7 sampled responses to test whether additional samples correct isolated mistakes or amplify template sensitivity.Its benefit depends on whether model errors are noisy or systematic.
- Test-Time Compute by Majority Voting: For GPT-OSS 20B, TTC reduces perturbed Q1 from 76.30% to 32.04% and Q2 from 54.06% to 31.11%.Similar perturbed Q1 drops occur for Qwen3-14B and Phi-4-Reasoning.
- Test-Time Compute by Majority Voting: For Qwen3-14B, FT+TTC reaches 81.67% on perturbed Q3, the strongest perturbed localization result.Training appears to provide a better verification policy, while TTC reduces residual uncertainty.
- Test-Time Compute by Majority Voting: TTC should be treated as a calibration tool rather than a substitute for learning trace-validity semantics.Voting can reinforce an incorrect heuristic when the model applies it consistently.
D. Limitations and Future Work
The benchmark is intentionally diagnostic, using a narrow algebraic setting for exact validation while leaving transfer to richer domains unestablished.
- Limitations and Future Work: The study is limited to single-variable linear equations with unique rational solutions.This setting enables exact label validation and unambiguous first-error indices.
- Limitations and Future Work: The results do not establish that the same behavior holds for inequalities, systems of equations, proofs, programs, or scientific workflows.The benchmark isolates process-level verification rather than representing all mathematical or scientific verification tasks.
- Limitations and Future Work: Future work should extend procedural evaluation to richer mathematical and non-mathematical domains and test held-out perturbation families.Such studies could examine whether models learn general trace validity or adapt to specific transformation templates.
- Limitations and Future Work: Single-run adaptation results warrant caution when interpreting small differences between variants.The authors also identify teacher model, prompt design, and filtering policy as variables affecting Gemini-enhanced distillation.
V. CONCLUSION
The benchmark shows that strong canonical-solution evaluation does not guarantee robust verification of admissible non-canonical traces, particularly for first-error localization. The results therefore support evaluating reasoning-process robustness separately from final-answer accuracy and extending process-focused evaluation to trustworthy applications.
- Base models that perform well on canonical solutions often fail on admissible non-canonical traces, especially when localizing the first invalid step.
- Evaluator robustness is a distinct capability from final-answer accuracy.
- Trustworthy LLM evaluation requires attention to reasoning processes, not only outcomes.
- Adaptation can reduce the robustness gap, but its benefits are model dependent, and majority-vote test-time compute can amplify systematic mistakes.
- Controlled process-level benchmarks can help measure and improve LLM reliability where reasoning paths must be verified.