PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification

Fateme Mazdarani, Carlos Toxtli

arXiv:2608.28725v1cs.AI

TL;DR

Most mathematical evaluations emphasize final answers, leaving unclear whether LLMs can verify valid non-canonical traces. The paper introduces a controlled linear-equation benchmark for final-answer, step-level, and first-error evaluation, finding that base models perform well on canonical traces but degrade sharply on perturbed ones, especially for localization.

  • Problem

    Most mathematical benchmarks emphasize final-answer accuracy, providing limited evidence about whether evaluators accept valid non-canonical solution traces.

    Source: §I · p. 1

  • Method

    The paper generates controlled linear-equation traces with exact labels and evaluates final answers, trace correctness, and first-error localization across canonical and perturbed solutions.

    Source: §I · p. 1, D. Perturbation Families · p. 3, C. Trace Semantics and Validation · p. 3

  • Results

    Base models perform well on canonical traces but degrade sharply under perturbation, with valid perturbed T1 false-rejection rates of 75.56% for GPT-OSS 20B, 83.89% for Qwen3-14B, and 85.28% for Phi-4-Reasoning.

    Source: A. Base-Model Robustness Gap · p. 4

  • Takeaways & Limitations

    Evaluator robustness is distinct from final-answer accuracy, and adaptation can reduce the gap although benefits are model dependent and majority-vote test-time compute can amplify systematic mistakes.

    Source: §V · p. 6

    The study is limited to single-variable linear equations with unique rational solutions and does not establish that the behavior generalizes to richer mathematical or scientific tasks.

    Source: D. Limitations and Future Work · p. 6

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final-answer accuracy. This can obscure whether a model can verify a non-canonical but valid solution trace. We introduce a controlled linear-equation benchmark for evaluating LLMs in the evaluator role. Each instance asks the model to judge final-answer correctness, step-level trace correctness, and the first incorrect step. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Across GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, base models perform well on canonical traces but degrade substantially on perturbed traces, especially for error localization. On valid perturbed traces, base-model false-rejection rates reach 75.6-85.3%, showing strong sensitivity to canonical solution form. Supervised fine-tuning, distillation, and test-time compute improve robustness in some settings, but gains are model dependent and can trade off against canonical performance. The results show that reliable process-level verification remains challenging, and evaluator robustness should be measured separately from solver accuracy, even in a simple algebraic domain with exact ground truth.

I. INTRODUCTION

The paper frames evaluator robustness as the ability to distinguish valid non-canonical solution paths from genuine errors, not merely check final answers. It introduces a controlled benchmark and three-part protocol to measure this capability in linear equations.

  • Correct final answers can still contain invalid steps, while valid solutions may follow non-standard paths that should not be penalized.Source: §I · p. 1
  • Template sensitivity describes reliance on familiar solution forms rather than step-level validity.Source: §I · p. 1
  • The benchmark uses fixed linear equations and perturbed traces with exact labels for correct, incorrect, and tricky solutions.Source: §I · pp. 1–2
  • The protocol separately evaluates final-answer correctness, step-level correctness, and first-error localization.Source: §I · p. 2
  • The study evaluates GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning, then tests supervised fine-tuning, distillation, and test-time compute.Source: §I · p. 2

II. RELATED WORK

Prior mathematical LLM benchmarks primarily measure solving performance, while verifier research studies process scoring and intermediate reasoning. This work focuses specifically on grading solution traces under controlled variability.

C. Evaluating Worked Solutions

The benchmark generates worked solutions for fixed linear equations across three trace types, separating final-answer checking from process checking and error localization.

B. Reproducibility Details

The benchmark uses exact, checker-based semantics to validate algebraic traces and distinguish equivalence-preserving transformations from candidate-generating steps. Its figures emphasize that some transformations require explicit verification obligations.

Reproducibility Details

Trace Semantics and Validation

D. Perturbation Families

The benchmark tests whether evaluators can handle valid but unconventional solution paths and distinguish them from genuine errors. It uses controlled perturbations and three separate judgments: final-answer correctness, full-trace validity, and first-error localization.

Perturbation design

  • Six perturbation families generate valid but unconventional traces while holding the original equation fixed.Four preserve equivalence directly; multiplying by an expression involving x and squaring both sides generate obligations checked by substitution.Source: D. Perturbation Families · p. 3

Evaluation outputs

  • The model receives the original equation and complete worked trace, then returns Q1, Q2, and Q3.Q1 checks the final answer, Q2 checks the full trace, and Q3 identifies the first invalid step or returns −1.Source: E. Evaluation Framework · p. 3

Metrics

  • Q2 false-rejection rate measures rejection of valid T1 traces, while Q2 error rates on T2 and T3 indicate false acceptance of invalid processes.Localization is strict and requires exact agreement with the first-error index.Source: E. Evaluation Framework · p. 3

F. Model Selection and Adaptation

The study evaluates three open-weight reasoning models and tests parameter-efficient fine-tuning, explanation-based distillation, and test-time voting as robustness adaptations. These interventions target acceptance of valid non-canonical traces, rejection and localization of invalid ones, or correction of isolated response errors.

Model selection

  • The evaluation covers GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning as open-weight reasoning-oriented models.The models were screened to reduce confounding from inability to follow mathematical notation.Source: F. Model Selection and Adaptation · p. 3

Fine-tuning

  • LoRA fine-tuning teaches acceptance of valid non-canonical traces and rejection or localization of invalid transitions.Training uses T1 samples for acceptance and T2/T3 samples for rejection and localization.Source: F. Model Selection and Adaptation · p. 3

Distillation

  • Gemini-enhanced distillation retains 4,000 samples whose explanations preserve ground-truth Q1, Q2, and Q3 labels.The intervention tests whether richer natural-language rationales improve verifier behavior beyond label-only supervision.Source: F. Model Selection and Adaptation · p. 3

Test-time compute

  • Test-time compute aggregates seven sampled responses by majority vote for each output field.This tests whether additional samples correct isolated mistakes or amplify systematic template sensitivity.Source: F. Model Selection and Adaptation · p. 3

G. Data Composition and Partitioning

The benchmark uses balanced canonical and perturbed evaluation sets across three trace types, enabling comparisons between standard and altered solution forms. Results show strong canonical performance but sharp degradation under perturbation, particularly for valid T1 traces.

Data composition

  • The training set contains 27,000 samples, including 9,000 canonical and 18,000 perturbed traces.Canonical traces are evenly split across T1/T2/T3, while perturbed traces are balanced across trace type and perturbation family.Source: G. Data Composition and Partitioning · p. 3

Partitioning

Perturbed-trace results

  • 75.56% of valid perturbed T1 traces are falsely rejected by GPT-OSS 20B, compared with 83.89% for Qwen3-14B and 85.28% for Phi-4-Reasoning.These Q2 false-rejection rates quantify the models’ sensitivity to non-canonical solution forms.Source: A. Base-Model Robustness Gap · p. 4

Canonical-versus-perturbed comparison

  • GPT-OSS 20B, Qwen3-14B, and Phi-4-Reasoning achieve 98.33%, 93.89%, and 92.22% Q2 accuracy on baseline T1 traces, but only 24.44%, 16.11%, and 14.72% on perturbed T1 traces.The models often treat the first unfamiliar transformation as an error despite deterministic acceptance of every transition.Source: 1) Template Sensitivity · p. 4

2) Outcome and Process Are Entangled:

Perturbations entangle outcome and process judgments: models can misjudge correct final answers or valid reasoning when derivations look unfamiliar, while adaptation can improve robustness with model-dependent trade-offs.

Outcome and Process Are Entangled:

  • On T3 traces, base-model Q1 accuracy drops from 96.67% to 67.78% for GPT-OSS 20B, from 71.39% to 63.06% for Qwen3-14B, and from 99.44% to 56.94% for Phi-4-Reasoning.The final answer remains correct by construction, so unfamiliar derivations can contaminate outcome judgment.Source: 2) Outcome and Process Are Entangled · p. 4
  • GED-FT improves GPT-OSS 20B perturbed Q2 from 54.06% to 76.94% and Q3 from 24.07% to 58.98%.On valid T1 perturbations, Q2 rises from 24.44% to 94.72%, indicating improved acceptance of valid non-canonical derivations.Source: B. Impact of Supervised Fine-Tuning… · pp. 4–5
  • Qwen3-14B’s best perturbed result comes from FT+TTC, raising localization from 14.35% to 81.67%, whereas GED-FT underperforms.The usefulness of distilled explanations depends on the student model and explanation style.Source: B. Impact of Supervised Fine-Tuning… · p. 5
  • GED-FT is most effective for Phi-4-Reasoning, reaching 96.39% on perturbed Q1, 96.76% on Q2, and 93.70% on Q3.Standard FT can over-accept traces, so valid and invalid examples must remain balanced.Source: B. Impact of Supervised Fine-Tuning… · p. 5

C. Test-Time Compute by Majority Voting

Majority-vote test-time compute helps when errors are noisy but can stabilize systematic mistakes; after adaptation, it can reduce residual uncertainty.

Test-Time Compute by Majority Voting

D. Limitations and Future Work

The benchmark is intentionally diagnostic, using a narrow algebraic setting for exact validation while leaving transfer to richer domains unestablished.

Limitations and Future Work

  • The study is limited to single-variable linear equations with unique rational solutions.This setting enables exact label validation and unambiguous first-error indices.Source: D. Limitations and Future Work · p. 6
  • The results do not establish that the same behavior holds for inequalities, systems of equations, proofs, programs, or scientific workflows.The benchmark isolates process-level verification rather than representing all mathematical or scientific verification tasks.Source: D. Limitations and Future Work · p. 6
  • Future work should extend procedural evaluation to richer mathematical and non-mathematical domains and test held-out perturbation families.Such studies could examine whether models learn general trace validity or adapt to specific transformation templates.Source: D. Limitations and Future Work · p. 6
  • Single-run adaptation results warrant caution when interpreting small differences between variants.The authors also identify teacher model, prompt design, and filtering policy as variables affecting Gemini-enhanced distillation.Source: D. Limitations and Future Work · p. 6

V. CONCLUSION

The benchmark shows that strong canonical-solution evaluation does not guarantee robust verification of admissible non-canonical traces, particularly for first-error localization. The results therefore support evaluating reasoning-process robustness separately from final-answer accuracy and extending process-focused evaluation to trustworthy applications.

  • Base models that perform well on canonical solutions often fail on admissible non-canonical traces, especially when localizing the first invalid step.Source: §V · p. 6
  • Evaluator robustness is a distinct capability from final-answer accuracy.Source: §V · p. 6
  • Trustworthy LLM evaluation requires attention to reasoning processes, not only outcomes.Source: §V · p. 6
  • Adaptation can reduce the robustness gap, but its benefits are model dependent, and majority-vote test-time compute can amplify systematic mistakes.Source: §V · p. 6
  • Controlled process-level benchmarks can help measure and improve LLM reliability where reasoning paths must be verified.Source: §V · p. 6

Related papers

Loading 2608.28725v1…