PaperlayerView exact revision on arXiv

Source-linked AI summary

SCI-Verifier: Scientific Verifier with Thinking

Shenghe Zheng, Chenyu Huang, Fangchen Yu, Junchi Yao, Jingqi Ye, Tao Chen, Yun Luo, Ning Ding, LEI BAI, Ganqu Cui, Peng Ye

arXiv:2509.24285v1cs.AIcs.CLcs.LG

TL;DR

Scientific answer verification is challenged by complex reasoning and diverse equivalent answer forms, alongside limited benchmark coverage and limitations in existing methods. The paper constructs a cross-disciplinary benchmark and post-trains a reasoning-augmented verifier; SCI-Verifier reports strong performance on challenging samples and cross-disciplinary generalization, with its 8B version on par with GPT-5.

  • Problem

    Scientific verification is difficult because answers require multi-step reasoning and may have diverse equivalent formulations, while benchmarks and methods have coverage and reasoning limitations.

    Source: §1 · p. 1

  • Method

    The paper builds SCI-VerifyBench from LLM responses and domain-specific equivalence transformations with model and expert annotations, and develops SCI-Verifier through reasoning-focused post-training.

    Source: §1 · p. 2

  • Results

    SCI-Verifier improves accuracy on challenging, confusable samples, generalizes across disciplines, and its 8B version performs on par with GPT-5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Together, the benchmark and verifier provide a framework for assessing cross-disciplinary scientific verification and a practical path for improving LLM reliability and reasoning in scientific domains.

    Source: §1 · p. 2, §6 · p. 9

    Scenarios requiring both high accuracy and extreme efficiency remain a future challenge; the authors propose improving implicit reasoning to reduce reliance on detailed explicit steps.

    Source: C LIMITATIONS AND FUTURE WORK · p. 37

Abstract

from arXiv · show

As large language models (LLMs) are increasingly applied to scientific reasoning, the complexity of answer formats and the diversity of equivalent expressions make answer verification a critical yet challenging task. Existing verification studies in scientific domains suffer from two major limitations: (a) the absence of systematic evaluation standards and insufficient disciplinary coverage, which hinders their comprehensive assessment; and (b) heavy reliance on cumbersome rule design or prompt engineering, which reduces their effectiveness in complex reasoning scenarios or limits their cross-disciplinary generalization. To address these challenges, we propose solutions at both the data and model levels. On the data side, we construct SCI-VerifyBench, a cross-disciplinary benchmark covering mathematics, physics, biology, chemistry, and general scientific QA. The benchmark is built from real LLM responses and enhanced with domain-specific equivalence transformations that generate challenging and realistic data. Model-based and expert annotations ensure both quality and diversity, enabling rigorous evaluation of verification ability. On the model side, we emphasize the importance of reasoning for verification and introduce SCI-Verifier, a unified reasoning-augmented verifier for scientific domains. Through post-training, SCI-Verifier demonstrates strong logical reasoning and equivalence judgment capabilities while maintaining concise and stable outputs. Together, SCI-VerifyBench and SCI-Verifier provide a principled framework for scientific verification, offering both systematic evaluation and practical pathways to enhance the reliability and applicability of LLMs in scientific domains.

1 INTRODUCTION

Scientific answer verification is difficult because scientific responses involve multi-step reasoning and equivalent formulations, while existing benchmarks and methods have notable coverage and reasoning limitations. The paper addresses these gaps with a cross-disciplinary benchmark and a reasoning-augmented verifier.

1 INTRODUCTION

  • Scientific verification faces limited systematic benchmarks and disciplinary coverage, while existing approaches struggle with complex reasoning or require labor-intensive rules and extensive prompt engineering.The paper also identifies unstable outputs and limited cross-disciplinary applicability among current methods.Source: §1 · p. 1
  • SCI-VerifyBench covers mathematics, physics, biology, chemistry, and general scientific QA using LLM responses, domain-specific equivalence transformations, and model and expert annotations.The transformations include formula rewrites and logical equivalence substitutions.Source: §1 · p. 2
  • SCI-Verifier improves accuracy on challenging, confusable samples, generalizes across disciplines, and its 8B version performs on par with GPT-5.The paper reports this comparison with current verifiers and the state-of-the-art closed-source model GPT-5.Source: §1 · p. 2
  • SCI-Verifier combines supervised fine-tuning and reinforcement learning to support logical reasoning and complex equivalence judgments with concise, stable outputs.The paper presents this as a two-stage post-training pipeline for scientific verification.Source: §1 · p. 2

2 RELATED WORKS

Related work covers scientific verification benchmarks, verifier methods, and reward models. The discussion distinguishes correctness verification from response-quality ranking and identifies coverage and reasoning challenges for scientific verification.

Verification Benchmark

  • The paper says existing verification benchmarks have pointless samples, including multiple-choice questions needing no specially designed verifiers, and limited disciplinary coverage that restricts assessment of scientific-domain generalization.VerifyBench (Yan et al., 2025) has 4,000 expert-level questions across STEM domains; VerifierBench (Liu et al., 2025) aggregates model outputs with manual meta-error analysis across math, science, knowledge, and general reasoning tasks.Source: §2 · p. 3

Verification Models

  • Prior verifiers range from efficient methods lacking reasoning to template-based approaches, while the paper describes complex scientific reasoning and prompt instability as unresolved challenges.The discussion names xVerify (Chen et al., 2025a), General-Verifier, and CompassVerifier (Liu et al., 2025).Source: §2 · p. 3

Reward Models

  • Reward models rank response quality rather than verify correctness, a difference the paper says creates distinct data-construction and training challenges.The discussion cites discriminative and reasoning-based reward-model approaches, including J1 (Whitehouse et al., 2025), Think-J, and Compass-Judger2.Source: §2 · p. 3

3 SCI-VERIFYBENCH

SCI-VerifyBench combines broad scientific-domain coverage with equivalence-focused data and difficulty controls. Its construction uses real LLM responses, synthetic equivalent answers, and hybrid annotation and filtering.

3.1 DATA OVERVIEW

  • SCI-VerifyBench spans more domains than existing benchmarks and includes challenging, commonly encountered equivalence transformations with difficulty controls.Table 1 compares verification benchmarks; Table 2 presents static data analyses.Source: §3.1 · p. 3, Table 1 · p. 3, Table 2 · p. 3

3.2 DATA COLLECTION

  • The data collection generates over 100K responses from eight models using more than 15K question-answer pairs across mathematics, physics, chemistry, biology, and general QA.Response length is controlled to support adaptation to different question and answer styles.Source: §3.2 · pp. 3–4
  • For each domain, the authors select 500 questions suited to equivalence transformations and generate five equivalent answers per question, screening invalid cases with five LLMs.Rejected cases are regenerated when multiple models agree that an equivalence is invalid.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Five LLMs label answers, and 2,500 samples with the greatest model disagreement receive expert review by at least two annotators, with a third consulted when needed.Answers are considered equivalent when they can be transformed into each other.Source: §3.2 · p. 4
  • The 2,500-sample test set mixes 350 real responses and 150 synthetic responses per domain, while the 14K training set uses mostly LLM annotations.Test samples require full expert agreement, with human–LLM disagreements preferentially included; substantial LLM disagreement is filtered from training data.Source: §3.2 · p. 4

4 SCI-VERIFIER

SCI-Verifier incorporates reasoning because scientific answers can be complex and equivalent in multiple forms. Its two-stage post-training combines supervised fine-tuning and reinforcement learning to produce concise reasoning for scientific verification.

4.1 MOTIVATION

  • Reasoning brings substantial gains in scientific verification, where complex responses and multiple equivalent forms require assessing equivalence from different perspectives.The authors compare final-answer-only outputs with outputs that include intermediate reasoning.Source: §4.1 · p. 5

4.2 POST-TRAINING

  • SFT uses rejection-sampled reasoning traces, filters for valuable concise examples, and fine-tunes a smaller model to transfer domain-specific verification knowledge.For reasoning models, only the conclusive summary is kept; overly long or unstructured responses from non-reasoning models are discarded.Source: §4.2 · p. 5
  • After SFT, DAPO dynamically filters overly simple or difficult samples and adds a length penalty to encourage concise reasoning.DAPO is described as a refined GRPO method; the reward combines alignment reward with an overlong penalty.Source: §4.2 · pp. 5–6
  • RL training rebalances samples to equal positive and negative examples to reduce reliance on label priors instead of reasoning.The authors identify class imbalance as a risk for this binary classification task.Source: §4.2 · p. 6

4 SCI-VERIFIER

  • The two-stage post-training yields a concise reasoning verifier applicable across scientific domains and is reported to improve evaluation reliability and provide a semantically clear reward function.The reward function is intended to facilitate training stronger reasoning-oriented language models.Source: §4.2 · p. 6

5 EXPERIMENTS

The experiments compare SCI-Verifier with model and verifier baselines on SCI-VerifyBench and two established benchmarks, then examine its training components. SCI-Verifier performs strongly across benchmarks and on equivalent-answer cases, while results vary with domain and training choices.

5.1 BASELINES AND SETUP

  • The evaluation includes SCI-Verifier 4B and 8B, four baseline categories, and Accuracy on SCI-VerifyBench plus Accuracy and F1 on two established datasets.The baselines include closed-source, open-source instruct, open-source reasoning, and specialized verifier models.Source: §5.1 · p. 6

5.2 EVALUATION AND ANALYSIS OF SCI-VERIFYBENCH

  • GPT-5 exceeds current open-source models by more than 5%, although SCI-Verifier performs comparably to GPT-5 on scientific verification.The authors describe a remaining proprietary-model edge despite open-source models closing the gap.Source: §5.2 · p. 7
  • Scaling does not consistently improve verification, and mathematics and physics score lower than other subjects, where complex transformations make judgments subtler.The paper points to factorization and Taylor expansions as examples of transformations that add difficulty.Source: §5.2 · pp. 7–8
  • On equivalent-answer tests, GPT-5 scores below 50% in mathematics and physics, while SCI-Verifier 4B and 8B perform substantially better.The authors attribute SCI-Verifier's advantage to targeted optimization for equivalence verification.Source: Figure 3 · p. 7, §5.2 · p. 7

5.3 EVALUATION AND ANALYSIS OF SCI-VERIFIER

  • Across SCI-VerifyBench, VerifierBench, and VerifyBench-Hard, both SCI-Verifier sizes perform strongly, comparably to GPT-5 and above current open-source models on all SCI-VerifyBench disciplines.The authors report consistent advantages across all three benchmarks and characterize this as evidence of cross-task generalization.Source: Table 4 · p. 7, §5.3 · p. 8

5.4 ABLATION STUDY

  • SFT alone and RL from a reasoning model are competitive, but RL directly on the base model performs poorly; combining SFT and RL yields the strongest cross-dataset generalization.The authors suggest SFT warm-up helps the base model acquire targeted reasoning abilities.Source: §5.4 · pp. 8–9
  • The constructed training data performs strongly across three benchmarks under both SFT and SFT+RL, while heterogeneous quality in the larger RM dataset slows RL improvement.The authors say RM exceeds 180K samples and that filtering is necessary in practice.Source: §5.4 · p. 9
  • Short CoT traces provide sufficient reasoning, whereas complete CoT adds output length without improving performance and omitting CoT reduces accuracy despite more efficient inference.The authors characterize concise reasoning from fixed perspectives as suitable for this relatively simple verification task.Source: §5.4 · p. 9

6 CONCLUSION

The conclusion presents SCI-VerifyBench as a cross-disciplinary evaluation framework and SCI-Verifier as a concise-reasoning verifier for scientific answer verification.

6 CONCLUSION

  • SCI-VerifyBench spans mathematics, physics, chemistry, biology, and commonsense scientific QA to assess cross-disciplinary verification.The conclusion describes it as a diverse benchmark for rigorous and systematic evaluation.Source: §6 · p. 9
  • The paper concludes that chain-of-thought reasoning is essential, particularly for scientific answers that are complex or have multiple equivalent forms.Source: §6 · p. 9
  • SCI-Verifier combines concise reasoning tailored to verification with SCI-VerifyBench as a framework and practical solution for scientific verification.The authors say the pair has potential to guide the reliability and advancement of LLMs in scientific reasoning.Source: §6 · p. 9

A DETAILS OF SCI-VERIFYBENCH

The appendix section outlines the process and locations of details for constructing SCI-VerifyBench.

A DETAILS OF SCI-VERIFYBENCH

A.1 DATA GENERATION

This section describes generating a dataset of LLM-produced answers to existing questions, checked against references and paired with subject-specific equivalent forms. It covers prompts and transformations tailored to scientific subjects and tasks, including biology and general scientific QA, that vary answer representation while preserving meaning or outcome.

Data generation

  • Two-part generation combines reference-checked answers to existing questions with subject-specific equivalent answers, using candidates from eight LLMs.The listed models include Qwen3 variants, LLaMa3.3-70B-Instruct, GPT-oss-20B, Qwen2.5-32B-Instruct, and Gemma-3-27b-it.Source: §A.1 · p. 13
  • Math equivalence prompts cover interval, expression, and equation answers through transformations such as inequality rewrites, algebraic manipulation, and domain restrictions.The prompts request exactly 10 boxed forms for each answer.Source: §A.1 · pp. 13–15
  • Physics prompts generate equivalent numerical, expression, and equation answers using arithmetic, notation and unit conversions, algebraic transformations, and dimensional adjustments.The prompts request exactly 10 boxed forms; equation transformations also include domain restrictions and approximate solutions.Source: §A.1 · pp. 13, 15, 16, 17
  • Chemistry prompts span solvent, property, yield, temperature, product, and retrosynthesis answers, generating alternatives through naming, notation, unit, range, and route changes.Examples include solvent names, formulas, and SMILES; property units or descriptive labels; and alternative valid retrosynthetic routes.Source: §A.1 · pp. 18–23
  • Other chemistry prompts convert between molecular structures and textual captions or names, using equivalent representations, naming conventions, and applicable salt or hydrate forms.The tasks include mol2caption, caption2mol, and chemical name conversion.Source: §A.1 · pp. 24–25

Protein inverse folding

  • Protein-sequence prompts generate equivalent forms through code changes, FASTA or array encodings, domain grouping, and tolerated substitutions or mutations intended to preserve structure.The prompt requests exactly 10 forms, each wrapped in LaTeX boxed notation.Source: §A.1 · p. 26

Protein structure prediction

  • Protein-structure prompts vary coordinate and file formats, structural abstractions, chain or atom ordering, and visual encodings to represent equivalent structures.Examples include PDB-to-mmCIF conversion, contact maps, distance matrices, torsion angles, and coarse-grained models.Source: §A.1 · p. 27

Transformed agentclinic problems

  • Agent-trajectory prompts express equivalent solutions as alternative action sequences, macro-actions, state transitions, structured logs, or outcome summaries.They also permit reordering independent actions and adding no-op actions that do not affect the outcome.Source: §A.1 · p. 28

RNA structure prediction

  • RNA-structure prompts recast structures across notation, pair lists, matrices, coordinates, annotations, and graph or visual forms, including near-optimal variants.The prompt asks for exactly 10 boxed forms.Source: §A.1 · p. 29

RNA inverse folding

  • RNA-sequence prompts generate equivalent sequences using RNA/DNA notation, case, alignment gaps, alternative nucleotides, region labels, and sequence encodings or chunking.These transformations are framed as preserving the target fold.Source: §A.1 · p. 30

General QA problems

  • General QA prompts preserve answer meaning through paraphrases, reordered phrasing, alternate formats, formal expressions, and selected numeric or unit conversions.The prompt requests exactly 10 boxed forms, separated by newlines.Source: §A.1 · p. 31

A.2 DATA ANNOTATION AND EVALUATION

The section specifies the evaluation inputs and output, then contrasts inference prompts with and without thinking and details the answer-judgment criteria.

A.2 DATA ANNOTATION AND EVALUATION

  • Evaluation takes a question, reference answer, and candidate answer, returns a correctness judgment, and uses temperature 0 for stable outputs.The section points to separate prompts for CoT, no-CoT, and prompt-stability experiments.Source: §A.2 · p. 31
  • The thinking prompt requests step-by-step comparison before a structured judgment, whereas the no-thinking prompt requests only the final structured judgment.Both prompt variants instruct the verifier to compare rather than regenerate answers.Source: §A.2 · pp. 32–33
  • The rubric first flags incomplete, repetitive, or refusal responses, then checks answer completeness, strict requirements, and permitted equivalences.It treats partial matches for multi-part answers as incorrect and allows equivalent mathematical forms where appropriate.Source: §A.2 · p. 33
  • The thinking prompt treats an indicated correct option as correct for explicit-choice questions and requires content and meaning to align when options are absent.Mathematically equivalent answers can be correct despite differing formats.Source: §A.2 · p. 34

A.3 DATA DETAILS

The appendix illustrates generated equivalent-answer cases across mathematics, physics, chemistry, biology, and general question answering.

A.3 DATA DETAILS

  • Examples span five fields: mathematical and physical problems, chemical answers, biological structure notation, and general QA.The section identifies these as examples of equivalent forms generated for the benchmark.Source: §A.3 · p. 34
  • A math example pairs a function-equation problem with its gold answer, while a physics example gives a prism question with a 7.16 mm gold answer.The physical example asks for the separation between o-ray and e-ray spectral lines.Source: §A.3 · pp. 34–35
  • A chemistry example asks for a SMILES representation of an IUPAC compound name, and a biology example gives an RNA sequence with dot-bracket gold notation.The chemical example includes the target SMILES string; the biological example includes the reference structure notation.Source: §A.3 · pp. 35–36
  • The QA example asks for a fictional president’s name and presents equivalent phrasings that identify Kelly Wade.The variants describe the character as president, incumbent U.S. leader, or serving as president in Season 1.Source: §A.3 · p. 36

B DETAILS OF TRAINING

Training has two parts: supervised fine-tuning and reinforcement learning, with full fine-tuning for SFT and a modified GRPO method for RL.

B DETAILS OF TRAINING

  • Training uses full fine-tuning for SFT and a modified version of GRPO for RL, with configurations reported in Tables 6 and 7.The section identifies SFT and RL as the two main training parts.Source: B DETAILS OF TRAINING · p. 36

C LIMITATIONS AND FUTURE WORK

Some scientific verification scenarios require both high accuracy and extreme efficiency, motivating future work on implicit reasoning. The paper also reports human reannotation and filtering of selected LLM-generated data.

C LIMITATIONS AND FUTURE WORK

D THE USE OF LARGE LANGUAGE MODELS

Loading 2509.24285v1…