PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Data and Reward Design for Scientific Reasoning in Large Language Models

Zijie Chen, Zhenghao Lin, Xiao Liu, Zhenzhong Lan, Yeyun Gong, Peng Cheng

arXiv:2602.08321v2cs.CL

TL;DR

Open-ended scientific reasoning is limited by unreliable supervision, evaluation, and reward design. The paper introduces the Dr. SCI dataset and a three-part post-training pipeline that expands reasoning exploration, adapts difficulty, and uses rubric-guided rewards with explicit answer correctness. The resulting compact-backbone system delivers substantial gains and consistently surpasses strong post-trained baselines.

  • Problem

    Open-ended science questions lack reliable supervision and evaluation because answers are free-form, making stable post-training and reinforcement learning difficult.

  • Method

    Dr. SCI combines a structured scientific dataset with Exploration-Expanding SFT, Dynamic Difficulty Curriculum, and SciRubric-Guided RL.

  • Results

    Dr. SCI delivers substantial scientific reasoning gains from a compact 4B backbone and consistently surpasses strong post-trained baselines, including much larger and proprietary models.

  • Takeaways & Limitations

    Principled data processing and correctness-aware rewards are critical for advancing open-ended scientific reasoning and provide a practical recipe for future research.

  • Takeaways & Limitations

    Models with stronger scientific reasoning may still generate plausible but incorrect explanations, and the work does not solve factuality, misuse, or overreliance on automated scientific advice.

Abstract

from arXiv · show

Solving open-ended science questions remains challenging for large language models, particularly due to inherently unreliable supervision and evaluation. The bottleneck lies in the data construction and reward design for scientific post-training. We develop a large-scale, systematic data processing pipeline that transforms heterogeneous open-source science data into Dr. SCI dataset, which comprises of 1M questions across eight STEM subjects, with explicit verifiable/open-ended splits, scalable difficulty annotation, and fine-grained rubrics that operationalize evaluation for open-ended answers. Building on this dataset, we propose the Dr. SCI post-training pipeline, which redesigns the standard SFT -> RL workflow through three components: (i) Exploration-Expanding SFT, which broadens the model's reasoning pattern coverage prior to RL; (ii) Dynamic Difficulty Curriculum, which adapts training data to the model's evolving scientific capability; and (iii) SciRubric-Guided RL, which enables stable reinforcement learning on open-ended scientific questions via rubric-based evaluation with explicit answer correctness. Qwen3-4B-Base trained using Dr. SCI pipeline achieves 63.2 on GPQA-diamond and 32.4 on GPQA-general, consistently improves over strong post-trained baselines such as o1-mini and GPT-4o, demonstrating substantial gains in scientific reasoning, especially in open-ended settings.

1. Introduction

Open-ended scientific question answering remains difficult because supervision and evaluation are unreliable. Dr. SCI addresses this bottleneck through systematic data construction and a post-training pipeline combining exploration, adaptive difficulty, and rubric-guided rewards.

  • Motivation: Open-ended science questions span broad STEM knowledge and heterogeneous reasoning patterns, while free-form answers make references, verification, and reinforcement-learning rewards unreliable.Existing pipelines therefore struggle to elicit high-quality scientific reasoning in open-ended settings.
  • Dr. SCI Dataset: 1,006,701 curated problems across eight STEM subjects form Dr. SCI, with 461K rule-verifiable and 545K open-ended instances.The dataset also includes quality control, scalable difficulty annotation, and fine-grained rubrics for open-ended responses.
  • Post-Training Pipeline: Dr. SCI redesigns post-training through Exploration-Expanding SFT, Dynamic Difficulty Curriculum, and SciRubric-Guided RL.The components broaden reasoning-pattern coverage, adapt training data to capability, and combine rubric evaluation with explicit final-answer correctness.
  • Results: A compact 4B backbone trained with Dr. SCI achieves substantial scientific reasoning gains and consistently surpasses strong post-trained baselines including o1-mini and GPT-4o.The comparison is reported across diverse scientific reasoning benchmarks and shown in Figure 1.

2. Dr. SCI Dataset

Dr. SCI converts heterogeneous scientific resources into a structured dataset with broad STEM coverage, explicit verification regimes, scalable difficulty estimates, and rubric supervision for open-ended questions. Its statistics are designed to support efficient, reliable post-training.

  • Data Processing: Dr. SCI transforms heterogeneous open-source scientific corpora into a systematically cleaned and structured dataset spanning eight STEM subjects.Processing includes malformed-answer removal, subject assignment, deduplication, conflict resolution, and benchmark-contamination removal.
  • Verification Structure: Questions are partitioned into mutually exclusive verifiable and open-ended classes, with canonicalized answers for deterministic validation.Verifiable items include numerical, mathematical, and multiple-choice answers; open-ended mathematics questions are discarded because they are predominantly proof-based.
  • Difficulty and Rubrics: The dataset uses eight independent rollouts to estimate difficulty and removes 413K questions solved in all attempts as trivial.The resulting difficulty distribution is J-shaped, retaining easier examples for early learning and harder instances for scientific reasoning.
  • Difficulty and Rubrics: Open-ended questions receive 7–20 atomic rubric items labeled Essential, Important, Optional, or Pitfall.These criteria characterize high-quality responses and provide structured supervision for rubric-guided reinforcement learning.
  • Dataset Statistics: 1,006,701 questions cover eight STEM subjects, with every non-dominant subject contributing more than 47K instances.Mathematics and physics dominate many source collections, but the final dataset retains broad domain coverage.
  • Dataset Statistics: 461K questions are verifiable and 545K are open-ended, enabling rule-based and rubric-guided assessment in one training regime.The split directly supports distinct evaluation mechanisms within unified post-training.
  • Dataset Statistics: Questions average 72.7 tokens and reference answers average 30.1 tokens, while only 0.3% of reference answers exceed 250 tokens.The length profile supports large-scale automated verification and reinforcement learning while retaining complex explanations.

3. Dr. SCI Post Training

The Dr. SCI post-training pipeline redesigns standard SFT followed by verifiable-reward RL for scientific reasoning, especially open-ended questions. It expands exploration, adapts difficulty, and supplies rubric-based rewards constrained by final-answer correctness.

  • Pipeline Overview: Standard SFT followed by verifiable-reward RL is poorly suited to open-ended science because free-form explanations lack reliable verification signals.This makes both supervision and reinforcement learning difficult to apply stably and systematically.
  • Pipeline Overview: Dr. SCI integrates Exploration-Expanding SFT, Dynamic Difficulty Curriculum, and SciRubric-Guided RL to redesign each post-training stage.Together, the components target exploration breadth, capability-matched training data, and reliable open-ended rewards.
  • Exploration-Expanding SFT: Exploration-Expanding SFT greedily selects examples with maximal incremental 4-gram coverage, producing a more lexically and structurally diverse SFT set.The procedure broadens reasoning-pattern exposure before reinforcement learning.
  • Dynamic Difficulty Curriculum: The dynamic curriculum removes trivial instances, defers currently too-difficult questions, and replaces mastered samples with the easiest remaining deferred instances.Training difficulty therefore increases as the model improves while rewards remain informative.
  • SciRubric-Guided RL: SciRubric-Guided RL evaluates each open-ended response against fine-grained rubric criteria and separately checks the extracted final answer against the reference.Weighted aggregation uses category-derived importance weights, and partial rubric satisfaction cannot compensate for an incorrect final answer.
  • SciRubric-Guided RL: Combining rubric-guided rewards for open-ended questions with exact rule-based rewards for verifiable questions yields a unified scientific post-training framework.The authors report more stable training and stronger performance than prior reward formulations.

4. Experiments

The experiments evaluate Dr. SCI across scientific reasoning benchmarks and ablate its three components. Results show strong gains, especially on open-ended questions, with improvements in exploration coverage, curriculum efficiency, and rubric-guided RL stability.

  • Overall results: 32.4 on GPQA-General is achieved by Dr. SCI-4B-think versus 5.62 for the base model, while Dr. SCI-4B-instruct achieves 24.3.These results rank best among thinking and instruct models at comparable scale.
  • Overall results: Dr. SCI-4B-think exceeds o1-mini on GPQA-Diamond, SuperGPQA, and HLE, while Dr. SCI-4B-instruct outperforms GPT-4o on GPQA-Diamond, GPQA-General, and HLE.Across benchmarks, the 4B models consistently outperform same-backbone post-trained baselines and often surpass models up to 32B parameters.
  • Exploration-Expanding SFT: EESFT selects substantially more unique 4-grams than random sampling, reaching 139.6M versus 78.23M at 50K in thinking mode.The expanded reasoning-pattern coverage is associated with stronger SFT checkpoints and larger performance gains after RL.
  • Exploration-Expanding SFT: Scaling EESFT to 1M examples accumulates 1.564B unique 4-grams in thinking mode and 488.7M in instruct mode, alongside the highest final performance.The results support EESFT’s role in raising the exploration ceiling and unlocking larger RL gains.
  • Dynamic Difficulty Curriculum: The dynamic curriculum matches the performance of No Easy and Hard Only while using only 13.1K examples per epoch instead of 100K.Scaling the pool while keeping per-epoch compute comparable yields further gains, as training shifts toward harder data over time.
  • SciRubric-Guided RL: SciRubric-Guided RL delivers consistent gains, particularly on GPQA-General, while unified training on verifiable and open-ended questions achieves the best overall performance.The results support structured correctness-aware rewards and unified supervision for open-ended scientific reasoning.

5. Conclusion

The conclusion presents Dr. SCI as a foundation for scientific reasoning post-training that combines systematic data processing with a unified training framework. Its compact 4B models achieve substantial gains and surpass strong larger and proprietary baselines.

  • Contributions: Dr. SCI addresses fragile data curation, poorly calibrated difficulty, and unreliable evaluation through scalable processing, fine-grained supervision, quality control, and three integrated post-training components.The framework combines Exploration-Expanding SFT, Dynamic Difficulty Curriculum, and SciRubric-Guided RL across verifiable and open-ended settings.
  • Conclusion: The dynamic difficulty curriculum adjusts training difficulty with model capabilities and yields steady scientific-reasoning performance growth.Figure 5 summarizes the curriculum’s changing average difficulty and corresponding performance dynamics.
  • Conclusion: Dr. SCI delivers substantial gains from a compact 4B backbone and consistently surpasses strong post-trained baselines, including much larger and proprietary models.The authors identify principled data processing and correctness-aware rewards as critical for advancing open-ended scientific reasoning.

Impact Statement

The paper targets more reliable scientific reasoning post-training through structured data and reward design, while recognizing deployment risks and broader limitations. Its resource is intended to support RL beyond strictly verifiable settings, but responsible use still requires safeguards and human involvement.

  • Impact Statement: The work may support education, scientific research assistance, and other knowledge-intensive applications requiring structured reasoning and faithful explanations.These are described as potential downstream benefits rather than demonstrated deployment outcomes.
  • Impact Statement: The authors state that the work does not solve broader challenges involving factuality, misuse, or overreliance on automated scientific advice.Models could still generate plausible-sounding but incorrect explanations without safeguards or human oversight.
  • Impact Statement: Responsible deployment requires transparency about model limitations and continued human involvement in high-stakes scientific decision-making.The methods operate at model-training level and the dataset uses publicly available sources.
  • Impact Statement: Existing science resources differ in supervision and often lack standardized verification signals for stable reinforcement learning across heterogeneous questions.SFT-oriented corpora provide teacher responses, while larger reference-answer mixtures may lack quality control or complete RL metadata.
  • Impact Statement: Dr. SCI provides 1M challenging scientific questions with verifiable/open-ended splits, difficulty annotations, and fine-grained rubrics for RL-ready evaluation.The dataset is designed to support stable RL beyond strictly rule-verifiable settings.

A.2. Scientific Reasoning Post-training

Scientific RL has expanded from rule-based rewards toward model-based and rubric-based verification, but open-ended scientific answers remain vulnerable to noisy rewards and reward hacking. Dr. SCI responds by jointly redesigning data, curriculum, and reward design for verifiable and open-ended questions.

  • A.2. Scientific Reasoning Post-training: Rule-based rewards work when scientific correctness is straightforward, but struggle with broader reasoning tasks involving less structured answers.Early science RL often uses multiple-choice questions where correctness reduces to selecting an option.
  • A.2. Scientific Reasoning Post-training: Model-based rewards can be exploited by policies because the reward model becomes an optimization target, producing reward hacking and spurious reward correlations.These vulnerabilities limit the reliability of learned verification for scientific post-training.
  • A.2. Scientific Reasoning Post-training: Likelihood-based reference-answer rewards remain noisy for open-ended science because long natural-language explanations are not well represented by a single reference likelihood.VeriFree removes a separate verifier but retains this reward-signal limitation.
  • A.2. Scientific Reasoning Post-training: Rubric-based RL decomposes answer quality into structured criteria, yielding more interpretable reward signals for open-ended tasks.Prior work reports gains over LLM-as-judge baselines in science and medical settings.
  • A.2. Scientific Reasoning Post-training: Dr. SCI jointly redesigns data, curriculum, and reward to enable stable optimization across verifiable and open-ended scientific questions.Its supervision combines fine-grained rubrics with explicit answer checks.

B. Further Implementation Details

The implementation uses a simple instruction template across training and evaluation, while additional SFT and RL hyperparameters are documented in tables. The template requests step-by-step solutions and a standardized final-answer format.

  • B. Further Implementation Details: Additional implementation details include hyperparameters for coverage-inspired SFT, standard SFT baselines, and RL training.These settings are referenced in Tables 5 and 6.
  • B. Further Implementation Details: The same simple instruction format is used for all training data and evaluation questions.The template includes subject and question placeholders.
  • B. Further Implementation Details: The instruction asks the model to solve each subject problem step by step.The subject defaults to science when no more specific subject is supplied.
  • B. Further Implementation Details: Responses must end with the standardized form `The final answer is: \textbackslash boxed\{ANSWER\}`.This format explicitly separates the final answer from the preceding solution.

C. Further Experiment Results

Further experiments attribute performance gains to both Exploration-Expanding SFT and the subsequent RL stage rather than solely to teacher-model distillation. The reported improvements are supported by checkpoint comparisons, while RL hyperparameters are listed separately.

  • C. Further Experiment Results: The reported growth is presented as evidence that Dr. SCI performance is not solely due to distillation from strong teacher models.The comparison includes Dr. SCI-4B-think, Dr. SCI-4B-instruct, and representative baselines.
  • C. Further Experiment Results: RL hyperparameters are provided in Table 6 for the reported training configuration.The passage identifies the table but does not enumerate individual settings.
  • C. Further Experiment Results: 16.0 average performance growth for thinking mode and 9.7 for instruct mode is achieved by the Exploration-Expanding SFT checkpoint over the base model.The subsequent RL stage further improves performance beyond the strongest baselines.

D. GPQA-General Construction

GPQA-general is constructed by converting GPQA-diamond into an open-ended benchmark while preserving reliable reference-answer criteria. GPT-4o classifies question types, checks correctness criteria, and rewrites the questions.

  • The original multiple-choice questions are then rewritten into open-ended format to create GPQA-general.
  • GPT-4o classifies GPQA-diamond questions as Type 1 with one correct answer or Type 2 where answers satisfying criteria are correct.
  • For Type 1 questions, the reference answer is the correct choice; for Type 2 questions, it is the criterion extracted from the explanation.
  • GPT-4o double-checks whether each correct choice satisfies the extracted Type 2 criterion, producing reliable reference answers.

E. Examples of Dr. SCI dataset

Dr. SCI examples illustrate its two verification settings: deterministic checking for verifiable questions and rubric-based evaluation for open-ended questions. The examples span physics and evolutionary biology.

  • Dr. SCI contains reference answers, subject categories, difficulty annotations, verification splits, and fine-grained rubrics for approximately one million scientific reasoning questions.
  • Verifiable data: The verifiable example asks for black-hole accretion efficiency and provides a numerical reference answer of ε ∼ 1/12.
  • Open-ended data: The open-ended archosaur example requires explanations of plesiomorphies, four-chambered hearts, pneumatic bones, comparative distinctions, physiological effects, and evolutionary context.
  • Open-ended data: Its rubric marks definitions, anatomical features, comparisons, physiological advantages, trade-offs, evolutionary links, and mechanistic accuracy as essential or important criteria.

F. Qualitative Examples of Rewarding Open-ended Questions

Qualitative reward examples expose weaknesses in generic model-based and decomposed rubric rewards. SciRubric-Guided RL addresses these issues by emphasizing final-answer correctness while retaining detailed feedback.

  • GenRM failures: GenRM can accept meaningless placeholders, reject sufficiently correct answers, and reward incorrect answers, revealing unreliable reward assignment.
  • RaR failure: RaR’s weighted rubric scoring encourages partial-credit gaming, producing unnecessarily long responses and weaker advantage signals.
  • Reward comparison: SciRubric-Guided RL produces larger reward divergence and more effective advantage signals, supporting stable reinforcement learning and stronger overall performance.
  • SciRubric-Guided RL: SciRubric-Guided RL combines fine-grained rubric scores with a dominant final-answer correctness term to tie rewards to problem solving rather than verbosity.
Loading 2602.08321v2…