PaperlayerView exact revision on arXiv

Source-linked AI summary

SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models

Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, Jie Tang

arXiv:2401.07950v3cs.CL

TL;DR

LLMs struggle with scientific concepts, symbolic equations, and numerical calculations, while scientific instruction data is scarce and often lacks reasoning traces. SciInstruct uses self-reflective LLM annotation to build a diverse science and mathematics dataset. Fine-tuning improves scientific and mathematical benchmark performance without sacrificing general-language understanding.

  • Problem

    LLMs struggle with intricate scientific concepts, symbolic equation derivation, and advanced numerical calculations, while scientific instruction data is scarce and often lacks detailed reasoning steps.

    Source: §1 · pp. 1–2

  • Method

    SciInstruct combines physics, chemistry, mathematics, and formal proofs with an LLM framework that generates, critiques, and revises reasoning traces.

    Source: §1 · pp. 2–3

  • Results

    Fine-tuning improves scientific and mathematical benchmark performance without sacrificing general-language understanding; ChatGLM gains are 4.87% (6B) and 2.67% (32B).

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings underscore the significance of diverse training data, LLM self-annotation, and correctness for enhancing general reasoning in hard domains such as science.

    Source: §5 · p. 9

    The approximately 254k dataset and experiments at 6∼8B and 32B parameters leave performance constrained by dataset and model scale.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise in assisting scientific discovery. However, such applications are currently limited by LLMs' deficiencies in understanding intricate scientific concepts, deriving symbolic equations, and solving advanced numerical calculations. To bridge these gaps, we introduce SciInstruct, a suite of scientific instructions for training scientific language models capable of college-level scientific reasoning. Central to our approach is a novel self-reflective instruction annotation framework to address the data scarcity challenge in the science domain. This framework leverages existing LLMs to generate step-by-step reasoning for unlabelled scientific questions, followed by a process of self-reflective critic-and-revise. Applying this framework, we curated a diverse and high-quality dataset encompassing physics, chemistry, math, and formal proofs. We analyze the curated SciInstruct from multiple interesting perspectives (e.g., domain, scale, source, question type, answer length, etc.). To verify the effectiveness of SciInstruct, we fine-tuned different language models with SciInstruct, i.e., ChatGLM3 (6B and 32B), Llama3-8B-Instruct, and Mistral-7B: MetaMath, enhancing their scientific and mathematical reasoning capabilities, without sacrificing the language understanding capabilities of the base model. We release all codes and SciInstruct at https://github.com/THUDM/SciGLM.

1 Introduction

LLMs struggle with college-level science questions, while accessible scientific instruction data often lacks detailed reasoning traces. SciInstruct combines diverse scientific data with self-reflective annotation and improves benchmark performance without sacrificing general-language understanding.

1 Introduction

  • Scientific questions demand concept selection, equation deduction, and rigorous numerical computation, while available science instructions are comparatively scarce and often lack reasoning steps.The introduction notes that GPT-3.5 and GPT-4 achieved 28.52% accuracy on some college-level textbook questions.Source: §1 · pp. 1–2
  • 4.87% improvement over the 6B model and 2.67% improvement over the 32B model follow SciInstruct tuning, with gains on scientific benchmarks.The reported comparison accompanies Figure 3's average accuracies across CEval-Sci, SciEval, SciBench, MATH, and SAT-Math.Source: Figure 3 · p. 2, §1 · p. 2
  • The dataset combines physics and chemistry questions, advanced mathematical calculations, and Lean proofs to cover distinct scientific problem-solving skills.The authors state that this mixture is intended to avoid overfitting to one subject and support general skills across scientific tasks.Source: §1 · pp. 2–3
  • SciInstruct uses LLMs to generate, critique, and revise reasoning traces for scientific questions, with answer checking and minimal human intervention.The framework addresses incorrect answer predictions through error identification and revision until the answer is correct.Source: §1 · p. 2
  • Fine-tuning the evaluated models improves scientific and mathematical benchmarks without sacrificing general language understanding.The evaluated models include ChatGLM3 (6B and 32B), Llama3-8B-Instruct, and Mistral-7B: MetaMATH.Source: §1 · pp. 2–3

2 SciInstruct

SciInstruct combines broad scientific and mathematical data with self-reflective reasoning annotation and quality filtering. The resulting dataset is used to fine-tune several language models.

2 SciInstruct

  • SciInstruct’s 254,051 verified instructions were assembled through self-reflective annotation and noisy-instruction filtering.The framework targets a shortage of scientific instructions with detailed reasoning traces.Source: §2 · p. 3, §2.3 · p. 6

2.1 Diverse Instruction Collection

  • The collection covers physics, chemistry, mathematics, and Lean formal proofs, drawing questions from textbooks, pedagogical materials, problem sets, and public Q&A sites.The physics collection includes college-level topics, while Lean data was formatted around successive deduction.Source: §2.1 · pp. 3–4
  • 257,143 raw questions were gathered, most without reasoning traces; translating some Chinese questions into English yielded correct solutions where the original attempts had not.The authors attribute this improvement as likely related to the higher-quality English corpus used in pretraining.Source: §2.1 · p. 4

2.2 Self-Reflective Instruction Annotation

  • GPT-4 first generates and labels solutions, then retries incorrect ones with reflective prompting and a real-answer hint, filtering solutions after each stage.The first filtering stage retained 19,824 correct solutions; the authors use an outcome reward model rather than a costly process reward model.Source: §2.2 · p. 5

2.3 Filtering Out Low-Quality Instruction

  • Because correct answers can still contain faulty reasoning and OCR can corrupt source material, a classifier ranks instruction reliability and filters low-quality data.The classifier uses ChatGLM3-6B-Base features; the filter ablation excludes the lowest-scored 10% and reports average weighted accuracy.Source: §2.3 · pp. 5–6, Table 2 · p. 5

2.4 Instruction-Tuning with SciInstruct

  • Fine-tuning ChatGLM3 (6B and 32B), Llama3-8B-Instruct, and Mistral-7B: MetaMATH on SciInstruct consistently outperforms the base models across parameter scales.The models were trained for two epochs using a learning rate of 3e-6.Source: Table 4 · p. 6, §2.4 · p. 6

3 Benchmark on SciInstruct

The authors evaluate SciInstruct-tuned models on scientific, mathematical, and general tasks. They report improved scientific reasoning and transfer to mathematics across model scales, without sacrificing most general language performance.

3.1 Experimental Setup

  • Evaluation covers out-of-domain science and math benchmarks, MMLU and CEval knowledge tasks, and MBPP code generation, using accuracy except pass@1 for code.Experiments use chain-of-thought settings and standardized baseline comparisons.Source: §3.1 · pp. 6–7
  • The authors report no overlap between SciInstruct’s training set and the evaluation benchmarks.Source: §3.1 · p. 7

3.2 Main Results and Analysis

  • Fine-tuning on SciInstruct improves scientific reasoning and transfers to mathematical tasks across 6B and 32B models.The reported scientific benchmarks include CEval-Sci, SciEval, SciBench, and MMLU-Sci.Source: §3.2 · p. 8
  • SciGLM retains general language understanding, slightly improves on MMLU and CEval, and is overall consistent on MBPP despite a small performance drop.The authors report this pattern for SciGLM fine-tuned at different parameter scales.Source: §3.2 · p. 8

4 SciInstruct Analysis

The analyses examine subject mixture, data scale, and repeated-sample performance. They report cross-domain benefits, gains from initial data additions with a later threshold, and maintained sample diversity after fine-tuning.

General language tasks

  • Fine-tuning preserves most language-task performance, with a small drop on code generation.Source: Table 6 · p. 8

Influence of Data Mixture

  • Physics and Chemistry data aid CEval-Math, while Math and Formal Proof improve SciBench in the leave-one-out subject analysis.The authors interpret this cross-task pattern as evidence that the mixture supports general reasoning rather than merely fitting particular task distributions.Source: §4 · p. 8, Figure 6 · p. 8

Influence of Data Scaling

  • 10% data augmentation improves performance, with no significant further gains until above 50%; the authors hypothesize early gains reflect basic reasoning and task formatting.They suggest equation deduction requires more data and that improved data quality could lower the requirement.Source: §4 · p. 8, Figure 7 · p. 9

Pass@K Analysis on Sample Diversity

  • K=25 on SciBench and K=3 on SciEval yield performance comparable to GPT-4, while fine-tuning does not influence sample diversity.The evaluation generated 30 solutions per question at temperature 1.0; Figure 8 reports improved performance even at large K.Source: §4 · p. 9, Figure 8 · p. 9

5 Conclusion

The paper presents SciInstruct as a self-instructive framework and dataset for scientific language models. Fine-tuning with it improves scientific and mathematical benchmarks, and the authors emphasize diverse data and self-annotation for general reasoning.

5 Conclusion

  • The self-instructive annotation framework produces SciInstruct to enrich LLMs’ scientific knowledge.Source: §5 · p. 9
  • Models trained with SciInstruct improve many scientific and mathematical benchmarks and outperform many state-of-the-art LLMs with an order of magnitude more parameters.Source: §5 · p. 9
  • The authors underscore diverse training data, LLM self-annotation, and correctness as important to general reasoning in hard domains such as science.Source: §5 · p. 9

Limitation

The limitations concern the scale of the dataset and models used, as well as the scope for further gains from instruction-quality classification.

Scale of Dataset and Model

  • Experiments use an approximately 254k-question dataset and models from 6–8B and 32B parameters, so the reported performance is constrained by model scale.The authors identify larger datasets and models as directions for exploring further performance improvements.Source: Limitation · p. 9

Using Data Classifier to Enhance the Generation of Models

  • The instruction-quality classifier improves performance, but the authors expect it could provide greater benefits through data bootstrapping.Bootstrapping data is proposed as a way to improve the base model's ability.Source: Limitation · p. 9

Broader Impact

The paper presents self-reflective instruction preparation as a source of insight for training-data construction, while acknowledging the relatively small scale of its dataset and models. It also situates the work among scientific evaluation, continued pre-training, and dataset-specific fine-tuning approaches.

Broader Impact

  • Collecting diverse instructions, annotating them self-reflectively, and filtering low-quality instructions provide researchers insights for preparing training datasets.The paper describes its aim as improving LLMs' scientific reasoning for college-level questions.Source: Broader Impact · p. 9
  • The authors identify the relatively small training dataset and model scale as a drawback and propose bootstrapping a larger dataset.They state that they believe the benefits of their data-generation approach outweigh this downside.Source: Broader Impact · p. 9

A.1 Related Works

  • SciBench [7] and GPQA [8] evaluate college- and graduate-level scientific reasoning, while SciEval [6] addresses data leakage and subjective question/answer evaluation issues.These are presented as high-level evaluation approaches.Source: §A.1 · p. 16
  • Galactica [1] and MINERVA [42] continue pre-training on web corpora that include science- or math-related text, an approach described as computationally expensive.The passage also says this approach contributes open-source models for the scientific community.Source: §A.1 · p. 16
  • RFT [38], WizardMath [43], MAmmoTH [10], and MetaMath [11] fine-tune on math datasets, with MAmmoTH also reported to generalize beyond in-domain tasks.The passage characterizes this series of methods as focused mainly on mathematical reasoning.Source: §A.1 · p. 16

A.2 SciInstruct Analysis

The appendix describes SciInstruct analyses, instruction formats for scientific problems and formal math, and the GPT-4 labeling process used in classifier-related work.

A.2 SciInstruct Analysis

  • Figure 9 presents the proportions of question types across all subjects and within each subject.The caption specifies these distributions but does not report their values.Source: Figure 9 · p. 16
  • Figure 10 presents question counts by question length alongside the average answer length for each question length.The caption describes the plotted quantities without giving numerical values.Source: Figure 10 · p. 17

A.3 Instruction Examples

  • The physics, chemistry, and math template moves from identifying relevant concepts to step-by-step reasoning and a summarized answer.The generic example uses placeholders for the problem, analysis, steps, and final answer.Source: §A.3 · p. 17
  • The appendix also gives a Lean formal-math example using the theorem gcd_self and a proof script.The example states gcd n n = n and gives a proof using cases and simp.Source: §A.3 · p. 17

A.4 GPT Labeling

  • GPT-4 labeling takes a prompt, question, solution, and real answer as inputs to produce a final label; the authors use an outcome reward model as a basic implementation.They note that accurate assessment of reasoning steps, such as with a process reward model, typically requires costly human annotations for complex scientific reasoning.Source: §A.4 · p. 17

A.5 Dataset of Instruction-quality Classifier

  • Table 7 is described as reporting the data statistics used to train the instruction-quality classifier.Its caption characterizes the data as composed from limited labeled examples to train a classifier and filter noisy instructions.Source: §A.5 · p. 17

A.6 Detailed Error Analysis

The supplied appendix passages point to detailed error examples, a GPT-4 labeling workflow, and classifier-training data assembled from limited labeled examples.

A.6 Detailed Error Analysis

  • Figure 12 contains detailed error examples, which the paper says are summarized in Section 2.3.The passage does not state the error categories or outcomes.Source: §A.6 · p. 17
  • Figure 11 shows GPT-4 labeling from a prompt, question, solution, and real answer to a final label.The caption identifies the workflow inputs and output, but not its results.Source: Figure 11 · p. 18
  • Table 7 describes data composed from a limited set of labeled examples to train a classifier and filter noisy instructions.The passage does not provide the table's values or composition breakdown.Source: Table 7 · p. 18

A.7 Evaluation Tasks

The evaluation suite covers scientific and mathematical benchmarks, while baselines include general models, continued pre-training, dataset-specific tuning, and GPT-4.

A.7 Evaluation Tasks

  • Scientific evaluations include CEval-Hard, CEval-Sci, MMLU-Sci, and SciEval, with the scientific tasks spanning physical, chemical, and mathematical subjects.Table 8 also marks subject categories including physics, chemistry, and biology.Source: §A.7 · pp. 18, 20, Table 8 · p. 18
  • Mathematical evaluations include MATH, Mathematics, SAT-Math, MMLU-Math, and CEval-MATH, covering topics such as algebra, probability, number theory, and calculus.The supplied descriptions specify the component subjects for Mathematics, SAT-Math, and MMLU-Math.Source: §A.7 · p. 20

A.8 Evaluation Baselines

  • Baselines comprise GPT-4 with CoT prompting, ChatGLM and LLaMA base models, Galactica, and dataset-specific tuning methods WizardMath, MAmmoTH-CoT, and MetaMath.The dataset-specific methods adapt to MATH; Galactica is included although its continued pre-training does not necessarily include college-level scientific reasoning.Source: §A.8 · p. 20

A.9 Detailed Experimental Results

The appendix reports domain-wise results and illustrative outputs, gives examples of SciInstruct data, and situates the dataset among prior reasoning and instruction-tuning work.

A.9 Detailed Experimental Results

  • SciGLM consistently improves ChatGLM3-Base across scientific tasks without sacrificing general language understanding, with detailed physical and chemical subject results reported separately.Figure 13 gives the overall comparison; Tables 9 and 10 provide physical- and chemical-subject results.Source: Figure 13 · p. 20, §A.9 · p. 20, Table 9 · p. 20

A.10 Output Examples of SciGLM

  • A SciBench quantum-chemistry example illustrates SciGLM (32B) analyzing required knowledge before solving step by step with correct formulas and calculations.The appendix also presents SciBench output examples from SciGLM (32B) and ChatGLM3-32B-Base.Source: Figure 14 · p. 21, §A.10 · p. 21

A.11 Examples of SciInstruct

  • SciInstruct examples cover math, physics, chemistry, and formal proofs, alongside statistics and quantum-chemistry SciBench examples.The listed figures include a baseball-series math problem and a hydrogen-atom uncertainty problem.Source: §A.11 · pp. 21–22, Figure 15 · p. 22, Figure 16 · p. 22

A.12 Scientific Reasoning Datasets

  • SciInstruct targets more complex and diverse college-level math problems than earlier datasets focused mainly on elementary through high-school mathematics.The related-work discussion also describes SciBench, SciEval, and GPQA as scientific reasoning or evaluation datasets.Source: §A.12 · p. 27

A.13 General Reasoning with LLMs

  • Prior work reports reasoning improvements from Chain-of-Thought and Tree-of-Thought prompting, while other approaches use stepwise processes, tools, or programs as thought processes.The discussion names self-critic, self-eval, and plan-and-solve as methods for improving program-of-thought reasoning.Source: §A.13 · p. 27

A.14 Instruction Tuning in LLMs

  • Instruction tuning progressed from human-annotated general-purpose datasets such as FLAN and T0 to larger and LLM-distilled datasets, while Platypus uses domain-specialized data.The discussion describes instruction tuning as aligning models to respond to human preferences and objectives.Source: §A.14 · p. 27

B Datasheet

The datasheet describes SciInstruct's purpose, composition, collection, release, maintenance, and uses, and records the authors' checklist responses.

B.1 Motivation

  • SciInstruct was created to address science-domain data scarcity and provide instructions for training models in college-level scientific reasoning.Source: §B.1 · p. 28

B.2 Distribution; B.3 Maintenance

  • The dataset is public, distributed through several hosting services, and released under CC BY 4.0; Zhipu AI and THUDM support and maintain it.The listed distribution channels include Hugging Face, Google Drive, Tsinghua Cloud, and GitHub for baseline code.Source: §B.2 · p. 28, §B.3 · p. 28

B.3 Maintenance; 4. Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete

  • The authors say they will update the dataset when necessary, publish errata if errors are found, and continue hosting older versions.Source: §B.3 · p. 28, §4 · pp. 28–29

7. If others want to extend/augment/build on/contribute to the dataset, is there a mechanisms

  • The authors direct contributors to submit extensions through GitHub pull requests.Source: §7 · p. 29

1. What do the instance that comprise the dataset represent (e.g., documents, photos, people,

  • SciInstruct contains 254,051 instructions: 123,869 physics and chemistry, 89,934 math, and 40,248 Lean formal-proof examples.Each instance includes a question, corresponding answer, and subject; all instances are used for training, with public benchmarks used as test sets.Source: §1 · p. 29

B.5 Collection Process

  • Questions come from textbooks, pedagogical materials, and problem sets; OCR extracts questions and final answers, an LLM API generates reasoning, and traces with incorrect predicted answers are filtered.The data-collection process involved LLM researchers at Tsinghua and Caltech.Source: §B.5 · p. 29

B.6 Uses

  • The dataset has been used to create code-formatted calculation data and fine-tune code-based language models; the authors describe its current composition as self-sufficient for training a scientific language model.Source: §B.6 · p. 30

Checklist

  • The checklist marks the dataset URL, training details, compute information, and discussion of limitations and potential negative societal impacts as provided.Source: Checklist · p. 31
Loading 2401.07950v3…