PaperlayerView exact revision on arXiv

Source-linked AI summary

SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation

Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, Ian Foster

arXiv:2405.09939v2cs.CLcs.AI

TL;DR

Existing science QA datasets often rely on multiple-choice questions, textbook content, or costly expert-driven construction. SciQAG generates and evaluates open-ended science QA from scientific papers, and fine-tuning on its dataset improves performance on open-ended QA and scientific tasks.

  • Problem

    Existing science QA datasets often lack question variety, draw mainly on textbooks, and can be costly to construct, limiting assessment of research-level science QA.

    Source: §1 · p. 2

  • Method

    SciQAG uses an LLM-based generator and evaluator to create and filter open-ended science QA pairs from scientific papers.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Fine-tuning with SciQAG science QA data significantly improves models’ ability to understand and apply scientific knowledge, including on scientific tasks.

    Source: §1 · p. 2, §7 · pp. 8–9

  • Takeaways & Limitations

    SciQAG-24D provides training and test data for fine-tuning and evaluating LLMs on open-ended, closed-book science QA.

    Source: §7 · p. 8

    Questions may retain implicit dependencies on source-specific terminology, hallucinations remain possible, and RACAR performs better with commercial models and separate dimension scoring at greater resource cost.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We introduce SciQAG, a novel framework for automatically generating high-quality science question-answer pairs from a large corpus of scientific literature based on large language models (LLMs). SciQAG consists of a QA generator and a QA evaluator, which work together to extract diverse and research-level questions and answers from scientific papers. Utilizing this framework, we construct a large-scale, high-quality, open-ended science QA dataset containing 188,042 QA pairs extracted from 22,743 scientific papers across 24 scientific domains. We also introduce SciQAG-24D, a new benchmark task designed to evaluate the science question-answering ability of LLMs. Extensive experiments demonstrate that fine-tuning LLMs on the SciQAG dataset significantly improves their performance on both open-ended question answering and scientific tasks. To foster research and collaboration, we make the datasets, models, and evaluation codes publicly available, contributing to the advancement of science question answering and developing more interpretable and reasoning-capable AI systems.

1 Introduction

Existing science QA datasets often rely on multiple-choice questions and textbook content, while expert- or crowd-driven construction can be costly. SciQAG generates and evaluates open-ended science QA from scientific articles, and its dataset supports fine-tuning that improves performance on scientific tasks.

1 Introduction

  • Existing science QA datasets often use multiple-choice questions and textbook content, and expert- or crowd-driven construction can be costly.The paper focuses on open-ended, closed-book science QA, where questions have no fixed answers and no external context is provided.Source: §1 · p. 2
  • SciQAG combines an LLM-based QA generator with an evaluator that assesses or filters generated pairs from scientific papers.The resulting SciQAG-24D dataset contains 188,042 QA pairs.Source: §1 · p. 2
  • Fine-tuning with science QA data significantly improves models’ ability to understand and apply scientific knowledge, including on scientific tasks.The paper also explores zero-shot performance on SciQAG-24D and fine-tuning effects on unseen QA pairs, SciQ, and scientific tasks.Source: §1 · p. 2

2 Related Work

Prior science QA benchmarks are often manually collected, elementary-level, and multiple-choice, while automatic QA generation and evaluation methods retain constraints such as predefined answers, knowledge-graph dependence, or reference-text requirements. The paper positions its approach as generating open-ended science QA from scientific texts without manually curated answers.

2.1 Science QA benchmarks

  • Popular science QA benchmarks often use manually collected, elementary-level multiple-choice questions, while open-ended long-form datasets cover relatively narrow domains.The cited examples include a biomedical dataset restricted to yes, no, or maybe answers.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 Automatic QA generation

  • Many automatic question-generation methods require predefined answers or an existing knowledge graph, whereas this paper aims to generate open-ended QA from scientific texts without manually curated answers.Earlier methods include syntactic transformation templates and systems adaptable to question types and domains.Source: §2.2 · p. 3

2.3 Evaluation metrics for open-ended QA

  • Open-ended QA evaluation is difficult because assessments are subjective, and many text metrics require references; this paper adopts a multi-perspective LLM-based approach.The cited reference-based methods include BLEU, ROUGE, METEOR, F1, BERTScore, sentence mover similarity, and MAUVE.Source: §2.3 · p. 3

3 SciQAG framework

SciQAG trains a generator to turn scientific papers into closed-book QA pairs and uses an evaluator to filter low-quality pairs. The generator uses keyword-guided question creation, while RACAR evaluates QA quality across five dimensions.

3 SciQAG framework

  • SciQAG generates closed-book QA pairs from full-text scientific papers and uses an evaluator to filter pairs that fail quality standards.The filtering standard can be a score threshold or selection of the top k pairs by overall ranking.Source: §3 · p. 3, §3.2 · p. 4

3.1 QA generator

  • For each input text, the generator produces 15 keywords and then generates 10 question-answer pairs focused on those keywords.Prompts instruct generation to avoid depending on or referring to the source paper, since the resulting dataset is intended for closed-book use.Source: §3.1 · pp. 3–4

3.2 QA Evaluator

  • RACAR evaluates relevance, agnosticism, completeness, accuracy, and reasonableness; GPT-4 scores QA pairs on each dimension from 1 to 5.The dimensions assess source relevance, context independence, answer coverage, factual support, and internal logic.Source: §3.2 · p. 4
  • Two domain experts scored 100 QA pairs with RACAR, and Figure 2 reports average Spearman and Pearson correlations with GPT-4 scores.The supplied passages identify the comparison but do not state the correlation values.Source: §3.2 · p. 4, Figure 2 · p. 4

4 SciQAG-24D Dataset

SciQAG-24D is built from selected scientific papers and generated QA pairs, then assessed for answer coverage, numerical sourcing, and question diversity. Its filtered training and test sets span 24 scientific domains.

4.1 Paper collection

  • The source collection contains 22,743 English full-text papers from 24 selected Web of Science categories in physical science disciplines.Figure 3 shows the category proportions in the training and test sets.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2 Using the SciQAG Framework

  • A generator fine-tuned on 4,260 GPT-4 seed QA pairs produced 227,430 pairs, after which GPT-4 scoring and rule-based filtering formed the test and training sets.The seed pairs came from 426 randomly selected papers; test pairs with any RACAR score below 3 were removed, and training pairs mentioning “this paper” or “this study” were excluded.Source: §4.2 · p. 5

4.3 Statistics of SciQAG-24D

  • After filtering, SciQAG-24D contains 179,511 training QA pairs from 21,529 papers and 8,531 test QA pairs from 1,199 papers.The original collection contains 22,743 papers, with 21,543 assigned to training and 1,200 to testing.Source: §4.3 · p. 5
  • 99.15% of numeric values found in generated answers also occur in the source text, while 27.29% of answers contain numerical information.The authors treat numeric values absent from the source as uncertain in authenticity.Source: §4.3 · p. 6
  • 87.29% of 2,250 question-pair similarity scores fell below 0.3, 2.27% exceeded 0.7, and the average was 0.186.Figure 4 visualizes similarities among sets of 10 questions, with higher scores shown in redder cells and scores above 0.7 marked by red dots.Source: §4.3 · p. 6, Figure 4 · p. 6
  • Answers had an average coverage rate of 78.6% across 50 sampled papers, measuring whether source-relevant sentences occur across the paper’s ten chunks.The method selects sentences among the top 15% most similar to any answer, using e5-mistral-7b-instruct contextual embeddings for similarity.Source: §4.3 · p. 6

5 Experiments

The experiments compare zero-shot science QA performance across five models and test fine-tuning LLaMA1 on SciQAG-24D. They evaluate generated answers with CAR and use task-specific metrics for other benchmarks.

5.1 Experimental setup

  • The setup compares five models on 1,000 test questions and evaluates SciQAG-24D-fine-tuned LLaMA1 against the original model and on other scientific tasks.The zero-shot models are LLaMA1, LLaMA2-chat, GPT-3.5, GPT-4, and Claude3; additional evaluations include SciQ and tasks spanning classification, regression, and transformation.Source: §5.1 · p. 7

5.2 Implementation details

  • LLaMA1 was fine-tuned for two epochs with batch size 1, sequence length 512, learning rate 2e-5, and DeepSpeed Stage 3 across eight AMD GPUs.Inference used temperature 0.8 and top_p 0.75.Source: §5.2 · p. 7

5.3 Evaluation metrics

  • Answer quality in zero-shot experiments uses CAR; SciQ uses accuracy, classification uses F1, regression uses MAE, and transformation uses KL divergence.CAR is a modified RACAR framework that excludes Relevance and Agnosticism.Source: §5.3 · p. 7

6 Results and Analysis

Zero-shot results on SciQAG-24D favor commercial models, while fine-tuning LLaMA1 on SciQAG-24D improves its benchmark, SciQ, and scientific-task performance. The reported gains include better QA scores, fewer unparseable MCQ outputs, and improvements on scientific prediction tasks.

6.1 Zero-shot Evaluation

  • GPT-4 led in Completeness (4.9) and Reasonableness (4.9896), while Claude3 led in Accuracy (4.95) on SciQAG-24D.GPT-3.5 scored closely behind them, and LLaMA1 scored lowest across all three dimensions.Source: §6.1 · p. 7
  • LLaMA2-chat improved substantially over LLaMA1 across all three SciQAG-24D metrics, while commercial models had superior zero-shot performance overall.Table 2 summarizes zero-shot performance on the SciQAG-24D test and includes LLaMA1-QA in its last line for comparison.Source: §6.1 · p. 7, Table 2 · p. 8

6.2 Impact of Fine-tuning

  • Fine-tuning improved LLaMA1-QA over LLaMA1 by over 13% in Completeness and over 30% in Accuracy and Reasonableness on SciQAG-24D.The authors also report a slight improvement on SciQ (+1%) and a drop in unparseable outputs from 4.1% to 1.7%.Source: §6.2 · p. 8
  • The Steels regression task showed the largest reported scientific-task gain, with MAE falling from 463.96 to 185.32.LLaMA1-QA also surpassed AMMExpress v2020 (0.4161) on Bandgap and outperformed the Diversity deep-learning baseline (0.3198).Source: §6.2 · p. 8, Table 3 · p. 8

7 Conclusion

The paper presents SciQAG and RACAR for generating, evaluating, and filtering science QA pairs from scientific literature. The resulting SciQAG-24D dataset supports open-ended, closed-book QA evaluation, and fine-tuning on its training set improves performance across several reported tasks.

7 Conclusion

  • SciQAG generates QA pairs from scientific literature, while RACAR metrics evaluate and filter them.The framework produced a dataset of 188,042 QA pairs; fine-tuning LLaMA1 on its training set improved performance on unseen SciQAG-24D questions, SciQ, and scientific tasks.Source: §7 · pp. 8–9
  • SciQAG-24D provides a test set for open-ended, closed-book science QA and a training set for fine-tuning LLMs with scientific knowledge.The authors report LLaMA1-QA results comparable to or surpassing machine-learning baselines on some scientific tasks.Source: §7 · pp. 8–9

Limitations

The authors identify limitations in question dependencies, answer factuality, RACAR evaluation costs, and SciQAG-24D's domain coverage.

Limitations

  • Questions may implicitly depend on author-specific terminology or names, and even a perfect RACAR score does not guarantee factual answers.The authors also report that hallucinations remain possible and that RACAR performs better with commercial models and with dimensions assessed individually, at greater resource cost.Source: Limitations · p. 9
  • SciQAG-24D covers a limited range of scientific domains, which the authors say should be expanded through community contributions.They also call for a wider variety of scientific disciplines and question types.Source: Limitations · p. 9

Ethics Statement

The authors report no major ethical concerns. The listed Web of Science categories span materials science, physics, chemistry, and energy.

Ethics Statement

A Complete list of 24 WoS categories and original distribution

  • The listed 24 Web of Science categories include materials science, physics, chemistry, and energy and fuels.The list includes multidisciplinary, applied, and specialized categories within those fields.Source: A Complete list of 24 WoS categories… · p. 14

B Statistics of SciQAG-24D dataset

This section presents SciQAG-24D statistics and documents the prompts used to generate and evaluate question–answer pairs. The evaluation prompts assess individual QA quality and distinguish question similarity by intent.

B Statistics of SciQAG-24D dataset

C Prompts for QA generator

  • The generator prompt extracts 15 paper-specific keywords, then requests 10 diverse, self-contained Q&A pairs with detailed, concise answers and no simple definitional questions.It instructs the model not to refer to figures, tables, or external sources.Source: C Prompts for QA generator · p. 14

D GPT-4 prompts for RACAR metric

  • RACAR prompts assess relevance, context independence, answer completeness, answer accuracy, and statement reasonableness on 1–3 scales.The prompts define low scores for irrelevant or figure-dependent questions and assess whether answers are supported by the paper.Source: D GPT-4 prompts for RACAR metric · p. 16

E Prompts for similarity of questions

  • The similarity prompt scores whether two questions seek the same information, distinguishing shared intent from related wording about opposite aspects.Its examples score questions about zeolite applications as 100 and questions about pros versus drawbacks as 0.Source: E Prompts for similarity of questions · pp. 16–17

F Self-questioning vs. SciQAG

SciQAG’s comparison with self-questioning highlights differences in question framing, answer specificity, and source coverage. The section also gives generated-question examples and visualizes how answers relate to source-paper sentences.

F Self-questioning vs. SciQAG

  • SciQAG prompts request 10 diverse, self-contained Q&A pairs based on extracted keywords, while the self-questioning prompt asks five questions directed to a paragraph.The section includes an example completion with questions about experimental models, metabolites, and observed changes.Source: F Self-questioning vs. SciQAG · pp. 17–18
  • SciQAG generates diverse, context-independent questions and specific answers, avoiding self-referential wording but potentially reducing compatibility with open-book QA.Answers may include precise details such as chemical elements and numerical values from source papers.Source: F Self-questioning vs. SciQAG · p. 19

G 10 generated questions example

  • The random Ivleva et al. (2009) example includes questions on rapid assays, antibody performance, extraction procedures, and country-specific mycotoxin limits..Source: G 10 generated questions example · p. 19

H Pairwise visualization

  • Figure 6 shows top answer-to-sentence similarity matches and heatmaps; the authors report extensive source-paper coverage and diverse answer sources.Some sentences support multiple answers, but sources are generally distinct across answers.Source: H Pairwise visualization · pp. 19–20

I Details of generating SciQAG-24D dataset

This section gives dataset-generation settings, describes the CAR answer-evaluation criteria, and identifies additional dataset and task summaries. It also shows similarity visualizations for two scientific articles.

I Details of generating SciQAG-24D dataset

  • Dataset generation and RACAR evaluation use GPT-4; Vicuna-7B is fine-tuned for five epochs on eight A800 GPUs with a maximum sequence length of 16,000.The stated learning rate is 2e-5, with inference temperature 0.8 and top_p 0.75.Source: I Details of generating SciQAG-24D… · p. 20

J Prompts for CAR metric

  • The CAR prompt scores answer accuracy, completeness, and reasonableness from 1 to 5, with comments explaining each score.The prompt describes accuracy in relation to the question and completeness in terms of addressing key points.Source: J Prompts for CAR metric · p. 20

K Summary of datasets in scientific tasks

  • Table 5 summarizes datasets and the tasks used to derive instructions, while Figure 6 displays sentence-similarity matches and heatmaps for two papers.The caption identifies classification, regression, and transformation tasks.Source: Figure 6 · p. 21, Table 5 · p. 22
Loading 2405.09939v2…