PaperlayerView exact revision on arXiv

Source-linked AI summary

NSF-SciFy: Mining the NSF Awards Database for Scientific Claims

Delip Rao, Weiqiu You, Eric Wong, Chris Callison-Burch

arXiv:2503.08600v3cs.CL

TL;DR

Scientific claim datasets have been limited, especially for claims and investigation proposals in grants. NSF-SciFy extracts these from NSF award abstracts at large scale and shows that fine-tuning improves claim and proposal extraction, often by over 100% relatively.

  • Problem

    Existing claim datasets derive mainly from scientific papers, leaving claims and investigation proposals in grants largely unstudied.

  • Method

    NSF-SciFy uses zero-shot prompting to jointly extract claims and investigation proposals from NSF award abstracts, producing large cross-disciplinary and focused subsets.

  • Results

    Relative performance gains often exceeded 100% for scientific claim and investigation proposal extraction, while non-technical abstract generation improved modestly.

  • Takeaways & Limitations

    NSF-SciFy supports large-scale claim verification, scientific discovery tracking, and meta-scientific analysis using claims at their origin.

  • Takeaways & Limitations

    The dataset excludes claims from unfunded proposals and international contexts.

Abstract

from arXiv · show

We introduce NSF-SciFy, a comprehensive dataset of scientific claims and investigation proposals extracted from National Science Foundation award abstracts. While previous scientific claim verification datasets have been limited in size and scope, NSF-SciFy represents a significant advance with 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines. We present two focused subsets: NSF-SciFy-MatSci with 114,000 claims from materials science awards, and NSF-SciFy-20K with 135,000 claims across five NSF directorates. Using zero-shot prompting, we develop a scalable approach for joint extraction of scientific claims and investigation proposals. We demonstrate the dataset's utility through three downstream tasks: non-technical abstract generation, claim extraction, and investigation proposal extraction. Fine-tuning language models on our dataset yields substantial improvements, with relative gains often exceeding 100%, particularly for claim and proposal extraction tasks. Our error analysis reveals that extracted claims exhibit high precision but lower recall, suggesting opportunities for further methodological refinement. NSF-SciFy enables new research directions in large-scale claim verification, scientific discovery tracking, and meta-scientific analysis. Code and data are available at https://github.com/darpa-scify/NSFSciFy.

1 Introduction

The rapid growth of scientific literature makes manual claim verification increasingly intractable, while existing datasets are limited in scale and scope. NSF-SciFy addresses this gap by extracting claims and investigation proposals from NSF award abstracts at unprecedented scale.

  • Motivation: 4% annual publication growth and a 17-year doubling time intensify the challenge of distinguishing substantiated scientific claims from spurious ones.Researchers, reviewers, and the public struggle to assess claims amid the expanding literature.
  • Motivation: Manual verification of the growing body of scientific claims has become intractable, with increasingly severe economic and societal consequences for unverified claims.The introduction illustrates this problem through controversies involving quantum supremacy, LK-99 superconductors, and microplastic leaches.
  • Prior limitations: Existing scientific claim datasets have largely focused on healthcare or modest-sized collections derived from scientific publications and fact-checking sources.SciFACT introduced scientific claim verification, but follow-up work remained concentrated in healthcare and limited in scale.
  • Contributions: NSF-SciFy introduces claims and investigation proposals extracted from NSF award abstracts, establishing grant proposals as a source for scientific claim extraction.NSF abstracts provide broad scientific coverage, expert-reviewed funded proposals, and publicly available source material.
  • Contributions: 2.8M claims from 400K NSF award abstracts make NSF-SciFy the largest scientific claim dataset to date.The dataset is intended to support robust scientific claim verification and analysis of scientific practice and evolving research questions.
  • Contributions: 114K extracted claims from 16K materials science abstracts form NSF-SciFy-MatSci, described as the first materials science claim dataset.The materials science subset provides a focused dataset alongside the broader NSF-SciFy resource.

2 Related Work

Prior scientific claim datasets largely draw on published literature, journalism, and fact-checking, whereas NSF-SciFy mines grant award abstracts at substantially larger scale and across all science and mathematics domains.

  • Scientific Claim Datasets: Scientific claim extraction and verification has grown as scientific literature expands, with prior work focusing mainly on published papers, fact-checking sites, and news articles.
  • Scientific Claim Datasets: Existing scientific claim datasets focus on published literature, whereas NSF-SciFy extracts claims from grant award abstracts.
  • Scientific Claim Datasets: 2.8 million claims in NSF-SciFy, 135,000 in NSF-SciFy-20K, and 114,000 in NSF-SciFy-MatSci establish a substantially larger contribution spanning all science and mathematics.
  • Meta Science and Social Science: Prior grant-data studies examined interdisciplinary grants, supported-paper impact, research funding, and team structure in meta-science and social-science contexts.

3 Building NSF-SCIFY

NSF-SciFY was built by parsing the NSF Awards database into 412,155 awards, with focused materials-science and cross-directorate subsets. A zero-shot Claude-3.5 prompting pipeline jointly extracts scientific claims and investigation proposals, improving claim relevance by reducing confusion between factual and forward-looking statements.

  • Dataset construction: 412,155 parseable awards from the NSF database, covering records from 1970 through September 2024, constitute NSF-SciFY.The source XML contained more than 0.5 million awards before parsing.
  • Dataset construction: 16,031 Division of Materials Research awards form NSF-SciFY-MatSci, while NSF-SciFY-20K contains 20,000 awards spanning five NSF directorates.The materials-science subset represents approximately 3.2% of the full NSF awards database.
  • Dataset construction: Each record typically includes award metadata, technical and non-technical abstracts, extracted claims and investigation proposals, and resulting publications when available.Non-technical abstracts occur in approximately 81% of awards; publication updates are primarily available from 2014 onward, with DOIs resolved to publication metadata.
  • Claim and proposal extraction: A zero-shot Claude-3.5 prompt extracts claims and forward-looking investigation proposals into a JSON object, using temperature zero for consistency.Claims are statements presented as true or assumed, whereas investigation proposals specify proposed research activities.
  • Claim and proposal extraction: Jointly extracting claims and investigation proposals maintained claim relevance by reducing confusion between proposed investigations and factual claims.When proposals were omitted, the model often mislabeled forward-looking investigation statements as factual claims.

4 Dataset Analysis

NSF-SciFy comprises 412,155 award abstracts and 2.8 million claims, with focused Materials Science and cross-directorate subsets. Analysis shows technical and non-technical abstracts differ substantially, claims emphasize applications and knowledge gaps, and claim extraction has high precision but lower recall.

  • Dataset Composition: 412,155 award abstracts spanning 1970–2024 contain 2.8 million scientific claims and corresponding investigation proposals.The full dataset covers NSF awards across science and mathematics disciplines.
  • Dataset Composition: The Materials Science subset contains 16,042 awards, 114K claims, 145K investigation proposals, and 2,953 awards with linked publications.Each award has technical and non-technical abstracts; linked awards represent 18.4% of the dataset and have 1–4 publications.
  • Dataset Composition: The NSF-SciFy-20K subset contains 20,001 awards stratified across five NSF directorates, with 135K claims and 139K investigation proposals.Each award includes a technical and non-technical abstract, averaging 7 ± 2 claims and 7 ± 2 proposals per abstract-pair.
  • Abstract Analysis: Only 202 (1.5%) of 13,025 technical/non-technical abstract pairs exceeded a symmetric BLEU similarity score of 0.6.This finding suggests that non-technical abstracts are not simply copied from technical abstracts; linear SVM classification achieved F1 scores of 90.99, 88.42, and 89.99 using SPECTER, STEL, and concatenated embeddings, respectively.
  • Claim and Proposal Characterization: Claims most commonly represented Capability/Application of Technology/Method (32.8%), Statement of Problem/Knowledge Gap (21.0%), and Observed Phenomenon/Property (18.9%).Investigation proposals were led by Theoretical Analysis and Computational Modeling (36.9%), Experimental Technique and Tool Development (16.8%), and Academic Training and Curriculum Development (12.8%).
  • Extraction Evaluation: Claim extraction achieved consistently high precision but lower recall, whereas investigation-proposal precision, recall, and F1 were more balanced across NSF areas.Overall extraction quality was high, but improving claim recall remained an important direction.

5 Tasks, Metrics, and Experiments

The section evaluates fine-tuned smaller language models on three NLP tasks: non-technical abstract generation, scientific claim extraction, and investigation-proposal extraction. It defines task-specific metrics and uses LLM-based claim and proposal judges validated against human judgments for scalable evaluation.

  • Tasks: Three tasks assess non-technical abstract generation, scientific-claim extraction, and investigation-proposal extraction.The tasks target accessible science communication, identification of verifiable assertions, and extraction of proposed investigations.
  • Models and fine-tuning: Two 7B-parameter models—Mistral-7B-instruct-v0.3 and Qwen2.5-7B-Instruct—were fine-tuned across the three tasks.The models were fine-tuned using LoRA for three epochs on an A100 GPU.
  • Metrics: Task 1 combines BERTScore and ROUGE to measure semantic alignment, lexical overlap, and structural similarity in generated abstracts.Reported BERTScore metrics include precision, recall, and F1; ROUGE metrics include ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-L-sum.
  • Metrics: Task 2 evaluates generated claim sets against gold claim sets using GPT-4o-mini’s zero-shot pairwise support judge to compute precision and recall.The procedure removes repeated or near-repeated generated claims and explicitly penalizes unsupported claims.
  • Validation: The claim-evaluation judge showed near-perfect correlation with human judgments on a subset of 120 awards, supporting application to the full dataset.GPT-4o-mini was restricted to pairwise sentence comparison during validation.
  • Results: Fine-tuning produced modest improvements for abstract generation but large improvements for claim extraction, while Mistral outperformed Qwen on almost all claim-extraction metrics.These findings are reported in the performance summaries for Tasks 1 and 2.

6 Results

Fine-tuning improved performance across non-technical abstract generation, claim extraction, and proposal extraction, with Mistral generally outperforming Qwen. Gains were especially substantial for the extraction tasks.

  • Task 1: Mistral achieved a BERTScore-F1 of 0.8561 after fine-tuning (+0.36% relative improvement) on non-technical abstract generation, outperforming Qwen on almost all metrics.ROUGE scores remained generally low at 0.01-0.22, reflecting stylistic differences between technical and non-technical abstracts.
  • Task 2: 116.7% relative improvement raised fine-tuned Mistral claim-extraction precision to 0.7450, with recall of 0.7098 (+59.5%) and F1 of 0.7097 (+101.8%).Fine-tuning substantially improved both Mistral and Qwen, while Mistral consistently performed better.
  • Task 3: 127.24% relative improvement raised fine-tuned Mistral proposal-extraction recall to 0.7539, alongside precision of 0.7351 (+18.24%) and F1 of 0.7261 (+90.97%).Qwen showed larger relative improvements, but Mistral still performed better overall.
  • Additional experiment: A Mistral-only version was also trained on NSF-SciFy-20K, spanning all NSF directorates, with results reported in Appendix F.This experiment followed Mistral’s observed edge over Qwen2.5 across the tasks.

7 Error Analysis

Error analysis found low error rates for both claim and investigation-proposal extraction, with recurring failures involving unsupported additions, omissions, mismatches, and misinterpretations. Manual review and proposed mitigations suggest reported accuracy may be conservative.

  • Claims: 2.6% error rate was found across 802 claims generated from 120 test-set awards by fine-tuned Mistral-7B models.The analysis covered NSF-SciFy-MatSci and NSF-SciFy-20K test sets.
  • Claims: Claim errors comprised overconfidence, mixing information, overgeneralization, information omission, and administrative hallucinations.Overconfidence misrepresented hedged statements, while other errors involved cross-sentence combinations, unsupported extensions, lost qualifiers, or invented funding and institutional information.
  • Claims: Overconfidence and overgeneralization were most common, while Claude-extracted claims had a 2.1% error rate, mostly administrative hallucinations.The comparison was between the fine-tuned Mistral-7B analysis and Claude-extracted claims.
  • Investigation Proposals: 2.4% error rate was found across 833 investigation proposals, including absent proposals, content mismatches, overspecification, and descriptions of existing work.The four categories captured missing proposals, altered or omitted key elements, unsupported details, and non-forward-looking descriptions.
  • Mitigation and Validation: Uncertainty calibration and stricter source-text alignment were proposed, while manual review found over half of labeled errors were not actual errors.Review of 20 examples also found that most claims labeled correct were indeed correct, suggesting higher true accuracy.

8 Discussion and Conclusion

NSF-SciFy is a large, comprehensive dataset of scientific claims and proposals extracted from NSF grant abstracts across science and mathematics. Its scale, coverage, temporal span, and publication links support improved extraction and new research in claim verification and meta-scientific analysis.

  • Dataset contributions: NSF-SciFy contains 2.8 million scientific claims and proposals from 400,000 NSF grant abstracts spanning all science and mathematics disciplines.Its focused subsets include 114,000 materials science claims and 135,000 claims from five NSF directorates.
  • Empirical impact: Fine-tuning language models on NSF-SciFy significantly improves scientific claim and proposal extraction.The passage reports relative performance gains often exceeding 10…
  • Dataset advantages: NSF-SciFy provides expert-reviewed quality, comprehensive scientific-domain coverage, a 1970–2024 temporal span, and links from recent grants to resulting publications.These properties enable longitudinal studies and connect awards with later research outputs.
  • Research opportunities: NSF-SciFy enables large-scale claim verification, scientific discovery tracking, and meta-scientific analysis of scientific assertions at their origin.The dataset is positioned as a resource for studying how scientific assertions emerge in funded research.

Limitations … E Stylistic Differences between Technical and Nontechinal Abstracts

The paper’s limitations concern source coverage, extraction recall, evaluation validation, outcome tracking, generalizability, and baseline breadth. Appendices document released resources, extraction and evaluation prompts, and stylistic differences between technical and nontechnical abstracts.

  • Limitations: NSF-SciFy excludes unfunded proposals and international contexts, although NSF covers approximately 25% of U.S. federally supported basic research across disciplines.The authors attribute these exclusions partly to limited public availability and the rarity and dispersion of international proposals.
  • Limitations: The zero-shot extraction pipeline achieves high precision but lower recall, reflecting a deliberate prioritization of precision during bootstrapping.The authors propose multi-pass extraction, ensembling, and targeted active annotation as routes to improve recall.
  • Limitations: LLM-based claim and proposal metrics correlate well with human judgment in samples, but broader validation across scientific domains remains necessary.The public dataset and code are intended to support community validation.
  • Limitations: Systematic outcome tracking is limited for older awards, restricting longitudinal claim-evolution analysis; generalizability and baseline comparisons beyond NSF abstracts and two models remain open.The framework could extend to other agencies, patents, or scientific news, while broader model comparisons are left for future work.
  • Appendix — A Reproducibility Statement: The authors release NSF-SciFy datasets, training code, trained models, and model checkpoints to support large-scale claim-extraction research.The NSF-SciFy superset spans the NSF awards database and uses claims and proposals extracted by fine-tuned models.
  • B Complete Prompt for Extracting Claims and Investigation Proposals: The extraction prompt requests JSON containing award identifiers, technical and nontechnical abstracts, claims, and investigation proposals, with explicit definitions for claims and forward-looking proposals.The prompt also requires valid JSON output and thorough extraction.
  • C Prompt for Task 2 evaluation function Φclaim: The evaluation prompts define claim support as YES when the second claim contains all information in the first without conflict, allowing additional content.
  • D Prompt for Task 3 evaluation function ΦIP: The investigation-proposal evaluation prompt applies the same containment-and-no-conflict criterion to proposed investigations, allowing additional proposals.

F Evaluation results for NSF-SCIFY-20K … I Error Analysis Examples

The appendix reports NSF-SCIFY-20K results for three generation tasks and provides tables with examples of extracted claims, investigation proposals, and their categories. The supplied passages do not include substantive error-analysis findings.

  • F Evaluation results for NSF-SCIFY-20K: Tables A5–A7 summarize results for the three generation tasks defined in Section 5.These tables cover the NSF-SCIFY-20K evaluation results.
  • I Error Analysis Examples: No substantive error-analysis example is provided in the supplied passages.The input names an error-analysis subsection but includes no corresponding passage text.
  • F Evaluation results for NSF-SCIFY-20K: Table A5 covers the Technical to Non-Technical Abstract Task using Mistral-7B.The passage identifies the table and model but supplies no metric values.
  • F Evaluation results for NSF-SCIFY-20K: Table A6 covers the Abstract to Claims Task using Mistral-7B.The passage identifies the table and model but supplies no metric values.
  • F Evaluation results for NSF-SCIFY-20K: Table A7 covers the Abstract to Investigation Proposals Task using Mistral-7B.The passage identifies the table and model but supplies no metric values.
  • G Examples of Extracted Claims and Investigation Proposals: Tables A8 and A9 provide a sampling of extracted claims and investigation proposals.The supplied passage does not describe the individual examples.
  • H Examples of Scientific Claim and Investigation Proposal Categories: Tables A10 and A11 contain examples related to scientific claim and investigation proposal categories.The supplied passage refers readers to the tables without listing their contents.

I.1 Claims … K AI Writing/Coding Assistance Disclosure

Error analyses found low claim and proposal error rates but identified recurring grounding failures, while NSF-SciFy’s automatic construction creates validation and bias risks. The authors disclose limited, verified use of generative AI for language editing and LaTeX assistance.

  • I.1 Claims: 2.6% of 802 generated claims were incorrect in the Mistral-7B error analysis.The analysis used 120 test awards from NSF-SCIFY-MATSCI and NSF-SCIFY-20K.
  • I.1 Claims: Claim errors included overconfidence, information mixing, overgeneralization, omission, and hallucinated administrative metadata.Scientific claims were rarely hallucinated; all observed hallucinations concerned administrative metadata.
  • I.2 Investigation Proposals: 2.4% of 833 generated investigation proposals were incorrect in the corresponding Mistral-7B analysis.The proposals were generated from 120 test awards using a model fine-tuned on NSF-SCIFY-20K.
  • I.2 Investigation Proposals: Investigation-proposal errors included generating proposals when none existed, content mismatch, overspecification, and treating existing work as forward-looking.Examples included misrepresenting conference activities, adding unsupported nanomachine integration, and converting a past application into a proposed activity.
  • J Potential Risks: NSF-SciFY’s automatic construction can leave extraction errors or omissions and may encode domain-specific or institutional biases.The authors call for careful validation and appropriate use to avoid generating or disseminating unverified claims.
  • K AI Writing/Coding Assistance Disclosure: The authors used generative AI only for language assistance, including proofreading, grammar, spelling, LaTeX fixes, and table generation.GPT-4o outputs were verified by the first author for correctness.
Loading 2503.08600v3…