Source-linked AI summary
The citation advantage of linking publications to research data
Giovanni Colavizza, Iain Hrynaszkiewicz, Isla Staden, Kirstie Whitaker, Barbara McGillivray
TL;DR
The paper asks whether DAS commonly provide repository links and whether those links are associated with citation impact. It classifies DAS in 531,889 PLOS and BMC articles and analyzes citation differences across categories. Repository-linked DAS are uncommon but associated with up to 25.36% higher average citation impact.
Problem
It is unclear how often DAS contain well-formed repository links and whether linking supporting data is associated with higher citation impact.
Method
The study classifies DAS from 531,889 PLOS and BMC articles into four categories and uses citation prediction regression to compare their citation impact.
Results
Up to 25.36% (± 1.07%) higher citation impact is associated with category 3 DAS containing repository links, while such statements comprise 20.8% of PLOS and 12.2% of BMC publications in 2017 and 2018.
Takeaways & Limitations
The results provide evidence of a potential citation benefit for providing research data through repositories and support stronger, consistent data-sharing policies.
Takeaways & Limitations
The PubMed OA collection includes only a fraction of all published literature, constraining the study’s data choices.
Abstract
from arXiv · showhide
Efforts to make research results open and reproducible are increasingly reflected by journal policies encouraging or mandating authors to provide data availability statements. As a consequence of this, there has been a strong uptake of data availability statements in recent literature. Nevertheless, it is still unclear what proportion of these statements actually contain well-formed links to data, for example via a URL or permanent identifier, and if there is an added value in providing such links. We consider 531,889 journal articles published by PLOS and BMC, develop an automatic system for labelling their data availability statements according to four categories based on their content and the type of data availability they display, and finally analyze the citation advantage of different statement categories via regression. We find that, following mandated publisher policies, data availability statements become very common. In 2018 93.7% of 21,793 PLOS articles and 88.2% of 31,956 BMC articles had data availability statements. Data availability statements containing a link to data in a repository -- rather than being available on request or included as supporting information files -- are a fraction of the total. In 2017 and 2018, 20.8% of PLOS publications and 12.2% of BMC publications provided DAS containing a link to data in a repository. We also find an association between articles that include statements that link to data in a repository and up to 25.36% ($\pm$~1.07%) higher citation impact on average, using a citation prediction model. We discuss the potential implications of these results for authors (researchers) and journal publishers who make the effort of sharing their data in repositories. All our data and code are made available in order to reproduce and extend our results.
Introduction
The study examines whether publisher data-availability policies produce informative statements and whether repository links are associated with greater citation impact. It focuses on classifying DAS content and comparing citation associations across categories.
- Motivation: Journal and funder policies aim to improve reproducibility, research quality, data reuse, and credit for data sharing.Citation counts also remain an important measure of research impact and reuse.
- Motivation: Data Availability Statements report whether supporting research data are public, supplementary, available on request, or unavailable.They are increasingly used to establish and assess compliance with data policies.
- Research gap: Repository links remain uncommon in DAS despite evidence from several disciplines associating data sharing or publication-data links with increased citations.Earlier work had not established whether DAS links specifically relate to citation impact across broad journal coverage.
- Research questions: The study analyzes PLOS and BMC articles to determine whether DAS adoption follows publisher policies and whether statement categories differ in citation impact.The categories include data available on request, data in the article or supplementary materials, and data linked directly through a repository.
Materials and methods
The authors construct an open-access publication dataset, isolate PLOS and BMC research articles, extract DAS, and classify their contents automatically. They then evaluate classifier performance and prepare the categorized data for analysis.
- Dataset construction: 531,889 PLOS and BMC journal articles remained after filtering the PubMed OA collection and removing reviews and editorials.The broader collection contained 1,969,175 publications with known identifiers, dates, and references.
- DAS classification: Four DAS categories distinguish unavailable or restricted data, data available on request, data in the article or supplementary information, and repository-linked data.Repository-linked statements provide a direct URL or preferably a persistent identifier.
- DAS classification: 380 statements were manually categorized, with 304 used for training and 76 for testing several text classifiers.The categories were selected to support reliable classification while covering the most common statement types.
- Classifier evaluation: The selected SVM classifier achieved 0.99 test accuracy, with support-weighted precision, recall, and F1-score also equal to 0.99.Accuracy reached 1.00 for the 250 most frequent test statements and for frequency-weighted accuracy.
Results
Mandated policies made data availability statements common, but repository links remained a minority and were associated with higher citation impact. Statement categories varied substantially between publishers and research domains.
- Statement uptake: 93.7% of 21,793 PLOS articles and 88.2% of 31,956 BMC articles had data availability statements in 2018.Both publishers showed clear adoption after introducing mandates.
- Statement categories: 6.0% of 31,965 BMC Series articles had statements during the encouraged-policy period, and 65.9% of those were category 3.Authors who supplied statements before they were required most often reported repository availability.
- Domain differences: BMC Genomics showed strong representation of category 3 statements, while Trials had a high proportion of category 1 statements.Parasites and Vectors illustrated substantial variability among statement categories.
- Citation impact: The regression controlled for publication timing, authors, references, publisher, policy status, research area, and author H-index.The citation analysis used a three-year accrual window and a dataset limited to publications through 2015.
- Citation impact: 22.65% (± 0.96%) higher citation impact was associated with category 3 statements after three years, rising to 25.36% (± 1.07%) using the whole-dataset average.After controlling for individual journals, category 3 was the only category significantly associated with positive citation impact.
Discussion
The study’s implications include evidence of a potential citation benefit from repository data sharing, while its reproducibility-focused design limits coverage and measurement. Future work should use broader literature and more detailed assessment of statement content and data accuracy.
- The reproducible design uses the PubMed Open Access collection, which covers only a fraction of published literature.
- The study does not assess whether repositories actually contain the data needed to reproduce reported results.
- More granular classification could examine templated versus non-standard statements and whether statements accurately identify reproducibility-relevant data.
- A potential increase in citations provides stakeholders with further evidence supporting access to research data and stronger, consistent data policies.
- Automated DAS detection and classification can support policy-compliance monitoring across multiple journals and publishers in open access literature.
Conclusion
The study finds that data availability statements rapidly become common after publisher mandates, but repository links remain a minority of statements. Articles with repository-linked statements show a substantial citation advantage in the analysis.
- DAS adoption rapidly increases after mandates in journals from both publishers.
- 12.2% of BMC and 20.8% of PLOS publications provided category 3 DAS containing repository links.
- BMC publications mostly use category 1, whereas PLOS publications mostly use category 2.
- Up to 25.36% (± 1.07%) higher citation impact is associated with category 3 DAS containing repository links.
Funding
The study received support from The Alan Turing Institute and Macmillan Education Ltd, part of Springer Nature.
- The study was supported by The Alan Turing Institute under EPSRC grant EP/N510129/1.
- Macmillan Education Ltd, part of Springer Nature, supported the work through grant RG92108 on data sharing policies and citation counts.
Competing interests
The authors report employment relationships with PLOS and Springer Nature and state that all other authors declared no competing interests.
- One author was employed by PLOS at publication and previously by Springer Nature during the study and manuscript preparation.
- All other authors declared that no competing interests exist.
Regressions table
Table 6 presents OLS and robust OLS estimates for a citation prediction model, including data-availability-statement categories, author and journal controls, and model-fit statistics.
- Estimation: Table 6 reports OLS and robust LS estimates, with coefficient standard errors given in parentheses.The accompanying method description states that robust-regression results did not differ significantly from standard OLS results.
- Author and article controls: 0.218 and 0.204 are the coefficients for mean author H-index, while median author H-index has coefficients of 0.007 and 0.008.The model also includes total references and number of authors as article-level controls.
- DAS categories: 0.252 and 0.271 are the largest coefficients among the three DAS-category indicators, corresponding to category 3.Categories 1 and 2 have coefficients of 0.085 and 0.059 in the first specification, and 0.072 and 0.057 in the second.
- Policy and journal controls: 0.073 and 0.070 are the coefficients for required DAS policies, while the PLOS indicator has coefficients of 0.211 and 0.213.The journal-field controls vary across categories, including negative coefficients for General Science & Technology and Psychology & Cognitive Sciences.
- Model fit: 367,836 observations are used in both specifications, with R2 of 0.144 and adjusted R2 of 0.144.The table reports residual standard errors of 0.593 and 0.665 and F statistics of 2,285.393, marked significant at p<0.01.