Source-linked AI summary
Reference Coverage Analysis of OpenAlex compared to Web of Science and Scopus
Jack Culbert, Anne Hobert, Najko Jahn, Nick Haupka, Marion Schmidt, Paul Donner, Philipp Mayr
TL;DR
The paper asks whether OpenAlex can provide trustworthy, reproducible bibliometric data comparable to Web of Science and Scopus despite its rapid evolution. It compares reference coverage and selected metadata across the three databases, finding comparable reference coverage in a shared corpus, higher ORCID coverage but fewer abstracts, and similar Open Access coverage in OpenAlex. These findings are bounded by the late-2023 state of the databases and by the absence of ground truth for reference counts.
Problem
The paper examines whether OpenAlex can serve as a trustworthy, reproducible alternative to proprietary bibliometric databases whose licensing limits data dissemination.
Method
The study compares reference counts and internal reference coverage, plus selected metadata, across OpenAlex, Web of Science, and Scopus using cleaned and matched records.
Results
OpenAlex is comparable to Web of Science and Scopus in reference coverage for newer shared records, while showing higher ORCID coverage and lower abstract coverage.
Takeaways & Limitations
OpenAlex is on par with commercial databases for internal citation coverage in contemporary shared publications, but its utility is limited by incomplete cited-reference data.
Takeaways & Limitations
The findings reflect late-2023 database states and lack ground truth for whether Web of Science and Scopus reference counts exactly match publication reference lists.
Abstract
from arXiv · showhide
OpenAlex is a promising open source of scholarly metadata, and competitor to established proprietary sources, such as the Web of Science and Scopus. As OpenAlex provides its data freely and openly, it permits researchers to perform bibliometric studies that can be reproduced in the community without licensing barriers. However, as OpenAlex is a rapidly evolving source and the data contained within is expanding and also quickly changing, the question naturally arises as to the trustworthiness of its data. In this report, we will study the reference coverage and selected metadata within each database and compare them with each other to help address this open question in bibliometrics. In our large-scale study, we demonstrate that, when restricted to a cleaned dataset of 16.8 million recent publications shared by all three databases, OpenAlex has average source reference numbers and internal coverage rates comparable to both Web of Science and Scopus. We further analyse the metadata in OpenAlex, the Web of Science and Scopus by journal, finding a similarity in the distribution of source reference counts in the Web of Science and Scopus as compared to OpenAlex. We also demonstrate that the comparison of other core metadata covered by OpenAlex shows mixed results when broken down by journal, capturing more ORCID identifiers, fewer abstracts and a similar number of Open Access status indicators per article when compared to both the Web of Science and Scopus.
1 Introduction
OpenAlex is evaluated as a freely available alternative to Web of Science and Scopus, with the study comparing reference coverage and selected metadata. The analysis uses internal reference coverage and journal-level metadata comparisons while recognizing that OpenAlex is dynamic and that internal coverage is only a partial, source-biased measure.
- Motivation: OpenAlex’s open licensing supports reproducible bibliometrics, unlike proprietary databases whose licensing limits data dissemination.
- Research scope: The study compares OpenAlex with Web of Science and Scopus on reference coverage, abstracts, ORCIDs, funding information, and Open Access status.
- Scope and limitations: OpenAlex’s changing contents mean the report’s assessments represent the state of the data in late 2023 rather than a stable endpoint.
- Reference coverage: Internal coverage measures the proportion of cited references that are themselves indexed as source items in the database.
- Reference coverage: Internal reference coverage is replicable without external data but remains partial and potentially source-biased because it depends on the assessed database’s own reference data.
- Reference coverage: The comparison emphasizes relative performance among databases rather than absolute coverage values because internal coverage depends on database size, disciplinary profile, and reference matching accuracy.
2 Data and Methodology
The study constructs a time- and DOI-restricted Shared Corpus to compare OpenAlex, Web of Science, and Scopus, using database snapshots and reference-count measures. It addresses matching errors and database differences by excluding ambiguous DOI records and restricting reference years.
- Shared Corpus: The Shared Corpus contains records common to all three databases, matched by exact unique DOI and restricted to publications from 2015–2022.Records without DOIs or with duplicate DOI assignments were excluded from matching.
- Database versions: The analysis uses WoS and Scopus snapshots captured in April 2023 and an OpenAlex version released in August 2023.Because the database versions differ in date, the Shared Corpus is restricted to publications through 31 December 2022.
- Reference restrictions: References are restricted to publication years 1996–2022 to reduce bias from differences in database coverage.Scopus mainly contains items from 1996 onward, whereas WoS and OpenAlex have no such restriction.
- Corpus comparison: The Shared Corpus represents 23.6% of WoS records, 25.6% of Scopus records, and 6.9% of OpenAlex records, while containing 41.1%, 35.8%, and 31.7% of their references, respectively.These values apply after DOI deduplication.
- Reference measures: The study compares total reference counts and source reference counts for complete databases, article subsets, and the Shared Corpus.OpenAlex source reference counts are related to corresponding counts from WoS and Scopus because OpenAlex provides source references rather than total references.
- Matching limitations: DOI matching can underestimate database size because records without DOIs are excluded and records sharing a DOI are counted once.Publication-year differences between databases, including early-access and print-date handling, also affect the common corpus.
3 Results
Across the shared corpus, OpenAlex shows competitive reference coverage, while journal-level metadata comparisons reveal both strengths and gaps relative to Web of Science and Scopus. Reference-count discrepancies in Scopus and OpenAlex also require attention when interpreting averages.
- Reference coverage: 7.6 versus 16.9 and 18.7 references per record in naïve averaging, but OpenAlex becomes competitive within the 2015–2022 corpus shared by all three databases.The underlying corpus definition substantially affects the comparison.
- Reference coverage: 83.2% against Web of Science and 83.6% against Scopus are OpenAlex’s inferred internal coverage rates for references restricted to publication years 1996–2022.These values lie between the corresponding Web of Science and Scopus values.
- Reference-count discrepancies: Reported and pre-calculated reference counts diverge from actual database entries in Scopus and OpenAlex because of inconsistent supplier ingestions and deleted OpenAlex items.The pre-calculated OpenAlex values exclude references to items deleted from OpenAlex.
- Reference-count discrepancies: Reference-count discrepancies should be considered in OpenAlex analyses because averages may differ significantly when databases are not judiciously curated.The authors call for deeper analysis of OpenAlex discrepancies as new versions are released.
- Metadata by journal: OpenAlex’s journal-level source-reference distributions are broadly similar to those of Web of Science and Scopus, but it overcounts some journals and substantially undercounts others.Greater density below the y = x line indicates more OpenAlex source references in some journals, while lesser density above it indicates undercounting in others.
- Metadata by journal: Over 92% of Web of Science and Scopus articles have abstracts versus 87% in OpenAlex, although OpenAlex has higher abstract coverage for some journals.The comparison is based on journal-level coverage percentages.
- Metadata by journal: OpenAlex contains at least one ORCID for 92% of articles, compared with 16% in Web of Science and 32% in Scopus, but its generous author disambiguation can produce questionable links.Some Chinese-named authors were linked to more than 10,000 publications.
4 Discussion
OpenAlex has reference coverage comparable to Web of Science and Scopus within a shared core corpus, but its broader database and metadata remain constrained by incomplete references, data errors, and curation challenges.
- OpenAlex’s source reference coverage is comparable to Web of Science and Scopus for newer records shared by all three databases.This holds both overall and when restricting references to 1996 onward.
- OpenAlex’s larger corpus does not yield clearly higher internal reference coverage, suggesting that its additional records represent a different publication spectrum or have lower data quality.The authors therefore suggest restricting comparative bibliometric analyses to a core corpus.
- 6.4 to 6.2 references per publication are captured by Web of Science and Scopus respectively but not by OpenAlex in the shared corpus.OpenAlex’s incomplete inclusion of non-source references and full reference strings limits independent reference matching and analysis.
- OpenAlex shows higher ORCID coverage but fewer abstracts than Web of Science and Scopus, while its abstract coverage remains higher than Crossref.Over 90% of OpenAlex articles had at least one ORCID, although extreme assignments suggest author-disambiguation problems.
- OpenAlex’s metadata requires caution because it combines heterogeneous sources, faces legal constraints, and changes through ongoing curation and data-quality corrections.The report specifically notes challenges in disambiguation, standardisation, abstract representation, and reference handling.
5 Limitations and Outlook
The study’s conclusions are bounded by changing database contents, absent ground truth for reference accuracy, unassessed matching accuracy, and limitations in ORCID and DOI validation.
- The dataset represents late 2023 and may not reflect the state of OpenAlex, Web of Science, or Scopus at publication.At least 151 million references were added to OpenAlex after data collection, underscoring its volatility.
- The study lacks ground truth for determining whether Web of Science and Scopus reference counts exactly match publication reference lists.The authors checked agreement between delivered and pre-calculated counts and references, but not their correspondence to the publications themselves.
- The study does not assess the accuracy of database reference-to-publication matching, although that accuracy underpins the internal coverage indicator.The authors propose deeper comparisons of source and non-source references and matching algorithms as future work.
- ORCID analysis counts whether an article has at least one ORCID, not whether all co-authors are represented or accurately matched.Observed assignment of one ORCID to tens of thousands of articles motivates further validation of author disambiguation.
- 39,481 records with duplicate DOI associations were excluded from the shared corpus, and DOI duplication, omission, or error can require title and author comparisons.The authors note that DOIs are imperfect identifiers for dataset combination.
Declarations
The authors acknowledge project funding, institutional support, and an earlier preprint version of the work.
- The work was funded by Germany’s Federal Ministry of Education and Research through OpenBib and Competence Network for Bibliometrics grants.
- Jack Culbert and Philipp Mayr received additional European Union funding through the Horizon Europe OMINO project.
- The findings were initially reported in an arXiv preprint containing a longer version of the results.