Source-linked AI summary
Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations' COCI: a multidisciplinary comparison of coverage via citations
Alberto Martín-Martín, Mike Thelwall, Enrique Orduna-Malea, Emilio Delgado López-Cózar
TL;DR
The paper addresses the lack of systematic evidence about how six citation databases overlap across subject categories. It compares citations to a common sample of highly cited 2006 documents and finds Google Scholar most comprehensive, with Microsoft Academic and Dimensions often viable alternatives to Scopus and WoS.
Problem
The study asks how much Google Scholar, Microsoft Academic, Scopus, Dimensions, WoS, and COCI overlap in citations and whether overlap varies by subject.
Method
The authors compare citing documents retrieved from six sources for 2,515 highly cited 2006 documents spanning 252 subject categories, using conservative citation matching.
Results
Google Scholar found 88% of all citations, Microsoft Academic 60%, Dimensions 54%, and COCI 28%, with coverage patterns varying across subject categories.
Takeaways & Limitations
Google Scholar remains the most comprehensive source, while Microsoft Academic and Dimensions are good coverage alternatives to Scopus and WoS in many categories.
Takeaways & Limitations
COCI results cannot reflect CrossRef’s full coverage because reference lists are incompletely available, and citation-based coverage estimates are affected by confounding factors.
Abstract
from arXiv · showhide
New sources of citation data have recently become available, such as Microsoft Academic, Dimensions, and the OpenCitations Index of CrossRef open DOI-to-DOI citations (COCI). Although these have been compared to the Web of Science (WoS), Scopus, or Google Scholar, there is no systematic evidence of their differences across subject categories. In response, this paper investigates 3,073,351 citations found by these six data sources to 2,515 English-language highly-cited documents published in 2006 from 252 subject categories, expanding and updating the largest previous study. Google Scholar found 88% of all citations, many of which were not found by the other sources, and nearly all citations found by the remaining sources (89%-94%). A similar pattern held within most subject categories. Microsoft Academic is the second largest overall (60% of all citations), including 82% of Scopus citations and 86% of Web of Science citations. In most categories, Microsoft Academic found more citations than Scopus and WoS (182 and 223 subject categories, respectively), but had coverage gaps in some areas, such as Physics and some Humanities categories. After Scopus, Dimensions is fourth largest (54% of all citations), including 84% of Scopus citations and 88% of WoS citations. It found more citations than Scopus in 36 categories, more than WoS in 185, and displays some coverage gaps, especially in the Humanities. Following WoS, COCI is the smallest, with 28% of all citations. Google Scholar is still the most comprehensive source. In many subject categories Microsoft Academic and Dimensions are good alternatives to Scopus and WoS in terms of coverage.
1. Introduction
The study responds to changing citation-index infrastructure by systematically comparing six sources across 252 subject categories. It asks how much their citation overlap varies by subject and evaluates coverage differences relevant to information needs.
- 1.2 Previous analyses of coverage: Coverage differences matter because greater citation coverage should produce higher citation counts when citations can be extracted from all documents.Search, analysis, export functionality, transparency, and cost are also relevant but are outside this study’s analysis.
- 1.1 Timeline: Six citation sources—including Google Scholar, Microsoft Academic, Scopus, Dimensions, WoS, and COCI—form the study’s comparative landscape.The newer sources expanded the range of openly or freely accessible citation data.
- 1.3 Objective: Previous studies compared heterogeneous samples, disciplines, and sources, leaving no systematic cross-subject comparison of all six databases.This paper expands and updates an earlier Google Scholar, Scopus, and WoS analysis.
- 1.3 Objective: The research question concerns overlap among the six sources’ citations to academic documents and whether that overlap varies by subject.The comparison covers 252 subject categories using a relatively large citation sample.
2. Methods
Because complete document-coverage lists were unavailable for all sources, the study compares citing documents retrieved for a common seed sample. It then matches citations conservatively across sources and evaluates coverage differences by category.
- 2.1. Direct coverage comparison vs. comparison of citations: Access restrictions prevented direct matching of complete document lists across all databases.Scopus and WoS charge for this access, while Google Scholar does not share its database.
- 2.1. Direct coverage comparison vs. comparison of citations: The study therefore selects documents covered by all sources and compares the citing documents each source retrieves for those seeds.This approach assumes citation-detection effectiveness is sufficiently comparable across sources.
- 2.2. Selection of seed sample: The seed sample contains highly cited 2006 documents selected from Google Scholar’s Classic Papers across 252 subject categories.The sample uses the top 10 documents per category, except French Studies, which has five.
- 2.2. Selection of seed sample: The sample was chosen because it provided Google Scholar article-level subject classifications and efficient citation extraction for highly cited documents.At data collection, no other readily available sample offered the same article-level classification.
- 2.4. Analysis of citation data: Citing documents were merged into a master citation list using conservative metadata matching across sources.The final merged dataset contained 3,073,351 citations, while duplicate citations within sources were not removed.
3. Results
Coverage varies substantially across citation databases overall and by subject area. Google Scholar is the most comprehensive, while Microsoft Academic, Scopus, Dimensions, WoS, and COCI show differing overlaps and field-specific coverage.
- 88% of all citations were found by Google Scholar, compared with 60%-52% for Microsoft Academic, Scopus, Dimensions, and WoS, and 28% for COCI.
- Google Scholar found 89%-94% of citations found by the other sources, whereas COCI found 30%-51% of citations found by the other sources.
- 26% of all citations were found only by Google Scholar, while 21% were found by all six sources and 18% by all except COCI.
- Scopus/WoS had the highest pairwise full citation overlap at 78%, followed by Scopus/Dimensions and Dimensions/WoS at 75%.
- Google Scholar found 85%-90% of citations across all eight broad subject areas, while COCI found 18%-20% in Humanities and Social Sciences and 27%-32% in STEM.
- Microsoft Academic, Scopus, Dimensions, and WoS covered 69%-72% in Chemical & Material Sciences but only 25%-39% in Humanities, Literature & Arts.
- Microsoft Academic generally had the most comprehensive coverage among the four comparable sources, except in Physics & Mathematics, where it found 57%.
- Across subject areas, Dimensions/COCI had the highest relative overlap, while Microsoft Academic showed lower overlap in Physics & Mathematics.
D. Engineering & Computer Science
Table 4 reports relative pairwise overlaps between data sources for broad subject areas, including the continuation presented in this table.
- D. Engineering & Computer Science: Table 4 reports relative pairwise overlaps between data sources.
- D. Engineering & Computer Science: The table reports overlap values as percentages.
- D. Engineering & Computer Science: The table presents results overall and by broad subject areas.
H. Chemical & Material Sciences
Citation overlap and relative coverage vary across subject areas and categories. In Chemical and Material Sciences, the three-source overlap is comparatively high, while Microsoft Academic and Dimensions still show category-level differences from Scopus and WoS.
- Google Scholar, Microsoft Academic, and Dimensions: 63% of citations in Chemical and Material Sciences were found by Google Scholar, Microsoft Academic, and Dimensions.This is the highest reported full-overlap value across the STEM areas described.
- Scopus, Microsoft Academic, and Dimensions: Microsoft Academic, Scopus, and Dimensions vary in relative coverage by subject area, with Scopus sometimes exceeding Microsoft Academic in Chemical and Material Sciences.Dimensions provides coverage similar to Scopus across subject areas, but each source also finds citations the others miss.
- Scopus, Microsoft Academic, and Dimensions: Large Microsoft Academic/Scopus and Scopus/Dimensions overlaps occur in most categories, especially in STEM, where overlap almost always exceeds 50%.Scopus/Dimensions overlap exceeds 70% in 148 of 252 categories, compared with 66 categories for Microsoft Academic/Scopus.
- Web of Science, Microsoft Academic, and Dimensions: Microsoft Academic found more citations than WoS in 223 of 252 categories, but its coverage was lower in Physics and Mathematics and, to a lesser degree, Chemical and Material Sciences.The category-level pattern includes strong Microsoft Academic coverage in Computing Systems and Software Systems, alongside gaps in some Physics categories.
- Microsoft Academic and Dimensions: Microsoft Academic found more citations than Dimensions in 226 of 252 categories, while Dimensions contributed distinctive coverage in several Physics and Mathematics categories.Dimensions-only citations were below 10% in 209 categories but reached 26%-45% in three cited Physics and Mathematics categories.
4. Discussion
The discussion situates the findings alongside prior comparisons while identifying limits on generalization and on what coverage alone can establish. The sample, source-selection effects, and unexamined coverage dimensions constrain interpretation.
- Limitations: The study samples highly cited English-language documents published in 2006, so generalization assumes their citing populations resemble those of all citing documents within each subject category.That assumption may fail when highly cited topics are overrepresented, and the results may not represent non-English or regionally relevant literature.
- Limitations: Google Scholar may have an advantage because the seed sample was selected from documents most highly cited in Google Scholar, although the authors report evidence that this advantage is not substantial.The potential advantage concerns citation-detection accuracy for this selected sample relative to Google Scholar documents generally.
- Limitations: Document type, language, publication date, and indexing speed are not analysed, although users may need these coverage dimensions when selecting data sources.The authors identify these aspects as topics for future studies.
- Comparison with previous studies: The results generally agree with previous studies showing that Microsoft Academic detects more citations than WoS and Scopus, especially in Humanities, Social Sciences, and Business & Economics.Its advantage is not uniform: coverage can be lower in Physics and Chemistry.
- Comparison with previous studies: Dimensions offers similar coverage to Scopus and WoS in the reported comparisons, while each source still detects a non-negligible share of citations the others miss.The discussion notes that Scopus is slightly larger, followed by Dimensions, in the cited comparison.
5. Conclusions
Google Scholar is the most comprehensive of the six sources across overall results and most subject areas, while Microsoft Academic and Dimensions often match or exceed Scopus and Web of Science but retain coverage gaps. Source choice therefore depends on whether users prioritize comprehensive counts, complete citing-source lists, search control, or combining sources.
- Comprehensiveness: Microsoft Academic and Dimensions provided at least as many citations as Scopus and Web of Science in many subject categories, but both had coverage gaps.Microsoft Academic gaps included Physics and some Humanities categories; Dimensions gaps were especially visible in the Humanities.
- Implications: Scopus and Web of Science provide advanced search and filtering suitable for systematic reviews despite lower coverage, slower indexing, limited document types, and subscription access.Their limitations reduce usefulness when fast and unrestricted access to current studies, including preprints, is important.
- Implications: The mismatch between comprehensiveness and search control may require using at least two sources for systematic reviews.Combining several sources is recommended when exhaustive coverage is needed and coverage divergence is large.
- Implications: Google Scholar offers the most comprehensive counts, whereas Microsoft Academic is the best free alternative when complete lists of citing sources are needed.The final choice may also depend on metadata quality and bulk-access options.
- Implications: New open citation sources may reduce reliance on expensive proprietary databases, although COCI remains too small to use alone.COCI may instead serve as an input to other sources rather than as a comprehensive standalone index.
Appendix 1
The first appendix provides links to the complete Venn-diagram materials used to examine citation overlap, both by individual subject category and after aggregation into eight subject areas.
- Venn diagrams: The appendix links to two-set and three-set Venn diagrams covering all subject categories without subject aggregation.The listed materials are hosted on OSF.
- Venn diagrams: It also links to Venn diagrams comparing Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and COCI after aggregation by eight subject areas.The listed comparisons include multiple three-source combinations.
Appendix 2
The appendices document the study’s supplementary overlap and boxplot materials, alongside multilingual statements of its research gap, sample, and principal coverage findings. They also record the manuscript’s submission, revision, acceptance, and publication timeline.
- Research scope: The study addresses the lack of systematic evidence about differences among six citation databases across subject categories.The Spanish appendix wording identifies Microsoft Academic, Dimensions, and COCI as newer sources compared with Web of Science, Scopus, and Google Scholar.
- Research scope: The analysis covers 3,073,353 citations to 2,515 English-language highly cited articles published in 2006 across 252 subject categories.The larger, more recent sample was intended to improve generalizability.
- Results: Google Scholar found 88% of all citations and 89%-94% of citations found by the other sources, while Microsoft Academic found 60% and Dimensions 54%.COCI was the smallest source, with 28% of all citations.
- Results: Microsoft Academic exceeded Scopus and Web of Science in 182 and 223 categories, respectively, while Dimensions exceeded them in 36 and 185 categories.Both sources nevertheless showed coverage gaps, especially in selected Physics and Humanities categories.
- Results: Google Scholar remained the highest-coverage source, and Microsoft Academic and Dimensions were good coverage alternatives to Scopus and Web of Science in many categories.This conclusion is stated in the multilingual appendix materials.