Source-linked AI summary
The coverage of Microsoft Academic: Analyzing the publication output of a university
Sven E. Hug, Martin P. Braendle
TL;DR
Earlier MA coverage studies provided limited empirical evidence and did not test recall against actual verified publication lists. Using a verified university list, this study compares MA with Scopus and WoS and finds similar overall coverage for MA and Scopus, with different strengths by document type.
Problem
Prior MA coverage studies offered limited empirical detail and did not measure recall against actual publication lists, leaving coverage of verified individual publications unresolved.
Source: Introduction · p. 3Method
The study compares a verified University of Zurich repository list with MA, Scopus, and WoS at publication level while examining citations, retrieval, and metadata quality.
Source: Introduction · p. 4, Zurich Open Archive and Repository… · p. 5Results
MA and Scopus cover ZORA similarly; MA leads in unique, book-related, and conference coverage, while Scopus is slightly stronger for journal articles, and citation rankings correlate highly.
Source: Discussion · pp. 22–23Takeaways & Limitations
The authors conclude that MA is nearing bibliometric-superpower status and that its features are also advantageous for literature search and library applications.
Source: Conclusion · pp. 23–24MA’s source coverage is opaque, metadata quality has not been investigated in detail, and the study covers one Swiss university’s 2008–2015 publications.
Source: Conclusion · p. 24
Abstract
from arXiv · showhide
This is the first detailed study on the coverage of Microsoft Academic (MA). Based on the complete and verified publication list of a university, the coverage of MA was assessed and compared with two benchmark databases, Scopus and Web of Science (WoS), on the level of individual publications. Citation counts were analyzed, and issues related to data retrieval and data quality were examined. A Perl script was written to retrieve metadata from MA based on publication titles. The script is freely available on GitHub. We find that MA covers journal articles, working papers, and conference items to a substantial extent and indexes more document types than the benchmark databases (e.g., working papers, dissertations). MA clearly surpasses Scopus and WoS in covering book-related document types and conference items but falls slightly behind Scopus in journal articles. The coverage of MA is favorable for evaluative bibliometrics in most research fields, including economics/business, computer/information sciences, and mathematics. However, MA shows biases similar to Scopus and WoS with regard to the coverage of the humanities, non-English publications, and open-access publications. Rank correlations of citation counts are high between MA and the benchmark databases. We find that the publication year is correct for 89.5% of all publications and the number of authors is correct for 95.1% of the journal articles. Given the fast and ongoing development of MA, we conclude that MA is on the verge of becoming a bibliometric superpower. However, comprehensive studies on the quality of MA metadata are still lacking.
Introduction
Earlier studies provided limited evidence about Microsoft Academic’s coverage and did not assess recall against actual verified publication lists. This study compares a university’s verified list with MA, Scopus, and Web of Science and examines coverage, citations, retrieval, and metadata quality.
Introduction
- MA accesses the Microsoft Academic Graph through a search engine and API, and the service was expanding its features and record count rapidly.The graph was updated weekly and was reported to be growing by 1.3 million records per month.Source: Introduction · p. 2
- MA had already attracted bibliometric research on visualization, mapping, indicators, algorithms, and evaluative applications, but publication-coverage evidence remained inconclusive.The cited prior coverage studies did not establish recall for actual publication lists.Source: Introduction · pp. 2–3
- Earlier MA coverage studies offered limited empirical detail and did not measure recall against actual publication lists, leaving coverage of verified individual publications unresolved.Some relied on small samples or author queries, while others compared databases without testing how well an actual list was represented.Source: Introduction · p. 3
- The study assesses a verified university publication list in MA, Scopus, and WoS, alongside citation counts, data retrieval, and metadata quality.The questions address document type, language, access status, year, research field, citation correlations, citations per publication, uncited publications, DOI coverage, author counts, and publication years.Source: Introduction · p. 4
- The paper is organized around repository and retrieval methods, followed by results on script performance, metadata, coverage, citations, and discussion.Source: Introduction · p. 4
Method
The study assessed Microsoft Academic’s retrieval, matching, and metadata quality using ZORA’s verified publication records, and compared its coverage and citation data with Scopus and Web of Science. It also examined how citation counts and citation-based rankings aligned across the databases.
Zurich Open Archive and Repository (ZORA)
- The ZORA source set was editorially checked and complete for the study’s purposes, and its 62,791-item 2008–2015 subset was restricted to institute researchers and five main document types.The full set contained 91,215 items and was used to assess retrieval and MA metadata; the subset supported database coverage comparisons.Source: Zurich Open Archive and Repository… · p. 5
Perl script for retrieving and matching Microsoft Academic data
- The Perl script queried MA’s Evaluate endpoint using title_words and title_exact modes, then matched returned items to ZORA by DOI, title, or bibliographic data.A DOI match had priority over title and bibliographic matches; the script and associated files were made available on GitHub.Source: Perl script for retrieving and… · pp. 6–8
Performance of the retrieval modes
- The title_exact mode achieved a higher F1 score than title_words (0.647 vs. 0.594), with corrected precision of 0.879 vs. 0.689 and recall of 51.2% vs. 51.4%.Combining both modes raised recall to 52.9% (48,231 items); precision estimates are upper estimates because queries returned at most ten items.Source: Recall, precision, and F1 score · p. 8, Table 1 · p. 9
Performance of the matching algorithm
- Across the two retrieval modes, 98% of jointly retrieved items had the same MA ID, while 2% differed and were likely false positives.Of the 29,586 items matched by DOI in both modes, 280 (0.9%) were likely false-positive DOI matches, suggesting duplicate MA records.Source: Performance of the matching algorithm · p. 9, Table 2 · p. 10
Quality of Microsoft Academic metadata
- MA’s publication year matched ZORA for 89.5% of matched items, and its author count matched for 95.1% of matched journal articles.MA included a DOI for 77.6% of matched items, with availability varying by field; all matched-item DOIs in MA were valid.Source: Quality of Microsoft Academic metadata · p. 10, Table 3 · p. 11
Coverage
- MA covered 52.5% of the expanded ZORA set, versus 52.0% for Scopus and 47.2% for WoS, and MA had the most exclusively covered items.For main document types, Scopus covered 57.9% and MA 56.6%; MA covered conference items better than Scopus and WoS and substantially more book-related items, though book coverage remained low.Source: Overall and unique coverage of… · p. 11, Table 4 · p. 12, Main document types · p. 12
- All three databases covered English publications and non-public full texts better than non-English, open-access, and book-related publications; MA’s citation counts were lower in several natural-science fields.MA, Scopus, and WoS had similar citations per publication in five of six major fields, but MA’s values were lower in physical, chemical, and computer/information sciences; uncitedness was 16.1% in MA, 15.2% in Scopus, and 12.2% in WoS.Source: Publication language and access status · p. 13, Table 5 · p. 14, Citations per publication, uncitedness · p. 19
- MA’s coverage of research fields was uneven: it performed well in several natural-science subfields and economics/business, but the databases covered humanities and some social-science fields poorly.MA’s title search missed technical-title publications in chemistry and physics; resolving those cases was estimated to raise MA coverage from 83.7% to 95.2% in chemistry and from 67.5% to 93.8% in physics, but DOI queries were suggested as a more practical improvement.Source: Publication language and access status · pp. 16–18, Figure 5 · p. 17, Figure 6 · p. 17
- Pearson correlations were r=0.99 (p<0.01) for Scopus–WoS and r=0.73 (p<0.01) for both MA–Scopus and MA–WoS.The scatterplots also reflect these correlation patterns.Source: Correlation of citations · pp. 19–20, Figure 9 · p. 20
- Citation-based rankings were very similar across MA, Scopus, and WoS, with rank correlations slightly lower for MA versus either benchmark than for Scopus versus WoS.Source: Correlation of citations · p. 20, Table 6 · p. 20
- The rank-correlation pattern held across all FOS fields, although correlations were somewhat weaker in the humanities.Source: Correlation of citations · p. 20
Discussion
MA and Scopus cover the university repository similarly, but their relative performance varies by document type and field. Citation rankings are similar across databases, while retrieval and metadata issues qualify the comparisons.
Discussion
- Title-based ti_ex and ti_wo retrieval had equal recall, but ti_ex had higher precision; technical terms led to estimated coverage undercounts of 11.5% in chemistry and 26.3% in physics.The authors estimate a 0.8% undercount for the repository overall and suggest combining both retrieval modes to maximize retrieved items.Source: Discussion · p. 21
- 89.5% of MA items have correct publication years, 95.1% of retrieved journal articles have the correct author count, and 12.6% of DOI-bearing ZORA publications lack a DOI in MA.MA’s API limits authors per publication to 50, which the authors say explains occasional undercounts.Source: Discussion · pp. 21–22
- MA and Scopus cover ZORA similarly; MA has the most exclusive coverage, both outperform WoS overall and uniquely, Scopus slightly leads journal articles, and MA leads conference and book-related types.MA also indexes document types such as working papers and dissertations that Scopus and WoS do not.Source: Discussion · p. 22
- MA substantially covers working papers, journal articles, and conference items, yet none of ZORA’s newspaper articles were found in MA.The authors interpret the absence of newspaper articles as indicating that MA separates scholarly from non-scholarly content.Source: Discussion · p. 22
- The databases cover English publications highly but non-English publications very little, and cover non-publicly accessible publications more than open-access publications.MA shows language and open-access biases similar to those reported for Scopus and WoS.Source: Discussion · p. 22
- MA covers at least two-thirds of ZORA2008-2015 publications in natural sciences, engineering/technology, medical/health sciences, psychology, and economics/business.The natural-sciences coverage includes mathematics and computer/information sciences.Source: Discussion · pp. 22–23
- Humanities coverage is poor across all three databases, and MA does not outperform Scopus or WoS there; other social-science subfields range from low to medium MA coverage.MA outperforms benchmark databases in most social-science subfields, with psychology and economics/business as stronger exceptions.Source: Discussion · p. 23
- MA, Scopus, and WoS rank ZORA2008-2015 publications very similarly by citations, but MA has lower CPP in physical, chemical, and computer/information sciences.The authors say further work is needed to determine whether those field-specific CPP observations are robust or sample-specific.Source: Discussion · p. 23
Conclusion
MA combines rich, structured metadata and a functional API with promising coverage and citation performance. However, source transparency, metadata quality, and the study’s narrow publication scope constrain conclusions and support using MA alongside established databases.
Conclusion
- MA’s rich, structured metadata and high-functionality API may serve bibliometrics, literature search, and library applications such as repository enrichment and collection assessment.The cited library applications use reference information.Source: Conclusion · pp. 23–24
- MA’s sources, coverage policy, and data curation remain insufficiently transparent, and metadata quality has not been investigated in detail; the authors recommend using MA with established databases until metadata are evaluated.The recommendation includes Scopus, WoS, or a disciplinary database.Source: Conclusion · p. 24
- The study covers publications from one Swiss university during 2008–2015, and the authors call for broader sets, including non-Western-language publications.Source: Conclusion · p. 24