Source-linked AI summary
Google Scholar as a data source for research assessment
Emilio Delgado López-Cózar, Enrique Orduna-Malea, Alberto Martín-Martín
TL;DR
GS was not designed for bibliometric analysis, although its broad academic-web coverage and citation data attracted interest as a research-assessment source. This chapter reviews how GS works and evaluates its coverage, growth, limitations, and citation-data relationships; a cited survey found it was the preferred literature-search tool among respondents.
Problem
GS's developers did not intend it for bibliometric analysis, prompting the need to assess its suitability as a research-assessment data source.
Source: §2.1 · p. 3Method
The chapter reviews GS and its secondary products, measures database characteristics, analyzes limitations, and compares citation data with traditional bibliographic databases.
Source: §2.1 · p. 3Results
89% of survey respondents selected GS for literature searches, compared with WoS (41%), Pubmed (40%), Others (36%), and Scopus (26%).
Source: §2.1 · p. 2Takeaways & Limitations
GS extends beyond the strict scientific-knowledge scope of WoS and Scopus, with 69.7% of its content hosted on generic top-level domains.
Source: Source coverage · p. 17, §2.5 · p. 35GS's opacity and lack of quality control leave it open to publication or citation manipulation, and its coverage and ranking mechanisms lack transparency.
Source: §2.6 · p. 38
Abstract
from arXiv · showhide
The launch of Google Scholar (GS) marked the beginning of a revolution in the scientific information market. This search engine, unlike traditional databases, automatically indexes information from the academic web. Its ease of use, together with its wide coverage and fast indexing speed, have made it the first tool most scientists currently turn to when they need to carry out a literature search. Additionally, the fact that its search results were accompanied from the beginning by citation counts, as well as the later development of secondary products which leverage this citation data (such as Google Scholar Metrics and Google Scholar Citations), made many scientists wonder about its potential as a source of data for bibliometric analyses. The goal of this chapter is to lay the foundations for the use of GS as a supplementary source (and in some disciplines, arguably the best alternative) for scientific evaluation. First, we present a general overview of how GS works. Second, we present empirical evidences about its main characteristics (size, coverage, and growth rate). Third, we carry out a systematic analysis of the main limitations this search engine presents as a tool for the evaluation of scientific performance. Lastly, we discuss the main differences between GS and other more traditional bibliographic databases in light of the correlations found between their citation data. We conclude that Google Scholar presents a broader view of the academic world because it has brought to light a great amount of sources that were not previously visible.
2.1 Introduction
Google Scholar (GS) challenged subscription-based citation databases by automatically indexing a broad, changing range of academic web content. Its search utility and citation-based secondary products also drew interest in bibliometric assessment.
2.1 Introduction
- GS crawls academic websites and indexes diverse document types across subjects and languages; its records can disappear when their web sources become unavailable.Unlike the cumulative, selective approach of WoS and Scopus, GS reflects what its search robots can see on the web at a given time.Source: §2.1 · p. 2
- 89% of surveyed respondents chose GS for literature searches, ahead of WoS (41%), Pubmed (40%), Others (36%), and Scopus (26%).The survey received more than 20,000 responses from researchers, students, librarians, and other scholarly-community members.Source: §2.1 · p. 2
- GSC and GSM repurpose GS citation data for researcher profiles and rankings of scholarly publications, extending citation counts beyond their original role as search aids.Until these products appeared, citation counts were used to rank search results and help users find cited documents.Source: §2.1 · pp. 2–3
- The chapter evaluates GS as a possible bibliometric source despite that use not being an intended application, reviewing its operation, empirical properties, limitations, and citation-data relationships with traditional databases.Its coverage includes size, growth, indexing speed, sources, subjects, languages, and document types.Source: §2.1 · p. 3
2.2 Basic functioning of Google Scholar
- The section concisely describes how the GS search engine works, presents its main inclusion criteria for sources and especially documents, and briefly outlines Google Scholar Citations (GSC) and Google Scholar Metrics (GSM).Source: §2.2 · p. 3
2.2.1 The Academic Search Engine
Traditional databases select sources in advance, whereas GS searches the academic web through largely unsupervised indexing. This broad approach supports discovery but leaves ranking details undisclosed.
2.2.1 The Academic Search Engine
- Traditional databases select sources through whitelists, often prioritizing core journals for efficiency and their perceived role in research evaluation.This approach has historically favored journals over other channels such as books, reports, and conference proceedings.Source: §2.2.1 · pp. 3–4
- GS instead indexes scholarly documents it finds across the academic web, regardless of quality or peer-review status, to facilitate literature discovery.Its crawlers locate academic resources, extract metadata and, when possible, full text, then create records for search results.Source: §2.2.1 · p. 4
- When GS parses references, it links recognized cited works or creates a [CITATION] record when it cannot match a reference to an existing record.A citation-only record may later be merged with a full record if another web version is found.Source: §2.2.1 · pp. 4–5
- GS ranks search results using factors including document text, publication venue, authorship, and citation frequency and recency, but does not disclose their weights.The detailed ranking parameters are not publicly available.Source: §2.2.1 · p. 5
2.2.2 What sources does Google Scholar index?
GS indexes web sources rather than relying solely on selected journals, then automatically identifies and processes academic documents within them. Its inclusion depends on both source accessibility and document-level conditions.
2.2.2 What sources does Google Scholar index?
- GS crawls a wide variety of sources and indexes documents it identifies as academic, across subjects, languages, countries, years, and document types.The process starts with compiling sources likely to host academic content, including institutional websites, repositories, and discovery tools.Source: §2.2.2 · p. 5, Step 1: Compilation of sources · p. 5
- Websites can request inclusion, but GS expects accessible content without log-ins or other unreasonable access barriers, and may reject sites that lack abstracts or show error pages.Common repository and journal platforms, as well as personal websites, may be considered.Source: Step 1: Compilation of sources · p. 5
- GS describes scholarly articles, reports, dissertations, and preprints as appropriate, but shorter items such as book reviews may or may not be included.The stated inclusion criteria therefore leave room for variation in practice.Source: Step 2: Document types · p. 6
- GS identifies academic documents through document-structure parsing or by indexing all documents in locations such as institutional repositories.Because document-type metadata may be absent and source-level inclusion can override type distinctions, GS may include book reviews despite their stated inappropriateness.Source: Step 2: Document types · p. 6
- Document processing can fail when files exceed 5 MB, lack searchable text, or do not meet URL and metadata requirements.For PDFs, scanned-image text is not processed, and URLs are expected to end in .pdf; missing metadata may prompt GS to parse the document itself.Source: Step 3: Documents · p. 7
2.2.3 Google Scholar’s official bibliometric products
Google Scholar Citations and Google Scholar Metrics provide researcher- and journal-focused tools built on GS publication and citation data. Their indicators and rankings are automatically derived from that database, with defined time windows and inclusion rules.
Google Scholar Citations
- GSC profiles list researchers' publications and citation data alongside author-level indicators calculated for both all-time and recent five-year citation windows.The calculations are automatic and use the publications in each profile without human supervision.Source: Google Scholar Citations · p. 8, Table 1 · p. 8
- GSC lets researchers find publications, merge versions, and correct bibliographic errors in their own profiles, but those edits do not affect coauthors' profiles or GS Search.The profiles also support alerts and monitoring of researchers and papers.Source: Google Scholar Citations · pp. 8–9
- GSC lacks visible document-type identification and document-level affiliations, limiting its usefulness for bibliometric studies.A profile displays only one affiliation at a time, although a researcher's publications may span institutions.Source: Google Scholar Citations · p. 9
Google Scholar Metrics
- GSM ranks journals using citation indicators from recently published articles and sorts sources by h5-index rather than a journal-impact-factor-style formula.It displays the articles contributing to the h5-index, with their citation counts and citing-document lists.Source: Google Scholar Metrics · p. 9, Table 2 · p. 10
- GSM includes journals with at least 100 articles in five years and at least one citation, plus some Computer Science and Electrical Engineering proceedings and preprint repositories.It excludes court opinions, books, and dissertations; rankings are organized by language and, for English sources, subject categories.Source: Google Scholar Metrics · pp. 9–10, Table 3 · p. 10
2.3 Radiographing a ‘big data’ bibliographic source
- The following section examines GS's size, coverage, and growth rate as bibliographic properties.Source: §2.3 · p. 11
2.3.1 Size
Estimating Google Scholar’s size is difficult because the academic web is dynamic and unstable. Comparisons indicate a broader document and citation base than traditional databases, with differences varying by discipline and measure.
2.3.1 Size
- GS’s size is hard to determine because it reflects what is available on a dynamic, unstable, and uncontrollable academic web at a particular moment.Search-engine estimates also face stability, precision, and resource-persistence problems.Source: §2.3.1 · p. 11
- Direct size estimates use web-domain, year, or absurd queries, while indirect estimates use capture/recapture or English-language proportions against external sources.Web-domain and absurd-query methods have yielded similar, higher figures.Source: §2.3.1 · p. 11
- The article collection includes source documents and cited references, which can appear in the same results list, unlike in WoS and Scopus.Cited references may be linked records with basic bibliographic information or unlinked records found only in citing documents.Source: §2.3.1 · pp. 11–12
- The 63-study comparison results depend on discipline, geographic and linguistic scope, and sample size, which was generally small.The comparison also spans different measurement units, including citations, publications, and h-index.Source: §2.3.1 · p. 14
- GS finds more citations than WoS or Scopus in most studies, with the greatest differences in the Humanities and Social Sciences and smaller differences in STEM.Comparisons based on unique citations favor GS even more; differences are also greater for works in languages other than English.Source: §2.3.1 · p. 15
- GS citations were 1.54 times WoS citations for a sample of 2.32 million articles and reviews, and 1.80 times WoS citations for 69,261 highly cited documents.These document-level ratios are reported in the comparison of two large samples.Source: §2.3.1 · p. 15, Table 6 · p. 15
2.3.2 Coverage
Because GS does not publish a master list of sources, its coverage is investigated through the web domains and providers from which it indexes academic documents. Domain-based estimates show broad source diversity, but the resulting totals and source comparisons have important qualifications.
Source coverage
- GS provides no master list of sources, so studies identify its web domains and estimate contributions from providers such as publishers, repositories, and other Google products.Earlier studies also identified universities and other content providers among the sources of highly cited documents.Source: Source coverage · p. 16
- China accounts for 12.12% of content among geographic domains, commercial companies lead generic domains with 45.39%, and generic domains host 69.7% of content overall.The domain counts were obtained with site: queries across 268 top-level domains in March 2017.Source: Source coverage · pp. 16–17, Table 7 · p. 16
- The 197,194,092 documents counted across 268 domains support an estimate of around 200 million source documents, but duplicates may inflate it and primary-version domain mismatches may lower it.The figure is similar to estimates from the absurd-query method and Aguillo’s domain-query method.Source: Source coverage · p. 17
- The source tables cover digital libraries, publishers, learned societies, government and international organizations, and universities, using primary versions rather than ranking sources by total size.The tables aim to show source diversity and the quantity of records GS finds for each source.Source: Source coverage · p. 17, Table 8 · p. 18
SOURCE TYPE
The source-type material names examples of digital libraries and bibliographic information systems, alongside separate tables for publishers and learned and professional societies.
SOURCE TYPE
- The listed digital-library and information-system examples include Europe PubMed Central, the Astrophysics Data System, and Semantic Scholar.They are labeled respectively as a subject repository, database, and search engine.Source: RANK DL & BIS URL · p. 18
- A separate table covers publisher sources of Google Scholar.Source: Table 9 · p. 18
- Another table covers learned and professional societies among Google Scholar sources.Source: Table 10 · p. 18
RAN K
This section examines Google Scholar’s coverage and growth, alongside limitations in its search counts, indexing, document classification, and bibliographic and citation records.
RAN K
- Google Scholar draws on bibliographic systems, publishers, search engines, repositories, social platforms, Google Books, agencies, associations, and universities, but source hit counts likely underestimate coverage because site: searches match only primary versions.Results for universities can also reflect bibliographic products hosted on their domains; hit counts are approximate, and GS groups document versions under a primary record.Source: RAN K · pp. 19–20
Geographic coverage
- Among Google Scholar results, excluding cited references and patents, 49.8% were English and 33.7% Chinese; English represented 90% of WoS and 80% of Scopus results.Source: Geographic coverage · pp. 23–24
Discipline coverage
- 95.2% (1,038) of 1,090 Spanish Social Sciences journals in IN-RECS were covered by GS, while GSM covered 36.97%; a broader journal sample found GSM covered 20.8%, below SJR’s 32%.GSM’s inclusion criteria require at least 100 articles in five years and one citation; reported citation coverage ratios are higher in Social Sciences, Humanities, and Engineering than in STEM, and Chemistry’s earlier coverage problems were resolved.Source: Discipline coverage · pp. 25–26, Figure 8 · p. 26
Document type coverage
- Journal articles make up 28% to 70% of GS documents across samples, compared with 90% in WoS and Scopus, reflecting GS’s more diverse document types.A 861,843-document query sample identified types for 53.8% of documents; its maximum of 1,000 relevance-ranked results per query biases the sample toward highly cited documents.Source: Document type coverage · pp. 26–28, Figure 9 · p. 27
2.3.3 Growth rate
- Citation counts grew 4.6% for 20 Nobel Prize winners and 4.4% for 146 University of Melbourne senior researchers over roughly a year; GS indexed JASIST articles in one to three days in a small study.Across Scopus-covered journals, the median delay for GS indexing was about two months relative to Scopus; in the JASIST sample, Scopus indexed through March 13, while WoS had indexed none of the sampled articles before formal publication.Source: §2.3.3 · pp. 28–30, Table 20 · p. 29
2.4 Google Scholar’s data for scientometric analyses
- The chapter distinguishes errors in GS’s processing, such as duplicate citations or incomplete authorship, from design limitations that may make it unsuitable for bibliometric analysis.Those limitations arise when the search engine is used for a purpose other than its original one: searching academic material.Source: §2.4 · p. 30
2.4.1 Errors in Google Scholar
- Published studies of Google Scholar errors are sparse, often anecdotal and unsystematic, difficult to compare, and quickly outdated as the service changes.They rarely use representative samples or measure how pervasively errors occur across the document base.Source: §2.4.1 · pp. 30–31
- Following Jacsó, the section proposes four error groups—search-related, parsing-related, matching, and source-related—that can affect bibliographic records and citations.A more in-depth discussion was published by Orduna-Malea, Martín-Martín, and Delgado López-Cózar.Source: §2.4.1 · p. 31
TYPE OF ERROR
The chapter presents Google Scholar as a broader bibliometric source than traditional databases while documenting limitations in search, data quality, reuse, and transparency. Its comparison evidence reports high citation correlations alongside higher Google Scholar citation counts in one large matched sample.
2.4.2 Google Scholar limitations
- The limitations analysis uses descriptive sheets based on observation and empirical tests, covering source coverage, search interface, data quality, and reuse; search results and filters are restricted.The platform has no public master list of indexed sources, and its query interface limits displayed results and offers few filtering options.Source: §2.4.2 · p. 31, GOOGLE SCHOLAR · p. 32
GOOGLE SCHOLAR
- Missing affiliation, language, and document-type data, parsing errors, duplicate records, and limited bulk exporting complicate bibliometric analysis and data reuse.Duplicate records can split citation counts, and records must undergo substantial cleaning and normalization.Source: GOOGLE SCHOLAR · pp. 33–34
2.5 The expanded academic world of Google Scholar
- Google Scholar’s size, coverage, and growth support treating it as a big-data bibliographic tool that gives a broader picture of academic activity across countries, languages, and document types.The chapter characterizes its coverage as extending beyond scientific sources to academic and professional activity.Source: §2.5 · pp. 34–35
- Unlike WoS and Scopus, Google Scholar automatically indexes academically structured content without quality-based selection, bringing reviewed and unreviewed sources together.The chapter says it covers prestigious publications found in WoS and Scopus, as well as other sources, some of comparable or higher quality in certain disciplines.Source: §2.5 · pp. 35–36, Figure 11 · p. 35
- Across studies, average correlations were 0.76 for Google Scholar versus WoS and 0.81 versus Scopus; correlations tend to be highest in STEM and lowest in Humanities and Social Sciences.The authors caution that samples and methods vary, and correlations may use different techniques.Source: §2.5 · p. 36
- A Spearman correlation of Rs=0.9 was found for 69,279 highly cited documents covered by both Google Scholar and WoS, and Google Scholar often found more citations for the same documents.The sample came from 861,843 highly cited documents in 13 languages; 69,279 (8%) were covered by WoS and had at least one citation.Source: §2.5 · p. 37, Figure 12 · p. 37
2.6 Final remarks
- Google Scholar can reveal document types and researchers previously overlooked by traditional databases, potentially changing how research impact is assessed.The chapter specifically identifies monographs, theses, reports, conference communications, and book chapters as newly visible sources.Source: §2.6 · pp. 37–38
- Data collection is slow and costly, records omit important fields, and opaque indexing permits empirically documented publication or citation manipulation.The authors identify manipulation risk and limited transparency about coverage, updates, and ranking as particularly serious concerns.Source: §2.6 · p. 38