Source-linked AI summary
The Google Scholar Experiment: how to index false papers and manipulate bibliometric indicators
Emilio Delgado López-Cózar, Nicolás Robinson-Garcia, Daniel Torres-Salinas
TL;DR
The paper examines how easily Google Scholar citation data can be manipulated, amid pressure on researchers and journal editors to increase impact. Its experiment finds that citation profiles are easy to modify, with effects extending to Google Scholar Metrics and varying by researcher experience.
Problem
The paper addresses concerns that pressure to advance academic careers and increase impact makes Google Scholar citation data susceptible to manipulation.
Method
The authors conduct an experiment to demonstrate how easily Google Scholar citation profiles can be modified and how publications can influence its indicators.
Results
The experiment shows that Google Scholar citation profiles are easy to modify, indirectly affecting Google Scholar Metrics, while h-index variation differs between experienced and young researchers.
Takeaways & Limitations
The findings raise serious concerns about using Google Scholar and its products as research evaluation tools and alert readers to a main shortcoming of Google Scholar Citations.
Takeaways & Limitations
The paper limits its concern to Google Scholar's susceptibility to citation-data manipulation and its implications for research evaluation.
Abstract
from arXiv · showhide
Google Scholar has been well received by the research community. Its promises of free, universal and easy access to scientific literature as well as the perception that it covers better than other traditional multidisciplinary databases the areas of the Social Sciences and the Humanities have contributed to the quick expansion of Google Scholar Citations and Google Scholar Metrics: two new bibliometric products that offer citation data at the individual level and at journal level. In this paper we show the results of a experiment undertaken to analyze Google Scholar's capacity to detect citation counting manipulation. For this, six documents were uploaded to an institutional web domain authored by a false researcher and referencing all the publications of the members of the EC3 research group at the University of Granada. The detection of Google Scholar of these papers outburst the citations included in the Google Scholar Citations profiles of the authors. We discuss the effects of such outburst and how it could affect the future development of such products not only at individual level but also at journal level, especially if Google Scholar persists with its lack of transparency.
Introduction
Google Scholar’s free access and broad coverage helped popularize its researcher- and journal-level bibliometric products, especially in the Social Sciences and Humanities. The introduction frames concerns that uncontrolled indexing and limited transparency make citation counting vulnerable to manipulation and research-evaluation risks.
- Introduction: Google Scholar’s free, universal access and easy interface contributed to the popularity of Google Scholar Citations and Google Scholar Metrics.Google Scholar Citations provides researcher citation profiles, while Google Scholar Metrics ranks journals by h-index for publications from the previous five years.
- Introduction: Open Access repositories supplied Google Scholar with distinctive content, including preprints and theses, while gaining visibility in return.The introduction presents this relationship as mutually beneficial and as part of broader changes in scholarly communication.
- Introduction: Researchers under pressure to demonstrate impact, particularly in the Social Sciences and Humanities, were attracted to tools offering greater visibility.The text links this attraction to funding and career demands and to the perceived limitations of traditional multidisciplinary databases.
- Introduction: Prior studies criticized Google Scholar for inconsistent citation counts, metadata errors, lack of quality control, and insufficient transparency.The lack of transparency prevents users from certifying that the information is correct, especially when interpreting unusual bibliometric patterns.
- Introduction: Google Scholar automatically retrieves and indexes uploaded scientific material without prior external control, allowing users to modify output that affects bibliometric performance.The introduction connects this design to ethical and sociological dilemmas in research evaluation.
- Introduction: Earlier experiments showed that fake papers, altered references, duplicates, and other procedures could manipulate Google Scholar results and improve visibility.These warnings predated Google Scholar Citations and Google Scholar Metrics, while related research examined whether systems could distinguish academic from fabricated content.
- Introduction: This paper tests how easily Google Scholar’s tools can be manipulated by uploading repeated fake documents citing a research group’s publications.The study focuses on consequences for research evaluation rather than the technical details of gaming, and also examines Google’s handling of retracted documents.
- Introduction: The experiment examines effects on researchers’ Google Scholar Citations profiles and discusses possible consequences for Google Scholar Metrics rankings.The paper reports the inclusion of false documents, their effects on bibliometric profiles, Google Scholar’s reaction, and implications of limited transparency.
Material and methods
The experiment tested whether Google Scholar could automatically index false documents and propagate their citations into researcher and journal metrics. Researchers uploaded one test paper and then six documents by a fabricated author, each citing EC3 publications.
- The experiment examined Google Scholar’s difficulty in automatically including false documents and the consequences for Google Scholar Citations and Metrics.
- The researchers first uploaded a paper referencing 36 publications by Daniel Torres-Salinas; Google Scholar Citations then notified him that all his papers had been cited.
- They divided the text into six documents, formatted as papers with titles, abstracts, authors, figures, and references to the EC3 group’s publications.
- A fabricated researcher named Marco Alberto Pantani-Contador authored the documents, which were publicly available rather than intended as published papers.
- Each document cited 129 EC3 papers, producing an expected total increase of 774 citations.
- The documents were linked from an HTML webpage under the University of Granada institutional domain, while repositories were excluded because the study did not target their bibliographic filters.
Effects and consequences on the manipulation of citation data
Google Scholar indexed the false documents and their citations substantially altered researcher profiles, journal rankings, and coauthor records. The effects were strongest for less-cited researchers and varied across bibliometric indicators.
- Google Scholar indexed the false documents nearly one month after upload, on May 12, 2012.
- NRG multiplied received citations by 7.25, DTS doubled them, and EDLC increased citations by 1.5.
- El Profesional de la Información would have risen from position 20 to 5 because seven papers surpassed the 12-citation threshold.
- Revista Española de Documentación Científica would have risen from position 74 to 54 because one article surpassed the 9-citation threshold.
- The manipulation affected 51 journals and a total of 47 authors, including coauthors outside the targeted research group.Examples include 60 additional citations across 10 Scientometrics papers, 18 across three JASIST papers, and six for one British Medical Journal paper.
Detection and suppression of false documents
Google Scholar removed the fabricated researcher and cleaned the affected profiles after the experiment became public, but the response followed a complaint rather than apparent automatic detection. An initial test record remained cached.
- The results were made public on May 29, 2012 through a working paper in the University of Granada repository and the research group’s blog.
- Google later erased Pantani-Contador and quarantined the paper authors’ Google Scholar Citations profiles before cleaning and republishing them.
- The false record was eventually removed from the authors’ profiles after restitution, while the initial testing document remained cached after its PDF disappeared from the web.
- The authors interpret the suppression as a reaction to their complaint rather than evidence that Google Scholar had uncovered the deception.
Discussion
The experiment indicates that Google Scholar’s bibliometric products are vulnerable to citation manipulation, with effects on researcher and journal indicators. The discussion therefore calls for stronger monitoring and more rigid evaluation criteria.
- Google Scholar’s citation profiles are easy to modify, raising concerns about using GS products as research-evaluation tools.The discussion links this vulnerability to citation manipulation and insufficient quality control.
- Citation manipulation can affect both individual researcher indicators and journal rankings, especially when small metric changes alter ranking positions.The discussion notes that journal h-index values can have substantial ranking effects when underlying figures are small.
- Because GS Citations and GS Metrics are evaluation tools, the authors argue that they should include filters, monitoring tools, and more rigid criteria.The paper contrasts these needs with Google Scholar’s broad indexing and retrieval remit.
- The discussion attributes the vulnerability partly to methodological and technical errors, including citation-identification problems, unclear coverage, and limited quality control.It also notes that susceptibility to citation manipulation had been identified previously without subsequent correction.
- The h-index remains relatively stable for experienced researchers but varies significantly for young researchers under the manipulation examined.The variation is described as being influenced by researchers’ previous performance.
- The i10-index shows the opposite pattern, varying more significantly for experienced researchers.
Concluding remarks
The experiment shows that Google Scholar citation counting is easy to manipulate and that its lack of transparency obstructs research-evaluation oversight. The authors recommend clearer citation and document-source information while recognizing that researchers’ ethical values remain the strongest control against fraud.
- Google Scholar’s main shortcoming is the ease with which its citation counting can be manipulated.
- The experiment used six false documents attributed to a false researcher and citing a research group’s publications, demonstrating that even clumsy manipulation can affect Google Scholar citations.
- A more refined manipulation could have gone unnoticed, and excluding self-citations alone may not detect such malpractice.
- Google Scholar deleted alerted false documents without notifying affected authors while leaving the initial testing document, making its response worrisome.
- Greater transparency should include self-citation counts, h-index values excluding self-citations, and filters distinguishing document types and retrieval sources.
- The authors argue that researchers’ ethical values are the most efficient safeguard, while Google Scholar should reduce the temptation to manipulate metrics through transparency.