PaperlayerView exact revision on arXiv

Source-linked AI summary

The distorted mirror of Wikipedia: a quantitative analysis of Wikipedia coverage of academics

Anna Samoilenko, Taha Yasseri

arXiv:1310.8508v2physics.soc-phcs.CYcs.DLcs.SIphysics.data-an

TL;DR

Prior research on Wikipedia’s academic coverage relied on small samples limited to one discipline, while Wikipedia is a highly visible source of information. The study sampled 400 Wikipedia biographies across four fields and compared Wikipedia and Scopus metrics for scholars with Scopus IDs. No tested Wikipedia–Scopus metric pair was significantly correlated; in all four fields, F-scores fell below random assignment and highly cited researcher coverage stayed below 50%.

  • Problem

    Prior research on Wikipedia’s academic coverage relied on small samples limited to one discipline, while Wikipedia is a highly visible source of information.

    Source: Introduction · pp. 2–3

  • Method

    The study sampled 400 Wikipedia biographies across four fields and compared Wikipedia and Scopus metrics for scholars with Scopus IDs.

    Source: Results · p. 5, Data Sources and Collection · p. 9

  • Results

    No tested Wikipedia–Scopus metric pair was significantly correlated; in all four fields, F-scores fell below random assignment and highly cited researcher coverage stayed below 50%.

    Source: Results · p. 5, Discussion · p. 8

  • Takeaways & Limitations

    Wikipedia offers a different image of researchers from academic prominence and may introduce subjectivity into the public image of science.

    Source: Discussion · pp. 7–8

    The study measures scholarly notability only through citation metrics, while Scopus excludes publications before 1966 and covers 12,850 reviewed journals.

    Source: Discussion · p. 8

Abstract

from arXiv · show

Activity of modern scholarship creates online footprints galore. Along with traditional metrics of research quality, such as citation counts, online images of researchers and institutions increasingly matter in evaluating academic impact, decisions about grant allocation, and promotion. We examined 400 biographical Wikipedia articles on academics from four scientific fields to test if being featured in the world's largest online encyclopedia is correlated with higher academic notability (assessed through citation counts). We found no statistically significant correlation between Wikipedia articles metrics (length, number of edits, number of incoming links from other articles, etc.) and academic notability of the mentioned researchers. We also did not find any evidence that the scientists with better WP representation are necessarily more prominent in their fields. In addition, we inspected the Wikipedia coverage of notable scientists sampled from Thomson Reuters list of "highly cited researchers". In each of the examined fields, Wikipedia failed in covering notable scholars properly. Both findings imply that Wikipedia might be producing an inaccurate image of academics on the front end of science. By shedding light on how public perception of academic progress is formed, this study alerts that a subjective element might have been introduced into the hitherto structured system of academic evaluation.

Introduction

The study examines whether Wikipedia representation tracks academic notability across four fields, against a backdrop of fragmented prior evidence and Wikipedia’s visibility. It finds poor alignment between encyclopedia presence and citation-based prominence.

Introduction

  • Prior research on Wikipedia’s coverage of academic topics was based on small samples limited to one discipline, leaving a fragmented view.Wikipedia is a prominent information source: 29.6% of academics in a cited survey preferred it to online library catalogues, and it ranked in the top five UK Google results in 96% of cases.Source: Introduction · pp. 2–3
  • The study examined 400 living academics’ Wikipedia biographies across Biology, Physics, Computer Science, and Psychology and Psychiatry, comparing article metrics with Scopus citation metrics.It also examined Wikipedia coverage of a random sample of 219 academics from Thomson Reuters’ list of highly cited researchers.Source: Introduction · p. 4, Results · p. 5

Results

  • Only 36% of Biologists, 31% of Computer Scientists, 24% of Psychologists and Psychiatrists, and 22% of Physicists on Wikipedia exceeded their field’s average h-index.The comparison uses field-specific averages from previous research.Source: Results · p. 4
  • No tested Wikipedia–Scopus metric pair was significantly correlated, and F-scores fell below random assignment in all four fields.The authors interpret this as failure on both dimensions: Wikipedia presence does not signify academic notability, and notability does not guarantee coverage.Source: Results · p. 5, Table 3 · p. 7
  • Wikipedia omitted 52% of sampled prominent Computer Scientists, 62% of Psychologists and Psychiatrists, 67% of Biologists and Biochemists, and 78% of Physicists.The highly cited researchers were sampled from a list published in 2010, and the authors say the omissions could not be attributed to time constraints.Source: Results · p. 5

Discussion

  • Wikipedia’s representation offers a different image of researchers from that suggested by academic prominence, and its metrics provide no reliable cues to notability.The authors conclude that Wikipedia metrics and citation metrics measure different phenomena; they suggest Wikipedia may reflect social impact or public engagement instead.Source: Discussion · pp. 7–8
  • The study measures scholarly notability only through citation metrics, and Scopus excludes publications before 1966 and covers 12,850 reviewed journals.Source: Discussion · p. 8

Data Sources and Collection

The study sampled living academics from four English Wikipedia categories, screened articles for eligibility, and recorded Wikipedia article metrics.

Data Sources and Collection

  • Researchers were selected from category lists for biologists, physicists, computer scientists, and psychologists and psychiatrists.The authors randomly sampled 300 entries per list, then manually selected the first 100 eligible articles in each field.Source: Data Sources and Collection · p. 9
  • Eligible articles featured living researchers, lacked biography-notability warnings, and matched the assigned category’s field.Source: Data Sources and Collection · p. 9
  • Wikipedia measures included article length, views, language editions, links, and counts of editors and edits, with bot activity also separated.The study collected editor and edit counts both including and excluding bots.Source: Data Sources and Collection · p. 9

Scopus

The authors matched researchers from the selected Wikipedia articles to Scopus profiles and manually verified the citation data.

Scopus

  • Scopus metrics included documents, citations, co-authors, years active, h-index, and mean citations per paper.Author ID was also collected.Source: Scopus · p. 9
  • Researchers were searched in Scopus, and profile data were manually collected and checked to address name ambiguity.Source: Scopus · p. 9
  • When a researcher had two Scopus profiles, the authors used the profile with the most citations.Source: Scopus · p. 9
  • The complete datasets are available in Additional Dataset A1.Source: Scopus · p. 9

Prominent researchers

The study sampled highly cited researchers in four fields and checked whether each had a Wikipedia article.

Prominent researchers

  • Researchers were drawn from Thomson Reuters’ ISI Highly Cited Research list, based on highly cited Web of Science publications from 1981–2008.Source: Prominent researchers · p. 10
  • Systematic random samples included 49 biology and biochemistry researchers, 63 computer scientists, 54 physicists, and 53 psychology and psychiatry researchers.The authors selected every sixth entry from alphabetized field lists.Source: Prominent researchers · p. 10
  • Each sampled researcher was searched on Wikipedia for a corresponding article, with data collected in August 2013.Source: Prominent researchers · p. 10

Data Analysis

The authors tested associations between Wikipedia and Scopus statistics using transformed data and linear regression.

Data Analysis

  • Histograms were inspected for normality, variables were log-transformed for skewness, and R^2 from linear regression measured associations across variable pairs.Source: Data Analysis · p. 10
  • Researchers without Scopus IDs were examined as separate cases.Source: Data Analysis · p. 10

Competing interests

Author’s contributions

Additional Files

The additional files provide two datasets: one compiling Wikipedia metrics and Scopus data across four disciplines, and another sampling prominent researchers by field from Thomson Reuters’ “ISI Highly Cited Research” list.

Additional file 2 (.xls) — Dataset A1

Additional file 3 (.xls) — Dataset A2

Loading 1310.8508v2…