Source-linked AI summary
Over-Optimization of Academic Publishing Metrics: Observing Goodhart's Law in Action
Michael Fire, Carlos Guestrin
TL;DR
Academic publishing metrics are increasingly used as targets even as publication volume, author collaboration, citation practices, and journal structures change. Using large-scale scholarly datasets, the study analyzes these changes across more than 120 million papers and thousands of research fields. It finds that citation-based measures have lost validity and usefulness, especially for comparing academic entities across fields, motivating careful development of new measures.
Problem
Established metrics such as publication counts, citation counts, h-index, and impact factor have remained standard despite major changes in academic publishing.
Method
The study analyzes over 120 million publications using large-scale datasets and an open-source framework to examine publishing trends from researchers to research fields.
Results
Citation-based measures are compromised by publishing-volume growth, self-citations, changing journal patterns, and wide variation across more than 2600 research fields.
Takeaways & Limitations
Citation number, h-index, and impact factor are no longer good measures for comparing researchers across different fields or even within some subfields.
Takeaways & Limitations
The study notes that missing keywords in the datasets may underestimate the actual number of multidisciplinary papers.
Abstract
from arXiv · showhide
The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which "when a measure becomes a target, it ceases to be a good measure." In this study, we analyzed over 120 million papers to examine how the academic publishing world has evolved over the last century. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of over 2600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Academic publishing has changed considerably; now we need to reconsider how we measure success.
Introduction
Academic publishing has expanded rapidly while established success metrics increasingly function as targets. An analysis of large-scale publishing trends examines how Goodhart’s Law affects papers, authors, journals, and research fields.
- Introduction: 174,000 papers were published in 1950 versus over 7 million in 2014, alongside faster dissemination through new venues.Researchers now publish through traditional venues, preprint repositories, and mega-journals.
- Introduction: The study analyzes over 120 million publications to assess publishing changes from individual researchers to large research domains.The authors developed an open-source framework using large-scale datasets containing references and authors.
- Introduction: Papers became shorter while titles, abstracts, author lists, references, self-citations, and uncited papers increased over time.The study reports a sharp increase in papers receiving no citations at all.
- Introduction: Authors multiplied, published more rapidly early in their careers, and collaborated with more coauthors over time.The study observed a negative relation between career age and publication number.
- Introduction: Ranked journals proliferated, their citation-based indicators shifted, and returning authors became more common.In Cell, about 80% of 2016 papers included at least one returning author, compared with less than 40% in 1980.
- Introduction: These patterns support the view that paper counts, h-index, and citation counts have become targets that researchers can optimize.Shorter papers, more coauthors, self-citations, longer titles and abstracts, and more references can raise these measures.
- Introduction: Citation-based measures are losing validity and usefulness, illustrating Goodhart’s Law in academic publishing.The paper argues that established measures should be reconsidered as indicators of academic success.
2. Related Work
Related work frames this study within scientometrics and documents changing publication practices, established impact metrics, their manipulation, and proposed alternatives.
- 2. Related Work: Scientometrics, or the science of science, studies quantitative features and characteristics of scientific research.The reviewed literature covers publication changes and metrics used to measure impact.
- 2. Related Work: Publication trends include more preprints, mega-journals, and papers with hundreds of authors, alongside concerns about honorary and ghost authorship.The cited LIGO paper had over 1,000 authors.
- 2. Related Work: Researchers have documented longer titles and greater online attention for positively framed, interestingly phrased titles.These findings motivate examining how presentation relates to scholarly attention.
- 2. Related Work: Common success measures include publication counts, citation counts, impact factor, and h-index, but citation practices differ across fields.The literature notes that citations are not equivalent and cross-field comparison is problematic.
- 2. Related Work: Goodhart-related metric manipulation includes self-citation, slicing studies into minimal publishable units, false-paper indexing, and other practices.Survey evidence found widespread misattribution, researcher disapproval, and pressure to participate.
- 2. Related Work: Alternatives include q-index, w-index, SJR, citation distributions, altmetrics, and Semantic Scholar measures, though traditional metrics remain widely used.Journal Citation Reports continues to publish widely followed impact-factor rankings.
3.1 Datasets
The study combines large scholarly datasets to analyze papers, authors, citations, journals, and research fields across historical publishing records.
- 3.1 Datasets: The Microsoft Academic Graph contains over 120 million papers and relationships among publications, authors, institutions, journals, conferences, and fields.It also records author sequence numbers and a four-level field-of-study hierarchy.
- 3.1 Datasets: The analysis focused on 120.7 million papers published through the end of 2014 to use the most comprehensive years of the dataset.The dataset also includes news items, response letters, and comments that may affect trend interpretation.
- 3.1 Datasets: AMiner supplied paper attributes and abstracts for estimating paper lengths and validating observed patterns against MAG.The dataset contains titles, keywords, abstracts, venues, languages, and ISSNs.
- 3.1 Datasets: The SJR dataset covers over 23,000 journals from 1999 to 2016 with yearly ranking, publication, h-index, citation, and quartile information.These fields support analysis of journal publication and impact trends.
- 3.1 Datasets: MAG and AMiner were joined using unique DOI values, then matched to SJR through ISSN values.This linked publication records with journal ranking measures such as h-index and SJR.
3.2 Code Framework
The open-source code framework combines scalable dataframe analysis with MongoDB queries to compute publication, author, venue, and field properties.
- 3.2 Code Framework: SFrame enabled big-data analysis on tens of millions of records and calculation of changing features such as author averages and title lengths.It was useful for statistics over all-paper features.
- 3.2 Code Framework: SFrame was less computationally effective for complex queries, including journal- and year-specific author-age calculations.The framework therefore used another database for more complicated analyses.
- 3.2 Code Framework: MongoDB collections supported straightforward calculations over papers, authors, paper collections, venues, and research fields.Examples include average coauthors per author for career cohorts in specified years.
- 3.2 Code Framework: Jupyter Notebook tutorials document construction of the SFrame and MongoDB collections to support reproducibility.The tutorials use the MAG, AMiner, and SJR datasets.
3.3 Analysis of Publication Trends
The study measured how papers, authors, journals, and research fields changed over time using large publication datasets and systematically calculated structural, citation, authorship, journal, and field-level features.
- Paper trends: The analysis tracked paper counts, language, title structure, punctuation, authorship, abstracts, keywords, references, citations, and self-citations over time.
- Author trends: Author trends were measured through new-author counts, career-stage publication rates, venue-specific output, coauthor counts, author sequence numbers, and first-author percentages.
- Definitions and scope: Self-citations were defined as citations between papers sharing at least one author, while analyses applied dataset and paper-selection restrictions.
- Journal trends: Journal trends used SJR data from 1999–2016 to examine prior top-journal publishing by first and last authors, journal activity, output, and changing metrics.
- Research fields: The study additionally matched papers to hierarchical research fields and calculated field-level publication, authorship, reference, and five-year citation features.
4. Results
Across the study period, academic publishing expanded while papers became shorter but their titles, abstracts, references, and author lists grew. Self-citations and uncited-paper totals increased, publication rates and coauthor counts rose, and journal output and citation-related metrics shifted substantially.
- Paper trends: Over 7 million papers were published annually in recent years, while papers with non-English titles also increased.
- Paper trends: Title length rose from 8.71 words in 1900 to 11.83 words in 2014, while average title-word length increased from 5.95 to 6.6 characters.
- Paper trends: The average number of authors per paper increased from 1.41 to 4.51 between 1900 and 2014, and alphabetical ordering declined from 43.5% to 21%.
- Paper trends: Abstract length increased from 116.3 words in 1970 to 179.8 words in 2014, while longer abstracts and more keywords became increasingly common.
- Citation trends: Self-citations increased from 3.67% of papers in 1950 to 8.29% in 2014, while the yearly maximum rose from 10 to over 250 self-citations.
- Paper trends: Average and median paper lengths decreased, with average length falling from 14.4 pages in 1950 to 8.4 pages in 2014.
- Citation trends: In 2009, 72.1% of all papers and 25.6% of papers with at least 5 references received no citations after five years.
- Author trends: Younger author cohorts published more papers, coauthor counts increased, and early-career researchers appeared as first authors less often than earlier generations.
5. Discussion
Academic publishing has changed substantially in its paper, author, journal, and field-level patterns. These changes compromise the usefulness of publication and citation-based measures as indicators of research impact and comparisons.
- Paper trends: Paper structures changed as papers became shorter while titles, abstracts, references, and author lists became longer.The number of keywords and papers without citations also increased over time.
- Paper trends: Citation numbers became less reliable as self-citations increased, more than 72% of papers had no citations after five years, and citation distributions differed across decades.These patterns make comparisons between researchers publishing in different periods challenging.
- Author trends: Researchers increasingly collaborated and published early in their careers, alongside exponential growth in the number of new researchers.Younger researchers also published more quickly and collaborated more than researchers in previous generations.
- Journal trends: Journal expansion and rising publication volumes weakened the meaning of quartiles and h-index measures while average citations and SJR increased.The number of ranked journals exceeded 20,000, and some journals published over 1,000 papers annually.
- Journal trends: Top journals published more papers involving older and returning authors, suggesting that impact-factor competition may favor established researchers.The selected journals saw sharp increases in paper counts, author career age, and returning-author percentages.
- Fields-of-research trends: Across more than 2,600 fields, differences in publication volume, references, and citation counts make citation number, h-index, and impact factor unsuitable for cross-field comparisons.The authors also report that these measures can be problematic even within the same subfield and may affect resource allocation.
6. Conclusions
A large-scale analysis shows that academic publishing and its evaluation environment have changed substantially, while citation-based measures have become less meaningful and useful as targets. The study argues for developing carefully evaluated, field-specific measures using data-science tools and open datasets.
- 120 million papers and over 20,000 journals were analyzed to document major changes in papers, authors, journals, paper lengths, and references.The analysis covers trends over the last century, especially the most recent decades.
- Citation-based measures have not kept pace with the changed publishing environment and have become less meaningful, useful, or accurate when treated as targets.The paper identifies this pattern as an illustration of Goodhart’s Law.
- Citation-based impact measures overlook unequal citation significance, journal-based distortions, extensive coauthorship, and large citation differences across research fields.The study cautions against comparing papers or researchers across fields and even within the same parent field.
- Limited-impact papers, self-citations, and concentration of publications among recurring authors in some top journals are among the undesirable effects associated with target-driven citation measures.The paper describes some top journals as “old boys’ clubs” publishing mainly papers from the same researchers.
- The authors propose using data-science tools and open datasets to develop more accurate field-specific impact measures, while evaluating them carefully to avoid creating new targets.The proposed measures should assess impact within a specific research field and be checked against Goodhart’s Law.
A.1 Additional Results
The additional results document broad changes in publication volume, paper structure, authorship, journals, and citation patterns. They also show substantial variation across fields and subfields in publishing, authorship, references, and five-year citations.
- Publication trends: Paper volume increased drastically across fields, with substantial diversity in the number of papers published by different research fields and subfields.The figures report this pattern for L0 fields, biology subfields, and genetics subfields.
- Field differences: Median five-year citation counts showed notable or high variance across L0 fields, biology and genetics subfields, and L3 fields grouped by parent fields.The figures consistently report cross-field variation in citation distributions.
- Authors: The average number of authors rose over time, while the maximum increased from 520 authors in 2000 to over 3100 in 2010.Average authorship also varied across L0 fields, biology subfields, and genetics subfields.
- References: The average number of references increased sharply, and papers with relatively high reference counts became more common over time.Reference counts also varied across L0 fields, biology subfields, and genetics subfields.
- Citations: In 2009, over 72.1% of published papers had no citations after five years, despite the share of uncited papers decreasing over time.Self-citations and the percentage of papers containing them both increased significantly over time.
- Author careers: Publication rates increased with each decade, new-author counts rose, and median author sequence numbers increased over time.The figures associate higher sequence numbers with senior researchers.
- Journals: The number of ranked journals increased drastically, with hundreds of new ranked journals published each year.Journal h-index values decreased over time, whereas SJR values increased.
- Top journals: Top journals experienced sharp increases in papers and authors, while papers with at least five references and papers containing returning authors also became more common.In many selected journals, returning authors appeared in more than 60% of papers and in some cases more than 80%.