Source-linked AI summary
Popular and/or Prestigious? Measures of Scholarly Esteem
Ying Ding, Blaise Cronin
TL;DR
Citation counts usually treat all citations equally, even though popularity and prestige may reflect different forms of scholarly esteem. The paper compares author popularity and prestige among leading information-retrieval researchers over time and finds that some rank highly on prestige but not popularity, and vice versa. It also examines how these measures relate to researcher characteristics and honors.
Problem
Citation analysis generally does not account for the quality of citing articles, limiting distinction between popularity based on total citations and prestige based on citations from highly cited papers.
Method
The study compares popularity and prestige rankings for 40 leading information-retrieval researchers across time and relates them to researcher characteristics and honors.
Results
Some authors rank highly on prestige but not popularity, while others rank highly on popularity but not prestige.
Takeaways & Limitations
Prestige and popularity rankings measure slightly different dimensions of peer esteem.
Takeaways & Limitations
The model does not consider the graph topology of citation networks and is planned for further application at the paper level.
Abstract
from arXiv · showhide
Citation analysis does not generally take the quality of citations into account: all citations are weighted equally irrespective of source. However, a scholar may be highly cited but not highly regarded: popularity and prestige are not identical measures of esteem. In this study we define popularity as the number of times an author is cited and prestige as the number of times an author is cited by highly cited papers. Information Retrieval (IR) is the test field. We compare the 40 leading researchers in terms of their popularity and prestige over time. Some authors are ranked high on prestige but not on popularity, while others are ranked high on popularity but not on prestige. We also relate measures of popularity and prestige to date of Ph.D. award, number of key publications, organizational affiliation, receipt of prizes/honors, and gender.
1. Introduction
The paper distinguishes scholarly popularity from prestige by considering not only how often researchers are cited but also the quality of citing articles. Using information retrieval as its test field, it examines how these measures vary over time and relate to researcher characteristics.
- Conceptual distinction: Popularity and prestige are related but are not identical measures of scholarly esteem.A scholar may rank highly on one measure without necessarily ranking highly on the other.
- Study contribution: The study incorporates the quality of citing articles when assessing researchers’ standing in information retrieval.This approach treats citations from highly cited papers as a distinct indicator of prestige.
- Conceptual distinction: Popularity counts an author’s citations from all papers, whereas prestige counts citations from highly cited papers.The distinction is based on the presumptive quality of citations.
- Study contribution: The authors track changes in researchers’ popularity and prestige rankings over time.The study compares the two ranking dimensions longitudinally.
- Study contribution: The study relates popularity and prestige to Ph.D. award date, honors and prizes, key publications, organizational affiliation, and gender.These variables are examined as correlates of the two esteem measures.
2. Related Work
Related citation research ranges from simple counts to weighted influence measures such as PageRank, h-index variants, and network-based analyses. The paper addresses a recurring gap: author-level studies have generally not distinguished raw citation popularity from prestige based on citing-article quality.
- Citation-analysis approaches: Citation analysis includes straightforward counts as well as co-citation, social-network, PageRank, and h-index approaches.These methods assess influence at author, paper, journal, field, or community levels.
- Applications: Prior studies have examined citation patterns in fields including Alzheimer’s research, biomedical research, and information science.Examples include h-index calculations, award comparisons, productivity, impact, creativity, and professional age.
- Research gap: Most citation studies still ignore the variable quality of citing articles, and few apply weighted approaches at the author or paper level.This motivates the paper’s author-level weighted citation-counting approach.
- Weighted measures: Weighted citation methods assign greater influence to citations from prestigious or highly influential journals.Examples include weighted impact factors, eigenvalue-based measures, and weighted PageRank.
- Weighted measures: Journal-level studies found discrepancies between popularity measures such as impact factor and prestige measures such as weighted PageRank.Popular journals may have high impact factors but low weighted PageRank, whereas prestigious journals show the reverse pattern.
- Research gap: The paper describes weighted citation counting at the author level to differentiate scholarly prestige from popularity.It extends a distinction previously developed primarily for journals.
3. Methods
The study builds an information-retrieval citation dataset from Web of Science and divides it into four periods from 1956 to 2008. It calculates popularity from all citations and prestige from citations supplied by highly cited papers, using period-specific matching and counting procedures.
- Data collection: The dataset contains 15,370 information-retrieval papers and 341,871 cited references collected from Web of Science.Records include first author, year, source, volume, and page number.
- Data collection: The data are divided into four periods: 1956-1980, 1981-1990, 1991-2000, and 2001-2008.The periods support comparisons of popularity and prestige over time.
- Measures: Popularity is the number of times a researcher is cited by all papers, while prestige is the number of times cited by highly cited papers.The measures are computed separately for each period.
- Prestige calculation: Highly cited papers are selected within each period to represent roughly 20% of total citations.Thresholds vary by period because citation volumes and publication counts differ substantially over time.
- Prestige calculation: The procedure matches highly cited papers to the IR paper dataset using author, year, volume, and beginning-page fields.The matched papers contribute cited references to the core cited-reference datasets.
- Prestige calculation: Prestige rankings are calculated by counting how often each author appears in the core cited-reference datasets.The same process is applied across all four time periods.
4. Results and discussion
The rankings show substantial turnover among leading Information Retrieval authors, with only a small core maintaining top-40 positions across time. Popularity and prestige overlap for some researchers but diverge for others, reflecting different dimensions of esteem.
- Shifting measures of esteem: Gerard Salton was consistently ranked highest on both prestige and popularity.The paper describes Salton as singular in this respect and links his standing to fundamental contributions to Information Retrieval.
- Popularity over time: Only four authors remained in the popularity top 40 across all four periods.The continuously ranked authors were Salton G, Jones KS, Van Rijsbergen CJ, and Robertson SE.
- Popularity over time: 40% of the authors in the top 40 were roughly new entrants in each phase.Retention also declined across successive transitions: 16, 10, and 5 phase-1 authors remained ranked in phases 2, 3, and 4, respectively.
- Prestige over time: Ten authors had a continuous prestige presence, including the four authors continuously ranked for popularity.Prestige rankings therefore showed a larger persistent group than popularity rankings.
5. Popularity, prestige, and other indicators of esteem
The study relates popularity and prestige rankings to researchers’ careers, affiliations, honors, gender, and publication timing. Prestige aligns more closely than popularity with awards and perceived scholarly significance, while the two measures identify different highly cited publications.
- Organizational affiliation: Almost all top-40 authors work or have worked at leading universities or research laboratories.The listed affiliations include major universities and organizations such as IBM, Microsoft, and Yahoo!.
- Gender and education: Six of the top 40 authors, or 15%, are female.The top 10 individuals received their Ph.D. degrees from illustrious institutions, with five in the USA and five in the UK.
- Awards and honors: For 2001-2008, highly prestigious authors were more strongly represented among award winners than highly popular authors.Among first authors receiving the ASIS&T Best Book Award, seven appeared among the most prestigious authors but only one among the most popular.
- Awards and honors: Six of nine Gerard Salton Award winners, or 67%, ranked among the top 10 most prestigious authors, compared with two, or 22%, among the top 10 most popular.The authors interpret this pattern as suggesting that prestige ranking better reflects perceived scholarly significance than popularity ranking.
- Career timing: The majority of key publications appeared 10-20 years after the authors’ Ph.D. awards.Key publications were defined as works cited at least 40 times; three were books, all listed in the popularity column.
- Publication comparisons: Only three of the 10 most highly cited publications were shared between popularity- and prestige-based lists.The comparison further suggests that popularity and prestige measures are not interchangeable.
6. Summary and concluding remarks
The study evaluates scholarly esteem in Information Retrieval by distinguishing popularity from prestige through weighted citation analysis and comparing leading authors across four time periods. Popularity rankings changed more than prestige rankings, while author prominence was associated with organizational affiliation, doctoral training, awards, and career timing.
- Longitudinal rankings: Prestige rankings were more stable than popularity rankings: 10 authors remained in the top 40 for prestige across all four phases, and high-prestige authors often retained status for 20 or 30 years.Popularity rankings changed over time, with only four scholars maintaining a presence across the periods and an approximate 40% churn rate between phases.
- Popularity and prestige: Authors could rank highly on prestige without ranking highly on popularity, and vice versa, showing that the two measures capture different dimensions of scholarly esteem.Popularity reflects citations from all other papers, whereas prestige emphasizes citations from relatively highly cited papers.
- Author characteristics: Top-ranked IR scholars typically published their key work approximately 10 to 20 years after completing their doctorates.This result relates scholarly prominence to the timing of key publications in the authors’ careers.
- Study design: The study applies weighted citation-based evaluation to Information Retrieval papers from 1956–2008, ranking the top 40 authors by popularity and prestige across four time bands.It also collects biographical and professional data, including gender and organizational affiliation.
- Scope and limitations: The proposed weighting gives more influence to citations from highly cited papers but does not account for the topology of citation networks.The authors identify applying the model at the paper level and testing PageRank and HITS as further work.