PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Predictability of Future Impact in Science

Orion Penner, Raj Kumar Pan, Alexander M. Petersen, Kimmo Kaski, Santo Fortunato

arXiv:1306.0114v2physics.soc-phcs.DL

TL;DR

Career evaluations focus on future impact, but most measures capture past work. Testing linear models on scientists’ careers, the paper finds that cumulative h-index correlations inflate apparent predictive power and that predictions vary with career age. The authors conclude that such models have far less predictive power than previously reported.

  • Problem

    Career decisions depend on future impact, while most measures quantify past work and past success matters chiefly insofar as it correlates with future success.

    Source: §I · p. 1

  • Method

    The authors analyze career data and use synthetic careers with scrambled information to test whether cumulative-measure correlation inflates linear models’ apparent predictive power.

    Source: §I · p. 2, D. Empirical evidence of overestimation · p. 4

  • Results

    Cumulative measures’ intrinsic correlation makes R2 misleading, and linear models predicting h-index increases within age cohorts do a poor job of predicting future impact.

    Source: §III · p. 7

  • Takeaways & Limitations

    Future predictive modeling should target applications in career advancement more directly and use approaches sensitive enough to investigate links between past and future research impact.

    Source: §III · p. 7

    Evaluation also considers institutional prestige, supervisors, service, teaching, and mentoring, while heavy reliance on cumulative measures may reinforce stratification and cumulative advantage.

    Source: §III · p. 7

Abstract

from arXiv · show

Correctly assessing a scientist's past research impact and potential for future impact is key in recruitment decisions and other evaluation processes. While a candidate's future impact is the main concern for these decisions, most measures only quantify the impact of previous work. Recently, it has been argued that linear regression models are capable of predicting a scientist's future impact. By applying that future impact model to 762 careers drawn from three disciplines: physics, biology, and mathematics, we identify a number of subtle, but critical, flaws in current models. Specifically, cumulative non-decreasing measures like the h-index contain intrinsic autocorrelation, resulting in significant overestimation of their "predictive power". Moreover, the predictive power of these models depend heavily upon scientists' career age, producing least accurate estimates for young researchers. Our results place in doubt the suitability of such models, and indicate further investigation is required before they can be used in recruiting decisions.

I. INTRODUCTION

Career decisions turn on future achievement, yet evaluations often emphasize past accomplishments; linear h-index prediction has been proposed to connect a researcher’s current record with future impact. This paper argues that such models overstate predictability and handle career-stage variation poorly.

I. INTRODUCTION

  • Evaluations must rank applicants by expected future impact, although past success matters chiefly insofar as it correlates with future success.The paper frames this challenge in recruitment, grants, and other career-advancement decisions.Source: §I · p. 1
  • Acuna et al. proposed linear regression using five metrics to predict future h-index, addressing a decision goal focused on what researchers will produce.The h-index itself measures past accomplishments.Source: §I · p. 1, A. Modeling cumulative measures · p. 2
  • Cumulative measures such as h-index retain past information, which can inflate R2, while mixing career ages handles different levels of uncertainty poorly.The authors identify both as methodological flaws in regression models for future impact.Source: §I · p. 1
  • Across physics, biology, and mathematics, current models do not predict future impact well enough for confident career-advancement use.The analysis includes both established and junior scientists.Source: §I · p. 2

A. Modeling cumulative measures

  • For physicists and biologists, pooled-career-age models report R2 values of 0.75 and 0.76, respectively, even 15 years ahead.These results can give the impression that future h-index is well predicted.Source: A. Modeling cumulative measures · p. 2

B. Age-dependent cumulative model

The authors refit the h-index model separately at fixed career ages to test its robustness across career stages. Early-career predictions are weaker than pooled-age predictions, and model coefficients vary across ages and data splits.

B. Age-dependent cumulative model

  • The model is refit separately for each career age using the same regression variables, with career age omitted as a predictor in each age-specific fit.The predicted outcome remains the h-index Δt years in the future.Source: B. Age-dependent cumulative model · p. 3
  • Early-career predictions have lower R2 than pooled-age predictions, though biologists retain R2 = 0.63 at t = 3 and R2 = 0.73 at t = 5 ten years ahead.The authors caution that these values can still make early career trajectories seem easily predictable.Source: B. Age-dependent cumulative model · p. 3
  • Regression coefficients differ across career ages: the h-index coefficient remains largest, while high-impact-journal papers can have a comparable coefficient, especially for biologists.The h-index coefficient also increases in variation over time, unlike in the pooled-age model.Source: B. Age-dependent cumulative model · p. 3
  • Splitting a dataset into two groups changes both R2 values and regression coefficients, further weakening the stability of age-specific analyses.The authors report this pattern in additional figure comparisons.Source: B. Age-dependent cumulative model · p. 3

C. Non-stationary time series

Because the h-index accumulates past achievements, it is non-stationary and intrinsically correlated with its future values. This correlation depends on career age, helping explain why future h-index estimates are less reliable for young careers.

C. Non-stationary time series

  • Because h-index is cumulative and non-stationary, its intrinsic correlation with future values can overestimate models’ predictive power.The authors also say this correlation helps explain career-age dependence.Source: C. Non-stationary time series · p. 3
  • The authors model a researcher’s h-index as the sum of yearly independent random increments, whose mean, variance, and autocovariance depend on career age.The accompanying figure shows the distribution of consecutive-year h-index increments.Source: C. Non-stationary time series · p. 3, Figure 3 · p. 4
  • When the forecast interval is small relative to career age, current and future h-index are completely correlated; as that ratio grows without bound, they are completely uncorrelated.The authors use this relationship to explain why models cannot perform as well for young careers as for old ones.Source: C. Non-stationary time series · p. 3
  • The correlation between current and future h-index increases with career age for a fixed forecast interval, making young careers harder to predict.The authors argue that this age dependence can partly explain why prediction models perform less well for young careers.Source: C. Non-stationary time series · pp. 3–4

D. Empirical evidence of overestimation

Null models with scrambled career-specific information still produce high R² values, supporting the authors’ conclusion that cumulative h-index autocorrelation inflates apparent predictive power. A financial time-series example likewise distinguishes index-level correlation from the faster-decaying autocorrelation of returns.

D. Empirical evidence of overestimation

  • The ∆h null model generates synthetic careers from randomly sampled yearly h-index increases, 98% of which are less than 5.Applying the simple regression to these careers produces high R² values.Source: D. Empirical evidence of overestimation · p. 4
  • High R² values from both null models do not indicate predictive power; the authors attribute them to intrinsic correlation in cumulative h-index.The null models scramble information a regression would need to predict an individual’s future h-index.Source: D. Empirical evidence of overestimation · p. 4
  • The paper shuffle null model preserves each researcher’s yearly publication count but randomly redistributes papers, and it also produces high R² values.In this model, each career has the same probability of including a high-impact paper.Source: D. Empirical evidence of overestimation · p. 4
  • Stock market index values remain highly correlated even after 100 days, while autocorrelation of index returns decays quickly.The authors use this contrast to support the efficient market hypothesis.Source: D. Empirical evidence of overestimation · p. 4

E. Modeling non-cumulative measures

When the model predicts h-index increments rather than cumulative measures, its predictive power is significantly lower. Predictive power also varies with career age and discipline, and coefficients differ across career stages.

E. Modeling non-cumulative measures

  • Figure 4 compares mean R² and standard coefficients across prediction horizons for established and younger scientist groups.Its averages cover career ages 2–15 for established scientists and 2–8 for younger researchers.Source: Figure 4 · p. 5
  • The incremental h-index model has significantly less predictive power than the cumulative-measure model.The analysis focuses on ∆h(t, ∆t), which avoids the intrinsic autocorrelation of cumulative measures.Source: E. Modeling non-cumulative measures · pp. 4–5
  • Young researchers show much lower R² than prominent senior scientists, whose results provide an upper bound on predictability.The young-scientist groups include physics assistant professors, young biologists, and graphene researchers.Source: E. Modeling non-cumulative measures · p. 5
  • For incremental h-index predictions, coefficients vary across disciplines and career stages, so a single parameter set is unlikely to work across cases.The number-of-papers coefficient declines as the measurement horizon grows; for biologists, the top-journal publication coefficient is larger later in careers.Source: E. Modeling non-cumulative measures · p. 6

F. Correlating past and true future

The authors compare impact measures from consecutive, non-overlapping career periods to examine how well earlier research impact relates to later work. They find low but significant correlations, with stronger correlations for mid-to-late than early-to-mid career and lower correlations for younger researchers.

F. Correlating past and true future

  • The analysis compares three non-overlapping periods: years 1–5, 6–10, and 11–15.For each period, the measures count only papers published in that period and citations those papers received within the same period.Source: F. Correlating past and true future · p. 6
  • Past and future impact measures are similarly good—or similarly poor—predictors, with low but significant correlations.This pattern holds across publications, square-root citations, and non-cumulative h-index; the prominent-scientist results provide an upper bound for all careers.Source: F. Correlating past and true future · p. 6
  • Mid-to-late career correlations are slightly higher than early-to-mid career correlations.The authors suggest this is consistent with greater fluctuation in early careers than among more established scientists.Source: F. Correlating past and true future · p. 6
  • Correlations between past and future impact are much smaller for the younger researcher groups than for prominent scientists.The younger groups include physics assistant professors, young biologists, and graphene researchers.Source: F. Correlating past and true future · p. 6

III. DISCUSSION

The discussion cautions that cumulative measures can make future-impact models appear more predictive than they are, and that metrics should not replace broader evaluation. It calls for models better suited to career decisions and further research on their calibration and data.

III. DISCUSSION

  • Heavy reliance on cumulative achievement measures may reinforce stratification and cumulative advantage, increasing the risk burden on young careers.The discussion situates these measures within science’s reputation systems.Source: §III · p. 7
  • Citation counts are only one dimension of academic success and should be used alongside, not instead of, other evaluations.The discussion notes that comparative evaluations should account for metric biases and potentially correct them.Source: §III · p. 7
  • The authors call for studying how model weights vary across disciplines and career ages, examining additional predictors, and improving career-data availability.Advisor prestige is given as an example of a predictor to investigate.Source: §III · p. 7
  • Career-advancement models should rank researchers precisely, assign confidence to rankings, and be easy to calibrate across disciplines.The authors also call for separately predicting impact from future citations to past papers and to future papers.Source: §III · p. 7
  • Cumulative measures are inappropriate prediction targets because their construction creates trivial correlation, and current models have less predictive power than previously reported.The authors conclude that future-impact models need to focus more directly on career-advancement applications.Source: §III · p. 7

IV. METHODS

The study analyzes publication and citation profiles for 762 scientists across physics, cell biology, and pure mathematics. It uses elastic net regularization to address correlated predictors in linear regression.

A. Data description

  • The dataset contains 762 scientists: 476 physicists, 236 cell biologists, and 50 pure mathematicians.The prominent-scientist datasets comprise the top-cited scientists in each field; profiles came from the Thomson Reuters Web of Knowledge database.Source: A. Data description · p. 7

B. Elastic net regularization for linear regression

  • Elastic net regularization is used when regression predictors are correlated, with α controlling collinearity and λ controlling model complexity.The analysis sets α = 0.2.Source: B. Elastic net regularization for… · p. 8

Appendix A: Methods

The study uses disambiguated author profiles from Thomson Reuters Web of Knowledge and ResearcherID to analyze distinct scientists’ publication records.

Appendix A: Methods

  • Thomson Reuters matching algorithms identify distinct authors’ publication profiles in Web of Knowledge.The authors also performed further case-by-case disambiguation for some datasets.Source: §1 · p. 8
  • ResearcherID lets users upload and maintain their publication profiles.The study describes the ISI online database as comprehensive and suited to developing testable models of scientific impact and career achievement.Source: §1 · p. 8
  • The study uses disambiguated profiles to reduce the risk of multiple authors being treated as one author.The disambiguation problem includes people with the same initials or even the same complete name.Source: §1 · p. 8

2. Selection of scientists

The selection covers 762 scientists across physics, biology, and mathematics, combining prominent senior researchers with younger scientists and broader ResearcherID profiles.

2. Selection of scientists

  • The sample includes 476 physicists, 236 biologists, and 50 mathematicians across six datasets.The analysis totals 762 scientists.Source: §2 · p. 9
  • The senior-scientist datasets select prominent physicists from Physical Review Letters, cell biologists from CELL, and mathematicians from Annals of Mathematics.The mathematics sample is limited to 50 because productivity and collaboration vary less there than in experimental and theoretical natural sciences.Source: §2 · p. 9
  • The younger samples include 100 physics assistant professors selected from top U.S. physics and astronomy departments and 174 graphene researchers plus two field leaders.The physics assistant professors were drawn as two from each of 50 departments.Source: §2 · p. 9
  • The ResearcherID group combines 60 molecular biology and 76 neuroscience scientists, assuming their uploaded publications are representative and approximately complete.The group’s average h-index is 17 ± 15.Source: §2 · p. 9
  • Figures S1 and S2 show standard coefficients over time across disciplines, with 95% confidence intervals, for pooled career ages and cohorts t = 3 and 7.Figure S3 separates top and prolific physicists across career ages and reports significant overall regression models.Source: Figure S1 · p. 9, Figure S2 · p. 9, Figure S3 · p. 10
  • For physicists, the 100 top-cited-share researchers showed slightly greater h-index model “predictive power” than the 100 prolific researchers, especially at larger ∆t.Both models were significant at p < 10^-6.Source: Figure S3 · p. 10

Appendix B: Elastic Net regularization

The appendix describes ordinary least squares and elastic net regularization, including how the latter shrinks coefficients to address collinearity and model complexity.

Appendix B: Elastic Net regularization

  • Ordinary least squares fits a linear model by minimizing the residual sum of squares between observed and predicted responses.The model’s predicted value is expressed as an intercept plus a weighted sum of independent variables.Source: Appendix B · p. 10
  • Ordinary least-squares coefficient estimates rely on independent model terms and become sensitive to response errors when terms are correlated.The resulting coefficient variance can be large.Source: Appendix B · p. 10
  • Elastic net regularization shrinks coefficients to reduce the number of variables on which the solution depends.The analysis sets α = 0.2 and selects λ by cross-validation; results were qualitatively similar at α = 0.1.Source: Appendix B · p. 10

Appendix C: Null models and randomized careers

The appendix constructs randomized career profiles as null benchmarks and describes supplementary comparisons across career ages, disciplines, and impact measures.

Appendix C: Null models and randomized careers

  • The paper-shuffle null reallocates papers within each career year while preserving each scientist’s observed annual paper count.It approximately retains collaboration patterns and uses a pooled repository of 200 prestigious physicists to generate 40,000 synthetic careers.Source: Appendix C · p. 10
  • The ∆h null generates careers by randomly drawing yearly h-index increases from the dataset’s observed distribution while keeping career length fixed.Unlike paper shuffling, this model does not use each scientist’s publication count.Source: Appendix C · p. 10
  • For top and prolific physicists, coefficient spread within each group and differences between groups indicate that one group’s parameters are unsuitable for predicting the other.Source: Figure S4 · p. 11
  • For young researchers, regression results at t > 8 are non-significant because too few careers are available, leading to overfitting.The figure reports significant overall regression models across the other cases.Source: Figure S5 · p. 11
  • Figure S6 tracks factor coefficients at early, mid-range, and late career ages, while Figure S5 plots h-index increment “predictive power” across disciplines and age cohorts.Source: Figure S5 · p. 11, Figure S6 · p. 12
  • Figures S7–S11 compare past and true future counts of papers, citations, and non-cumulative h-index using non-overlapping early-, mid-, and late-career paper sets.For mathematicians, low impact fluctuations place many observations on top of one another.Source: Figure S7 · p. 12, Figure S8 · p. 12, Figure S9 · p. 12
Loading 1306.0114v2…