PaperlayerView exact revision on arXiv

Source-linked AI summary

Regression for citation data: An evaluation of different methods

Mike Thelwall, Paul Wilson

arXiv:1510.08877v1cs.DL

TL;DR

Citation counts are used in research evaluation, but their skewed, discrete-lognormal pattern leaves the best regression strategy uncertain. The paper compares candidate regressions on simulated data and finds that negative binomial regression can produce excess false positives, while log-transformed general linear models and continuous-lognormal generalized linear models avoid that problem. The authors recommend these alternatives, while noting limits to the simulation scope.

  • Problem

    Citation counts are highly skewed and appear to follow a discrete lognormal distribution, leaving uncertainty about whether negative binomial regression is the appropriate analysis.

    Source: Introduction · pp. 2–3

  • Method

    The study compares five regression variants on simulated discrete-lognormal citation data with known factor relationships and null effects.

    Source: Research question · p. 5, Results · p. 6

  • Results

    Negative binomial regression identifies non-existent factors more often than the significance level predicts, whereas log-transformed general linear models and continuous-lognormal generalized linear models do not show that problem in the reported simulations.

    Source: Results · p. 6, Discussion and conclusions · p. 9

  • Takeaways & Limitations

    When discrete-lognormal citation data are suspected, the authors identify log-transformed general linear models or continuous-lognormal generalized linear models as reasonable alternatives, with zero-removal approaches less powerful than using the full data.

    Source: Discussion and conclusions · p. 10

    The tests cover only a single factor, and the simulation may overstate negative binomial unreliability because real citation data can be bounded.

    Source: Discussion and conclusions · p. 9

Abstract

from arXiv · show

Citations are increasingly used for research evaluations. It is therefore important to identify factors affecting citation scores that are unrelated to scholarly quality or usefulness so that these can be taken into account. Regression is the most powerful statistical technique to identify these factors and hence it is important to identify the best regression strategy for citation data. Citation counts tend to follow a discrete lognormal distribution and, in the absence of alternatives, have been investigated with negative binomial regression. Using simulated discrete lognormal data (continuous lognormal data rounded to the nearest integer) this article shows that a better strategy is to add one to the citations, take their log and then use the general linear (ordinary least squares) model for regression (e.g., multiple linear regression, ANOVA), or to use the generalised linear model without the log. Reasonable results can also be obtained if all the zero citations are discarded, the log is taken of the remaining citation counts and then the general linear model is used, or if the generalised linear model is used with the continuous lognormal distribution. Similar approaches are recommended for altmetric data, if it proves to be lognormally distributed.

Introduction

Research performance monitoring has expanded, and citations are used in some evaluation settings despite concerns that they reflect factors beyond scholarly quality. Regression can examine multiple factors together, but skewed citation counts and uncertainty about the best method motivate testing alternative approaches.

Introduction

  • Citations are considered in some national research evaluation exercises and are also used in formal and informal assessments of academics.The Journal Impact Factor is another widely used citation metric.Source: Introduction · p. 1
  • Regression can examine multiple variables simultaneously, helping distinguish a factor's association with citations from the influence of another factor.The passage gives international collaboration and collaboration with the USA as an example.Source: Introduction · p. 2
  • Citation counts are highly skewed, making tests that rely on normally distributed raw counts inappropriate.Logistic regression instead predicts membership in a highly cited group rather than citation counts directly.Source: Introduction · p. 2
  • The discrete lognormal distribution appears to fit citation counts for collections of articles, leaving the suitability of negative binomial regression uncertain.The passage also identifies the hooked power law as a reasonable fit, while noting that a power-law tail alone is not helpful for analyses of all articles.Source: Introduction · p. 3
  • The study compares regression strategies to identify powerful, reliable approaches for citations and similar data, including altmetrics.Its motivation is to improve statistical analyses of factors associated with citation counts.Source: Introduction · p. 3

Citation distributions and statistical tests

The section explains why regression error distributions matter for skewed citation data. It contrasts generalized linear models with log-transformed general linear models and describes the biases and assumptions involved.

Citation distributions and statistical tests

  • Generalized linear models can specify non-normal error distributions and use maximum likelihood estimation to fit regression equations.The passage gives negative binomial, Poisson, and lognormal distributions as examples.Source: Citation distributions and statistical… · p. 4
  • Negative binomial regression may fit citation data poorly because its residual distribution differs from the lognormal distribution that citations are thought to follow.At the time of writing, statistical software did not offer a discrete-lognormal generalized linear model; a continuous-lognormal option was available.Source: Citation distributions and statistical… · p. 4
  • Taking logs of citation counts and using the general linear model assumes the transformed errors are close enough to normal for fitting.Because the log is undefined at zero, the alternatives described are discarding zero counts or adding one before taking logs.Source: Citation distributions and statistical… · p. 4
  • Removing zero-citation articles truncates one side of the distribution, potentially biasing estimated means upward and reducing sample size and test power.The passage says this may especially affect factors associated with lower citation scores.Source: Citation distributions and statistical… · p. 4
  • The authors regard a mixture of other distributions that combines into a lognormal citation distribution as theoretically possible but unlikely given tested article sets.They note that this possibility has not been fully tested.Source: Citation distributions and statistical… · p. 4

Research question

The research question asks which available regression strategy best analyzes discrete lognormal data without inflating false positives while retaining power to detect small effects.

Research question

  • The comparison seeks a method for discrete lognormal data that avoids inflated false positives while remaining powerful enough to detect minor factors.The question arises because a discrete-lognormal generalized linear model was not available in software.Source: Research question · p. 5
  • Five candidate analyses are listed: raw-data negative binomial regression, continuous-lognormal generalized linear models with zeros removed or counts increased by one, and log-transformed general linear models under both zero-handling choices.Source: Research question · p. 5

Methods

The methods use simulated citation-like data with a binary factor and known relationships. They compare five regression models across parameter settings, estimating correct detection and false-positive behavior from repeated datasets.

Methods

  • The simulations use a binary factor whose two groups have different lognormal means but the same standard deviation, with both related and unrelated data conditions.The authors chose this single-factor setup as the simplest case for clear results.Source: Methods · p. 5
  • Each simulated dataset contains citation counts and a binary factor, with half the sample assigned to each factor value and varying differences between group means.Source: Methods · p. 5
  • Five models are implemented: negative binomial regression, continuous-lognormal generalized linear models, and ANOVA on log counts, each using the stated zero-handling variants.The code also tests false positives using data where the group means are identical.Source: Methods · p. 5
  • For each parameter setting, the procedure generates 1,000 datasets and calculates the proportion yielding correct results for each model.The procedure is repeated across sample sizes, means, and standard deviations to reduce dependence on any one parameter choice.Source: Methods · p. 5

Results

In null simulations, ANOVA and continuous-lognormal models maintained an approximately 5% false-positive rate, while negative binomial regression increasingly flagged nonexistent factors as dispersion rose. With a binary factor, modified ANOVA and lognormal approaches gained apparent power, while negative binomial’s power edge was consistent with its excess false positives.

Results

  • Negative binomial false positives rose from about triple the 0.05 rate at log SD 1 to about half at log SD 2; alternatives stayed near 5%.The authors report the pattern across other tested log means as well.Source: Figure 1 · p. 6, Results · p. 6, Figure 2 · p. 7
  • For log means 0.50 versus 0.55, all methods needed larger samples for greater detection; negative binomial appeared most powerful, but its advantage matched its false-positive excess.The authors attribute its apparent edge to false positives rather than greater power.Source: Results · pp. 6–7, Figure 3 · p. 8
  • ANOVA+1 and lognormal+1 gained apparent power, which the authors attribute presumably to retaining zeros, increasing sample size and factor separation.Their power advantage was over standard ANOVA and the standard continuous-lognormal model.Source: Results · p. 7, Figure 3 · p. 8
  • Doubling log SD to 2 made the factor harder to detect but preserved the methods’ relative pattern; the pattern also held at log means 1 and 1.5.The comparison of methods was also reported as consistent across these means.Source: Results · p. 7, Figure 4 · p. 8
  • With log-mean separation 0.1 rather than 0.05, modified ANOVA and lognormal had a more pronounced advantage over their standard versions.This comparison used simulated data with log SD 1.Source: Results · p. 7, Figure 5 · p. 9

Discussion and conclusions

In simulated discrete-lognormal citation data, negative binomial regression produced excess false positives, while log-transformed general linear models and related generalized linear models were more trustworthy. The authors recommend checking distributional fit and qualify their conclusions by the single-factor simulations and citation-data bounds.

Discussion and conclusions

  • The tests cover only a single factor, and bounded real citation counts may make negative-binomial unreliability appear greater than it is.The authors consider similar conclusions likely for more complex factors, but identify that extension as uncertain.Source: Discussion and conclusions · p. 9
  • Negative binomial regression on discrete-lognormal data identified nonexistent factors more often than the significance level, making its conclusions unreliable in this setting.The authors note that this unreliability may be overestimated because real citation data are bounded.Source: Discussion and conclusions · p. 9
  • Adding 1 before logging citations and using the general linear model is trustworthy and more powerful than approaches that discard zeroes.This strategy is equivalent to adding 1 and then using the generalized linear model; its greater power is attributed to larger samples and increased factor differences.Source: Discussion and conclusions · p. 9
  • When zero inflation is suspected, discarding zeroes and using a log-transformed general linear model or continuous-lognormal generalized linear model is reasonable but less powerful.These alternatives are unlikely to increase false relationship detection.Source: Discussion and conclusions · pp. 9–10
  • The recommendations apply to citation data following a discrete lognormal distribution; researchers should test for that distribution, and check altmetric distributions individually.The authors say this pattern seems likely for article sets with a common citation window, while some altmetrics may follow a power law.Source: Discussion and conclusions · p. 10
Loading 1510.08877v1…