Source-linked AI summary
Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data
Lutz Bornmann, Robin Haunschild
TL;DR
The paper asks whether altmetrics relate to scientific quality, since metrics unrelated to quality could rate bad or fraudulent research highly. It examines metric dimensions and their relationship with F1000Prime assessments. Citation-based metrics and readership counts were more strongly related to quality than tweets and the Altmetric attention score.
Problem
The study asks whether altmetrics relate to scientific quality, since metrics unrelated to quality risk rating bad or fraudulent research highly.
Source: §5 · p. 16Method
The study compares metric dimensions and uses negative binomial regression to relate them to F1000Prime quality assessments.
Source: §5 · p. 16Results
Citation-based metrics and reads were significantly more strongly related to FFa quality than tweets and the Altmetric attention score.
Source: §5 · p. 16Takeaways & Limitations
The findings question Twitter counts for research evaluation and indicate potential use for Mendeley reader counts.
Source: §5 · p. 17The results cannot be generalized to disciplines such as physics or social sciences, or to papers with lower quality levels.
Source: §5 · p. 17
Abstract
from arXiv · showhide
In this study, we address the question whether (and to what extent, respectively) altmetrics are related to the scientific quality of papers (as measured by peer assessments). Only a few studies have previously investigated the relationship between altmetrics and assessments by peers. In the first step, we analyse the underlying dimensions of measurement for traditional metrics (citation counts) and altmetrics - by using principal component analysis (PCA) and factor analysis (FA). In the second step, we test the relationship between the dimensions and quality of papers (as measured by the post-publication peer-review system of F1000Prime assessments) - using regression analysis. The results of the PCA and FA show that altmetrics operate along different dimensions, whereas Mendeley counts are related to citation counts, and tweets form a separate dimension. The results of the regression analysis indicate that citation-based metrics and readership counts are significantly more related to quality, than tweets. This result on the one hand questions the use of Twitter counts for research evaluation purposes and on the other hand indicates potential use of Mendeley reader counts.
1 Introduction
Although altmetrics are used in research evaluation, the meaning of social-web attention remains unclear; this study asks whether they relate to papers’ scientific quality as assessed by peers. It maps metric dimensions and tests their relationship with F1000Prime assessments.
1 Introduction
- Wider societal or cultural impacts may be undervalued by citation-based indicators, motivating other data sources for assessing wider research impact.Source: §1 · p. 3
- The study tests whether and to what extent altmetrics relate to scientific quality measured by peer assessments.The question follows from uncertainty about what frequent social-web mentions mean for research evaluation.Source: §1 · p. 3
- PCA and factor analysis map dimensions among citation counts and altmetrics, then regression tests those dimensions against F1000Prime assessments.The study treats peer-assessed F1000Prime quality as the outcome in the second step.Source: §1 · p. 4
- The authors argue that research-evaluation metrics should be related to paper quality.Source: §1 · p. 4
2 Previous literature
Earlier studies report different relationships between altmetrics and peer-assessed quality, varying by indicator and field. Evidence on the relationship between altmetrics and peer assessments remains limited.
2 Previous literature
- Bornmann’s meta-analysis reports negligible correlations with citations for microblogging, small correlations for blogs, and medium-to-large correlations for online reference-manager counts.Reported pooled correlations are 0.003 for microblogging, 0.12 for blogs, 0.23 for CiteULike, and 0.51 for Mendeley.Source: §2 · p. 4
- The authors found only two prior studies focused on altmetrics and peer-assessed quality, with earlier work using REF scores rather than F1000Prime quality scores.Source: §2 · p. 4
- In REF 2014 data, overall correlations with quality were higher for Mendeley readership (r = 0.19) than Twitter counts (r = 0.07), with results varying by field.Source: §2 · p. 5
- Ravenscroft et al. found a very low negative correlation between REF societal-impact scores and Altmetric attention scores (r = −0.0803); scores were available for only around half of publications.Source: §2 · pp. 5–6
3 Methods
The study combines F1000Prime assessments with citation and altmetrics data, retaining papers with suitable identifiers and indicator coverage. PCA and factor analysis characterize metric dimensions, followed by regression analyses relating them to F1000Prime scores.
3.1 Dataset used
- Four data sources were merged, and F1000Prime, citation, and altmetrics records were matched through DOI.Source: §3.1 · pp. 6–7
- The FFa is given as a total score in the F1000 database and is the sum of scores; F1000Prime papers are rated Good, Very good, or Exceptional (1, 2, or 3), often by several faculty members.Source: §3.1 · p. 6
- The analysis included 33,683 papers, restricting publication to 2011–2013 to ensure at least three years for citations and excluding papers absent from the altmetrics database and 132 papers without JIF.Source: §3.1 · p. 7
- Only Twitter counts and the Altmetric attention score had enough variance for PCA and factor analysis; other altmetrics were dominated by zero counts.Source: §3.1 · p. 8
3.2 Statistical procedures
- PCA creates uncorrelated components from indicator variables, while factor analysis provides a separate data-reduction method to validate the PCA findings.Source: §3.2 · pp. 8–9
- Because citation and altmetrics data were non-normal and affected by outliers, the authors used log transformation and robust covariance methods.They also ran robust negative binomial regressions for over-dispersed F1000Prime count scores and OLS analyses as validation.Source: §3.2 · pp. 9–10
4 Results
Citation and readership dimensions are more strongly related to F1000Prime quality than Twitter-focused dimensions. PCA and FA separate these metric patterns into distinct components or factors.
4.1 Principal components analysis (PCA) and factor analysis (FA)
- Arranged in decreasing order of variance, the first component explains 72% of total variance, and the common 80% cumulative-proportion cutoff is reached with two components.The first dimension is mainly based on Twitter counts and concerns academic impact.Source: §4.1 · pp. 10, 11, 13
- PCA groups citations and Mendeley readers on the first component, while Twitter counts define a second component to which the Altmetric attention score is scarcely related.The score may reflect a broad range of altmetrics, although it is mainly driven by Twitter counts.Source: Table 3 · p. 11, §4.1 · p. 12
- FA retains three factors: journal metrics form an additional factor, and the Altmetric attention score loads with Twitter counts, unlike in the PCA.The FA results validate the main PCA result; the attention-score loadings in PCA are close to but below the recommended threshold.Source: Table 4 · p. 12, §4.1 · p. 12
4.2 Negative binomial regression analysis (NBREG)
- In the FA regression, the citation-and-reader factor is strongly related to quality, while the tweets-and-Altmetric-attention factor is not; the journal-impact factor is scarcely related too.The FA results validate the PCA pattern, with the difference between the first and third factors larger than in the PCA analysis.Source: §4.2 · p. 14
- A standard-deviation change in the citation-and-reader component is related to an expected FFa increase of 0.62, versus 0.3 for the mainly Twitter component.The authors describe the first component as significantly more strongly related to paper quality in practical terms.Source: §4.2 · pp. 13–14, Table 5 · p. 14
- OLS analyses preserve the different relationships of single-paper citation impact and altmetrics with paper quality, though the contrast is less pronounced than in NBREG.The OLS analyses use PCA and FA scores, respectively.Source: §4.2 · p. 15
5 Discussion
The discussion finds that citations and Mendeley readership track F1000Prime-assessed quality more strongly than tweets, while PCA places tweets on a separate dimension from citation and readership impact. It cautions that Twitter-based evaluation needs quality standards and that the findings are bounded by a biomedical, largely high-quality F1000Prime dataset.
5 Discussion
- Altmetrics may also matter for assessing outputs such as research data or scientific software, which are not traditionally included in evaluations.The study frames its purpose as testing whether altmetrics can be used in research evaluation practice.Source: §5 · p. 16
- PCA placed citations and reads on similar dimensions but tweets on a different dimension, and factor analysis validated this pattern.The discussion interprets citations and readership as academic impact and tweets as wider impact, largely independent of citation impact.Source: §5 · p. 16
- Citations and reads were more strongly related to F1000Prime-assessed quality than tweets and the Altmetric attention score in both robust NBREG models.The authors therefore question tweets for research evaluation while indicating potential use for Mendeley reader counts.Source: §5 · pp. 16–17
- Because tweets often come from outside science and public users strongly influence their distribution, the authors recommend quality standards for Twitter-based evaluations.They extend the same recommendation, cautiously, to evaluations using the Altmetric attention score.Source: §5 · p. 17
- The authors argue that public attention and popularity studies can offer insights, but research evaluation should also consider scientific quality standards.They warn that metrics unrelated to quality could rate bad or fraudulent research highly.Source: §5 · pp. 16–17
- The study uses F1000Prime-recommended biomedical papers, which mostly have high quality levels by citation impact, limiting generalization to other disciplines and lower-quality papers.The authors encourage future peer-assessment studies in other disciplines using more diverse datasets.Source: §5 · pp. 17–18