Source-linked AI summary
Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics
Lutz Bornmann
TL;DR
Research on altmetrics needed an overview of how major alternative measures relate to traditional citations and research evaluation. This paper reviews Twitter, Mendeley, CiteULike, and blogging studies and applies meta-analysis to their citation correlations. The pooled correlations are negligible for microblogging, small for blogs, and medium to large for online reference-manager bookmarks.
Problem
The paper addresses limited consolidated evidence on the relationship between major altmetrics and traditional citation counts in research evaluation.
Method
The paper reviews research on Twitter, Mendeley, CiteULike, and blogging and meta-analyzes each metric's correlation with traditional citation counts.
Results
Pooled correlations are negligible for microblogging counts (pooled r=0.003), small for blog counts (pooled r=0.12), and medium to large for bookmarks: CiteULike pooled r=0.23; Mendeley pooled r=0.51.
Takeaways & Limitations
The findings distinguish the strength of altmetrics' correlations with traditional citations across microblogging, blogging, and online reference-manager bookmarks.
Takeaways & Limitations
The meta-analysis has limitations, including that altmetrics may measure something different from traditional citations and that the meta-analysis adds only limited evidence about their relationship.
Abstract
from arXiv · showhide
Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was particularly interested in the correlation between altmetrics counts and citation counts, this overview focuses particularly on this correlation. For each altmetric, a meta-analysis is calculated for its correlation with traditional citation counts. As the results of the meta-analyses show, the correlation with traditional citations for micro-blogging counts is negligible (pooled r=0.003), for blog counts it is small (pooled r=0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r=0.23; Mendeley pooled r=0.51).
1 Introduction
The paper situates altmetrics as a developing approach to measuring research impact more broadly than citations and reviews three major types in relation to research evaluation. It focuses especially on their correlations with citation counts while also considering their scholarly value and potential complementarity.
- Altmetrics use data from social media platforms to measure research impact more broadly than citations.
- The paper reviews microblogging through Twitter, online reference managers through Mendeley and CiteULike, and blogging.
- The study focuses on these three metrics because they have sufficient published data for meta-analysis.
- For each metric, the paper meta-analyzes its correlation with traditional citation counts using coefficients from multiple studies.
- The literature review also examines how altmetrics and citations differ in scholarly value and may capture impact for different audiences.
2 Methods
The literature search combined multiple discovery strategies, including bibliogram-derived keywords, database and web searches, citation chasing, journal contents, reviews, and grey literature. The authors then synthesized heterogeneous correlation studies quantitatively through meta-analysis alongside narrative review.
- The literature search used narrative-review references, journal contents, bibliographic databases, Internet search engines, and citation chasing.
- The search included journal articles, monographs, Internet documents, institutional reports, and case reports to reduce publication bias.
- A bibliogram ranked abstract words by frequency, and prominent terms such as altmetrics and Twitter guided subsequent searches.
- Meta-analysis statistically combines evidence across studies to estimate overall effects and make generalized statements despite study-specific differences.
- Because studies commonly reported correlation coefficients despite methodological heterogeneity, the coefficients could be synthesized meta-analytically.
- The analyses were conducted in Stata using the metan command, following related scientometric applications of pooled correlations.
3 Results
The section reviews Twitter, online reference managers, and blogs as alternative metrics, emphasizing their relationships with traditional citation counts. Their pooled correlations differ substantially across metric types, from negligible for Twitter to stronger for reference-manager bookmarks.
- Microblogging (Twitter): Around 40% of Twitter citations appeared within one week of publication.Twitter citations can therefore occur much earlier than traditional citations, for which the section states one must wait at least three years.
- Microblogging (Twitter): Twitter citation counts were not correlated with traditional citation counts in the meta-analysis (pooled r=0.003).The studies produced very different coefficients, but the sample-size-weighted pooled result was negligible.
- Online reference managers: Mendeley’s large user population and coverage made it the most promising new source for measuring impact in and beyond academic research.Nature or Science papers were stored in over 90% of cases in Mendeley, compared with 73% in CiteULike.
- Online reference managers: Mendeley bookmarks had a pooled correlation of r=0.51 with traditional citations, compared with r=0.23 for CiteULike.The overall pooled correlation across the two reference-manager datasets was r=0.37; the higher Mendeley correlation most probably reflects better literature coverage.
- Blogging: Only about half of the blogs identified in a 2006 Nature study could still be accessed six years later.Two blogs were inaccessible, 16 were inactive, and three were hibernating.
- Blogging: Blog counts had a low correlation with traditional citation counts (overall pooled r=0.12).The meta-analysis in Figure 3 included all studies reviewed for this relationship.
4 Discussion
The discussion finds distinct relationships between altmetrics and traditional citations, while emphasizing that these findings do not yet establish how altmetrics should be used in research evaluation. It also identifies substantial methodological limitations in pooling heterogeneous and potentially dependent scientometric studies.
- Evaluation context: The overview evaluates these altmetrics in relation to research evaluation and focuses on their correlations with citation counts.It discusses advantages and disadvantages for evaluation and calculates meta-analyses of related correlation coefficients.
- Findings: Pooled correlations were negligible for microblogging, small for blogs, and medium to large for online-reference-manager bookmarks.The reported values were Twitter pooled r=0.003, blog pooled r=0.12, CiteULike pooled r=0.23, and Mendeley pooled r=0.51.
- Interpretation: Twitter and blog citations may measure something different from traditional citations because their pooled coefficients are very low.The discussion presents this as an interpretation, while acknowledging that low correlations could also indicate limited value.
- Limitations: The meta-analysis cannot determine the added value or exact construct represented by Twitter citations.Although Twitter citations appear to measure a different construct from traditional citations, the discussion says what that construct is remains unclear.
- Limitations: Pooling is problematic because relationships vary substantially across fields and years, especially as social-web uptake changes over time.The discussion recommends accounting for field and time dependencies in future meta-analytic models.
- Limitations: Underlying datasets can be dependent or reused, causing some cases to be counted multiple times and receive excessive weight in pooled analyses.Examples include repeated analysis of the same dataset and multiple correlation coefficients from related datasets.
5 Conclusions and future research
The meta-analyses show that altmetric–citation correlations vary with how research-focused the user community is. Lower correlations may indicate relevance for measuring research impact beyond science, motivating more targeted future studies.
- Conclusions: Higher research focus in an altmetric’s community corresponds to higher correlation with traditional citations.Reference managers are associated with research-affine users, while microblogging shows the reverse pattern.
- Conclusions: Bookmark counts from online reference managers show the highest correlation with traditional citations among the compared altmetrics.The reported correlation is medium to large for bookmark counts.
- Conclusions: Microblogging shows a low pooled correlation, contrasting with the stronger relationship observed for bookmark counts.The paper interprets this reversal in relation to the audiences using these services.
- Conclusions: Frequent correlation studies and this meta-analysis should be treated as an initial step in altmetrics research.The authors frame the results as a basis for focusing the field’s extensive research on more productive avenues.
- Future research: Lower correlations may identify altmetrics of special interest for measuring research impact on societal areas beyond science.The paper distinguishes broad impact from impact on research itself.
- Future research: Future studies should examine users outside academia, their workplaces, paper use, literature preferences, and precisely defined user groups.Examples include politicians in a particular country and preferences for reviews rather than articles.