Source-linked AI summary
Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective
Rodrigo Costas, Zohreh Zahedi, Paul Wouters
TL;DR
The paper asks whether altmetrics relate to citations and can identify highly cited publications across disciplines. Using a large multidisciplinary Web of Science dataset linked to Altmetric.com indicators, it finds weak positive relationships and differentiated filtering strengths, supporting complementary rather than substitutive use.
Problem
The study addresses limited evidence about altmetrics’ relationship with citation indicators and their ability to identify highly cited publications across fields.
Method
The authors combine bibliometric, correlation, and precision-recall analyses of 718,315 Web of Science publications linked to Altmetric.com indicators.
Results
Altmetrics show weak positive relationships with citations, while their scores—particularly blog mentions—provide higher precision but lower recall than journal citation scores for identifying highly cited publications.
Takeaways & Limitations
Altmetrics are best understood as complementary to citation analysis because they capture different types of impact and provide selective but incomplete filtering of highly cited publications.
Takeaways & Limitations
The study is exploratory because it depends on Altmetric.com’s data quality and has stronger coverage for publications from July 2011 onward.
Abstract
from arXiv · showhide
An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some altmetric activity and concentrating in the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities and the medical and life sciences show the highest presence of altmetrics, indicating their potential value and interest for these fields. The analysis of the relationships between altmetrics and citations confirms previous claims of positive correlations but relatively weak, thus supporting the idea that altmetrics do not reflect the same concept of impact as citations. Also, altmetric counts do not always present a better filtering of highly cited publications than journal citation scores. Altmetrics scores (particularly mentions in blogs) are able to identify highly cited publications with higher levels of precision than journal citation scores (JCS), but they have a lower level of recall. The value of altmetrics as a complementary tool of citation analysis is highlighted, although more research is suggested to disentangle the potential meaning and value of altmetric indicators for research evaluation.
1. Introduction
The paper examines how altmetrics are distributed across scientific fields and whether they relate to citations or improve identification of relevant publications. It frames altmetrics as complementary indicators that may capture impacts beyond traditional citation analysis.
- Altmetrics capture mentions of scientific outputs across social web tools, including Facebook, Twitter, blogs, news media, and reference-management platforms.
- Altmetrics seek to extend impact assessment beyond journal papers and traditional bibliometrics to outputs such as blogs and datasets.
- The dataset links 718,315 Web of Science publications with Altmetric.com indicators from blogs, Twitter, Facebook, Google+, news outlets, and other sources.
- The study analyzes altmetric presence and relationships with citations across publications and scientific fields at broad multidisciplinary scale.
- The analysis combines bibliometric approaches, correlation analyses, and precision-recall methods to test relationships and filtering performance.
2. Data and methodology
The study links Altmetric.com records to Web of Science publications, restricts the analysis to recent publications because altmetric coverage is strongly recency-biased, and compares bibliometric with altmetric indicators.
- 1,589,440 Altmetric.com records yielded 1,380,143 unique DOIs, of which 718,315 matched the Web of Science database.
- 7% of DOI-bearing Web of Science papers received an altmetric score in the unrestricted matched dataset.
- 10.8% of DOI-bearing publications from 2011, 23.8% from 2012, and above 25% from 2013 received some altmetric score.
- The analysis focuses on publications from July 2011 onward because Altmetric.com coverage is stronger for recent publications and a one-year citation window remains available.
- Bibliometric measures include citations, normalized citations, journal citation scores, journal-to-field impact scores, and top-1% publication status.
- Altmetric measures include Facebook walls, blogs, Twitter, Google+, news outlets, and a combined total-altmetrics indicator.
- Selecting top-1% publications by normalized citations addresses strong impact differences across disciplines and subdisciplines in the multidisciplinary dataset.
3. Results
Altmetric activity is uncommon and concentrated in recent publications, with the strongest presence in biomedical, health, social sciences, and humanities fields. Factor analysis separates bibliometric dimensions from two distinct altmetric dimensions.
- Presence across publications: 15% of 500,229 publications had at least one altmetric score.
- Presence across publications: Twitter covered 13% of publications, followed by Facebook at 2.5%, blogs at 1.9%, Google+ at 0.6%, and news outlets at 0.5%.
- Differences across fields: More than 22% of biomedical and health sciences and social sciences and humanities publications had altmetric scores, versus less than 10% in natural sciences and engineering and mathematics and computer science.
- Differences across fields: Social sciences and humanities had the highest altmetric density at 1.04, followed by biomedical and health sciences at 0.83 and life and earth sciences at 0.71.
- Differences across fields: Social sciences and humanities had higher altmetric than citation density, 1.04 versus 0.9, whereas other fields had lower altmetric density than citation density.
- Indicator structure: Factor analysis explained 80% of total variance and separated journal-based indicators, article-based indicators, and two altmetric dimensions.
Correlations among bibliometric and altmetric indicators
Bibliometric indicators correlate more strongly among themselves than with altmetrics, supporting separate article-impact and journal-impact dimensions. Twitter and blog mentions show the strongest altmetric relationships with citation-based indicators.
- Analysis: The correlation analysis used ranked indicators and 95% confidence intervals obtained through 1,000 bootstrap resamplings.
- Bibliometric indicators correlate better among themselves than with altmetric indicators, separating article-based and journal-impact dimensions.
- Total altmetrics correlates mostly with Twitter, which dominates the compound indicator, while both correlate only moderately with citations and journal indicators.
- Twitter and blog mentions have the strongest relationships with citations and journal indicators, while other metrics show negligible correlations with citation-based impact.
3.2. Comparison between altmetrics and citations
Publications with more altmetrics tend to have higher citation impact and journal scores, but paper-level correlations are weak and altmetrics do not consistently outperform JCS in identifying highly cited publications. Their relative precision and recall vary by indicator and field.
- Aggregate relationships: Average citations and JCS increase with total altmetrics, while field-normalized impact rises from below 1 to above 1 with one altmetric score.
- Aggregate relationships: Paper-level correlations are rather weak, despite stronger relationships when publications are aggregated.
- Filtering highly cited publications: Precision-recall analysis identified the top 1% most highly cited publications using ncs-ranked publications and compared total altmetrics with an independent journal indicator, JCS_0510_12.
- Filtering highly cited publications: At 10% recall, total altmetrics achieved around 25% precision, meaning 10% of top-1% publications were captured in a selection with 25% highly cited publications.
- Filtering highly cited publications: JCS generally outperformed altmetrics, although altmetrics performed better at approximately 5% recall.
- Field and indicator differences: Blog mentions had higher precision than JCS at lower recall levels, while life and earth sciences showed JCS outperforming altmetrics across the precision-recall range.
- Field and indicator differences: In social sciences and humanities, JCS led below 0.10 recall, after which both measures tended to merge at low precision.
- Field and indicator differences: Mathematics and computer science showed overlapping JCS and altmetrics curves, with both having poor filtering capacity for top publications.
3.3. Tight analysis
The tight analysis finds only marginal improvements in altmetrics–impact relationships, with the earlier pattern largely unchanged. Blogs improve recall while retaining higher precision than JCS.
- The analysis excludes publications without altmetrics, reducing the problem of missing early online-first publication identification.
- Only marginal improvements appear in the relationship between altmetrics and impact indicators.
- Blogs show increased recall while retaining higher precision than JCS for identifying highly cited publications.Figure 7 compares precision-recall curves for JCS and blogs.
- The tight analysis reproduces the main results observed in the broader analysis.
Limitations of the study
The study’s conclusions are constrained by the limited coverage and frequency of altmetric data, especially across fields and publication years. The authors therefore frame the work as an exploratory approach whose generalizability depends on these limitations not invalidating its main conclusions.
- Data coverage: Around 15% of 2011 publications and above 20% of 2012 publications had any altmetric measure, limiting information for most publications.The authors warn that altmetrics’ value remains limited if coverage does not rise substantially beyond 20–30%.
- Data coverage: Low altmetric presence, even in recent years, challenges the reliability of research-assessment indicators based on these data.The authors identify incomplete publication coverage as a boundary on using altmetrics for assessment.
- Filtering highly cited publications: Altmetrics identify fewer highly cited publications than journal citation scores, despite offering higher precision for selected top publications.Altmetrics, particularly blog mentions, may suit selecting a small number of highly cited publications, whereas JCS better supports broad retrieval.
- Interpretation of impact: Altmetric–citation relationships are positive but only moderate, limiting the interpretation of altmetrics as measures equivalent to citation impact.The weak correlations appear in both the general and tight analyses.
- Disciplinary scope: Field differences constrain generalization: mathematics, computer science, and natural and engineering sciences show the lowest altmetric presence, while medical and life sciences have lower altmetric than citation density.Social sciences and humanities show higher activity, whereas citation indicators remain more prominent in medical and life sciences.
Final conclusions
Altmetric presence remains low but is increasing and concentrated in recent publications. Weak correlations with citations support using altmetrics as complementary indicators rather than replacements, especially for broader forms of impact.
- Altmetric presence is increasing over time and is most valid for the most recent publications.
- Weak correlations with citations indicate that altmetrics do not capture the same impact concept as citations.
- Social-media altmetric presence and density remain low among scientific publications, challenging the reliability of indicators based on them.
- Altmetrics may complement citation analysis by informing societal or cultural impact, particularly in the humanities and social sciences.
- Further research should validate these potential impact types using quantitative and qualitative studies.
Appendix III. Precision-recall analysis of individual altmetric indicators (green lines) vs JCS (blue lines): extended analysis
The appendix compares individual altmetric indicators and total altmetrics with journal citation scores across disciplines. The displayed analysis includes precision-recall curves for identifying the 1% most highly cited publications and rankings for several indicators.
- The analysis distinguishes comparisons involving blogs, Google+, Twitter, Facebook, news, total altmetrics, and JCS.
- The correlation analysis identifies ranks for citation scores, normalized citation scores, journal indicators, and related journal-based measures.
- Precision-recall curves compare JCS with total altmetrics for identifying the 1% most highly cited publications across disciplines.
- The appendix covers biomedical and health sciences, life and earth sciences, mathematics and computer science, natural sciences and engineering, and social sciences and humanities.