Source-linked AI summary
Tweets vs. Mendeley readers: How do these two social media metrics differ?
Stefanie Haustein, Vincent Larivière, Mike Thelwall, Didier Amyot, Isabella Peters
TL;DR
The paper asks how Twitter and Mendeley social-media metrics differ and relate to traditional citations. It compares these measures across 1.4 million biomedical papers using coverage, activity rates, correlations, specialty frameworks, and exploratory user analyses. Mendeley coverage and citation association were substantially stronger than Twitter’s, while the platforms reflected distinct patterns of scholarly and broader interest.
Problem
Little is known about how social media metrics differ and what kinds of impact they reflect, motivating comparison of Twitter and Mendeley with citations.
Method
The study analyzes 1.4 million papers using platform coverage, activity rates, citation correlations, specialty-level frameworks, and exploratory analyses of popular papers and users.
Results
66.2% of papers had at least one Mendeley reader versus 9.4% with a tweet; citation correlations were ρ=0.456 for Mendeley and ρ=0.157 for Twitter.
Takeaways & Limitations
Mendeley measures impact similar but not identical to citations, whereas tweets and citations are only very weakly associated.
Takeaways & Limitations
Twitter coverage is likely underestimated because collection captured only identifier-containing tweets during a limited period, especially affecting older papers.
Abstract
from arXiv · showhide
A set of 1.4 million biomedical papers was analyzed with regards to how often articles are mentioned on Twitter or saved by users on Mendeley. While Twitter is a microblogging platform used by a general audience to distribute information, Mendeley is a reference manager targeted at an academic user group to organize scholarly literature. Both platforms are used as sources for so-called altmetrics to measure a new kind of research impact. This analysis shows in how far they differ and compare to traditional citation impact metrics based on a large set of PubMed papers.
Keywords
The paper examines altmetrics based on social media, especially Twitter and Mendeley, alongside citation analysis.
- The study focuses on Twitter and Mendeley as social media sources for measuring scholarly communication and research impact.
Introduction
The introduction motivates studying social media metrics as faster, broader indicators of research impact than citations alone. It frames the study as a quantitative comparison of Twitter and Mendeley activity with citations across biomedical literature.
- Citations measure research evaluation through later scholarly use, whereas social media can reflect broader audiences and activity soon after publication.
- Little is known about how social media metrics differ or what kinds of impact they reflect.
- The study compares Twitter and Mendeley quantitatively using citations, Mendeley readers, and tweets across journal articles.It also explores highly read and tweeted documents in two research fields, including user demographics.
Literature Review
Mendeley and Twitter represent different audiences and forms of engagement, and prior evidence suggests their relationships with citations vary. The literature therefore supports treating them as distinct indicators rather than interchangeable measures.
- Mendeley stores and shares references for users including academics, practitioners, and students, making readership a possible indicator of wider article use.
- Prior studies found statistically significant positive but low-to-moderate correlations between Mendeley readership and citation counts.
- Twitter reaches beyond academia but is also used by academics, so tweets may reflect public, educational, or scholarly interest.
- Prior tweet studies reported associations with later citations, although correlations were very low and sometimes negative across fields.
Methods
The study analyzes 1.4 million PubMed and Web of Science papers using platform coverage, activity rates, and citation correlations, supplemented by specialty-level and demographic analyses. Its automated data collection and matching procedures introduce documented coverage and precision limitations.
- Data collection: The dataset contains 1.4 million journal articles and reviews published between 2010 and 2012, covered by both PubMed and Web of Science.
- Data collection: Mendeley readership was retrieved through title, first-author surname, and publication-year matching, with manual checking of relevance-ranked results.
- Data analysis: The analysis calculated platform coverage, mean reader and Twitter rates, and Spearman correlations with citations for 2011 papers.
- Data analysis: The specialty framework crosses above- or below-average coverage with positive or negative citation correlations to define four activity patterns.Point size represents mean social-media counts among papers with at least one count.
- Exploratory analysis: General Biomedical Research and Public Health were examined through their most tweeted and read papers, topics, and user demographics.
- Limitations: Twitter coverage may be underestimated because collection captured only identifier-containing tweets from July 2011 through December 2012.
- Limitations: Mendeley coverage may be underestimated and matching included false positives, with manual testing finding 1.7% false positives and 0.7% false negatives.
Results and discussion
Mendeley reached far more PubMed papers than Twitter and tracked citation impact more closely, while Twitter activity was lower, more skewed, and more weakly associated with citations. Differences became sharper across specialties and reflected distinct audiences, with Twitter often capturing public interest and Mendeley showing stronger academic alignment.
- Overall coverage and activity: 66.2% of PubMed papers had at least one Mendeley reader versus 9.4% with at least one tweet.Twitter coverage increased from 2.4% to 20.4% across 2010–2012 papers, while Mendeley coverage declined from 70.3% to 57.3%.
- Overall coverage and activity: The mean Mendeley reader rate was 9.7, almost four times the mean Twitter citation rate of 2.5, and exceeded mean and median citation rates.Mendeley reader rates decreased from 10.7 to 7.6 across 2010–2012 papers.
- Associations with citations: Mendeley readership correlated moderately with citations (ρ=0.456**), whereas tweets correlated much less strongly (ρ=0.157**).Mendeley and Twitter themselves had a low correlation of ρ=0.275** among papers with both reader and tweet activity.
- Specialty-level differences: Across specialties, Mendeley correlations with citations were positive for all 119 specialties, ranging from ρ=0.038 in History to ρ=0.682** in Materials Science.Mendeley coverage ranged from below average in History and Psychoanalysis to 88.3% in Ecology.
- Specialty-level differences: Twitter activity was more skewed across specialties, with coverage highest in Communication (27.6%) and correlations with citations negative in 14 specialties.Twitter citation rates ranged from 1.1 in Polymers to 6.5 in General Biomedical Research.
- Audience patterns: Exploratory cases showed Twitter popularity often involved public audiences and topical events, whereas highly read Mendeley papers attracted students, researchers, and interdisciplinary academic readers.General Biomedical Research had highly read methodological papers, while Fukushima-related papers were retweeted predominantly by the Japanese general public.
Conclusion and outlook
Citations, Mendeley reader counts, and tweets indicate different types of impact and reach different social groups. Highly popular papers varied in academic and public relevance, while platform usage alone could not reliably identify scholarly versus non-scholarly impact.
- Citations reflect impact on the scientific community, whereas Mendeley readership reflects use by a broader but still largely academic audience.
- Tweets often represent discussions by members of the public, but scientists, teachers, students, librarians, funders, and politicians also tweet about scholarly content.
- Highly tweeted and read papers varied in whether they showed academic, scientific, general-public, or combined relevance.
- Platform-based metrics cannot by themselves distinguish scholarly from non-scholarly impact because audiences and engagement levels vary within and across platforms.
- Future research should combine quantitative analyses with content, context, and qualitative analyses to determine what social media metrics measure.
Appendix
The appendix presents scatterplots comparing citations with tweets and Mendeley readers for 2011 papers in General Biomedical Research and Public Health.
- For General Biomedical Research, citations correlate more strongly with Mendeley readers (ρ=0.677**) than with tweets (ρ=0.181**).The plots label the three most tweeted and three most-read papers; all values were increased by 1 for logarithmic representation.
- For Public Health, citations correlate more strongly with Mendeley readers (ρ=0.351**) than with tweets (ρ=0.074**).The plots label the three most tweeted and three most-read papers; all values were increased by 1 for logarithmic representation.
- The figures are available online through the supplied figshare link.