Source-linked AI summary
How well developed are altmetrics? A cross-disciplinary analysis of the presence of 'alternative metrics' in scientific publications
Zohreh Zahedi, Rodrigo Costas, Paul Wouters
TL;DR
The paper addresses limited evidence about how altmetrics appear across scientific fields and publication periods and how they relate to citations. It analyzes Impact Story data from a random Web of Science sample across fields, document types, and years. Mendeley provides the broadest coverage, while its readership counts show a moderate correlation with citation indicators, alongside unresolved questions about what altmetrics mean.
Problem
Evidence was limited on altmetrics presence across scientific fields and extended publication periods, as well as their relationship with citation indicators.
Method
The study analyzes Impact Story metrics for 20,000 randomly sampled DOI publications from Web of Science, comparing coverage and distributions with citation indicators.
Results
Mendeley readerships covered 62.6% of 19,772 publications, and correlated moderately with citation impact indicators (r=.49).
Takeaways & Limitations
Mendeley is the major and more useful altmetrics source for the studied publications, with higher coverage than Twitter, Wikipedia, and Delicious.
Takeaways & Limitations
The meaning and quality of altmetrics remain unclear, and readership accumulation over time is not fully observable.
Abstract
from arXiv · showhide
In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readerships for 62.6% of all the publications studied, other sources only provide marginal information. In terms of relation with citations, a moderate spearman correlation (r=0.49) has been found between Mendeley readership counts and citation indicators. Other possibilities and limitations of these indicators are discussed and future research lines are outlined.
Introduction
Traditional citations and peer review are widely used but capture only partial aspects of research impact, motivating broader alternative metrics. This study examines altmetrics across fields, document types, publication years, and their relationship with citations.
- Motivation: Citations and peer review are established research-evaluation approaches, but both have recognized limitations and capture only partial scientific impact.Citation indicators primarily reflect impact on subsequent scientific publications, while peer review has its own biases and limitations.
- Alternative metrics: Alternative metrics aim to provide broader and faster impact measures that complement traditional citation metrics.They include usage data, web citation and link analyses, and social-web analysis.
- Alternative metrics: Altmetrics cover mentions of scientific outputs in social media, news media, and reference-management tools, using diverse web-based tracking tools.The concept was introduced as an alternative way to measure broader research impacts in the social web.
- Research gap: Earlier studies mainly examined selected journals, biomedical literature, or bibliometrics communities rather than broad cross-disciplinary samples.The paper identifies limited investigation across scientific fields and extended publication periods.
- Research objectives: The study asks how Impact Story altmetrics are distributed across document types, subject fields, and publication years, and how they correlate with citation indicators.These questions guide an exploratory analysis of publications from science, social sciences, and humanities.
Research methodology
The study uses Impact Story to collect altmetric data for a random, cross-disciplinary sample of Web of Science publications. It analyzes coverage and distributions across publication characteristics and compares altmetrics with citation indicators.
- Data collection: Impact Story aggregates impact data from multiple external sources into a single report for research outputs identified by DOIs, URLs, or PubMed IDs.The tool queries external services for metrics associated with each publication or other artifact.
- Sample: 20,000 publications with DOIs, published between 2005 and 2011 across Web of Science disciplines, were randomly sampled.Random selection used the SQL command “NEW ID ()”.
- Data collection: Altmetric data were automatically collected through the Impact Story REST API during the last week of April 2013.Responses to DOI searches were downloaded separately in JSON format.
NCS JS NJS Mendeley Wikipedia Delicious Twitter
The study finds that Mendeley supplies the broadest altmetric coverage, while readerships show moderate positive relationships with citation impact. Coverage and interpretation vary across fields, document types, publication years, and data-collection conditions.
- Citation comparisons: Publications with altmetrics generally had higher citation scores than publications without metrics, although overlapping confidence intervals weaken some comparisons.The comparison used bibliometric indicators and 95% confidence intervals for publications with and without altmetrics.
- Citation comparisons: Citation impact tended to increase as Mendeley readerships per publication increased, with a stronger effect for average citations than for NJS.This relationship is shown using NCS and NJS impact measures.
- Mendeley and other sources: 62.6% of 19,772 publications had at least one Mendeley reader, making Mendeley the major altmetrics source in this study.Twitter, Wikipedia, and Delicious had lower coverage among the studied publications.
- Coverage patterns: Review papers and articles were proportionally most represented across data sources, while multidisciplinary publications had the highest citation and readership densities.Medical & Life Sciences and Natural Sciences accounted for large shares of accumulated altmetrics, with field-specific patterns in Mendeley attention.
- Coverage patterns: Recent publications may have fewer Mendeley readers because readership accumulates over time, and the study lacks readership histories to identify peak readership.The authors note a slight decrease in readership counts and the proportion of publications with Mendeley readers during the last two years.
- Citation comparisons: Mendeley readerships and citation impact indicators had a moderate Spearman correlation of r=.49, indicating related but distinct activities.The paper identifies reading and citing as related activities while distinguishing them conceptually.