Source-linked AI summary
Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics
Lutz Bornmann
TL;DR
The paper examines whether altmetrics can measure research’s broader societal impact beyond traditional scientific metrics. It reviews their potential and use while emphasizing that important problems remain unresolved and evidence for research evaluation is not conclusive.
Problem
The central question is how far altmetrics can measure research’s broader impact beyond traditional metrics.
Method
The paper reviews altmetrics’ potential for determining the impact of research products and stresses the importance of knowing who used a product and why, not merely how many users there were.
Results
Altmetrics are becoming an established and increasingly popular area of study, accompanied by growing discussion of possible applications.
Takeaways & Limitations
Altmetrics offer substantial potential for assessing impact, but their application requires continued critical discussion.
Takeaways & Limitations
Problems must be solved before altmetrics are used, and their significance for research evaluation lacks conclusive evidence.
Abstract
from arXiv · showhide
Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with research output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of altmetrics for measuring societal impact. It deals with the definition and classification of altmetrics. Furthermore, their benefits and disadvantages for measuring impact are discussed.
1 Introduction
Science policy increasingly expects research to demonstrate value beyond science, but societal impact lacks an accepted measurement framework. The section introduces the need for reliable, valid, feasible indicators that support interaction between research and societal stakeholders.
- Science policy has shifted from an automatic assumption of public benefit toward expectations that science demonstrate its value to society.
- Societal research impact remains unclear to measure, unlike research impact within science, where peer review and bibliometrics are standard.
- The case-study approach is favored for societal impact, but it does not meet all requirements of a general impact framework.
- Desired indicators should measure impact reliably and validly while supporting productive interaction and communication between research and societal stakeholders.
2 What are altmetrics?
Altmetrics are web-based measures of scholarly impact that use social-media and other online activity around research outputs. The section traces their development, data sources, and expanding use while emphasizing that the field remains fluid.
- Altmetrics describe web-based metrics for scholarly material, using social-media activity such as Twitter and Mendeley to assess engagement with research output.
- Article-level metrics count views, downloads, clicks, saves, tweets, shares, recommendations, bookmarks, discussions, and comments rather than only citations.
- Altmetrics developed from earlier web citation and download analyses as the social web enabled participation, interconnection, interaction, and user-generated content.
- The main data sources include bookmarking, reference managers, recommendation services, article comments, microblogging, Wikipedia, and blogging.
- Altmetric data sources remain in flux, with new tools becoming data sources while established tools lose appeal.
3 How can altmetrics be classified?
Altmetrics and article-level metrics can be organized by user activity, product type, data source, and application. Existing classifications cover the progression from viewing and storing research to discussing, recommending, and citing it, but boundaries between audiences are imperfect.
- ImpactStory and PLOS classifications cover viewing, storing, discussing, recommending, and citing research products across the user-engagement process.
- ImpactStory distinguishes scientific from societal impact, whereas PLOS places Wikipedia differently, showing that classification schemes are not identical.
- The scientific-versus-public distinction can be artificial because scholars may download PDFs and the public may use HTML versions.
- Altmetrics can be classified by Web data versus Web tools, including sharing services, social bookmarking services, and formal or informal social networks.
- Metrics also differ in information richness, with blog posts and Wikipedia links treated as content-rich and tweets or likes as content-poor.
4 What benefits do altmetrics offer?
Altmetrics may broaden, diversify, accelerate, and open impact measurement beyond traditional citations. The overview also identifies important boundaries: online attention does not by itself establish societal impact, and altmetric data and platforms remain imperfect.
- 4.1 Broadness: Altmetrics can measure impact beyond science, including attention from professionals, students, government, and the interested general public.
- The overview cautions that altmetrics may reveal hidden impact but overstates their possibilities when treating them as measures of true or full impact.
- 4.2 Diversity: Altmetrics can capture diverse scholarly products and activities, including datasets, software, algorithms, course materials, reading lists, and MOOCs.
- 4.3 Speed: Altmetrics can measure impact shortly after publication, whereas reliable citation measurement may require several years.
- 4.3 Speed: Real-time access through APIs allows researchers to track online activity and identify newly published studies of interest.
- 4.4 Openness: Altmetric data are relatively accessible through Web APIs and can support large publication sets instead of only case-specific societal-impact studies.
5 What are the disadvantages of altmetrics?
Altmetrics have disadvantages involving bias, ambiguous meanings, inconsistent measurement, weak replication, limited empirical evidence, and greater susceptibility to manipulation than traditional metrics.
- Bias: Commercial social-media incentives may promote communication in ways that bias altmetric counts, unlike traditional metrics.Providers benefit when users communicate frequently, but the extent of this bias has not been empirically investigated.
- Data quality: Altmetric counts often omit user-group information and may represent different forms of engagement, from simple mentions to detailed discussions.The same platform can yield different measures depending on whether researchers count mentions, wall posts, likes, or comments.
- Measurement standards: Multiple publication versions create ambiguity and redundancy, while inconsistent social-media mention practices make papers difficult to count reliably.Unlike formal citation rules, social platforms lack comparable standards governing when and how research links appear.
- Normalization and replication: Altmetric scores require normalization and stable data sources because newer or topic-specific papers may score higher and providers frequently change their services.Only normalized scores support comparisons across topics and periods, while provider changes make replication difficult.
- Evidence and manipulation: Evidence for altmetrics remains limited because large-scale reliability and validity studies are rare, methods are often inadequate, and scores are easy to manipulate.Reported problems include weak sampling, misinterpreted correlations, uncorrected multiple testing, fake accounts, and robot tweeting.
6 Discussion
Altmetrics are gaining significance as a possible way to assess broader research impact, but evidence remains inconclusive and important methodological problems remain. The literature supports using altmetrics alongside, not instead of, traditional metrics and informed peer review.
- Growing significance: Altmetrics are establishing themselves as a new subfield, despite continuing uncertainty about their significance for research evaluation.The literature reports growing attention, overviews, and critical discussion alongside a lack of conclusive evidence.
- Perceived potential: 86% of surveyed bibliometricians saw some potential for altmetrics in author or article evaluation, compared with 72% for paper downloads or views.Perceived potential was around 35% for platforms such as blogs or reference managers, significantly lower than the figures for typical altmetrics and downloads or views.
- Evidence limitations: The evidence base is limited because large-scale studies are rare, user motivations are poorly understood, and many studies only calculate correlations with citations.Medium-level correlations found by most studies are described as hardly meaningful and open to loose interpretation.
- Research priorities: Future research should identify who uses research products and why, including users’ scholarly status, geography, and career stage.The literature also recommends more sophisticated technologies for analysing user demographics and comparing altmetric counts with expert evaluations of societal impact.
- Validity and use: Altmetric data require safeguards against gaming and representativeness problems, such as whether relevant groups are adequately represented on a platform.For evaluating political impact, the political representation of the relevant social-media platform must be understood.
- Responsible evaluation: Altmetrics should inform expert peer review and complement traditional metrics rather than directly determine research funding or replace bibliometrics.The discussion recommends using altmetric results to help experts make decisions within an informed peer-review process.