Source-linked AI summary
Grand challenges in altmetrics: heterogeneity, data quality and dependencies
Stefanie Haustein
TL;DR
As social media enter scholarly communication and research evaluation, altmetrics promise broader evidence of scholarly and societal impact but lack a settled conceptual basis. The paper examines their heterogeneity, data-quality problems, and dependencies, finding that varied acts, dynamic events, and reliance on platforms and intermediaries constrain their interpretation and sustainability.
Problem
Altmetrics lack a common definition and theoretical foundation, while their data quality, coverage, and representativeness remain difficult to establish for research evaluation.
Method
The paper identifies and discusses heterogeneity, data quality, and dependencies as three major altmetrics challenges, emphasizing parallels with bibliometrics.
Results
Altmetrics comprise heterogeneous acts with markedly different coverage, while their accuracy, replicability, and sustainability depend on dynamic events, metadata, platforms, providers, and aggregators.
Takeaways & Limitations
Meaningful altmetric indicators require clearer conceptualization of underlying acts and attention to data quality, sustainability, and platform-related dependencies.
Takeaways & Limitations
Many altmetric acts remain shaped by technological affordances, and the influence of those affordances has not been systematically analyzed.
Abstract
from arXiv · showhide
As uptake among researchers is constantly increasing, social media are finding their way into scholarly communication and, under the umbrella term altmetrics, were introduced to research evaluation. Fueled by technological possibilities and an increasing demand to demonstrate impact beyond the scientific community, altmetrics received great attention as potential democratizers of the scientific reward system and indicators of societal impact. This paper focuses on current challenges of altmetrics. Heterogeneity, data quality and particular dependencies are identified as the three major issues and discussed in detail with a particular emphasis on past developments in bibliometrics. The heterogeneity of altmetrics mirrors the diversity of the types of underlying acts, most of which take place on social media platforms. This heterogeneity has made it difficult to establish a common definition or conceptual framework. Data quality issues become apparent in the lack of accuracy, consistency and replicability of various altmetrics, which is largely affected by the dynamic nature of social media events. It is further highlighted that altmetrics are shaped by technical possibilities and depend particularly on the availability of APIs and DOIs, are strongly dependent on data providers and aggregators, and potentially influenced by technical affordances of underlying platforms.
1 Introduction
Digital scholarly communication has expanded beyond the traditional journal through social media and online tools, creating both broader reach and greater information overload. Altmetrics emerged as new filters for this expanding scholarly ecosystem, but their rapid adoption also raises concerns about how impact indicators are used.
- 1 Introduction: Social media enable researchers to raise visibility, connect with others, and disseminate scholarly work alongside the still-dominant peer-reviewed journal.Digital technologies have increased collaboration and publishing speed, but the electronic article remains essentially similar to its print counterpart.
- 1 Introduction: Digital scholarly communication increases transparency through open discussion, visible peer review, online sharing and reuse of data and software, and social-media dissemination.This diversification allows researchers to distribute different forms of scholarly work and reach larger audiences.
- 1 Introduction: The diversification of scholarly communication creates an opportunity for broader dissemination while increasing information overload.New online scholarly tools provide additional filters for navigating the expanding literature.
- 1 Introduction: Altmetrics and citation indexing both use collective intelligence to identify relevant scholarly works.The paper frames citation indexing as an early, pre-social-web form of crowdsourcing.
- 1 Introduction: Bibliometric indicators and altmetrics can be adopted too quickly as measures of impact and scientific performance.Earlier bibliometric work emphasized complementarity, expert verification, and triangulation, but citations later became associated with scientific impact and quality.
2 Challenges of altmetrics
Altmetrics face three major challenges: heterogeneity, data quality, and dependencies. These challenges arise from diverse scholarly acts and platforms, dynamic and uneven data, and reliance on technical infrastructures, providers, and aggregators.
- Lack of a common definition: Altmetrics comprise heterogeneous metrics whose diverse acts, users, platforms, and motivations make a common definition and conceptual framework difficult to establish.Their meanings vary by specific indicator rather than forming one unified meaning.
- Lack of a common definition: Scholarly metrics indicate recorded acts related to scholarly documents or agents, while altmetrics are a heterogeneous subset of informetrics, scientometrics, and webometrics.Examples of acts include viewing, reading, saving, diffusing, mentioning, citing, reusing, and modifying.
- Heterogeneity of social media acts, users and motivations: Platform purposes and audiences produce different acts and coverage: blogs and F1000Prime reach less than 2% of recent publications, versus roughly 10–20% for Twitter and 60–80% for Mendeley.Lower coverage reflects both differences in engagement and differences in user-population size.
- Heterogeneity of social media acts, users and motivations: Recorded events can represent substantially different behaviors, may be automatically generated, and may fail to capture acts when identifiers are absent.A tweet may promote, diffuse, appraise, or criticize work, while a readership count may reflect either a quick look or intensive reading.
- 2.2 Data quality: Altmetrics data quality is threatened by accuracy, consistency, and replicability problems because dynamic sources can be altered or deleted and data issues occur across providers, aggregators, and users.Unlike static bibliographic sources, social-media events can change over time.
- 2.2 Data quality: Metadata and identifiers constrain meaningful tracking: publication year poorly represents fast social-media activity, document versions complicate aggregation, and DOI reliance creates field and geographic biases.DOI-related biases particularly affect the social sciences, humanities, and the Global South.
- 2.3 Dependencies: Altmetrics are strongly shaped by technology and depend on data providers and aggregators, with APIs and document identifiers determining which activities are captured.Platforms without APIs, such as ResearchGate or Zotero, are not used in the cited collection context.
- 2.3 Dependencies: Dependence on social-media platforms is deeper than dependence on citation databases because discontinuing Twitter or Mendeley would eliminate both the data source and the underlying acts.Platform affordances may also shape user behavior, although the paper presents these effects as hypotheses rather than systematic findings.
3 Conclusions and Outlook
Altmetrics face conceptual and evaluative challenges because their underlying online acts are heterogeneous, dynamic, and shaped by technological affordances. Their data quality and sustainability are also constrained by dependence on providers and aggregators, requiring careful indicator selection and safeguards against misuse.
- Conceptual challenges: Heterogeneous online acts require conceptualization before recorded events can become valid indicators of engagement or impact.Some events may function as useful filters, valid impact measures, or merely reflect buzz; social media activity does not equal social impact.
- Conceptual challenges: Altmetrics are difficult to interpret because their underlying acts are still forming and are shaped by technological affordances.This makes it challenging to establish a conceptual framework for what different recorded events measure.
- Data quality and dependencies: Dynamic events make altmetrics difficult to maintain with high accuracy, consistency, and reproducibility.Data quality and sustainability are further impeded by strong dependence on single data providers and aggregators.
- Data quality and dependencies: Most altmetric data being held by for-profit companies conflicts with the openness and transparency motivating altmetrics.This dependency constrains the sustainability and transparency of the data used for evaluation.
- Evaluation implications: Metrics should be chosen for the particular assessment aim rather than allowing any single indicator to dominate evaluation.The paper warns against replacing the impact factor with the Altmetric donut and calls for mitigating adverse effects caused by indicators.