Source-linked AI summary
Beyond citations: Scholars' visibility on the social Web
Judit Bar-Ilan, Stefanie Haustein, Isabella Peters, Jason Priem, Hadas Shema, Jens Terliesner
TL;DR
The paper examines whether scholars’ Web footprints broaden visibility measurement beyond publication and citation counts, and how social Web indicators relate to traditional metrics. It studies 57 STI 2010 presenters using Web, bibliometric, and social reference-manager data, finding widespread online presence and strong Mendeley coverage of sampled articles. The study also reports that Mendeley readership measures correlate with traditional citation counts.
Problem
Traditional publication and citation counts capture scholarly visibility within publishing networks but omit broader Web-based scholarly behavior.
Method
The study sampled 57 STI 2010 presenters and compared manually collected Web footprints and social reference-manager data with Scopus and Web of Science records.
Results
Mendeley covered 82% of 1,136 sampled articles, compared with 85% cited in Scopus and 74% cited in Web of Science.
Takeaways & Limitations
Social reference managers provide substantial reader-centered data that expands available views of authors’ and articles’ scholarly visibility.
Takeaways & Limitations
Article-level data from blogs, Twitter, and Wikipedia were not discussed because of space constraints, and incomplete DOI metadata may underestimate CiteULike coverage.
Abstract
from arXiv · showhide
Traditionally, scholarly impact and visibility have been measured by counting publications and citations in the scholarly literature. However, increasingly scholars are also visible on the Web, establishing presences in a growing variety of social ecosystems. But how wide and established is this presence, and how do measures of social Web impact relate to their more traditional counterparts? To answer this, we sampled 57 presenters from the 2010 Leiden STI Conference, gathering publication and citations counts as well as data from the presenters' Web "footprints." We found Web presence widespread and diverse: 84% of scholars had homepages, 70% were on LinkedIn, 23% had public Google Scholar profiles, and 16% were on Twitter. For sampled scholars' publications, social reference manager bookmarks were compared to Scopus and Web of Science citations; we found that Mendeley covers more than 80% of sampled articles, and that Mendeley bookmarks are significantly correlated (r=.45) to Scopus citation counts.
Introduction
The paper frames altmetrics as a way to broaden scholarly visibility measurement beyond publication and citation counts by examining researchers’ Web presence and social bookmarking data. It asks how widespread these footprints are, how alternative metrics compare with traditional indicators, and what qualitative information tags provide.
- Motivation: Citation counts capture authors’ impact on other publishing authors but omit aspects of scholarly behavior outside citation databases.The paper motivates social Web measures as a complement to traditional reference-based indicators.
- Altmetrics: Altmetrics combines social media with traditional scholarly practice to expand impact measurement using new data sources and metrics.The paper describes altmetrics as a subset of webometrics covering multiple stages and products of scholarly research.
- Platforms: Mendeley and CiteULike let users save, share, search, and tag scholarly references, enabling counts of bookmarks and reader-generated tags.Bookmarking actions provide usage measures, while tag frequencies reflect readers’ perceptions of article content and bookmarking motivations.
- Related work: Prior research found Mendeley bookmarks covered 92% of sampled articles and correlated moderately with Web of Science and Google Scholar citations.Reported correlations were r=.55 with Web of Science and r=.60 with Google Scholar, while CiteULike correlations were lower.
- Research questions: The study asks what researcher information is visible online, how Mendeley and CiteULike compare with publication and citation counts, and what tags reveal about authors.The article-level comparison focuses on coverage and correlations between readership and citations.
Data collection for the STI 2010 presenters
The study assembled a dataset of 57 STI 2010 presenters and manually combined Web-footprint, bibliometric, and social reference-manager information. The collection used Scopus as the main publication source, supplemented identifiers and checked researcher details, while acknowledging incomplete bibliographic coverage.
- Sample: 57 researchers remained after excluding ambiguous presenters, presenters without WoS or Scopus information, and one researcher outside bibliometrics and scientometrics.The exclusions totaled 10 names, with Herbert Marsh removed for topic mismatch.
- Web data: The researchers manually collected homepages, publication lists, LinkedIn and Google Scholar profiles, Twitter accounts, and Mendeley memberships.Collected information was verified against self-provided details including names, professions, institutions, publications, countries, and related fields.
- Bibliometric data: Publication searches used Web of Science and Scopus during February 2012, with Scopus as the main source and WoS and Crossref supplying missing DOIs.WoS category restrictions and Scopus first-name searches helped disambiguate authors.
- Altmetric data: Mendeley data came from manual searches, ReaderMeter, and total-impact, while total-impact also exposed usage across CiteULike, Wikipedia, Twitter, and other platforms.The tools retrieved author-level or article-level information through identifiers such as DOIs and PubMed IDs.
- Limitations: CiteULike retrieval used DOIs for 1,136 documents, but incomplete bibliographic data and only 77.6% correct DOI coverage may underestimate bookmarks.The paper notes that data quality limitations affected CuL, Mendeley, Scopus, and WoS, despite efforts to improve the dataset.
Results
The 57 presenters showed widespread professional Web visibility, while article-level Mendeley coverage was substantial and correlated significantly with Scopus citations. Social bookmarking also supplied reader-centered information beyond traditional citation databases.
- Author level metrics and footprints: 84% of presenters had institutional homepages, 70% had LinkedIn accounts, 23% had public Google Scholar profiles, and 16% had Twitter accounts.A narrow majority had complete publication lists, while another 33% displayed selected lists.
- Article level metrics – traditional and alternative: 928 of 1,136 articles (82%) had at least one Mendeley bookmark, compared with 319 (28%) bookmarked in CiteULike.Mendeley-bookmarked documents averaged 9.5 users, versus 2.4 bookmarks per CiteULike-bookmarked document.
- Article level metrics – traditional and alternative: Mendeley coverage was 44% for articles published before 1990 and 88% for those published since 2000.The results indicate lower bookmarking of older articles in this sample.
- Article level metrics – traditional and alternative: 19 authors had all searched publications bookmarked by at least one Mendeley reader, with Ed Noyons having all 23 documents bookmarked.Author-level results were aggregated by coverage and number of documents.
- Article level metrics – traditional and alternative: CiteULike tags provided a reader-centric view of authors’ publication histories and areas of expertise.For Harnad, the dominant “openaccess” tag suggested greater focus than Leydesdorff’s multiple research strands.
- Comparison of traditional and alternative metrics in terms of author visibility: Mendeley and Scopus had the highest correlation among article-level metrics, at ρ=.448, and the relationship was significant.The analysis used single articles to avoid potential misinterpretation of author-level correlations; relationships tightened as counts grew.
- Discussion & Future Work: The study is limited by its specific sample, messy social-media sources, and limited understanding of how and why scholars use altmetric tools.The authors characterize the findings as an early step toward understanding online impact data.