PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying Scientists on X

Philipp Meier, Katarina Boland, Laura Kallmeyer, Stefan Dietze

arXiv:2609.31264v1cs.CL

TL;DR

As science-related discourse grows and the traditional scientific knowledge order erodes, the paper addresses automatic identification of scientists and non-scientists on X. It uses tweets and user biographies, reaching F1 scores of 88% with explicit features and 96% with a DeBERTa ensemble.

  • Problem

    The paper addresses the lack of approaches identifying scientists across fields and topics using only tweet- and user-level information, without network features or X Lists.

    Source: §1 · p. 2

  • Method

    The approach classifies X accounts using linguistic features from tweets and biographies alongside pre-trained language models.

    Source: §1 · p. 2

  • Results

    F1 scores reach 88% with the explicit-feature model and 96% with the DeBERTa Ensemble model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach provides a basis for studying scientists' interactions with other user groups and perceived expertise in science-related discourse.

    Source: §1 · p. 1

    The Orcid matching method misses many scientist accounts and biases the dataset toward scientists who list an ORCID in their account description.

    Source: §3.1.1 · p. 3

Abstract

from arXiv · show

With the growing importance of science-related discourse on the Web and the erosion of the classical knowledge order, it is important to identify different user groups, such as scientists, automatically. This work proposes an approach for identifying scientists and non- scientists on X/Twitter based on their user biographies and tweets. We show that we are able to classify accounts as scientists and non- scientists on two different datasets, reaching an F1 score of up to 0.88 using Random Forests with linguistic features and up to 0.96 using a contrastively fine-tuned DeBERTa model in an ensemble setup. Furthermore, we provide two datasets with X users labeled as scientists or non scientists and their respective tweets and user biographies.

1 Introduction

As science-related discourse grows on X, scientists and users from other backgrounds participate in it, making automatic identification of scientists a useful basis for studying online discourse. The paper targets identification across fields and topics using tweets and biographies.

1 Introduction

  • Science communication on social networks gained importance as public interest in science increased, including during the COVID-19 pandemic.The paper describes online science discourse as involving users with varied backgrounds and occupations, where scientific insights may be oversimplified or instrumentalized.Source: §1 · p. 1
  • Earlier research covered specific disciplines or broad scientific fields, while communication-style studies often used small, manually annotated samples.Ke et al. [24] found mathematics and physics under-represented and social scientists over-represented in online discourse.Source: §1 · p. 1
  • Prior work reports that perceived experts can serve as information sources and become vectors of misinformation, underscoring the importance of identifying scientists during crises.False information spreads faster than correct information, as reported by Vosoughi et al..Source: §1 · p. 1
  • F1 scores reach 88% with the explicit-feature model and 96% with the DeBERTa Ensemble model.The paper also contributes the Orcid and Scholar datasets, with users labeled as scientists or non-scientists and their tweets and biographies.Source: §1 · p. 2
  • The proposed approach uses linguistic cues from tweets and biographies, with scientists defined to include PhD holders, current PhD candidates, and scientific entities.The work also uses lexical, syntactical, and semantical features and pre-trained language model-based models.Source: §1 · p. 2
  • The paper addresses a gap in approaches identifying scientists across fields and topics using only tweet- and user-level information, without network features or X Lists.It presents this work as a basis for studying scientists' interactions with other user groups, their roles, and perceived expertise.Source: §1 · p. 2

2 Related Work

Prior approaches identified scientists through discipline-specific sources, manual study, or list-based sampling. This paper positions text-based scientist identification as author profiling using tweet and user information.

2 Related Work

  • Earlier work identified scientists in specific disciplines, including computer scientists and astrophysicists [20], or sampled across fields using Twitter/X lists [24].The list-based approach iteratively sampled accounts from seed users using list memberships and names containing scientist titles.Source: §2 · p. 2
  • The paper frames scientist identification as author profiling using tweet-level linguistic features, biography keywords, and PLM-based embeddings.Earlier user-group identification often relied on network information or X Lists, which the authors report was no longer available in their data sources.Source: §2 · p. 2

3 Data

The paper constructs labeled scientist and non-scientist corpora from X accounts and TweetsKB tweets for training identification classifiers. It describes how the accounts were selected, how the datasets were balanced or annotated, and what information the examples provide for classification.

3 Data

  • The two scientist datasets draw on TweetsKB tweets from December 2019 to May 2023, with Orcid and Scholar accounts identified through different sources.Source: §3 · p. 2, §3.1.1 · p. 3, §3.1.2 · p. 3

3.1.1 Orcid.

  • ORCID matching found 578 scientist accounts and 21,443 full-text tweets, but missed many scientists and biased the dataset toward users who disclose an ORCID in their biography.Source: §3.1.1 · p. 3

3.1 Scientists

  • Scholar scientist accounts came from publication-DOI matches to OpenAlex authors, with manual validation of a subset and DOI-containing tweets excluded to prevent shortcut learning.The resulting corpus contains 23,785 hydrated accounts and 189,852 full-text tweets; its tweet counts per user have a long-tail distribution.Source: §3.1.2 · p. 3

3.2 Non-Scientists

  • The Orcid non-scientist class uses 508 hydrated accounts and 6,737 tweets from a prior sample whose annotation status is uncertain.The source sampled 1,500 accounts from a million crawled users, so some negatives may be scientists from disciplines beyond computer science.Source: §3.2.1 · p. 3

3.2.2 Scholar.

  • 379,704 tweets from 47,570 accounts form a dataset balanced by scientist class; the Orcid dataset instead contains more than three times as many scientist tweets as non-scientist tweets.Source: Table 1 · p. 3, §3.2.2 · p. 3

3.2.1 Orcid.

  • Non-scientists were sampled from TweetsKB to match scientists' user-level tweet counts; annotators found two scientists among 400 random test-set accounts and treated unknown cases as non-scientists.The annotators agreed at κ = 0.67; a separate sample of 300 classifier-flagged accounts yielded 10 scientists and was treated as an upper-bound estimate.Source: §3.2.2 · p. 3

3.2.2 Scholar.

3.2.3 Data examples.

  • The examples show that biographies can identify occupations directly, tweets can suggest scientist or student status, and some accounts remain unknown when neither source is informative.Classification and manual annotation considered all tweets by each user.Source: §3.2.3 · p. 4

4 Methods

The methods combine tweet-level features and biography cues with contrastively or triplet-fine-tuned language models, then combine tweet- and biography-model predictions in an ensemble.

4.1 Feature-based models

  • The feature pipeline covers linguistic, X-based, sentiment, and topical features from tweets, while biographies contribute keyword and science-related emoji matches; four classifiers evaluate these features.The classifiers are SVM, Random Forest, Logistic Regression, and AdaBoost.Source: §4.1 · pp. 4–5

4.2 Finetuned Pre-Trained Language Models

  • DeBERTa models are fine-tuned with contrastive or triplet loss on concatenated user tweets or biographies, with tweet- and biography-specific models and a plain DeBERTa baseline.The two modality-specific predictions can be combined in an ensemble.Source: §4.2 · p. 5

4.2.1 Contrastive Loss.

  • Contrastive loss brings same-class user representations closer and separates different-class representations; at inference, logistic regression classifies embeddings as scientist or non-scientist.The paired inputs are concatenated tweets or biographies, and the encoder maps them to a lower-dimensional space.Source: §4.2.1 · p. 5

4.2.2 Triplet Loss.

  • Triplet loss trains on an anchor, a same-class positive, and a different-class negative representation, with a margin controlling their relative distances.The inference procedure follows the contrastive-loss setup, using the triplet-trained model.Source: §4.2.2 · p. 6

4.3 Ensemble of Language Models

  • The ensemble stacks tweet- and biography-model class probabilities as inputs to a logistic-regression model, combining predictions from both modalities.Users may have multiple biography texts across the study period, December 2019 to May 2023.Source: §4.3 · p. 6

5 Results

The results compare feature-based classifiers on the Scholar and Orcid datasets and contrastively fine-tuned models on Scholar, with the DeBERTa ensemble achieving the strongest reported result.

5 Results

  • 0.88 F1: Random Forest scores highest on Scholar with the full feature set, while Random Forest and SVM reach 0.84 without keyword features.Source: §5 · p. 6, Table 3 · p. 7
  • Keyword-only models assign every matched Scholar account to the scientist class, while adding keyword features to tweets improves performance on both datasets.Source: §5 · pp. 6–7
  • 0.82 F1: AdaBoost leads on Orcid with the complete feature set, narrowly ahead of Random Forest at 0.81; without keyword features, Logistic Regression leads at 0.79.Source: §5 · p. 7, Table 4 · p. 7
  • 0.88 F1: the contrastively fine-tuned DeBERTa ensemble achieves the highest results among the contrastive models on Scholar.The Random Forest model with keyword features scores 0.84 F1 in the ensemble comparison.Source: §5 · p. 7, Table 5 · p. 7
  • 0.96 accuracy: the approach matches the reported accuracy of Hadgu et al.'s computer-scientist model while covering unrestricted scientific domains.Source: §5 · p. 7
  • Random Forest offers feature-analysis transparency, whereas the contrastively fine-tuned DeBERTa model achieves the highest scores but lacks transparency.Source: §5 · p. 7

6 Feature Analysis

Feature permutation analysis identifies which linguistic cues Random Forests use, both with and without keyword features. Keywords dominate when included; without them, topical and lexical features remain prominent, while hedge features do not appear among the top ten.

6.1 Analysis with keyword features

  • Keyword match is the most important feature on both datasets, and topic score is also important; the two figures show the top ten features for Orcid and Scholar.The analysis uses mean feature-permutation importance over ten-fold cross-validation.Source: §6.1 · p. 7, Figure 3 · p. 8
  • Complex syntactical features help distinguish scientists from non-scientists, but their relative accuracy drops are considerably lower than keyword match.The Scholar model’s third-ranked feature differs from Orcid’s: corrected-type-token-ratio versus type-token-ratio; type-token-ratio is important on both datasets.Source: §6.1 · p. 8

6.2 Analysis without keyword features

  • Without keyword-match and regex-match features, the feature distribution changes: topic-related features become more prominent, and lexical features gain importance.For Orcid, type-token-ratio is around 0.007 and Flesch-Kincaid Reading Easiness around 0.0025; spelling errors and first-letter capitalization each score 0.005 on Scholar.Source: §6.2 · p. 8
  • Hedge count and hedge relative frequency are absent from the top ten, while topical-feature importance supports the hypothesis that scientists discuss specific topics.The comparison with prior work on spelling is noted, but does not establish that spelling causes classification differences.Source: §6.2 · p. 8

7 Discussion

The discussion reports strong performance from linguistic features, including without keywords, while noting limits on which scientists can be detected and uncertainty about transfer to other platforms.

7 Discussion

  • Linguistic features identify scientists on X even without keyword features, while contrastively fine-tuned DeBERTa achieves the highest results but is less transparent.The approach detects only scientists who share profession-related information in biographies or tweets about research or employment.Source: §7 · p. 8
  • Randomly sampled Scholar non-scientist accounts may include scientists; manual annotation suggests evaluation is not substantially biased by false negatives, but bias cannot be ruled out.The Orcid negative samples may also contain scientist accounts.Source: §7 · p. 8
  • Applicability to platforms such as Mastodon is currently unknown, and dataset access is restricted to reduce risks of repression or harassment.Source: §7 · p. 8
  • Figure 4 presents the top ten Random Forest features for Orcid and Scholar without keyword features.Source: Figure 4 · p. 9

8 Future Work

Future work will improve dataset annotations, broaden classification to journalists and politicians, and further develop feature-based and language-model approaches.

8 Future Work

  • The authors plan to improve dataset quality by adding annotations.Source: §8 · p. 9
  • Planned extensions add journalists and politicians as classification classes and examine which features help distinguish different user groups.Source: §8 · p. 9
  • The authors plan to improve contrastively fine-tuned models through methods such as data augmentation and combine feature-based approaches with PLM models in an ensemble.Source: §8 · p. 9

9 Conclusion

The conclusion presents linguistic cues in tweets as a means of detecting scientists, with lexical features and contrastive learning among the findings. It also notes that implicit-feature models outperform explicit-feature models and identifies other user groups for future application.

9 Conclusion

  • Scientists can be detected using linguistic cues in tweet texts.Source: §9 · p. 9
  • Keyword and regex matches in user biographies may improve models, but do not always do so.Source: §9 · p. 9
  • Lexical features play a substantial role in classifying scientists on both datasets, and the study observes potential in contrastive learning.Source: §9 · p. 9
  • Models using implicit features yield better results than models using explicit features; future work will examine other groups, including journalists and politicians.Source: §9 · p. 9

A.1 Parameters

The section specifies the BERTopic configuration used to embed, reduce, cluster, and vectorize documents.

A.1 Parameters

  • Documents were embedded with all-MiniLM-L6-v2, then reduced with UMap using 50 neighbors, two components, minimum distance 0.0, and Euclidean distance.The embedding model is cited as [43].Source: §A.1.1 · p. 10
  • Clustering used HDBSCAN with minimum cluster size 5, Euclidean distance, and the eom cluster-selection method.Source: §A.1.1 · p. 10
  • Vectorization used TF-IDF with BM25 weighting and reduced frequent words.Source: §A.1.1 · p. 10

A.1.1 BertTopic.

This appendix describes classifier hyperparameter searches and the construction and splitting of triplet and contrastive datasets for tweets and biographies.

A.1.1 BertTopic.

  • The feature-based classifier search varied Random Forest, logistic-regression, SVM, and AdaBoost settings across the listed hyperparameter options.Random Forest settings included 300 estimators and maximum depth None or 50; SVM kernels included rbf, linear, or poly, while AdaBoost used decision-tree estimators.Source: §A.1.2 · p. 10

A.2 Data creation

  • The datasets comprise 10 folds of 41,653 instances each, split 80% for training and 20% for validation without stratification.The tweet and biography datasets differ in author counts because more biographies were unusable and their authors were removed.Source: §A.2 · p. 10, Table 8 · p. 11, Table 6 · p. 11
  • Each author served as a triplet anchor, paired with a randomly chosen same-label positive and different-label negative, while reversed duplicate positive pairs were excluded.A negative author could be selected multiple times or not at all.Source: §A.2.1 · p. 11

A.2.2 Creation of contrastive Pairs.

  • Contrastive data used randomly paired authors with equal numbers of positive and negative pairs, and duplicate pair reversals were excluded.Each author was represented by concatenated tweets or biography text; unlike triplet creation, no author-level deduplication was performed.Source: §A.2.2 · p. 11

Related papers

Loading 2609.31264v1…