LMBot: Distilling Graph Knowledge into Language Model for Graph-less Deployment in Twitter Bot Detection

• The framework is motivated by bots’ role in disseminating misinformation, manipulating opinion, and facilitating fraudulent activities.Source: §1 · p. 1
• Graph-based inference requires increasingly many multi-hop neighbors, making API-based deployment time-consuming and vulnerable to sampling bias.Source: §1 · p. 2, Figure 1 · p. 2
• Finetuned language models outperform state-of-the-art non-graph-based methods and closely rival graph-based performance without neighbor information.Source: §1 · p. 2, Figure 2 · p. 2
• LMBot represents users as textual sequences, adapts the language model, and iteratively exchanges embeddings and soft labels between the LM and GNN.Source: §1 · p. 2
• Replacing the GNN with an MLP extends LMBot to datasets without graph structure.Source: §1 · p. 3
• LMBot achieves state-of-the-art performance on four Twitter bot detection benchmarks while outperforming 10 competitive baselines.Source: §1 · p. 3

2.1 Twitter Bot Detection

• Twitter bot detection research includes feature-based, text-based, graph-based, and multimodal methods using metadata, tweets, network structure, and combined information sources.Source: §2.1 · p. 3
• Graph-based methods exploit Twitter’s network structure and generally outperform non-graph methods, but require expensive neighborhood retrieval for deployment.Source: §2.1 · pp. 3–4

2.2 Knowledge Distillation

• Knowledge distillation typically transfers a trained teacher’s knowledge to a student model to reduce parameters and computation.Source: §2.2 · p. 4
• Existing cross-model work includes distilling GNN knowledge into MLPs and alternately training language models and GNNs through pseudo-labels.Source: §2.2 · p. 4
• LMBot addresses the under-explored problem of distilling text-rich heterogeneous network information into language models, with LM and GNN mutually enhancing one another.Source: §2.2 · p. 4, Figure 3 · p. 4

3 METHODOLOGY

• LMBot presents each user as a textual sequence, obtains LM embeddings, and iteratively transfers graph knowledge from GNNs into the LM through soft labels.The framework first performs LM domain adaptation, then alternates LM and GNN/MLP updates using knowledge distillation.Source: §3 · p. 4, §3.2 · p. 5, §3.2.1 · p. 5

3.1 User as a Textual Sequence

• Metadata, descriptions, and tweets are organized with special tokens into a unified user sequence that can be processed by the language model.The representation includes metadata, descriptions, and tweets, with noise such as hashtags, mentions, and URLs normalized before encoding.Source: §3.1 · pp. 4–5

3.2.1 LM domain adaptation finetuning.

• Domain-adaptation finetuning encodes user sequences, mean-pools LM outputs into 768-dimension embeddings, and trains a two-class prediction head with hard labels.The objective includes L2 regularization for LM parameters, while the adaptation step is intended to improve stability and convergence speed.Source: §3.2.1 · p. 5

3.2.2 GNN/MLP step.

• For graph-structured datasets, the GNN consumes LM embeddings, performs message passing and aggregation, and produces user prediction logits.For datasets without graph structure, LMBot replaces the GNN with an MLP operating on LM-generated embeddings.Source: §3.2.2 · p. 5
• After GNN/MLP training, the best validation model generates soft labels that serve as teacher signals for the subsequent LM update.The distillation temperature controls the difficulty of transferring this knowledge.Source: §3.2.2 · p. 5, §3.2.3 · p. 6

3.2.3 LM step.

• The LM step combines hard-label cross entropy with soft-label Kullback-Leibler divergence, balancing them with coefficient α.The best LM checkpoint on the validation set supplies embeddings for the next GNN training step.Source: §3.2.3 · p. 6

3.3 Training and Inference

• Training begins with LM domain adaptation and then alternates GNN/MLP and LM updates until validation performance stops improving.After convergence, both the LM and GNN/MLP can be used for inference as LMBot-LM and LMBot-GNN/MLP.Source: §3.3 · p. 6

4.1 Experimental Settings

• The experiments use four benchmarks and compare LMBot with ten feature-, text-, and graph-based baselines across repeated runs.Results are averaged over five runs with standard deviations.Source: §4.1 · p. 6

4.2 Main Results

• LMBot consistently outperforms all baseline methods across four datasets with or without graph structure, including graph-less inference.It surpasses state-of-the-art baselines by 0.37%, 0.97%, 0.67%, and 0.40% on the reported datasets.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Combination of Different GNNs and LMs

• LM performance remains relatively stable across GNN combinations, whereas GNN performance depends strongly on both the LM and GNN selected.T5 averages 84.94, close to 85.5 for the other models, while GAT reaches 84.46 with DeBERTa but 76.47 with T5.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

4.4 Convergence Analysis

• The iterative process mutually improves LM and GNN performance, and domain-adapted LMs converge in fewer iterations than untuned LMs.Even lower-performing models such as T5 require fine-tuning to converge to the optimal value.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5 Prediction Consistency

• LM and GNN/MLP predictions are highly consistent, with matching prediction regions comprising 99.59% of Cresci-2017 points and 95.77% of TwiBot-20 points.The remaining probability differences are attributed to model-specific inductive biases that can be learned through iteration.Source: §4.5 · p. 8, Figure 6 · p. 8

4.6 Data Efficiency Study

• LMBot remains relatively stable with only 10% to 100% of training labels and robust when retaining only 10% to 100% of graph edges.These experiments evaluate label-level and edge-level data efficiency separately.Source: §4.6 · p. 8, Figure 7 · p. 8

4.7 Ablation Study

• The full LMBot model outperforms ablations, while removing metadata has an especially large effect on performance.The ablations compare LM-only finetuning, GNN-only training, and replacing GNN with MLP during iteration.Source: Table 2 · p. 8, §4.7 · p. 8

• LMBot achieves state-of-the-art performance on four widely used Twitter bot detection benchmarks.Source: §5 · p. 8
• Graph-free LM inference addresses the time and sampling-bias issues caused by retrieving multi-hop neighbors.Source: §5 · p. 8
• Extensive studies report that LMBot is more robust, versatile, and efficient than previous state-of-the-art graph-based methods.Source: §5 · p. 8

• LMBot is designed as an assistive tool rather than the sole decision maker, so human judgment remains essential for deployment on real social networks.Source: §6 · p. 9
• False positives may identify legitimate accounts as bots and create serious implications for individuals or organizations spreading crucial information.Source: §6 · p. 9
• Both the LM and GNN may inherit biases, including stereotypes from pretraining data and discrimination against certain demographic groups.Source: §6 · p. 9

• The algorithm outputs optimized LM parameters ΘLM and optimized GNN/MLP parameters ΘGNN/MLP.Source: A ALGORITHM FOR TRAINING LMBOT · p. 11
• Training initializes ΘLM and ΘGNN/MLP before iterating over users in the dataset S.Source: A ALGORITHM FOR TRAINING LMBOT · p. 11
• For each user, the algorithm preprocesses the textual sequence t_i and obtains the user embedding z_i from Equation (1).Source: A ALGORITHM FOR TRAINING LMBOT · p. 11

• Table 3 reports statistics for the four selected datasets.Source: Table 3 · p. 11

• BotRGCN combines RoBERTa encodings of tweets and descriptions with numerical and categorical features in an RGCN classifier.Source: C BASELINE DETAILS · p. 11
• RGT uses graph transformers for each relationship type and semantic attention across relationships to learn user representations.Source: C BASELINE DETAILS · p. 11
• SGBot, BotHunter, Kudugunta et al., and LOBO use metadata or tweet features with random forests or boosting-based classifiers.Source: C BASELINE DETAILS · p. 11
• GLNN distills GNN knowledge into an MLP for fast inference without graph structure, while BIC and BotBuster combine multimodal or mixture-of-experts designs.Source: C BASELINE DETAILS · p. 11

• The implementation uses PyTorch, PyTorch Geometric, scikit-learn, and Transformers.Source: D IMPLEMENTATION DETAILS · p. 11
• Experiments run on one Tesla V100 GPU with 32 GB of memory.Source: D IMPLEMENTATION DETAILS · p. 11
• LMBot hyperparameter settings are provided in Table 4.Source: D IMPLEMENTATION DETAILS · p. 11, Table 4 · p. 11

• Training LMBot is more computationally intensive and time-consuming than graph-based methods, complicating application to large-scale datasets such as TwiBot-22 [9].Source: E LIMITATIONS AND FUTURE WORK · p. 11
• A 512-token maximum input length for RoBERTa [30] prevents LMBot from incorporating all user information and limits the proportion of included tweets.Source: E LIMITATIONS AND FUTURE WORK · p. 11
• Future work targets large-scale deployment, more comprehensive use of user information, and more effective textual information extraction.Source: E LIMITATIONS AND FUTURE WORK · p. 11