Source-linked AI summary
TweepFake: about Detecting Deepfake Tweets
Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, Maurizio Tesconi
TL;DR
Machine-generated social-media text raises a detection gap because real tweets are short, potentially deceptive, and produced by unknown generators. The paper introduces TweepFake, a balanced dataset of real bot and human tweets, and evaluates 13 detectors. Results indicate that GPT-2 tweets are particularly difficult to detect, while transformer fine-tuning achieves nearly 90% accuracy with a RoBERTa-based detector.
Problem
Detecting machine-generated social-media messages remains underexplored, especially for real short tweets produced by heterogeneous and unknown generators.
Method
The paper constructs TweepFake from real Twitter posts generated by multiple bot technologies, balances them with tweets from imitated humans, and evaluates 13 detection methods.
Results
GPT-2-generated tweets are especially difficult to detect, while fine-tuning a RoBERTa-based detector achieves nearly 90% accuracy.
Takeaways & Limitations
TweepFake provides a public benchmark for studying detection of short, real machine-generated social-media messages across heterogeneous generation techniques.
Takeaways & Limitations
The authors recommend further investigation of RNN-based detectors and human ability to distinguish machine-generated from human-written tweets.
Abstract
from arXiv · showhide
The recent advances in language modeling significantly improved the generative capabilities of deep neural models: in 2019 OpenAI released GPT-2, a pre-trained language model that can autonomously generate coherent, non-trivial and human-like text samples. Since then, ever more powerful text generative models have been developed. Adversaries can exploit these tremendous generative capabilities to enhance social bots that will have the ability to write plausible deepfake messages, hoping to contaminate public debate. To prevent this, it is crucial to develop deepfake social media messages detection systems. However, to the best of our knowledge no one has ever addressed the detection of machine-generated texts on social networks like Twitter or Facebook. With the aim of helping the research in this detection field, we collected the first dataset of \real deepfake tweets, TweepFake. It is real in the sense that each deepfake tweet was actually posted on Twitter. We collected tweets from a total of 23 bots, imitating 17 human accounts. The bots are based on various generation techniques, i.e., Markov Chains, RNN, RNN+Markov, LSTM, GPT-2. We also randomly selected tweets from the humans imitated by the bots to have an overall balanced dataset of 25,572 tweets (half human and half bots generated). The dataset is publicly available on Kaggle. Lastly, we evaluated 13 deepfake text detection methods (based on various state-of-the-art approaches) to both demonstrate the challenges that Tweepfake poses and create a solid baseline of detection techniques. We hope that TweepFake can offer the opportunity to tackle the deepfake detection on social media messages as well.
Introduction
TweepFake addresses the lack of evaluation for detecting machine-generated short social-media texts posted in real settings with unknown generators. It contributes a balanced, publicly available Twitter dataset and evaluates 13 detection methods to establish baseline challenges.
- Detection challenges: Existing detectors often learn statistical characteristics of neural text, but homoglyph substitutions and misspellings can alter those characteristics and complicate detection.The problem is especially challenging for short social-media messages, whose provenance is harder to identify than that of longer texts.
- Research gap: TweepFake targets a gap in detecting real deepfake social-media messages that are short, potentially altered, and generated by unknown models.Prior work mainly used ad-hoc generated texts, longer documents, or known adversarial generators.
- Dataset contribution: The dataset contains 25,572 tweets, balanced equally between human-written and bot-generated messages collected from 23 bots imitating 17 human accounts.The bots use Markov Chains, RNN, RNN+Markov, LSTM, and GPT-2; each deepfake tweet was actually posted on Twitter.
- Evaluation: The authors evaluated 13 deepfake text detection methods spanning text-representation classifiers, deep learning networks, and transformer fine-tuning.The experimental code is publicly available on GitHub.
- Related approaches: Detection methods generally comprise simple classifiers, zero-shot detection using pretrained-language-model outputs, or fine-tuning-based detection.Prior work also includes logistic-regression and likelihood-threshold baselines, visual human-aiding tools, and energy-based detectors.
DeepFake tweets generation
The dataset’s machine-generated tweets are produced token by token using Markov Chains, recurrent models, LSTM, GPT-2, and related account-collection procedures. These generators differ in how they select tokens and retain contextual information.
- Generation pipeline: The generation pipeline tokenizes human-written training corpora, then produces tokens until a stop token or predefined maximum length is reached.RNN, LSTM, and GPT-2 sample each next token from a multinomial vocabulary distribution using techniques such as maximum probability, top-k, or nucleus sampling.
- Markov Chains: Markov Chains select each next token from tokens following the current one, with probabilities proportional to their observed frequencies.The model’s transition probability depends only on the current state, where a state is a token.
- Recurrent models: RNNs use accumulated memory of previously encountered tokens to compute the next-token distribution, while LSTMs selectively retain relevant information and reduce vanishing-gradient effects.LSTM memory is described as longer than RNN memory.
- GPT-2: GPT-2 uses transformer attention to model how tokens relate across contexts, then generates text through multinomial next-token sampling like RNNs and LSTMs.Its attention mechanism supports more coherent and non-trivial generated text.
- Dataset construction: The dataset construction searched Twitter, GitHub, and the web for accounts describing automated text-generation technologies, then matched bot tweets with randomly sampled tweets from imitated humans.The resulting collection contains 25,572 tweets, half human and half bot generated.
DeepFake tweets detection
The study benchmarks machine-learning, deep-learning, character-level, and fine-tuned language-model approaches for detecting automatically generated tweets. It builds baselines using several text representations and neural architectures, including CNN, bidirectional GRU, and combined CNN-GRU networks.
- Experimental design: The study evaluates methods of increasing complexity to establish performance baselines and identify promising approaches for deepfake-tweet detection.Experiments and code were made publicly available through the project repository.
- Detection approaches: The experiments compare four approaches: BoW with TF-IDF, BERT representations, character-level deep learning, and fine-tuned pretrained language models.The BoW approach uses logistic regression, random forest, and SVM classifiers.
- Character-level networks: Character-level models encode each tweet as contiguous character IDs from a fixed vocabulary and learn embeddings during training.The resulting embedding matrix is passed to subsequent deep-learning layers.
- Character-level networks: The tested character architectures use either a multi-kernel CNN, a bidirectional GRU, or a combined CNN-GRU network.The combined architecture targets spatial and temporal feature extraction.
- Fine-tuned models: Fine-tuning adapts pretrained language-model weights to the classification task by adding a dense classification layer and training on the dataset.Four transformer-based language models were tested, with training limited to three epochs for computational reasons.
Results
The evaluation reports standard classification metrics on a test set and separately examines accuracy across human, GPT-2, RNN, and other account categories. Human examples were undersampled for the category-level comparison.
- Overall evaluation: The test-set evaluation reports precision, recall, F1, and accuracy for the proposed detection baselines.Because the dataset is balanced, accuracy is treated as the most reasonable overall effectiveness measure.
- Account categories: The account-level analysis divides examples into human, GPT-2, RNN, and others categories.The others category includes mixed or unknown generation approaches.
- Category-level analysis: Category-level accuracy is visualized in a heat-map alongside global performance over all account categories.The analysis is intended to show how the baselines behave across different account types.
- Category-level analysis: Humans were randomly undersampled to 484 examples, matching the maximum category size used for the comparison.The resulting counts were humans 484, GPT-2 384, RNN 412, and others 484.
Discussion
The results show a clear accuracy hierarchy across representation types, while revealing systematic differences across account categories. Fine-tuned language models achieve the strongest global accuracy, but GPT-2 tweets remain difficult to detect.
- Global performance: 0.90 accuracy for roberta ft outperforms character encodings up to 0.85, BERT methods around 0.83, and BoW methods around 0.80.The reported ordering favors methods using native language-model encoding.
- Global performance: Character encoding remains a relatively simple approach that generally performs well and can help when pretrained models are unavailable.The paper gives non-English languages as an example setting.
- Error patterns: Fine-tuning baselines generally balance precision and recall better than the other configurations, except XLNET.Across methods, precision is higher for human examples, while recall is higher for bot examples.
- Discussion: The findings suggest that GPT-2 can produce more human-like short texts than older generative methods such as RNNs.The authors manually found GPT-2 tweets harder to label as bot-generated and propose further investigation with human questioning.
Conclusion
The paper presents TweepFake, the first real deepfake-tweet dataset, and evaluates 13 detectors to establish detection baselines. Results indicate that transformer-generated short texts remain difficult to identify, while fine-tuned detectors achieve nearly 90% accuracy and RNN-based detectors warrant further study.
- TweepFake is presented as the first real deepfake tweets dataset for developing social-media deepfake detection techniques.Its tweets were actually posted on Twitter and are publicly available on Kaggle.
- 25,572 balanced tweets combine human and bot-generated messages from 23 bots imitating 17 human accounts.The bots use GPT-2, RNN, LSTM, Markov Chain, and related generation techniques.
- The study evaluates 13 detectors spanning representation-based machine-learning classifiers, deep-learning networks, and fine-tuned transformer classifiers.This evaluation is intended to test discrimination difficulty and provide detection baselines.
- Nearly 90% accuracy is achieved by the RoBERTa-based fine-tuning detector, outperforming text-representation-based approaches.The results also suggest that transformer-based language models provide strong word representations for both representation-based and fine-tuning detectors.
- Further investigation of RNN-based detectors is recommended because CHAR GRU best labels GPT2-generated tweets as bots.The paper also calls for studies of human discrimination ability and tweet humanness across generation methods.