Source-linked AI summary
Unsupervised User Stance Detection on Twitter
Kareem Darwish, Peter Stefanov, Michaël Aupetit, Preslav Nakov
TL;DR
The paper addresses the need for stance detection without extensive user labeling, topic expertise, or sensitivity to skewed data. It projects prolific Twitter users into a low-dimensional space and clusters them, finding small, high-purity groups across varied datasets. The resulting clusters can be labeled in bulk and used directly or for downstream classification, although the approach assumes users within each cluster share a stance and uses simplifying assumptions in some dataset labels.
Problem
Existing stance-detection methods rely on supervised or semi-supervised labeling, topic expertise, and handling skewed class distributions.
Method
The framework projects active users into a low-dimensional space using similarity features, then applies peak detection or clustering to identify core user groups for cluster-level labeling.
Results
The best setups use retweeted accounts, UMAP or Fruchterman-Reingold reduction, and Mean Shift to identify predominant-stance groups with more than 98% purity.
Takeaways & Limitations
The resulting user groups can be used directly, labeled in bulk, or used to train supervised classifiers and seed semi-supervised methods.
Takeaways & Limitations
The framework assumes users in each cluster share the target stance, and one dataset labels support using party affiliation as a simplifying assumption.
Abstract
from arXiv · showhide
We present a highly effective unsupervised framework for detecting the stance of prolific Twitter users with respect to controversial topics. In particular, we use dimensionality reduction to project users onto a low-dimensional space, followed by clustering, which allows us to find core users that are representative of the different stances. Our framework has three major advantages over pre-existing methods, which are based on supervised or semi-supervised classification. First, we do not require any prior labeling of users: instead, we create clusters, which are much easier to label manually afterwards, e.g., in a matter of seconds or minutes instead of hours. Second, there is no need for domain- or topic-level knowledge either to specify the relevant stances (labels) or to conduct the actual labeling. Third, our framework is robust in the face of data skewness, e.g., when some users or some stances have greater representation in the data. We experiment with different combinations of user similarity features, dataset sizes, dimensionality reduction methods, and clustering algorithms to ascertain the most effective and most computationally efficient combinations across three different datasets (in English and Turkish). We further verified our results on additional tweet sets covering six different controversial topics. Our best combination in terms of effectiveness and efficiency uses retweeted accounts as features, UMAP for dimensionality reduction, and Mean Shift for clustering, and yields a small number of high-quality user clusters, typically just 2--3, with more than 98\% purity. The resulting user clusters can be used to train downstream classifiers. Moreover, our framework is robust to variations in the hyper-parameter values and also with respect to random initialization.
Introduction
The paper introduces an unsupervised stance-detection framework that projects active Twitter users into a low-dimensional space and clusters them for efficient, topic-independent labeling. Across varied datasets and configurations, it identifies a small number of high-purity user groups and remains robust to important parameter and data changes.
- Motivation and contribution: The framework replaces user-level supervised labeling with automatic clustering followed by cheaper bulk labeling of homogeneous user groups.Analysts label clusters using representative retweeted accounts or hashtags rather than labeling individual users.
- Method: Users are projected into two dimensions according to similarity, then peak detection or clustering identifies core groups of similar users.The low-dimensional representation also supports visual assessment of stance separability and downstream clustering.
- Experimental design: The study compares three dimensionality-reduction methods, DBSCAN and Mean Shift clustering, and multiple feature and dataset configurations.The experiments assess both effectiveness and computational efficiency.
- Results: 2–3 user clusters composed of hundreds of users on average achieve purity in excess of 98% across datasets in different languages and topics.Results were further verified on tweet datasets covering six controversial topics.
- Robustness: The framework is robust to changes in dimensionality-reduction and clustering hyper-parameters, tweet-set size, similarity features, minimum user counts, and random initialization.These experiments identify minimum requirements for effective stance detection.
Background
Prior stance-detection work commonly relies on labeled users, while clustering in high-dimensional feature spaces is computationally difficult and often produces weak structure. The paper motivates dimensionality reduction and core-focused clustering as a way to make stance groups more interpretable and efficient to identify.
- Features: Candidate stance features include tweet text, hashtags, retweeted accounts, mentions, follow relationships, and profile information.The paper explores retweeted tweets, retweeted accounts, and hashtags as relevant input features.
- Prior classification methods: Existing user-level stance methods mainly use supervised or semi-supervised classification requiring an initial labeled seed set.Label propagation can have high precision but is sensitive to extreme views, thresholds, and imbalanced seed sets.
- Dimensionality reduction: High-dimensional clustering suffers from computational growth, memory demands, and less informative distances between points.Direct clustering in the high-dimensional feature space failed to produce meaningful clusters in the authors’ experiments.
- Dimensionality reduction: Two-dimensional projections can preserve similarity, reveal separability visually, and enable more efficient downstream clustering.The paper considers FD, t-SNE, and UMAP for this projection step.
- Clustering: DBSCAN and Mean Shift are selected because they can handle arbitrarily shaped groups and identify cluster cores while ignoring outliers.The focus on core clusters targets users likely to express strong stances rather than sparse noisy groups.
- Pipeline: The framework combines selected features, two-dimensional reduction, and clustering to find efficient stance clusters that experts can inspect and label.Visual inspection supports checking cluster quality beyond standard clustering statistics.
Finding Stance Clusters
The framework constructs user similarities from engagement features, projects users into two dimensions, and then identifies dense cluster cores for stance analysis. The supplied figures contrast a successful UMAP-plus-Mean Shift setup with an unsuccessful t-SNE projection.
- Feature selection: User similarity is computed from retweeted tweets, hashtags, retweeted accounts, or their concatenation, using cosine similarity between users.Retweeted-account vectors encode each account by its relative retweet frequency.
- Dimensionality reduction: FD, t-SNE, and UMAP reduce the similarity space to two dimensions before clustering.These methods differ in how they preserve or model relationships among users during projection.
- Clustering: After projection, user positions are scaled independently to [−1, 1] on both axes before cluster-core detection.The pipeline then applies density-based or peak-detection clustering.
- Clustering: DBSCAN identifies density-based clusters using ε and m, while Mean Shift detects density peaks controlled by its kernel bandwidth.DBSCAN can exclude points that are neither core members nor reachable from core sets; Mean Shift groups points converging to the same peak.
- Cluster labeling: Clusters are assumed to share a stance and can be labeled using salient retweeted accounts and hashtags.A valence-score variant identifies frequent group characteristics for assigning cluster labels.
- Visual comparison: Figure 2 compares UMAP embeddings, Mean Shift clusters, and true labels, whereas Figure 3 shows a t-SNE projection colored by true labels.Together, the figures provide a visual comparison between a successful and an unsuccessful setup.
Datasets
The study uses three labeled datasets in English and Turkish to tune the framework, then tests its best settings on fresh data covering six controversial U.S. topics. The datasets vary substantially in languages, topics, collection periods, tweet volumes, and labeling procedures.
- Dataset design: The experiments use labeled and unlabeled datasets: labeled data supports setup and hyperparameter testing, while fresh unlabeled data evaluates the selected settings.The unlabeled collection contains new topics not used in the initial tuning datasets.
- Labeled datasets: Three datasets span English and Turkish topics, including Kavanaugh, Trump and the 2018 midterm elections, and Erdoğan and the 2018 Turkish elections.The Kavanaugh and Trump datasets are English; the Erdoğan dataset is Turkish.
- Labeled datasets: The Kavanaugh collection contains 23 million tweets authored by 687,194 users and begins with manual labeling of the 50 most prolific users.The collection covers two periods surrounding the Supreme Court confirmation process.
- Labeled datasets: The Trump collection contains 4,152,381 tweets from 1,129,459 users, with 13,731 users automatically labeled from stance-associated hashtags.The labels distinguish pro-Trump users from users using several anti-Trump hashtags.
- Labeled datasets: The Erdoğan collection contains 19,856,692 tweets authored by 3,184,659 users and covers June 16–23, 2018.The dataset concerns Erdoğan and the June 24 Turkish elections.
- Unlabeled datasets: Fresh data covers six U.S. topics combining long-standing political issues, transient political controversies, and the non-political vaccine debate.The collection is filtered to users whose locations indicate the USA.
Experiments and Evaluation
The experiments vary tweet-set size, user features, dimensionality reduction, and clustering to identify effective and efficient configurations. Retweet-based similarity with FD or UMAP and Mean Shift produced high-purity clusters, while the unsupervised method favored precision over recall relative to supervised classification.
- Experimental Setup: The experiments vary dimensionality reduction, clustering, user count, similarity features, and tweet-set size across multiple configurations.The compared dimensionality-reduction methods are FD, t-SNE, and UMAP; clustering uses DBSCAN or Mean Shift.
- Evaluation Criteria: Effective configurations require at least two clusters, average purity of at least 80%, and labels assigned to at least 30% of available users.The evaluation also reports average cluster count, cluster size, and recall across the tweet subsets.
- Configuration Results: 99.6% purity was achieved with 1M tweets, retweets, FD, and Mean Shift, although this configuration was computationally expensive.Hashtags alone failed to meet the success criteria in all setups.
- Configuration Results: 99.1% purity with FD and 98.8% with UMAP were achieved on 250k tweets and 1,000 users using retweets and Mean Shift, with greater efficiency than the overall best setup.Reducing tweet-set size and the number of clustered users improved computational efficiency.
- Configuration Results: UMAP with Mean Shift produced similar purity and cluster counts to FD with Mean Shift while running more than twice as fast.FD with Mean Shift achieved the best purity with two clusters on average.
- Comparison to Supervised Classification: The unsupervised setup achieved 99.1% cluster purity versus 86.0% precision for SVM-based classification, but recall was 70.6% versus 95.3%.The comparison used a 250k-tweet subset, 1,000 users, retweets, UMAP, and Mean Shift for the unsupervised method.
- New Unlabeled Data: 98.0% average cluster purity and 86.5% average recall were obtained on additional unlabeled tweet sets covering six topics.The experiments used UMAP, Mean Shift, the top 1,000 users, and retweets to compute similarity.
Analysis: Refining in Search of Robustness
The experiments refine the framework toward robust, efficient configurations by testing dimensionality reduction, clustering, hyper-parameters, initialization, and cluster-labeling signals. FD or UMAP with Mean Shift performs best, while UMAP is faster and salient retweets and hashtags support cluster labeling.
- Robustness checks: Five repeated runs found very little effect of UMAP and Mean Shift settings on purity, cluster count, or clustered-user count.The repetitions assessed sensitivity to both hyper-parameter values and random initialization.
- Unsuccessful configurations: No t-SNE or DBSCAN configuration met the minimum criteria of purity ≥0.8 and at least two clusters.Lowering t-SNE perplexity enabled t-SNE with Mean Shift, but not the unsuccessful DBSCAN combinations.
- Robust configurations: Successful setups combine FD or UMAP dimensionality reduction with Mean Shift clustering and remain robust to hyper-parameter settings.The study identifies these combinations as best for cluster purity and recall.
- Efficiency: UMAP runs significantly faster than FD, while Mean Shift and DBSCAN have comparable execution times.This makes UMAP preferable when computational efficiency is important.
- Recommended setup: The recommended automatic setup is UMAP plus Mean Shift with scikit-learn’s default settings.The recommendation follows the observed effectiveness, robustness, and efficiency results.
- Cluster labeling: Salient retweeted accounts and hashtags provide cluster-level signals that can be used to assign stance labels.The salience score combines an element’s cluster frequency with its valence score, retaining elements with V(e) ≥0.8.
Conclusion and Future Work
The paper concludes that dimensionality reduction followed by clustering can identify stance-homogeneous Twitter-user groups without supervised labeling. Its strongest setups use retweeted accounts, FD or UMAP, and Mean Shift, while future work will profile widely retweeted users and shared content across polarizing topics.
- Conclusion: The framework addresses manual-labeling, topic-expertise, and skew-sensitivity shortcomings of supervised or semi-supervised stance detection.It identifies clusters of users with similar stances on controversial topics.
- Conclusion: Dimensionality reduction brings similar users together, separates dissimilar users, supports visualization, and enables downstream clustering.The paper evaluates FD, t-SNE, and UMAP as dimensionality-reduction methods.
- Results: More than 98% purity was achieved using retweeted accounts, FD or UMAP, and Mean Shift on 250k-tweet datasets with the most active 500 or 1,000 users.UMAP was significantly faster than FD, and the best setups were robust to hyper-parameter changes and random initialization.
- Future work: Future work will profile popularly retweeted users, cited websites, and shared media using valence scores across polarizing topics.The proposed extension applies the stance-detection technique beyond the currently analyzed user clusters.