Source-linked AI summary
User-level sentiment analysis incorporating social networks
Chenhao Tan, Lillian Lee, Jie Tang, Long Jiang, Ming Zhou, Ping Li
TL;DR
The paper asks whether social-network relationships can complement tweet text for semi-supervised, topic-dependent user-level sentiment classification. It models Twitter follower/followee and @-mention networks, finding statistically significant improvements over text-only SVMs, especially when connectedness strongly correlates with shared sentiment.
Problem
The paper addresses semi-supervised, topic-dependent user-level sentiment categorization by asking whether social relationships provide information beyond users’ utterances.
Method
The authors use graphical models and transductive learning with Twitter follower/followee or @-mention networks, including directed and mutual connections.
Results
Statistically significant sentiment-classification improvements over text-only SVMs were achieved by incorporating social-network information.
Takeaways & Limitations
Significant gains can occur even with very sparse graphs when connectedness strongly correlates with shared sentiment, although the best link type varies by topic.
Takeaways & Limitations
Text alone can outperform the graph-based model for prolific users with abundant textual data, and manual tweet-based labeling had unusably low inter-annotator agreement.
Abstract
from arXiv · showhide
We show that information about social relationships can be used to improve user-level sentiment analysis. The main motivation behind our approach is that users that are somehow "connected" may be more likely to hold similar opinions; therefore, relationship information can complement what we can extract about a user's viewpoints from their utterances. Employing Twitter as a source for our experimental data, and working within a semi-supervised framework, we propose models that are induced either from the Twitter follower/followee network or from the network in Twitter formed by users referring to each other using "@" mentions. Our transductive learning results reveal that incorporating social-network information can indeed lead to statistically significant sentiment-classification improvements over the performance of an approach based on Support Vector Machines having access only to textual features.
1. INTRODUCTION
The paper proposes semi-supervised user-level sentiment analysis that combines tweet evidence with social-network relationships. Experiments on Twitter show statistically significant improvements over text-only SVMs, while comparing multiple network and learning configurations.
- Motivation: Social relationships can complement textual evidence because connected users may hold similar opinions.The motivation draws on the availability of Twitter relationship data and the homophily principle.
- Approach: The proposed models use either Twitter’s follower/followee network or an @-mention network to connect users’ opinions.The follower/followee model links a user’s opinion to those of followed users, while the @ variant links it to mentioned users.
- Approach: The framework is semi-supervised and targets user-level rather than tweet-level sentiment.This setting addresses the time-consuming and error-prone acquisition of large sentiment-labeled datasets and treats tweet sentiment as a proxy for overall user opinion.
- Results: The paper empirically confirms that shared opinion probability correlates with whether users are connected.It presents this correlation as an empirical contribution alongside the classification improvements.
- Results: The models yield statistically significant improvements in user-level sentiment polarity classification over text-only SVMs.The paper also varies network type, parameter learning, baseline training, and directed versus mutual graph construction.
2. CONCRETE PROBLEM SETTING
The task is semi-supervised binary sentiment classification for users discussing a topic on Twitter. The paper defines four connection graphs by combining t-follow or @-mention relationships with directed or mutual connectivity.
- Task: For a query topic, the task classifies each user as Positive or Negative toward that topic.This differs from classifying individual tweets.
- Task: The setting assumes that a relatively small proportion of users in a topic-specific graph are labeled.The remaining users’ sentiment labels must be predicted under a semi-supervised learning paradigm.
- Connection definitions: A directed t-follow edge exists when one user follows the other, reflecting attention to that user.The reverse follow is not required.
- Connection definitions: An @-mention is used as a rough proxy for a personal connection, although the paper identifies it as a heuristic.The intuition is that users address people with whom they are having a conversation and thus know.
- Connection definitions: Mutual connections require both users to link to each other and may indicate stronger relationships.The paper contrasts mutual links with one-way links because attention and homophily may contribute differently to shared sentiment.
- Connection definitions: The four graph variants are directed t-follow, mutual t-follow, directed @, and mutual @ graphs.They result from crossing two relationship types with directed versus mutual connectivity.
3. DATA AND INITIAL OBSERVATIONS
The dataset uses conservatively labeled Twitter users and investigates whether network connections correlate with shared sentiment. The observations support correlation in both directions, with stronger patterns generally associated with t-follow links than @ links.
- Data collection: Manual labeling of arbitrary users produced unusably low inter-annotator agreement because tweets require topic-specific knowledge and cultural context.The example “#lakers b**tch!” was mistakenly labeled negative for the Lakers topic.
- Data collection: The study therefore labels users conservatively from interpretable profile metadata, retaining only clearly classifiable opinions.This produces trustworthy gold-standard labels but focuses the graphs on users with strong opinions.
- Data collection: The experimental graphs contain gold-standard users and the edges between them, using only on-topic tweets for the experiments.Topics include Obama, Sarah Palin, Glenn Beck, Fox News, and Lakers.
- Initial observations: Connected user pairs are more likely than chance to share sentiment, with the effect generally more pronounced in t-follow than @ graphs.Mutual links have higher shared-sentiment probability for Obama, Sarah Palin, and Glenn Beck, while directed links are higher for Lakers.
- Initial observations: Same-opinion users are much more likely to be connected in the t-follow graph than in the @ graph.The paper suggests that the difference may reflect greater sparsity in the @ graph.
- Initial observations: Users with the same sentiment are more likely to be connected than users with different sentiment.The analysis examines both shared sentiment conditioned on connectedness and connectedness conditioned on labels.
4. MODEL FRAMEWORK
The paper formulates semi-supervised user-level sentiment classification as a factor-graph problem that combines tweet evidence with social-network structure. It supports several network definitions, parameter-estimation strategies, and transductive inference over labeled and unlabeled users.
- Graph formulation: The model represents users and their topic-related tweets in a heterogeneous graph, with tweet edges and network-induced user-user edges.The network edges may encode directed or mutual t-follow and @-mention relationships.
- Graph formulation: User sentiment is assumed to depend only on the labels of the user’s tweets and immediate network neighbors.This Markov assumption yields user-tweet and user-user factors in the factor-graph model.
- Factor design: User-tweet factors encode configurations of user and tweet labels, assigning different confidence weights to initially labeled and unlabeled users.The tweet contribution is normalized by the number of tweets associated with each user.
- Factor design: User-user factors encode label configurations between neighboring users because network connections can correlate with sentiment agreement.The model’s neighbor definition follows directed edges and reflects which users may see another user’s tweets or mentions.
- Learning and inference: Inference uses loopy belief propagation, while parameters are estimated either from labeled-data statistics or with SampleRank in the semi-supervised setting.SampleRank compares label assignments using a labeled-data relative-performance function and the model log-likelihood ratio.
- Learning and inference: The approach assumes positive users post only positive on-topic tweets and negative users only negative on-topic tweets.This sets the user-tweet parameters to 1 when user and tweet labels match and 0 otherwise.
5. EXPERIMENTS
Experiments compare text-only SVM voting with heterogeneous graph models using Twitter relationships, across topics, graph types, parameter-learning strategies, and data conditions. Network information generally improves user-level sentiment classification, although gains depend on graph quality and can reverse for individual users or settings.
- Baseline variation: Combining all on-topic tweets into one document produced much worse baseline results than the majority-vote formulation.The comparison concerns an alternative SVM baseline tried in addition to the main training scheme.
- Case study: The case study shows graph-aware labels more coherently clustered and more often correct than text-only SVM labels.Network links can connect initially misclassified users to others with the same sentiment, helping resolve difficult or ambiguous tweet text.
- Case study: Text alone can be better for prolific users when abundant tweet evidence conflicts with a sparse network neighborhood.User 7 was correctly classified by SVM Vote but not HGM-Learning because the network method forced the user to share a neighbor’s label.
- Average performance: All graph-based approaches outperform the baseline in both Accuracy and Macro F1, although improvements are rather small for @ graphs.NoLearning and Learning perform quite similarly in the average analysis.
- Graph comparisons: t-follow graphs outperform @ graphs, while directed graphs outperform mutual graphs.The directed-versus-mutual difference may reflect approval or attention links, denser graphs, or both; the experiments do not distinguish these explanations.
- Per-topic analysis: Per-topic gains vary: t-follow graphs help most for Obama, Sarah Palin, and Glenn Beck, whereas @ graphs yield more significant improvements for Lakers and Fox News.Graph quality, rather than sparsity or graph size alone, helps explain these topic differences; high-quality graphs can help even when sparse.
- Unlabeled data: HGM-Learning is generally better than SVM Vote and at worst comparable as unlabeled data increases, while HGM-NoLearning degrades more.This pattern suggests learned parameter estimation is more robust to additional unlabeled users.
6. RELATED WORK
Prior work largely addressed sentiment at the tweet, document, or sentence level, while user-level opinion prediction generally relied on users’ text. Graph-based methods, including semi-supervised approaches, provide related foundations for using network structure.
- Twitter sentiment research has primarily focused on tweet-level classification, often ignoring links between users.
- Previous user-level opinion and ideology studies generally used only the text generated by users.
- Graph-based methods have been applied to document- and sentence-level sentiment analysis, including in semi-supervised settings.
- Using @-mentions resembles earlier sentiment-analysis work based on one speaker’s references to another.
7. CONCLUSIONS AND FUTURE WORK
The paper finds that social-network links can improve user-level sentiment analysis, while different network constructions capture attention and homophily with topic-dependent effectiveness. It also identifies sparse graphs, limited labeled data, and model choice as directions for further study.
- 7. CONCLUSIONS AND FUTURE WORK: User-level sentiment analysis significantly improves when social-network link information is incorporated.
- 7. CONCLUSIONS AND FUTURE WORK: Follower/followee versus @ networks and directed versus mutual links represent different mixtures of homophily and attention.
- 7. CONCLUSIONS AND FUTURE WORK: Considering both homophily and attention shows slight evidence of superiority over homophily alone, with exceptions.
- 7. CONCLUSIONS AND FUTURE WORK: Significant gains remain possible with very sparse graphs when connectedness strongly correlates with shared sentiment.
- 7. CONCLUSIONS AND FUTURE WORK: Future work includes larger labeled datasets, other social-media networks, alternative semi-supervised models, and theories of topic-dependent user correlation.
- 7. CONCLUSIONS AND FUTURE WORK: Applying a Markov Random Field formulation may be complicated by graph sparsity.