Source-linked AI summary
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora
William L. Hamilton, Kevin Clark, Jure Leskovec, Dan Jurafsky
TL;DR
Domain-general sentiment lexicons can miss how word polarity varies across domains, while manually constructing domain-specific resources is costly. The paper introduces SENTPROP, combining domain-specific embeddings with label propagation from small seed sets, and applies it to historical English and Reddit communities. The resulting studies show strong social and historical variation, while the method achieves state-of-the-art performance competitive with hand-curated approaches.
Problem
Sentiment depends on domain, but generic lexicons may be inaccurate for domain-specific texts and manual construction is expensive.
Method
SENTPROP combines domain-specific word embeddings with label propagation to induce lexicons from small seed sets and provides bootstrap-based confidence scores.
Results
SENTPROP achieves state-of-the-art performance competitive with hand-curated methods and supports studies of sentiment across 150 years of English and 250 Reddit communities.
Takeaways & Limitations
Sentiment lexicons should account for social and historical context, which the released domain-specific lexicons and toolkit support studying.
Takeaways & Limitations
The induced lexicons are not perfect, with uncertainty reflected in bootstrap-sampled estimates.
Abstract
from arXiv · showhide
A word's sentiment depends on the domain in which it is used. Computational social science research thus requires sentiment lexicons that are specific to the domains being studied. We combine domain-specific word embeddings with a label propagation framework to induce accurate domain-specific sentiment lexicons using small sets of seed words, achieving state-of-the-art performance competitive with approaches that rely on hand-curated resources. Using our framework we perform two large-scale empirical studies to quantify the extent to which sentiment varies across time and between communities. We induce and release historical sentiment lexicons for 150 years of English and community-specific sentiment lexicons for 250 online communities from the social media forum Reddit. The historical lexicons show that more than 5% of sentiment-bearing (non-neutral) English words completely switched polarity during the last 150 years, and the community-specific lexicons highlight how sentiment varies drastically between different communities.
1 Introduction
Sentiment varies with social and historical context, making domain-specific lexicons important for computational social science. SENTPROP addresses this need by inducing lexicons from small seed sets and domain-specific corpora, with confidence estimates.
- Motivation: Lexical sentiment can differ across communities and change over time, as illustrated by soft and terrific.soft has different sentiment in sports and toy-animal communities, while terrific shifted from negative to positive historically.
- Motivation: Generic lexicons can misassign sentiment in financial, historical, or tightly knit social-media domains.Such assignments may neglect genre, community vernacular, and demographic variation, introducing harmful biases.
- Approach: SENTPROP combines label propagation with domain-specific word embeddings to learn accurate lexicons from small seed sets and modestly sized corpora.The framework is designed for corpora of approximately 10^7 tokens and provides confidence scores.
- Contributions: The work contributes a bootstrap-sampling framework for confidence scores and two large-scale studies of social and historical sentiment variation.The studies cover 250 Reddit communities and 150 years of English.
- Resources: The authors release inferred lexicons, SENTPROP code, and a benchmark toolkit through the SOCIALSENT package.The package includes implementations of previously published induction algorithms.
2 Related work
Prior sentiment-lexicon research uses corpus patterns, distant supervision, or hand-curated dictionaries, but these approaches have different domain and quality constraints. This work emphasizes accurate sentiment induction for specific social contexts using improved embeddings and propagation.
- Corpus-based approaches: Corpus-based methods induce lexicons from seed words and unlabeled-corpus patterns, but syntactic patterns can be brittle in non-standard social media.Other corpus-based methods rely on general co-occurrence or distant-supervision signals.
- Dictionary-based approaches: Dictionary-based methods propagate sentiment through hand-curated resources such as WordNet, but such resources are unavailable for many domains.Historical texts are given as a key example of a domain lacking an applicable dictionary resource.
- Research goal: Most previous work prioritizes recall and enlarging existing lexicons, whereas this work targets high-quality lexicons accurate to a specific social context.The contrast reflects different goals: massive polarity coverage versus domain-specific accuracy.
- Relation to prior work: SENTPROP extends earlier propagation work with high-quality embeddings, a new graph construction method, an alternative propagation algorithm, and bootstrapped confidence values.The authors attribute particular importance to the high-quality word vectors.
3 Framework
SENTPROP induces sentiment lexicons by building lexical graphs from unlabeled-corpus embeddings and propagating seed labels, with bootstrap sampling providing confidence estimates. Its SVD-based embeddings and graph-based propagation support domain-specific sentiment scoring.
- Framework overview: SENTPROP combines high-quality word embeddings, lexical-graph construction, label propagation, and bootstrap sampling to induce interpretable and confidence-aware sentiment scores.The framework is resource-light, uses small seed sets, and does not rely on domain-specific signals.
- Lexical graph construction: Lexical graphs connect each word to its nearest k semantic neighbors using embeddings learned from unlabeled corpora.Edge weights are based on cosine similarity in the semantic space.
- Distributional word embeddings: The SVD-based embedding method significantly outperformed word2vec and GloVe on preliminary experiments with the domain-specific data.The method constructs smoothed positive pointwise mutual-information representations and applies truncated singular value decomposition.
- Polarity propagation: Random-walk propagation assigns polarity according to the probability of reaching a word from positive and negative seed sets.The algorithm iteratively updates scores, balancing local neighbor consistency against global consistency with seed labels through β.
- Confidence estimation: Bootstrap propagation over random subsets of seed words produces standard deviations that quantify confidence in the resulting polarity scores.This addresses the influence of seed selection and possible corpus artifacts correlated with seed words.
4 Recreating known lexicons
SENTPROP is evaluated against established sentiment lexicons and competing methods across standard English, Twitter, and Finance. It performs especially strongly with domain-specific embeddings while remaining effective on smaller corpora.
- Evaluation results: Corpus-based methods outperform WordNet on standard English when using word-vector embeddings learned from a massive corpus of 10^11 tokens.Both corpus-based methods outperform the WordNet-based approach overall.
- Evaluation results: SENTPROP significantly outperforms other methods when using domain-specific finance embeddings trained on ∼2×10^7 tokens.The finance lexicon contains binary sentiment labels and is used as the domain-specific evaluation target.
- Evaluation results: SENTPROP maintains strong performance on standard English with a 1000x reduction in corpus size using COHA embeddings.The embeddings were learned from the 2000s decade of COHA, containing only 2 × ∼10^7 tokens.
- Evaluation results: SENTPROP significantly outperforms baselines on all evaluation tasks and is competitive with Sentiment140 on Twitter.It also outperforms a WordNet-based state-of-the-art approach on standard English.
- Evaluation results: SENTPROP combines high-quality word embeddings with label propagation to achieve state-of-the-art performance, including competitiveness with hand-curated lexical graphs.Its accuracy remains high with modest-sized domain-specific corpora, whereas DENSIFIER performs extremely well with very large corpora and abundant training data.
- Evaluation results: Embedding choice strongly affects performance: replacing SVD-based embeddings with default word2vec SGNS vectors caused a >40% SENTPROP performance drop across all measures.DENSIFIER experienced a >10% drop in the same preliminary COHA experiments.
5 Inducing community-specific lexicons
The paper induces sentiment lexicons for the 250 largest Reddit communities to measure how word sentiment varies across social contexts. The resulting lexicons reveal opposing word polarities and broad differences between community pairs.
- 5.1 Examining the lexicons: Sentiment lexicons were induced for the top-250 Reddit subreddits using all 2014 comment data.Sentiment was induced for the top-5000 non-stop words in each subreddit by comment frequency.
- 5.1 Examining the lexicons: Words such as soft and animal are positive in r/TwoX but negative in r/sports, while crazy and insane show the opposite pattern.The communities represent female perspectives and gender issues versus sports, respectively.
- 5.1 Examining the lexicons: Across 1000 randomly selected community pairs, sentiment correlations are noticeably skewed toward highly uncorrelated values.Each pair overlapped in at least half of its top-5000-word vocabulary before sentiment correlations were computed.
- 5.1 Examining the lexicons: Sentiment correlation is higher for the conflicting communities r/TwoX and r/TheRedPill (τ25% = 0.58) than for the unrelated pair r/TwoX and r/sports (τ25% = 0.41).The comparison suggests that conflicting communities may use language with more similar sentiment than entirely unrelated communities.
- 5.1 Examining the lexicons: Community pairs range from very similar sentiment, such as r/sports and r/hockey, to drastically different sentiment, such as r/sports and r/TwoX.This variation forms a long tail of communities with highly different word sentiments.
6 Inducing diachronic sentiment lexicons
The paper uses COHA to induce historical sentiment lexicons and examine polarity change across 150 years. Many words changed sentiment, including examples of both amelioration and pejoration.
- 6.1 Examining the lexicons: >5% of sentiment-bearing words completely switched polarity during the 150-year period.>25% of all words changed their sentiment label when switches to or from neutral were included.
- 6.1 Examining the lexicons: Historical lexicons were constructed from the genre-balanced COHA corpus for adjectives and top-5000 non-stop words.The adjective lexicons included words with counts above 100 in each decade.
- 6.1 Examining the lexicons: The prevalence of full polarity switches underscores the importance of historical sentiment lexicons for studying diachronic linguistics and cultural change.The paper positions these lexicons as resources for NLP analyses of historical text, digital humanities, and history.
- 6.1 Examining the lexicons: Lean illustrates amelioration, shifting from negative associations with weakness toward positive associations with muscularity and fitness.Historical word-vector similarities were used to contextualize this shift.
- 6.1 Examining the lexicons: Pathetic illustrates pejoration, shifting from associations with passionate toward stronger negative associations with weakness and inadequacy.The shift is contextualized using semantic similarities from learned historical word vectors.
- 6.1 Examining the lexicons: Other captured changes include sorry shifting toward a neutral discourse marker and worldly toward an association with sophistication.Earlier uses linked sorry with serious negative states and worldly with materialism and religious impurity.
7 Conclusion
SENTPROP provides robust, uncertainty-aware domain-specific sentiment lexicons and supports large-scale analyses showing that sentiment depends strongly on social and historical context. Methodological comparisons find performance competitive with hand-curated dictionary approaches, while the authors acknowledge that induced lexicons remain imperfect.
- SENTPROP enables researchers to induce robust, domain-relevant sentiment lexicons from their study domains.The conclusion emphasizes their relevance to computational social science research.
- 1850-1880 and 1970-2000 are the historical periods compared in the conclusion’s reported analysis.
- Sentiment depends strongly on both social and historical context.The paper’s two large-scale studies provide evidence for this conclusion.
- State-of-the-art performance is competitive with methods relying on hand-curated dictionaries.The comparison attributes this result to combining label propagation with high-quality word vector embeddings.
- SENTPROP lexicons are not perfect, with uncertainty reflected in bootstrap-sampled estimates.The authors present quantified uncertainty as part of interpreting the induced sentiment values.