Source-linked AI summary
MoralStrength: Exploiting a Moral Lexicon and Embedding Similarity for Moral Foundations Prediction
Oscar Araque, Lorenzo Gatti, Kyriaki Kalimeri
TL;DR
The MFD is a key resource for analyzing moral values in text, but its vocabulary is limited and it does not measure moral strength. This paper expands it into MoralStrength and evaluates prediction models, reporting improved performance over the state of the art across moral dimensions.
Problem
The MFD is an essential NLP resource, but has few lemmas, includes rarely used radical terms, and lacks ratings of moral strength.
Source: §1 · p. 2Method
The authors expand the MFD using WordNet, add normative moral-valence ratings, and evaluate three models using lemma features and embedding-based semantic similarity.
Source: §1 · p. 2Results
All models improve prediction performance over the current state of the art for all moral dimensions, with prominent approaches combining textual and lexicon-based representations.
Source: §5 · p. 24Takeaways & Limitations
The authors conclude that moral lexicons can support moral-values classification from text and propose the framework for analyzing large-scale user-generated content and potentially nowcasting attitudes.
Source: §5 · p. 24Direct comparison with Rezapour et al. is not possible because the datasets and evaluation metrics differ.
Source: §4.2 · p. 15
Abstract
from arXiv · showhide
Moral rhetoric plays a fundamental role in how we perceive and interpret the information we receive, greatly influencing our decision-making process. Especially when it comes to controversial social and political issues, our opinions and attitudes are hardly ever based on evidence alone. The Moral Foundations Dictionary (MFD) was developed to operationalize moral values in the text. In this study, we present MoralStrength, a lexicon of approximately 1,000 lemmas, obtained as an extension of the Moral Foundations Dictionary, based on WordNet synsets. Moreover, for each lemma it provides with a crowdsourced numeric assessment of Moral Valence, indicating the strength with which a lemma is expressing the specific value. We evaluated the predictive potentials of this moral lexicon, defining three utilization approaches of increased complexity, ranging from lemmas' statistical properties to a deep learning approach of word embeddings based on semantic similarity. Logistic regression models trained on the features extracted from MoralStrength, significantly outperformed the current state-of-the-art, reaching an F1-score of 87.6% over the previous 62.4% (p-value<0.01), and an average F1-Score of 86.25% over six different datasets. Such findings pave the way for further research, allowing for an in-depth understanding of moral narratives in text for a wide range of social issues.
1. Introduction
The introduction identifies limitations in the Moral Foundations Dictionary and presents MoralStrength as an expanded lexicon with moral-valence ratings. It evaluates three increasingly complex prediction approaches and reports performance above the current state of the art.
1. Introduction
- The MFD has few lemmas, includes rarely used radical terms, and assigns vice or virtue without indicating moral strength.These limitations motivate expanding its vocabulary and adding normative ratings.Source: §1 · p. 2
- MoralStrength extends the MFD using WordNet and provides normative ratings, while the paper describes it as offering approximately three times more lemmas.The lexicon is intended to address the MFD's limitations in vocabulary and binary moral classification.Source: §1 · p. 2
- The framework tests three increasingly complex models, from lemma-frequency and statistical features to embedding-based semantic similarity, on seven MFTC Twitter datasets.The corpus contains approximately 35,000 tweets annotated according to MFT foundations.Source: §1 · p. 2
- The authors report that MoralStrength-based approaches outperform current state-of-the-art methods for predicting morality.They frame the results as supporting deeper analysis of moral judgments and attitudes in spontaneous digital data.Source: §1 · pp. 2–3
- The authors propose the resource and framework for analyzing large-scale user-generated communication and potentially nowcasting attitudes on controversial phenomena.They also describe these analyses as potentially informing communication campaigns that appeal to people's values.Source: §1 · p. 3
2. Related Literature
Prior work on moral values in text includes dictionary-based analysis, topic and semantic methods, and neural approaches. This study positions MoralStrength as an expanded MFD with moral-valence ratings, evaluated through feature-engineering and embedding-similarity models.
2. Related Literature
- Earlier dictionary-based work used the MFD for cultural comparisons, political-debate analysis, and tweet classification; Teernstra et al. found pure machine learning preferable to dictionary features under similar accuracy with fewer assumptions.The present study follows a similar approach while expanding the MFD and adding lemma-level moral valence.Source: §2 · pp. 3–4
- Topic and semantic approaches used MFD seeds with LDA or LSA, whereas this study uses pre-trained word embeddings to represent semantic similarity.The authors note that these embeddings encode language regularities but lack domain-specific knowledge.Source: §2 · p. 4
- Related approaches include DDR for moral sentiment around topics and an LSTM combined with MFD category percentages [9]; the present study uses embeddings to compute word similarity instead of feeding them directly to a neural network.The authors also cite expanded-MFD work that improved morality prediction accuracy.Source: §2 · pp. 4–5
- The study's central contribution is an expanded moral lexicon with moral-valence ratings, evaluated through approaches ranging from feature engineering to embedding-based semantic similarity.The authors distinguish their approach from methods that feed word embeddings directly into a neural network [9].Source: §2 · p. 5
3. Materials and Methods
The methods extend the Moral Foundations Dictionary (MFD), add crowdsourced moral-valence ratings, and evaluate three lexicon-based feature extractors on moral-foundation prediction. Experiments use seven MFTC datasets and logistic regression, with comparisons to unigram and original-MFD baselines.
3.1. Expansion of The Moral Foundations Dictionary (MFD)
- 1,148 words were classified as 520 virtues, 476 vices, and 152 general morality words after expanding the MFD.The expansion began with WordNet synsets and manually removed matches that did not relate to a moral trait.Source: §3.1 · p. 6
3.2. Moral Valence Annotation
- Crowdsourced ratings assign each lemma a moral-valence score on a 9-point scale, with the midpoint representing neutrality toward the moral dimension.At least five annotators rated each lemma; valence and arousal ratings were also collected.Source: §3.2 · pp. 6–7
3.3. Moral Lexicon Approaches
- The three feature extractors are Moral Freq, Moral Stats, and SIMON; only SIMON ignores the lexicon’s numeric moral values.Moral Freq uses thresholded normalized counts, Moral Stats summarizes valence distributions, and SIMON projects text using embedding similarity to lexicon words.Source: §3.3 · pp. 8–9
3.4. Data Collection and Preprocessing
- The evaluation uses seven MFTC Twitter datasets with expert moral-foundation labels, spanning political and nonpolitical topics.Only 82% of the original tweets were recovered; text was normalized, and retweets were removed from the Hurricane Sandy data.Source: §3.4 · pp. 9–10
3.5. Experimental Design
- Models are evaluated with 10-fold cross-validation using F1-score, with unigram and original-MFD frequency models as baselines.For comparison with Lin et al., the study reports both oversampling and undersampling results to assess sensitivity to class-imbalance handling.Source: §3.5 · pp. 10–12
4. Results and Discussion
Crowdsourced MoralStrength valence ratings showed agreement with other annotators, an external normative lexicon, and the MFD. Across datasets, moral-strength features improved prediction over unigram baselines, while the best model varied with the evaluation setting.
4.1. Evaluation of Moral Valence
- Valence AC2 ranged from 0.61 to 0.72, correlations with Warriner et al. from 0.79 to 0.95, and binarized MFD agreement reached 0.78 for Authority and 0.84 for fairness.Fairness initially had poor inter-annotator agreement (0.17), leading the authors to remove five annotators; aggregate fairness agreement with MFD was 0.84.Source: §4.1 · pp. 12–13, Table 3 · p. 13
4.2. Evaluation of MoralStrength Lexicon
- Across datasets, results improved when MoralStrength was added to unigram models; unigram + Moral Freq ranked best overall, while SIMON with MoralStrength led the direct comparison with MFD.The direct comparison averages F1 across datasets; its Friedman ranking favored SIMON with the proposed lexicon.Source: §4.2 · p. 15, Table 11 · p. 23, Table 12 · p. 23
- Combining unigrams and SIMON did not generally improve results, and the authors hypothesize that large feature vectors can lead to overfitting.The combination primarily improved metrics under over-sampling; increasing training data also improved its results.Source: §4.2 · p. 15
- The Baltimore Protest comparison with Rezapour et al. is not direct because the datasets and reported metrics differ.The paper also notes that performance varies with discourse topic and the amount of training data available for each trait.Source: §4.2 · pp. 15–16
- 87.6 average F1-score versus 62.4 reported by Lin et al. was a statistically significant improvement for unigram + SIMON on Hurricane Sandy with over-sampling.The approach was also consistently higher than the unigram model across moral dimensions.Source: §4.2 · p. 14, Table 4 · p. 16
- 85.0% average F1-score with under-sampling was below the 87.6% with over-sampling on Hurricane Sandy, although the best-performing models were consistent across sampling approaches.Under-sampling was described as preferable for avoiding overfitting to the most prevalent outcome.Source: §4.2 · p. 14, Table 5 · p. 17
5. Conclusions
MoralStrength extends moral-text analysis with a larger, graded lexicon. Its evaluation finds improved moral prediction when textual and lexicon-based representations are combined.
5. Conclusions
- MoralStrength contains approximately five times more lemmas than the Moral Foundations Dictionary and provides a moral-valence metric for each lemma.The resource also includes less radical and more frequently used lemmas.Source: §5 · p. 24
- The framework evaluates normalized lemma frequencies, statistical features, and embedding-based semantic similarity on six Twitter datasets annotated for moral content.Source: §5 · p. 24
- All evaluated models improve on current state-of-the-art across moral dimensions, with the most prominent approaches combining textual and lexicon-based representations.The evaluation covered six benchmark datasets from Twitter.Source: §5 · p. 24
- Future work could encode domain knowledge in word vectors, add moral-valence or sentiment features, and examine moral narratives across social issues and opinion change.Source: §5 · p. 24