Source-linked AI summary
Multilingual Contextual Affective Analysis of LGBT People Portrayals in Wikipedia
Chan Young Park, Xinru Yan, Anjalie Field, Yulia Tsvetkov
TL;DR
Existing manually annotated connotation lexicons were limited to English, despite the importance of contextual affective analysis for understanding how people are portrayed. The paper develops multilingual datasets and classifiers, then finds systematic differences in LGBT portrayals across English, Russian, and Spanish Wikipedia biographies.
Problem
Manually annotated affective lexicons existed only for English, with no in-language evaluation of connotation translatability or extensive analysis in other languages.
Source: §1 · p. 2Method
Results
LGBT biographies show more negative connotations than other pages in Russian, more positive or neutral ones in English, and generally neutral or mixed connotations in Spanish.
Source: §1 · p. 2Takeaways & Limitations
The methodology can help identify articles for further analysis and support investigation of social stereotypes in English, Russian, and Spanish narrative text.
Source: §1 · p. 1, §8 · p. 8The analysis focuses on the LGBT sub-community because data for other groups was scarce, and model outputs are intended to guide manual analysis rather than black-box inference.
Source: §1 · p. 1, §6.2 · p. 5
Abstract
from arXiv · showhide
Specific lexical choices in narrative text reflect both the writer's attitudes towards people in the narrative and influence the audience's reactions. Prior work has examined descriptions of people in English using contextual affective analysis, a natural language processing (NLP) technique that seeks to analyze how people are portrayed along dimensions of power, agency, and sentiment. Our work presents an extension of this methodology to multilingual settings, which is enabled by a new corpus that we collect and a new multilingual model. We additionally show how word connotations differ across languages and cultures, highlighting the difficulty of generalizing existing English datasets and methods. We then demonstrate the usefulness of our method by analyzing Wikipedia biography pages of members of the LGBT community across three languages: English, Russian, and Spanish. Our results show systematic differences in how the LGBT community is portrayed across languages, surfacing cultural differences in narratives and signs of social biases. Practically, this model can be used to identify Wikipedia articles for further manual analysis -- articles that might contain content gaps or an imbalanced representation of particular social groups.
1 Introduction
The paper extends contextual affective analysis to English, Spanish, and Russian, where verb connotations can shape different impressions of the same event. It combines multilingual data and models with an analysis of LGBT biographies on Wikipedia.
1 Introduction
- English, Spanish, and Russian verb choices can imply different agency and sentiment about the same event, potentially giving Wikipedia readers different impressions of Alan Turing’s experience.The English “accepted” suggests low agency, while Spanish “chose” and Russian “preferred” suggest high agency; “preferred” can imply positive sentiment.Source: §1 · p. 1
- Existing manually annotated affective lexicons were English-only, and prior work had not evaluated connotation translatability in-language or extensively analyzed other languages.The paper relates phrasing differences to social norms and perceptions of social roles, and notes that stereotypes on Wikipedia violate its Neutral Point of View policy.Source: §1 · pp. 1–2
- The authors collect connotation-frame annotations in three languages, analyze variation across contexts and languages, and develop multilingual classifiers for power, agency, and sentiment.The method is intended to support identifying potentially biased article sets for further analysis or editing.Source: §1 · pp. 1–2
- LGBT biography pages show more negative connotations than other pages in Russian, more positive or neutral ones in English, and generally neutral or mixed connotations in Spanish.The reported trends align with survey results about perceptions of LGBT people in English-, Spanish-, and Russian-speaking countries.Source: §1 · p. 2
- The work contributes annotated datasets, machine-learning models, and a general methodology for nuanced multilingual analysis of narratives about people.It also examines LGBT portrayals, beyond prior Wikipedia biography work focused primarily on male/female gender bias.Source: §1 · p. 2
Connotation Frames
Connotation frames capture implications of verbs for power, agency, and sentiment. This work extends prior lexicons with annotations in multiple languages and complete contexts.
Connotation Frames
- Connotation frames annotate how verbs imply power, agency, and sentiment for a sentence’s agent and theme.The agent is approximated as the grammatical subject and the theme as the object; the firefighter rescuing the boy illustrates power, active agency, and positive sentiment toward the subject.Source: Connotation Frames · p. 2
- Unlike prior lexicons, the dataset collects connotation frames in languages beyond English and annotates verbs in complete contexts rather than simplified tuples or placeholders.The authors describe these choices as two primary differences from existing lexicons.Source: Connotation Frames · p. 2
- The authors selected frequent transitive verbs and their three most common subject-object-verb contexts per language, requiring at least one human subject or object.They used News Crawl tuples and a person-noun list to select contexts for annotation.Source: Connotation Frames · p. 2
- For English, Russian, and Spanish, the study collected power, agency, and sentiment judgments for 300 verbs in three contexts each, with three annotators per instance.The resulting annotation total was 32,400 judgments.Source: Connotation Frames · p. 3
- Annotation agreement was modest, with Krippendorff’s alpha averaging 0.22 for English, 0.31 for Russian, and 0.26 for Spanish.The authors report that neutral-versus-nonneutral labels were the most common disagreement; annotations were mapped to scores and ternarized using thresholds.Source: Connotation Frames · p. 3
3 Crowdsourced Connotation Analysis
The analysis finds that both context and in-language annotation matter: verb-level and translated connotations often differ from context-level and original-language annotations.
3 Crowdsourced Connotation Analysis
- The translation comparison covers 125 Russian and 135 Spanish verbs that overlap with English annotations.The table assesses information lost when using translated rather than in-language annotations.Source: Table 2 · p. 3
- Ignoring context can produce an accuracy drop of more than 30%, and this pattern extends prior English sentiment findings to power and agency and to Spanish and Russian.The analysis compares a verb-level score averaged across contexts with scores averaged within each context.Source: §3 · p. 3
- Translated connotation scores are substantially less similar to in-language scores than the decontextualized comparisons, indicating information loss from translation.The comparison uses overlapping verbs after inaccurate or questionable translation pairs are discarded.Source: §3 · p. 3
4 Classification of Connotations
Because annotated contexts cover only a small subset of verb uses, the paper trains a classifier to predict connotation labels for verbs in context. It uses contextual multilingual representations to apply those predictions to target-language corpora.
4 Classification of Connotations
- Table 3 reports better results when training and test languages match than when they differ.Its caption describes macro F1 scores across source and target languages and connotation attributes.Source: Table 3 · p. 4
- The classifier predicts connotation-frame labels for unseen verbs in context, addressing the limited coverage of the 900 annotated contexts.It is trained on contextualized multilingual annotations and then used to predict labels for context-provided verbs in a target-language corpus.Source: §4 · p. 4
- The model uses XLM’s last hidden layer as multilingual verb-in-context representations and feeds them to a sample-weighted logistic regression classifier.The classifier is trained directly on contextual embeddings, unlike earlier models that used noncontextual or decontextualized embeddings.Source: §4 · p. 4
5 Connotation Classification Evaluation
Evaluation finds that combining in-language and out-of-language data can improve classification over non-augmented models. Language transfer alone and machine translation perform worse than in-language training.
5 Connotation Classification Evaluation
- Table 4’s machine-translation approach has lower macro F1 than the cross-lingual model reported in Table 3.The table caption states the comparison but supplies no specific score values.Source: Table 4 · p. 4
- Matching the training and test languages performs better; for Spanish power, English training lowers F1 by 19 points (44%) versus Spanish training.The authors conclude that these results support the importance of in-language connotations.Source: §5 · p. 4
- Machine translation performs strictly worse than training on in-language data, so it cannot replace in-language data for this evaluation.The experiment translates Spanish and Russian test sentences into English and applies a model trained on English annotations.Source: §5 · p. 4
- The best augmented models outperform non-augmented models for all connotation dimensions.The paper uses these best-performing models to analyze LGBT biography pages in §6.Source: §5 · p. 4
6 Case Study: Multilingual Affective Analysis
The case study uses LGBTBio, a multilingual corpus of LGBT biographies and matched controls, to examine portrayals across English, Russian, and Spanish. It finds language-specific differences and explores variation by nationality, birth year, and occupation, while treating model outputs as leads for manual investigation.
6.1 LGBTBio Corpus
- LGBTBio contains 1,340 LGBT biographies, each matched to a non-LGBT control to account for overall language differences when comparing portrayals.Controls were matched on similar characteristics in aggregate, rather than as perfect individual pairs.Source: §6.1 · p. 5
6.2 Contextual Affective Analysis of Narratives Describing LGBT People
- The model is intended to surface trends and examples for manual review, not to support black-box inferences; findings may not generalize beyond the sampled articles or collection period.Wikipedia articles changed during the study.Source: §6.2 · p. 5
- English biographies portray LGBT people more positively than controls, while Russian portrayals are significantly negative across affective dimensions and Spanish results are negative but mixed in significance.The analysis covers the same people across languages; Spanish agency is the only dimension significantly negative.Source: Figure 2 · p. 5, §6.2 · p. 6
- Across American and non-American subjects, portrayals are nearly identical, providing no evidence of the hypothesized local bias in the corpus.The authors avoid finer nationality subdivisions because the resulting groups would be small and potentially misleading.Source: §6.2 · p. 6, Figure 3 · p. 6
- English portrayals are neutral or insignificantly negative for people born before 1900, then significantly positive for those born 1900–1960; Russian and Spanish trends do not significantly change by birth year.The authors describe this as some evidence that changing global perceptions appear in English Wikipedia biographies, while noting that not all results are significant.Source: §6.2 · p. 6, Figure 4 · p. 7
- Occupation patterns differ by language: Spanish entertainers have significantly negative connotations, while Russian non-entertainers do, and English shows little occupational difference.The analysis focuses on the two most common occupations to maintain sufficient sample sizes.Source: §6.2 · p. 7, Figure 5 · p. 7
- Article-level examples show that language editions can differ in included information and phrasing, with model scores reflecting more negative portrayals in specific Russian and Spanish passages.Examples include Cleve Jones, Hugh Walpole, and Audrey Tang; the authors contrast Russian or Spanish descriptions with English versions.Source: §6.2 · pp. 7–8
7 Related Work
Prior work developed contextual affective analysis and applied it to portrayals across media, while multilingual connotation evaluation and automated analysis of LGBTQIA+ bias remain less developed.
7 Related Work
- Rashkin, Singh, and Choi (2016), Sap et al. (2017), and Field, Bhat, and Tsvetkov (2019) successively developed sentiment connotation frames, power and agency frames, and the CAA framework.Connotation frames have been applied to films, newspapers, and online stories.Source: §7 · p. 8
- Rashkin et al. (2017) extended connotation frames to other languages using mapped embeddings, but did not evaluate against in-language annotations or provide multilingual annotations.Source: §7 · p. 8
- Research has documented cross-cultural differences in social roles and cultural biases in Wikipedia biographies, alongside gender differences in biography portrayals across languages.The cited studies include Dong et al. (2019), Callahan and Herring (2011), and Wagner et al. (2015).Source: §7 · p. 8
- Automated analysis has examined LGBTQIA+ bias only to a limited extent, with Smith et al. (2015) studying gender, race, and LGBT portrayals in 700 popular films.Source: §7 · p. 8
8 Conclusion
The paper extends affective analysis to English, Russian, and Spanish narrative text through new methodology and datasets. The framework can support analyses of social stereotypes and offers directions for extending the model, languages, and data.
8 Conclusion
- The methodology and datasets extend affective analysis to English, Russian, and Spanish narrative text.Source: §8 · p. 8
- Used alongside other methods, the framework can help writers seek a neutral point of view and provide insight into social stereotypes.Source: §8 · p. 8
- Future work could improve the multilingual model, add languages, examine Wikipedia edit histories, and incorporate more connotations and linguistic databases.Source: §8 · p. 8
A Multilingual Annotation Collection
The annotation collection used language-specific recruitment and quality controls, with task design adapted to challenges such as defining agency and obtaining annotations in each target language.
A Multilingual Annotation Collection
- Crowdsourced through Figure 8, now Appen, the study dropped Chinese and French after receiving almost no annotations and substituted Spanish.English and Chinese underwent in-house pilot studies; task instructions were translated and checked by native speakers.Source: A Multilingual Annotation Collection · p. 9
- Because agency was difficult to define and lacked a single-word translation in Russian, Chinese, and French, annotators received three priming questions.Source: A Multilingual Annotation Collection · p. 9
- Annotators had to pass an eight-question quiz at 70% accuracy and maintain that score on ten interspersed questions to continue.The quiz questions checked that annotators read and understood the instructions.Source: A Multilingual Annotation Collection · p. 9
- Tasks were released in batches of 100–200 examples, and the Chrome translation plugin was disabled for non-English tasks.Smaller batches allowed the researchers to block annotators who failed earlier quizzes.Source: A Multilingual Annotation Collection · p. 9
- English annotators were from the United States, Russian annotators from Russia, and Spanish annotators from nine South American countries; agency pay was higher because its tasks included three extra priming questions.Table 6 reports task pay rates per five instances.Source: A Multilingual Annotation Collection · p. 9, Table 6 · p. 9
- After removing annotators whose disagreement exceeded one standard deviation above the mean, the final dataset retained instances with at least two judgments.Tables 7 and 8 report post-processing agreement scores and annotated-instance counts, respectively.Source: A Multilingual Annotation Collection · p. 9, Table 7 · p. 9, Table 8 · p. 9
B Model Details
The model details describe logistic-regression classifier size and a validation-based class-weight search, then compare the XLM-based model with prior systems on shared data.
B Model Details
- Each logistic-regression classification model has 3,075 parameters, and class weights were selected by grid search using validation-set F1.The trained models and hyperparameter configuration are released with the codebase.Source: B Model Details · p. 9
- The XLM-based model achieves performance comparable to prior work when evaluated on the same data as Rashkin, Singh, and Choi (2016) and Field, Bhat, and Tsvetkov (2019), with uncontextualized annotations and decontextualized XLM embeddings.Source: B Model Details · p. 9
- The XLM-based model performs similarly to Rashkin, Singh, and Choi (2016), but slightly worse than Field, Bhat, and Tsvetkov (2019), a drop described as the cost of language-agnostic modeling.Table 9 reports classification results in macro F1, including a majority baseline.Source: B Model Details · pp. 9–10, Table 9 · p. 10
C LGBTBio Corpus Construction
LGBTBio includes multilingual LGBT biographies and matched controls, with eligibility filters applied across English, Russian, and Spanish.
C LGBTBio Corpus Construction
- The corpus draws on Wikipedia lists and Wikidata LGBT-related properties, retaining people with pages in all three languages and at least three analyzable sentences per language.Analyzable sentences contain the person's name or pronoun as subject or object; pronouns were inferred from the more frequent of “he” or “she” in the article.Source: C LGBTBio Corpus Construction · p. 10
- 1,340 biography pages remain in the LGBT half of LGBTBio after filtering.Source: C LGBTBio Corpus Construction · p. 10
- Control biographies are matched using TF-IDF vectors of page categories, with pivot-slope correction tuned so controls and LGBT biographies have the same average number of non-LGBT-specific categories.The pivot is fixed at 0.1 after testing values from 0 to 0.5 in 0.1 increments.Source: C LGBTBio Corpus Construction · p. 10