PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-label Categorization of Accounts of Sexism using a Neural Framework

Pulkit Parikh, Harika Abburi, Pinkesh Badjatiya, Radhika Krishnan, Niyati Chhaya, Manish Gupta, Vasudeva Varma

arXiv:1910.04602v4cs.CL

TL;DR

Prior sexism classification used limited category sets and often assumed categories could not co-occur, despite accounts involving multiple forms. This paper builds a multi-label dataset and neural framework for categorizing such accounts, and reports that its framework outperforms many deep-learning and traditional machine-learning baselines.

  • Problem

    Prior sexism classification used limited category sets and often assumed categories were mutually exclusive, leaving multi-label categorization of accounts as a distinct task.

    Source: §1 · p. 1

  • Method

    The paper proposes a neural framework combining pretrained sentence representations with trainable word-embedding representations and adapts BERT using unlabeled sexism accounts.

    Source: §1 · p. 2, §4.1 · p. 5, §4.3 · p. 6

  • Results

    The proposed framework outperforms many deep-learning and traditional machine-learning baselines for multi-label sexism classification.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Automated categorization of online accounts may help social scientists and policy makers analyze sexism efficiently.

    Source: §1 · p. 1

    Dataset category frequencies do not represent real-world frequencies, in part because of short-post sampling bias and the dataset’s limited size.

    Source: §3 · p. 3

Abstract

from arXiv · show

Sexism, an injustice that subjects women and girls to enormous suffering, manifests in blatant as well as subtle ways. In the wake of growing documentation of experiences of sexism on the web, the automatic categorization of accounts of sexism has the potential to assist social scientists and policy makers in studying and countering sexism better. The existing work on sexism classification, which is different from sexism detection, has certain limitations in terms of the categories of sexism used and/or whether they can co-occur. To the best of our knowledge, this is the first work on the multi-label classification of sexism of any kind(s), and we contribute the largest dataset for sexism categorization. We develop a neural solution for this multi-label classification that can combine sentence representations obtained using models such as BERT with distributional and linguistic word embeddings using a flexible, hierarchical architecture involving recurrent components and optional convolutional ones. Further, we leverage unlabeled accounts of sexism to infuse domain-specific elements into our framework. The best proposed method outperforms several deep learning as well as traditional machine learning baselines by an appreciable margin.

1 Introduction

The paper addresses a gap between detecting sexism and categorizing accounts that may involve multiple types, presenting a multi-label task and a dataset of 13,023 accounts. It proposes a neural framework that combines sentence and word representations for this task.

1 Introduction

  • The dataset contains 13,023 accounts tagged with one or more of 23 categories, while experiments merge categories occurring in fewer than 400 records into 14.The dataset includes accounts by survivors and observers.Source: §1 · pp. 1–2
  • Prior sexism classification generally assumed mutually exclusive categories and used sets of 2–5 categories, while sexism detection did not categorize accounts.The paper frames categorizing accounts involving any type or types of sexism as a distinct multi-label problem.Source: §1 · p. 1
  • The proposed neural framework combines sentence representations from pretrained models with trainable word-embedding representations and linguistic features through hierarchical recurrent and optional convolutional operations.It also adapts BERT using unlabeled accounts of sexism.Source: §1 · p. 2
  • The framework yields results superior to many deep-learning and traditional machine-learning baselines.The paper presents this as a contribution to multi-label categorization of accounts of sexism.Source: §1 · p. 2

2 Related Work

The related work distinguishes sexism detection from categorizing accounts of sexism and identifies prior categorization work as narrower in scope and label coverage.

2 Related Work

  • The review situates the paper alongside work on hate-speech detection and sexism classification, while distinguishing its goal of categorizing accounts of sexism.The paper describes sexism as a specific form of discrimination or hate.Source: §2 · p. 2, §2.1 · p. 2

2.1 Hate Speech Detection

  • Hate-speech studies classify categories such as racist, sexist, or neither, but the paper distinguishes this detection work from categorizing accounts of sexism.The reviewed approaches include character n-grams, deep learning, skipped CNNs, and CNN-GRU combinations.Source: §2.1 · p. 2

2.2 Sexism Categorization

  • Prior sexism categorization addressed language styles, tweets, or sexual-harassment accounts; this paper presents multi-label categorization of accounts involving any types of sexism.The authors also describe their dataset as larger and more fine-grained than earlier work.Source: §2.2 · pp. 2–3

3 Dataset Construction

The authors construct a multi-label sexism dataset from Everyday Sexism Project entries through category formulation and a multi-phase annotation process. They also describe safeguards for privacy and for honoring removals from the source website.

3 Dataset Construction

  • Around 20,000 entries with at least seven words were shortlisted for annotation, with shorter entries prioritized and the website's tag distribution approximated.The framework is described as size-agnostic, with shorter entries selected in view of potential transfer learning to Twitter content.Source: §3 · p. 3
  • A social scientist guided the formulation of 23 categories, informed by gender-related discourse and campaigns and possible public-policy impact.Table 1 provides succinct descriptions of the categories.Source: §3 · p. 3, Table 1 · p. 4
  • Three annotation phases produced 13,023 accounts; experiments merge categories found in fewer than 400 records into 14 categories, while the dataset retains all 23.The accounts were labeled by at least two of 10 annotators; average Cohen’s Kappa across the first two phases was 0.584.Source: §3 · p. 3
  • Dataset category frequencies do not represent real-world prevalence, and labeling may contain inaccuracies or discrepancies despite efforts to improve reliability.The authors also acknowledge that their categorization scheme could be improved.Source: §3 · p. 3

3.1 Ethical Data Use and Release

  • The authors omit entry titles to protect privacy and provide academic requesters numerical placeholders, labels, a retrieval script, and annotation guidelines.The retrieval method is intended to ensure that content removed from the source website is also removed from the dataset.Source: §3.1 · p. 4

4 Sexism Categorization Approach

The framework predicts up to 14 applicable sexism categories per post and combines sentence representations derived from word embeddings with representations from general-purpose models. It also uses unlabeled sexism accounts and explores loss functions for multi-label prediction.

4.1 Sexism Categorization Architecture

  • The architecture flexibly combines word- and sentence-level representations, processes sentence sequences with bi-LSTM and attention, then predicts category probabilities.Configurable concatenation and optional CNN- and LSTM-based operations form sentence representations before post-level aggregation.Source: §4.1 · p. 5

4.2 Word and Sentence Representations

  • The framework explores ELMo, GloVe, and fastText word vectors, linguistic features, and BERT, USE, and InferSent sentence embeddings.The sentence encoders were chosen because their training corpora are considerably larger than the paper’s supervised data and likely contain greater semantic diversity.Source: §4.2 · pp. 5–6

4.3 Utilizing Unlabeled Data

  • Over 90,000 unlabeled Everyday Sexism Project entries tune BERT using masked language modeling and next sentence prediction, producing tBERT.Entries corresponding to validation and test posts are removed before tuning.Source: §4.3 · p. 6

4.4 Loss Function Choice

  • The authors explore alternatives to standard cross-entropy for multi-label prediction, using binary multihot target vectors for both.The alternatives are EBCE and NCE.Source: §4.4 · p. 6, §4.4.1 · p. 6, §4.4.2 · p. 6

4.4.1 Extended Binary Cross Entropy Loss

  • EBCE weights label-wise binary cross-entropy values to counter class imbalance.The target indicates separately whether each label applies to a post.Source: §4.4.1 · p. 6

4.4.2 Normalized Cross Entropy Loss

  • NCE is a normalized cross-entropy variant tailored to multi-label prediction that also mitigates class imbalance.It uses softmax-estimated label probabilities.Source: §4.4.2 · p. 6

4.4.3 Discussion on Single-label Transformations

  • Label Powerset can incur computational cost and cannot predict unseen label combinations, while binary relevance trains one classifier per label and ignores label correlations.These are trade-offs of transforming multi-label classification into single-label tasks.Source: §4.4.3 · p. 6

5 Experiments

The experiments compare traditional machine-learning and neural baselines with configurable neural-framework variants, using multi-label metrics. Several framework variants outperform all baselines, with results also varying by configuration and number of labels per post.

5.1 Evaluation Metrics

  • Results use FI, AccI, Fmacro, and Fmicro; Table 3 reports three-run averages with EBCE, and 15% of the data was reserved for testing and validation each.The NCE loss produced inferior results across multiple methods.Source: §5.1 · p. 7, §5.3 · p. 7, §5 · p. 7

5.2 Baselines

  • Baselines included SVM, Random Forest, and Logistic Regression feature sets, alongside recurrent, hierarchical, sentence-encoder, CNN, and CNN-LSTM neural architectures.Traditional methods used Label Powerset; logistic regression with averaged ELMo features performed best among those combinations.Source: §5.2 · p. 7, §5.3 · p. 7

5.3 Results

  • Several framework variants outperform all baselines; s(wl(ELMo), wl(GloVe), tBERT) is best by FI and Fmacro, and LSTM processing beats its CNN counterpart.Adding linguistic features slightly improves some metrics.Source: §5.3 · p. 8, Table 3 · p. 8
  • tBERT works better than vanilla BERT and other sentence encoders, while combining tBERT sentence representations with attended ELMo word vectors beats either individually.Sentence-level concatenation of ELMo and GloVe also performs better than word-level concatenation in the described configuration.Source: §5.3 · p. 8
  • Attention analysis lists test accounts correctly predicted by the best method but missed by the best baseline, with category labels and category-relevant attended words and sentences.Examples include words associated with role stereotyping, sexual harassment, and hostile work environments.Source: §5.3 · pp. 8–9, Table 4 · p. 9
  • Performance is strongest for posts with 2 to 4 labels; results for 6 and 7 labels are omitted because each group has fewer than 10 test samples.Table 6 reports one run across different numbers of labels per post.Source: §5.3 · p. 9, Table 6 · p. 9

6 Conclusion

The paper studies multi-label classification of sexism accounts with co-occurring categories, reports a neural framework that outperforms many deep-learning and traditional ML baselines, and provides the largest sexism-classification dataset linked with 23 categories.

  • The task classifies accounts reporting any kinds of sexism while allowing categories to co-occur.Source: §6 · p. 9
  • The proposed neural framework outperforms many deep-learning and traditional machine-learning baselines for multi-label sexism classification.Source: §6 · p. 9
  • The paper provides the largest dataset for sexism classification, linked with 23 categories.Source: §6 · p. 9
  • Future work includes improving classification accuracy, enhancing the categorization scheme, and developing other ways to help counter sexism.Source: §6 · p. 9
Loading 1910.04602v4…