Source-linked AI summary
Aggression-annotated Corpus of Hindi-English Code-mixed Data
Ritesh Kumar, Aishwarya N. Reganti, Akshit Bhatia, Tushar Maheshwari
TL;DR
Online aggression has expanded with web interaction, creating a need for safeguards and more nuanced automated recognition. The paper develops a hierarchical tagset and Hindi-English code-mixed corpus from Twitter and Facebook, with approximately 18k tweets and 21k Facebook comments released for further research.
Problem
Online aggression and related behaviors have grown in reach and impact, while automated recognition must handle overt, covert, ratified, and unratified behavior beyond dictionary lookup.
Method
The paper develops a hierarchical aggression annotation scheme with 3 top-level tags and 10 discursive-effect categories for Hindi-English code-mixed social-media data.
Results
The final dataset contains approximately 18k tweets and 21k Facebook comments, with top-level inter-annotator agreement slightly above 72% after guideline changes.
Takeaways & Limitations
The released corpus is intended to support research on understanding and automatically identifying aggression, trolling, and cyberbullying on social media.
Takeaways & Limitations
Initial automatic identification reached only an F1 score barely above 0.70 for top-level aggression classification, indicating that even basic classification remains complex.
Abstract
from arXiv · showhide
As the interaction over the web has increased, incidents of aggression and related events like trolling, cyberbullying, flaming, hate speech, etc. too have increased manifold across the globe. While most of these behaviour like bullying or hate speech have predated the Internet, the reach and extent of the Internet has given these an unprecedented power and influence to affect the lives of billions of people. So it is of utmost significance and importance that some preventive measures be taken to provide safeguard to the people using the web such that the web remains a viable medium of communication and connection, in general. In this paper, we discuss the development of an aggression tagset and an annotated corpus of Hindi-English code-mixed data from two of the most popular social networking and social media platforms in India, Twitter and Facebook. The corpus is annotated using a hierarchical tagset of 3 top-level tags and 10 level 2 tags. The final dataset contains approximately 18k tweets and 21k facebook comments and is being released for further research in the field.
1. Introduction
Rapid growth in user-generated web activity has expanded the scale and impact of online aggression, making manual moderation impractical and motivating nuanced automated detection.
- 1. Introduction: 25% more tweets and 22% more Facebook posts per minute were reported over three years, alongside hundreds of millions of daily online messages and posts.The cited estimates include approximately 500 million tweets and 4.3 billion Facebook messages per day.
- 1. Introduction: Online aggression, trolling, cyberbullying, flaming, and hate speech have increased as web interaction has expanded and can affect billions of people.The paper links these incidents with psychological harm and users deactivating their accounts.
- 1. Introduction: Manual monitoring and moderation have become almost completely impractical and ineffective because online data is produced at high volume and pace.The paper therefore calls for automatic or semi-automatic recognition and intervention.
- 1. Introduction: Useful automated systems must distinguish overt from covert aggression and ratified from unratified aggressive behavior rather than rely on traditional dictionary lookup.The paper emphasizes intelligent and nuanced recognition for large-scale cases.
2. Verbal Aggression
The paper frames verbal aggression as behavior that damages a target’s social identity and focuses on understanding its structure, forms, and relationships with related phenomena.
- 2. Verbal Aggression: Verbal aggression is linguistic behavior intended to damage a target’s social identity and lower their status and prestige.It is described as behavior that upsets social equilibrium.
- 2. Verbal Aggression: The paper focuses on unratified aggression and stresses distinguishing it from ratified behavior so detection systems flag the most serious cases.This distinction is presented as important for building automatic aggression-detection systems.
- 2. Verbal Aggression: Prior computational work addressed trolling, cyberbullying, flaming, insults, and abusive language, but Indian-specific theoretical insight and cross-phenomenon understanding remained limited.The paper identifies a gap in understanding how these related behaviors overlap and form.
3. Data Collection
The corpus was collected from public Facebook Pages and Twitter content centered on issues expected to attract discussion among Indians, especially in Hindi, before language filtering.
- 3. Data Collection: Public Facebook Pages and Twitter supplied data focused on issues expected to be discussed among Indians and in Hindi.The collection targeted social-media discussions rather than language-sampled content.
- 3. Data Collection: More than 40 Facebook Pages were crawled across news organizations, web portals, political groups, student organizations, and university-incident groups.Examples include NDTV, BJP, SFI, and groups surrounding incidents at HCU and JNU.
- 3. Data Collection: Twitter data was collected using popular hashtags about contentious themes including beef bans, India–Pakistan cricket, elections, and movie opinions.The hashtags were selected around topics expected to generate discussion.
- 3. Data Collection: Collection was not language-based, so English, Hindi, and other Indian-language data were initially included before handling non-Hindi and non-code-mixed material.The final processing stage addressed languages other than Hindi and Hindi-English code-mixing.
4. Aggression Typology
The paper organizes aggression by expression, target, and related distinctions, with the typology supporting an annotation scheme that captures overt and covert forms and potential physical threats.
- 4. Aggression Typology: Verbal aggression is divided into basic types according to how it is expressed and into four types according to its target.The supplied typology passages enumerate target categories including sexual, identity, geographical, and non-threatening aggression.
- 4. Aggression Typology: The typology distinguishes aggression from abuse and aggression analysis from sentiment analysis, with these distinctions informing the annotation scheme.The paper states that the categories and distinctions are discussed as part of the scheme’s development.
- 4. Aggression Typology: Overt aggression is expressed through aggressive lexical items, lexical features, or syntactic structures.The paper provides a Hindi-English example illustrating overt expression.
- 4. Aggression Typology: Covert aggression is not overtly expressed and may appear as an indirect attack packaged through insincere politeness, satire, or rhetorical questions.The paper illustrates this category with a rhetorical question.
- 4. Aggression Typology: Physical threat includes text threatening physical harm or death, as well as suicide intentions, mass killings, and verbal aggression that may transform into physical aggression.The category is intended to capture potentially physically aggressive language requiring recognition.
4.4 Sexual Threat / Aggression
Sexual threat/aggression covers verbal aggression that graphically depicts sex or threatens to carry out sexual acts against the victim.
- Sexual aggression includes graphic depictions of sex or threats to carry out sexual acts against the victim.
4.6 Non-threatening Aggression
Non-threatening aggression targets individual traits or choices and includes personal insults and cyberbullying, while abuse and aggression remain related but distinct categories.
- 4.6 Non-threatening Aggression: Non-threatening aggression targets individual traits and choices, such as house color or non-communal food preferences, without threatening the victim.
- 4.6 Non-threatening Aggression: Personal insults and cyberbullying are included among non-threatening aggression examples.
- 4.7 Aggression vs. Abuse: Abusive constructions can support banter or jocular mockery, so abuse does not entail aggression.
- 4.7 Aggression vs. Abuse: Aggression and abuse often co-occur, but the paper treats abuse as an aspect rather than a strict subtype of aggression.
- 4.9 Annotation Scheme: The tagset organizes aggression using top-level tags, discursive roles, and ten discursive effects.
4.10 Annotation Conventions
Annotation is performed at the document level with multilabel rules for discursive effects, abuse, general non-threatening aggression, and unintelligible languages.
- 4.10 Annotation Conventions: Annotations apply to complete posts, comments, or other single discourse units.
- 4.10 Annotation Conventions: Annotators select all discursive effects represented when a tweet or comment depicts more than one.
- 4.10 Annotation Conventions: Comments containing abuse must receive the abuse label plus at least one additional effect.
- 4.10 Annotation Conventions: General Non-threatening Aggression cannot be combined with other effects except abuse.
- 4.10 Annotation Conventions: Texts in languages other than English or Hindi, or not understood by annotators, are marked non-aggressive.
5. Inter-annotator agreement
The first agreement round produced low top-level agreement, prompting revised guidelines; a second crowdsourced round achieved higher agreement and supported continuing annotation.
- 5. Inter-annotator agreement: Krippendorff’s Alpha was 0.49 for top-level annotation in the first experiment with four annotators and approximately 500 test instances.
- 5. Inter-annotator agreement: The first round revealed disagreement from single-effect annotation and undefined discursive effects, leading to guideline revisions.
- 5. Inter-annotator agreement: The second round used approximately 1,100 test instances on Crowdflower, with each instance annotated by three annotators.
- 5. Inter-annotator agreement: Top-level agreement exceeded 72%, while agreement for the ten-class discursive-effect annotation was approximately 57%.
- 5. Inter-annotator agreement: Despite flexibility in interpreting this pragmatic phenomenon, the higher second-round scores led the authors to continue data annotation.
6. Final Dataset
The final dataset contains approximately 18k tweets and 21k Facebook comments annotated for aggression. Preliminary analysis compares platform-specific aggression, discursive effects, and the relationship between code-mixing and aggression.
- Dataset: Approximately 18k tweets and 21k Facebook comments comprise the annotated dataset.The comments and tweets were annotated with aggression level and discursive effects by four Hindi-English bilingual annotators.
- Platform comparison: Facebook users are more vocal and overtly aggressive, whereas Twitter users express aggression more subtly and covertly.The comparison is based on aggression levels shown for the two platforms.
- Discursive effects: A majority of tweets and Facebook comments in the dataset involve political aggression.This observation concerns discursive effects across both platforms.
- Code-mixing: A majority of code-mixed comments and tweets are aggressive, Hindi posts are equally distributed, and English posts are largely non-aggressive.The interaction between code-mixing and aggression is summarized in Figure 4.
7. Summing Up
The paper presents a publicly released aggression-annotated Hindi-English dataset for research on aggression and related online phenomena. Initial automatic classification results remain limited, indicating that even top-level aggression classification is complex.
- Contribution: The dataset contains approximately 18k tweets and 21k Facebook comments annotated with different levels and types of aggression.It is presented as the first dataset annotated with different levels and kinds of aggression, according to the authors.
- Contribution: The dataset is publicly released for free use in further research on aggression and related phenomena such as trolling and cyberbullying.The stated intended uses include understanding and automatically identifying aggression on social media.
- Limitations: An initial top-level classifier barely reaches an F1 score of 0.70 across overtly aggressive, covertly aggressive, and non-aggressive classes.The authors describe these results as not very encouraging and call for further investigation.