PaperlayerView exact revision on arXiv

Source-linked AI summary

Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement

Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, Sara Tonelli

arXiv:2109.13563v1cs.CLcs.AIcs.CY

TL;DR

Offensive-language datasets rely on crowdsourced judgments, but their quality and the treatment of disagreement matter in a subjective task. The authors create three English-tweet datasets with five judgments per tweet and study agreement levels. They find that low agreement can reflect genuine interpretive difficulty rather than poor annotation, and argue for including more such cases in benchmark test sets.

  • Problem

    Crowdsourced offensive-language datasets receive limited attention to annotation quality, even though annotator agreement is important in this subjective task.

    Source: §1 · p. 1

  • Method

    The authors create three English-tweet datasets with agreement levels, using a classifier ensemble to help select tweets and five crowd judgments per tweet.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Low-agreement examples represent genuine difficult interpretive cases and are usually not due to poor-quality annotation; such cases are minimally present in a popular benchmark dataset.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors argue that benchmark datasets, especially test sets, should better represent hard cases to account for differing viewpoints and support more robust system evaluation.

    Source: §6 · pp. 8–9

    Preliminary, unreported experiments found that BERT performed poorly at distinguishing ambiguous from unambiguous tweets and among offensive, non-offensive, and ambiguous classes.

    Source: §6 · p. 8

Abstract

from arXiv · show

Since state-of-the-art approaches to offensive language detection rely on supervised learning, it is crucial to quickly adapt them to the continuously evolving scenario of social media. While several approaches have been proposed to tackle the problem from an algorithmic perspective, so to reduce the need for annotated data, less attention has been paid to the quality of these data. Following a trend that has emerged recently, we focus on the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. Our study comprises the creation of three novel datasets of English tweets covering different topics and having five crowd-sourced judgments each. We also present an extensive set of experiments showing that selecting training and test data according to different levels of annotators' agreement has a strong effect on classifiers performance and robustness. Our findings are further validated in cross-domain experiments and studied using a popular benchmark dataset. We show that such hard cases, where low agreement is present, are not necessarily due to poor-quality annotation and we advocate for a higher presence of ambiguous cases in future datasets, particularly in test sets, to better account for the different points of view expressed online.

1 Introduction

Crowdsourced annotation can be affected by worker reliability, but disagreement in subjective social-computing tasks may reflect ambiguity and differing perspectives rather than poor-quality labels. This paper studies disagreement in offensive-language datasets and reports that low-agreement examples can still provide useful training signal.

1 Introduction

  • Crowdsourced benchmarks may be compromised by workers lacking relevant training or by workers who submit quick answers unrelated to the correct label.Prior responses include expertise and reliability controls, identifying spammers, and repeating labels on targeted examples.Source: §1 · p. 1
  • The study examines how different levels of annotator agreement affect offensive-language classifiers and whether low-agreement examples remain useful rather than being discarded as noise.It also investigates agreement in the Offenseval test set and how it relates to participating systems’ performance.Source: §1 · pp. 1–2
  • In subjective social-computing tasks such as offensive-language detection, ambiguity and bias can make annotator disagreement an inherent part of the task rather than an error to eliminate.The introduction contrasts these tasks with cases such as part-of-speech tagging or parsing, where disagreement may stem from unclear guidelines and be reconciled through adjudication.Source: §1 · p. 1
  • The authors release annotated datasets upon request, covering three domains with more than 10k tweets, labeled with 50k crowd-worker judgements and annotated with agreement levels.To the authors’ knowledge, this is the first dataset explicitly created to cover different agreement levels in a balanced way.Source: §1 · p. 2

2 Related Work

Earlier work has variously treated disagreement as annotation noise, filtered hard examples, or developed methods to represent annotator uncertainty. This paper instead centers dataset creation and controlled agreement levels for offensive-language evaluation.

2 Related Work

  • Prior studies have proposed treating disagreement as noise, modeling annotator reliability, incorporating agreement into learning, or filtering hard examples, with mixed findings on their effects.Some report poorer classification of easy examples or improved performance after filtering low-agreement instances.Source: §2 · p. 2
  • Other proposals embrace uncertainty by evaluating stable opinions separately from noise or by grouping annotators according to polarization.These approaches compile different gold-standard datasets for training separate classifiers.Source: §2 · p. 2
  • This study focuses on dataset creation and evaluation under controlled agreement levels, rather than primarily on evaluation metrics or classification strategies.It introduces a dataset with three agreement levels and balanced classes across three domains for in-domain, out-of-domain, and benchmark analyses.Source: §2 · p. 2
  • Existing crowdsourced toxic- and abusive-language datasets with disaggregated labels were not designed to analyze disagreement or balance judgments across dimensions.The authors identify this as a distinction from their agreement-focused dataset design.Source: §2 · p. 3

3 Data Selection and Annotation

The authors use a classifier ensemble to select tweets across agreement strata, collect five crowd judgments per tweet, and analyze the relationship between classifier and annotator agreement. Human disagreement includes cases of genuine interpretive difficulty and is not uniformly distributed across the sampled data.

3.1 Ensemble of classifiers to select data for annotation

  • The five-classifier BERT ensemble pre-screens tweets into unanimous, mild, and weak agreement strata, and about 92%, 5%, and 3% of the collected data fall into those respective classes.The ensemble uses models fine-tuned on the Founta et al. dataset; tweets are then sampled equally by ensemble agreement class and label for annotation.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 Data Annotation with AMT

  • They report 10,753 AMT-annotated tweets (3,472 Covid-19, 3,490 US elections, 3,791 BLM), with five US native speakers per tweet; failed-gold HITs and annotations from workers below 70% accuracy were removed.Tweets without five collected annotations were removed from the final dataset.Source: §3.2 · p. 4

3.3 Annotators and Ensemble Agreement

  • 43% of annotated tweets received full annotator consensus, 29.35% mild agreement, and 28.28% weak agreement, despite equal sampling across ensemble-agreement classes.Full consensus was mostly non-offensive (34.12% N++), while 8.05% were unanimously offensive (O++); weak-agreement tweets included 15.56% N0 and 12.92% O0.Source: §3.3 · p. 4
  • Classifier and annotator agreement have a moderate Pearson correlation (r = 0.51), suggesting ensemble pre-screening could help identify unambiguous or challenging domain-specific tweets.A BiLSTM ensemble yielded a similar correlation (r = 0.50).Source: §3.3 · p. 4

3.4 Qualitative analysis of (dis)agreement

  • Weak-agreement tweets often involve missing conversational context, untargeted anger or negative feelings, and rhetorical questions that are difficult for workers to interpret.The manual analysis identifies these as recurring phenomena in the A0 class.Source: §3.4 · pp. 4–5
  • Overall, disagreement appears to reflect genuine differences in tweet interpretation rather than poor annotation, although annotators’ opinions on recent events may also have influenced judgments.The authors mention BLM and US elections as topics where personal opinions may have played a role.Source: §3.4 · p. 5

4 Classification experiments

The experiments isolate how annotator agreement in training and test data relates to offensive-language classifier performance, controlling for agreement level, label distribution, and domain. They also test cross-domain transfer and whether low-agreement labels carry signal beyond random labels.

4.1 Impact of (dis)agreement in training data

  • 900 A++ tweets outperformed all 2,700 instances across agreement levels; adding A0 data to higher-agreement training configurations reduced performance.The reported averages also show that combining A++ and A+ was comparable to the best two-step fine-tuning configuration.Source: Table 2 · p. 6, §4.1 · p. 6

4.2 Impact of (dis)agreement in test data

  • Lower agreement in the test set brought a dramatic performance drop; training on A++ and A+ data performed best on ambiguous cases.The authors argue that benchmarks should control for test-set ambiguity so high scores are not driven by a lack of challenging examples.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Impact of (dis)agreement on out-of-domain data

  • In out-of-domain tests, high-agreement training performed well, while adding A0 data hurt performance; the authors report little in-domain overfitting.Each run trained on two domains and tested on the third; reported F1 averages the results across domains.Source: §4.3 · pp. 6–7, Table 4 · p. 7

4.4 (Dis)agreement versus Randomness

  • Replacing A0 labels with random labels worsened results, indicating that low-agreement labels contain useful, though challenging, signal rather than chance assignments.The authors caution that coin-flipping to resolve disagreement may lose information.Source: §4.4 · p. 7

5 Experiments on Offenseval dataset

The authors examine agreement levels in Offenseval and compare system performance across those levels. Their results indicate that the benchmark is dominated by high-agreement examples and that system scores vary with test-data agreement.

Benchmark performance

  • On balanced data, Transformer-based models had F1 ≈0.75, whereas the majority of submissions in the last Offenseval shared task for offensive language identification in English tweets achieved F1 > 0.90 on the binary classification task.Source: §5 · p. 7

Annotation procedure

  • The authors annotated a random sample of 1,173 tweets—30% of the test set—with five judgments per tweet, slightly adapting the guidelines to include profanity.The adaptation treated profanities as offensive, as in the Offenseval guidelines.Source: §5 · p. 7

Agreement distribution

  • More than 90% of the annotated Offenseval sample had A+ or A++ agreement, and its agreement distribution matched the projection on the initially collected 400k tweets.Only 6.4% of sample annotations differed from the original labels; half of those differences were in A0.Source: §5 · pp. 7–8, Table 6 · p. 8

Performance by agreement level

  • Across 81 Offenseval runs, average F1 increased with test agreement, and the authors infer that high benchmark scores were probably influenced by the prevalence of total-agreement tweets.The runs included systems that scored F1 > 0.75 in the task.Source: Table 7 · p. 8, §5 · p. 8

6 Discussion and Conclusions

The experiments show that agreement levels matter for offensive-language detection and that low-agreement cases are not necessarily poor-quality annotations. The authors argue for representing such challenging cases in benchmarks, especially test sets, while identifying automated disagreement classification as future work.

Conclusion

  • The authors recommend more hard, ambiguous cases in benchmark test sets to support robust evaluation and avoid overestimating classifier performance.They suggest combining oversampling of the minority offensive class with oversampling of minority agreement classes.Source: §6 · p. 8
  • The authors characterize challenging examples as real difficult decisions, usually not low-quality annotations, and propose treating disagreement as a signal rather than noise.They connect attention to disagreement with representing diverse points of view and reducing exclusion of minority voices and demographic misrepresentation.Source: §6 · pp. 8–9
  • An ensemble of classifiers can preliminarily select tweets likely to be unambiguous or challenging for annotation.The paper presents this as a way to identify candidate examples for further analysis.Source: §6 · p. 8

Discussion

  • The discussion notes that training-data quantity alone does not ensure classification quality, and proposes disagreement study as a possible partial explanation for cross-language performance differences.The cited examples include Offenseval 2020 results for Arabic, Greek, and Turkish.Source: §6 · p. 8

Future work

  • The authors plan better methods to classify disagreement; preliminary BERT experiments performed poorly on binary and three-way ambiguity classification.They suggest ambiguity involves lexical, semantic, and pragmatic aspects that are difficult for a language model to capture.Source: §6 · p. 8

7 Ethics Statement

The authors describe safeguards for crowd-workers and limits on dataset reuse, alongside steps intended to protect Twitter users and annotators.

Worker safeguards

  • Adult-content task requirements were implemented, and workers were not required to complete long sessions or label a minimum amount of data.Observed annotation sessions averaged 20 minutes; average compensation was US$6 per hour.Source: §7 · p. 9

Quality control

  • Completed hits were compensated even when annotations failed the 70% gold-standard accuracy threshold, and workers could discuss mismatches before being unlocked.The authors report strict quality control during data collection.Source: §7 · p. 9

Data access and privacy

  • Dataset access is restricted to non-commercial research, with tweet IDs rather than text and no crowd-worker IDs included.Requestors must declare compliance with user-protection conditions; tweet IDs are intended to respect users’ right to be forgotten.Source: §7 · p. 9

A Annotation Guidelines for AMT

The AMT instructions tailor a short prompt to each tweet domain, then give all workers a shared definition of offensive language and a quality check. Annotators are asked to judge abusive content independently of personal or political views.

Annotation prompts

  • For COVID-19, US Presidential campaign, and Black Lives Matter tweets, workers were asked to judge offensiveness based solely on abusive content, independently of personal views or political orientation.The US campaign prompt specifically names political orientation; the other prompts refer to opinion.Source: A Annotation Guidelines for AMT · p. 11

Shared task description

  • The shared definition includes profanity and hurtful or vulgar language intended to insult or debase a person or group, including targeted derogatory language and expressions meant to offend.Tweets outside that category are labeled “Normal.”Source: A Annotation Guidelines for AMT · p. 11

Quality check

Annotation prompts

Appendix results tables

  • The included tables distinguish single-domain tests from out-of-domain tests, where each domain is tested after training on the other two.Source: Table 8 · p. 12, Table 9 · p. 12

B Impact of (dis)agreement on classification - results in detail

Domain-specific results confirm that higher-agreement training data improve predictions, while lower-agreement data are detrimental. Classification takes about two minutes without the intermediate fine-tuning stage, which raises runtime to 1.5 hours.

Single-domain results

  • Higher-agreement training data improve predictions on individual domains, while lower-agreement training data are detrimental.These domain-specific results confirm the direction of the average across the three domains reported in the main document.Source: B Impact of (dis)agreement on… · p. 12

Runtime

C Impact of (dis)agreement on out-of-domain data - results in detail

In the out-of-domain setting, results are consistent with the main document’s average: higher-agreement training data improve predictions, while lower-agreement data are detrimental. Classification time was about the same as in the single-domain configuration.

Out-of-domain results

  • In out-of-domain tests, training on higher-agreement data improved predictions, while training on lower-agreement data was detrimental.The passage says these results are consistent with the average scores reported in the main document.Source: C Impact of (dis)agreement on… · p. 12

Evaluation setting

Runtime

D Twitter data collection

The authors collected tweets in real time using domain-specific hashtags and keywords. They randomly selected about 400,000 collected tweets for evaluation with the paper’s ensemble method.

Collection method

Collection periods

  • Collection periods were 25 January–9 November 2020 for COVID-19 and 24 May–16 June 2020 for Black Lives Matter; the cited passage lists US Elections keywords but not its dates.Source: D Twitter data collection · p. 12

Selection and evaluation

  • About 400,000 tweets were randomly selected from the collected data and evaluated with the ensemble method described in the main paper.Source: D Twitter data collection · p. 12

Related papers

Loading 2109.13563v1…