PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Annotator Group Bias in Crowdsourcing

Haochen Liu, Joseph Thekinen, Sinem Mollaoglu, Da Tang, Ji Yang, Youlong Cheng, Hui Liu, Jiliang Tang

arXiv:2110.08038v1cs.HCcs.LG

TL;DR

Crowdsourced annotations can be noisy, and prior work largely overlooked demographic group effects in annotator bias. The paper studies those effects and introduces GroupAnno, a probabilistic graphical framework trained with an extended EM algorithm. Experiments report improved bias estimation, truth inference, and classification compared with competitive baselines.

  • Problem

    Crowdsourced labels can be inconsistent, while prior studies mainly examined individual annotators and largely overlooked group effects in bias.

    Source: §1 · p. 1

  • Method

    GroupAnno models annotation production with demographic group bias in a probabilistic graphical framework and uses extended EM to estimate bias, infer labels, and optimize a classifier.

    Source: §1 · p. 2

  • Results

    Experiments on synthetic and real-world datasets report accurate group-bias estimates and more accurate truth inference and classification than competitive baselines.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Annotators within the same demographic groups tend to show similar bias, which the study considers when modeling annotation and learning classifiers.

    Source: §6 · p. 8

    For simplicity, the study considers binary classification, while stating that its findings and framework can be extended to multiclass classification.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Crowdsourcing has emerged as a popular approach for collecting annotated data to train supervised machine learning models. However, annotator bias can lead to defective annotations. Though there are a few works investigating individual annotator bias, the group effects in annotators are largely overlooked. In this work, we reveal that annotators within the same demographic group tend to show consistent group bias in annotation tasks and thus we conduct an initial study on annotator group bias. We first empirically verify the existence of annotator group bias in various real-world crowdsourcing datasets. Then, we develop a novel probabilistic graphical framework GroupAnno to capture annotator group bias with a new extended Expectation Maximization (EM) training algorithm. We conduct experiments on both synthetic and real-world datasets. Experimental results demonstrate the effectiveness of our model in modeling annotator group bias in label aggregation and model learning over competitive baselines.

1 Introduction

The introduction argues that group effects in annotator bias remain understudied, despite demographic groups showing different annotation tendencies. It presents GroupAnno, an extended-EM framework, and reports improved label inference and classification against competitive baselines.

  • 5.1% more comments were rated toxic by native than non-native English speakers, while annotators over 30 rated 2.5% more toxic than younger annotators.Source: §1 · p. 1
  • Prior annotator modeling mainly examined individual attributes, while annotator stereotypes and demographic group effects received comparatively little study.Source: §1 · p. 1
  • GroupAnno estimates annotator group bias, infers true labels, and optimizes the text classification model simultaneously using a probabilistic graphical model and an extended EM algorithm.Source: §1 · p. 2
  • GroupAnno accurately estimates group bias and outperforms competitive baselines in truth-label inference and classification on synthetic and real-world data.Source: §1 · p. 2

2 Understanding Annotator Group Bias

The study examines demographic differences in annotation behavior across three Wikipedia Detox datasets, then tests whether demographic categories significantly affect annotator sensitivity and specificity. It reports group differences in annotation tendencies and bias measures.

2.1 Data and Tasks

  • Three Wikipedia Detox datasets cover personal attack, aggression, and toxicity, with comments labeled by about ten annotators and demographic attributes collected.Source: §2.1 · p. 2

2.2 Empirical Study

  • Young and non-native annotators were less likely to label comments as attacking, aggressive, or toxic; native and non-native speakers showed larger annotation gaps than other group pairs.Source: §2.2 · p. 3
  • For binary classification, the study describes individual annotator bias using sensitivity and specificity, with group effects assessed by demographic category.Source: §2.2 · p. 3
  • Because true labels were unavailable, majority-vote labels approximated truth when estimating each annotator’s sensitivity and specificity for the demographic-group ANOVA.Source: §2.2 · p. 3
  • Gender, age, and language significantly affected annotator sensitivity and specificity in most cases across the datasets, supporting the existence of annotator group bias.Source: §2.2 · p. 3

3 Modeling Annotator Group Bias

This section defines group-level bias measures, formulates the joint estimation task, and presents GroupAnno and its extended EM training procedure.

3.1 Measurements

  • Group sensitivity and specificity extend the corresponding measures to each demographic group and category; the parameters collect these rates across categories.The section denotes the group-bias parameters for category p as πp = (αp,0, αp,1, βp,0, βp,1).Source: §3.1 · p. 4

3.2 Problem Statement

  • Given annotations and annotators’ demographic groups, the task is to estimate group bias, infer each instance’s true label, and learn a classifier.The model parameters comprise the classifier parameters and group-bias parameters.Source: §3.2 · p. 4

3.3 GroupAnno: The Probabilistic Graphical Model

  • GroupAnno models true labels with a classifier and annotator labels through individual bias, whose mean reflects demographic-group and individual effects.Annotator bias is modeled with a Beta prior.Source: §3.3 · p. 4

3.4 The extended EM algorithm

  • The extended EM algorithm addresses Beta-distributed annotator bias and unknown true labels by optimizing a lower bound on the likelihood.It updates posterior label probabilities in the E-step and model parameters in the M-step.Source: §3.4 · pp. 5–6

4 Experiment

Across Wikipedia Detox and Information Detection, GroupAnno outperforms the baselines on the reported outcomes. Synthetic experiments also assess group-bias estimation and truth-label inference, alongside experiments using real-world and simulated data.

4 Experiment

  • The experiments ask whether GroupAnno estimates group bias, infers true labels, and learns more accurate classifiers..Source: §4 · p. 6

4.1 Baselines

  • GroupAnno is compared with nine existing true-label inference methods, including majority voting, LFC-binary, and VI-BP and VI-MF..Source: §4.1 · p. 6

4.2 Data

  • The experiments use circle- and moon-shaped synthetic binary datasets, three Wikipedia Detox subsets, and a meeting-text information-detection dataset.The synthetic datasets each contain 400 instances per class; the meeting-text labels are converted to information exchange versus other.Source: §4.2 · p. 6

4.3 Results on Synthetic Data

  • The extended EM algorithm estimates the simulated group-bias parameters accurately within an acceptable error range.Table 3 compares real and estimated parameters on the synthetic datasets.Source: §4.3 · p. 6, Table 3 · p. 7
  • GroupAnno achieves the best synthetic truth-label inference performance, with group bias supplying prior information for estimating individual annotator bias.The authors attribute the improvement over LFC-binary to exploiting group bias as additional information.Source: §4.3 · pp. 6–7, Table 4 · p. 7

4.4 Results on Wikipedia Detox Dataset; 4.5 Results on Information Detection Dataset

  • GroupAnno outperforms state-of-the-art methods on Wikipedia Detox classifier performance and Information Detection truth inference and classifier performance.For Wikipedia Detox, the reported performance is for learned classifiers on test data; Information Detection reports both truth inference and classifier performance.Source: §4.4 · p. 7, §4.5 · p. 7

5 Related Work

Prior annotation research models worker reliability and other individual factors, while the paper positions demographic group effects as a comparatively overlooked source of annotator bias.

Bias and annotation reliability

  • Annotation labels can skew training-data distributions, while task difficulty, ambiguity, context, and annotator expertise influence annotation results.The passage situates these issues within broader concerns about data-labeling and machine-learning bias.Source: §5 · p. 8

Prior annotation-reliability work

  • Prior work models annotator ability, quality, numeric-task bias and variance, task-specific skills, instance-level reliability, and annotator identity.The cited methods include Demartini et al. (2012); Aydin et al. (2014), Whitehill et al. (2009); Li et al. (2014b), Welinder et al. (2010), Fan et al. (2015), Ma et al. (2015), Li et al. (2019), and Geva et al. (2019).Source: §5 · p. 8

Group effects

  • The reviewed studies are described as individual-focused and as ignoring group effects; this work instead studies systemic bias associated with demographic groups..Source: §5 · p. 8

6 Conclusion

The paper studies demographic group bias in crowdsourced annotations and introduces GroupAnno with an extended EM algorithm. Experiments report group-bias estimation, truth-inference, and classifier results, while work beyond classification remains future work.

Conclusion

  • Annotators from the same demographic groups tend to show similar bias in annotation tasks.The paper reports this finding from an empirical study on real-world crowdsourcing datasets.Source: §6 · p. 8
  • GroupAnno models group effects in the annotation process, and its extended EM algorithm estimates group bias, infers true labels, and optimizes the classifier..Source: §6 · p. 8
  • Experiments report accurate group-bias estimation, more accurate truth inference, and better classifiers than state-of-the-art true-label inference baselines; regression and text generation are future work..Source: §6 · p. 8

Related papers

Loading 2110.08038v1…