Source-linked AI summary
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets
Mor Geva, Yoav Goldberg, Jonathan Berant
TL;DR
Crowdsourced NLU datasets may encode annotator-specific language because a few workers generate many examples, raising questions about diversity and generalization. The paper tests this bias across three datasets and finds that annotator information improves prediction while generalization to unseen annotators often fails. It therefore recommends monitoring annotator bias and separating training-set from test-set annotators.
Problem
A small number of workers generate many free-text examples, raising concerns that NLU datasets encode annotator bias and limit data diversity and generalization.
Method
The paper runs experiments on three NLU datasets, adding annotator IDs as input features and evaluating models on annotator-disjoint splits.
Results
Annotator information improves model performance, models recognize productive annotators, and generalization to unseen annotators often fails.
Takeaways & Limitations
Annotator bias should be monitored during data collection, with test-set annotators separated from training-set annotators.
Abstract
from arXiv · showhide
Crowdsourcing has been the prevalent paradigm for creating natural language understanding datasets in recent years. A common crowdsourcing practice is to recruit a small number of high-quality workers, and have them massively generate examples. Having only a few workers generate the majority of examples raises concerns about data diversity, especially when workers freely generate sentences. In this paper, we perform a series of experiments showing these concerns are evident in three recent NLP datasets. We show that model performance improves when training with annotator identifiers as features, and that models are able to recognize the most productive annotators. Moreover, we show that often models do not generalize well to examples from annotators that did not contribute to the training set. Our findings suggest that annotator bias should be monitored during dataset creation, and that test set annotators should be disjoint from training set annotators.
1 Introduction
The paper investigates annotator bias in crowdsourced NLU datasets, where a small group of workers generates many examples. It tests whether models exploit annotator-specific language and fail to generalize to unseen annotators.
- Crowdsourcing commonly relies on a small group of high-quality workers to generate large numbers of NLU examples.
- Few-worker annotation raises concerns about data diversity and model generalization when workers freely generate text.
- Annotator-specific language patterns that correlate with labels can cause models to overestimate performance.
- The study investigates annotator bias across MNLI, OPENBOOKQA, and COMMONSENSEQA.
- Adding annotator identifiers improves performance, while models often fail to generalize to unseen annotators.
- The authors propose monitoring annotator bias during data collection and separating training-set annotators from test-set annotators.
2 Crowdsourcing Practice
Crowdsourced NLP datasets often depend on a small, unevenly distributed group of annotators. This concentration may reduce diversity and inflate apparent model performance.
- Crowdsourcing is a prominent method for creating datasets across question answering, reasoning, and natural language inference.
- Dataset construction often recruits and screens workers, leaving a small group responsible for most examples.
- Examples include dozens of workers creating 5,957 questions and 99 workers creating more than 100,000 examples.
- Datasets may contain approximately 1 annotator per 10^2–10^3 examples, with a few annotators producing the vast majority.
- This skew may affect data diversity and lead to over-estimated model performance in creative language-writing tasks.
3 Experimental Setup
The experiments examine three NLU datasets built from annotator-generated sentences and questions, using shared modeling procedures. Dataset statistics and annotator coverage characterize the concentration of contributions.
- The study focuses on crowdsourcing tasks in which workers produce full-length sentences.
- The datasets are MNLI, OPENBOOKQA, and COMMONSENSEQA, selected because annotator IDs are available.
- MNLI asks annotators to write hypotheses with entailment, contradiction, or neutrality labels.
- OPENBOOKQA workers write questions and distractors from scientific facts, while COMMONSENSEQA workers bridge concepts from CONCEPTNET.
- Figure 1 reports the fraction of examples covered by annotators sorted by the number of examples they annotated.
- Most examples in all datasets, especially OPENBOOKQA and COMMONSENSEQA, were generated by a small number of annotators.
- Models use pretrained BERT-base with a linear output layer and a common three-epoch fine-tuning procedure.
4 Experiments and Results
The experiments test whether annotator information is predictive and whether models generalize across annotators. Across three NLU datasets, annotator IDs improve performance, while unseen-annotator generalization varies and can fail substantially.
- The experiments ask whether annotator IDs improve prediction, whether models recognize annotators, and whether they generalize across annotators.
- Adding annotator IDs as input features improves model performance across all three datasets.The model receives a textual annotator identifier alongside each input sequence.
- In OPENBOOKQA, unseen-annotator performance drops by up to 23 accuracy points across all five multi-annotator splits.Drops reach up to 10 accuracy points in COMMONSENSEQA and 5 in MNLI, with more varied effects.
- In roughly half of the single-annotator cases, performance is lower than on an equal-sized random split.Because annotators vary in example difficulty, the authors introduce augmentation to separate difficulty from annotator bias.
- Moving a random fraction k = 0.1, 0.2, 0.3 of development examples into training tests whether exposure to an annotator’s language improves performance.Training-set size remains constant through random removal, and results are reported as mean and standard deviation.
- COMMONSENSEQA and OPENBOOKQA show rapid increases of 10-20 accuracy points after exposure to a small number of top annotators’ examples, whereas MNLI changes marginally.The authors interpret the first pattern as evidence that annotator language, rather than inherent example difficulty, affects performance.
5 Discussion and Conclusions
The study concludes that annotator bias is evident in two of three recent NLU datasets and is associated with difficulties generalizing to unseen annotators. It recommends monitoring this bias during data collection and separating training-set from test-set annotators.
- Annotator bias is evident in two of the three evaluated NLU datasets, with skewed distributions more prone to bias.The study links this pattern to datasets where a few annotators create most examples.
- The authors recommend monitoring annotator bias during free-text data collection by testing model performance on new annotators.
- They propose separating training-set annotators from test-set annotators to address annotator bias.