Source-linked AI summary
Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation
Joseph James
TL;DR
NLP needs reliable measures of agreement for increasingly varied annotation and evaluation tasks, but common approaches have differing assumptions and limitations. This paper surveys IAA measures by task type and reviews reporting practices, concluding that agreement should be interpreted in context with uncertainty and disagreement patterns. Its scope is deliberately broad rather than mathematically or domain-specific in depth.
Problem
NLP annotation and evaluation span diverse tasks, while agreement measures have differing assumptions and can be affected by imbalance, annotator number, granularity, and disagreement.
Method
The paper provides a broad overview of IAA measures across NLP tasks and discusses their assumptions, practical implications, reporting requirements, and limitations.
Results
The paper concludes that agreement is a context-dependent reliability indicator rather than an absolute measure of annotation quality.
Takeaways & Limitations
Reliable evaluation requires task- and rater-appropriate measures, explicit assumptions, uncertainty reporting, and analysis of disagreement patterns.
Takeaways & Limitations
The paper prioritizes breadth over depth, so detailed mathematical descriptions and domain-specific adaptations remain outside its scope.
Abstract
from arXiv · showhide
Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.
1. Introduction
Reliable NLP annotation and evaluation depend on consistent human judgments, but raw agreement can overestimate reliability. The paper therefore surveys task-appropriate IAA measures and emphasizes transparent reporting of uncertainty, rater design, missing data, and label prevalence.
- IAA quantifies how consistently multiple annotators or evaluators apply labels to the same items, affecting dataset and evaluation validity.High agreement can reflect clear guidelines and reproducibility, whereas low agreement may indicate underspecified instructions, insufficient training, or subjectivity.
- Raw agreement alone can overestimate reliability, motivating chance-corrected and task-appropriate agreement statistics.
- Point estimates without uncertainty can overstate precision and reduce comparability across annotation studies.Recommended reporting includes confidence intervals, rater design, missing data, and label prevalence because these affect interpretation.
- The paper reviews IAA approaches across settings and explains how their assumptions affect interpretation and comparability.It also addresses imbalance, missing annotations, and uncertainty in agreement estimates.
2. Categorical data
Categorical agreement measures range from raw matching proportions to chance-corrected coefficients, with different assumptions about annotator count, category prevalence, ordinal distance, and missing data. Metric choice therefore depends on the structure of the annotation task and its distributional conditions.
- Percentage agreement (Po) measures the proportion of items receiving matching labels, but can overestimate reliability when categories are imbalanced because it ignores chance agreement.It remains a useful baseline, especially in exploratory or crowdsourced studies, and is often reported with chance-corrected statistics.
- Bennett, Alpert, and Goldstein’s S coefficient corrects percentage agreement for chance under the assumption that all categories are equally likely.It does not account for annotator bias or imbalanced label distributions, limiting its suitability despite its historical importance.
- Cohen’s κ compares observed agreement between two annotators with agreement expected from their labelling distributions, reducing scores driven by coincidental category preferences.It ranges from 1 for perfect agreement to 0 for chance-level agreement and can become negative for systematic disagreement.
- Fleiss’ κ generalises chance-corrected agreement to any number of annotators assigning nominal categories to the same items.It assumes equal annotation counts per item and can be sensitive to class imbalance or uneven marginal distributions.
- Weighted κ extends Cohen’s κ to ordinal scales by giving partial credit to near agreements and penalising larger differences more heavily.Linear and quadratic weighting schemes are commonly used for Likert or other ordinal rating tasks.
- Krippendorff’s α supports nominal, ordinal, and interval data, any number of annotators, and missing data by modelling observed and chance disagreement.For nominal two-annotator data without missing values, α is mathematically equivalent to Cohen’s κ; for multiple annotators, it relates closely to Fleiss’ κ.
- Gwet’s AC1 and AC2 use an alternative expected-agreement formula intended to be more stable than κ under high agreement, class imbalance, or annotator bias.AC1 is for nominal data, whereas weighted AC2 is for ordinal data.
3. Structured annotations
Structured annotations require agreement measures that account for span overlap, boundary placement, and joint segmentation–categorisation decisions. Metric choice therefore depends on whether annotations are spans, boundaries, or aligned labelled units.
- Span labelling: Pairwise F1 measures exact overlap between annotators’ spans, rewarding shared entities and penalising missed entities or differing boundaries.It is commonly used for span-labelling tasks such as named entity recognition.
- Text segmentation: Pk evaluates segmentation agreement by sliding a fixed-size window and checking whether annotators agree on boundaries between points.It is designed for boundary-based segmentation tasks.
- Text segmentation: WindowDiff similarly evaluates boundary agreement but penalises near-misses less harshly than Pk.It is more forgiving when segment boundaries are slightly shifted.
- Text segmentation: Segmentation Similarity generalises Pk and WindowDiff by incorporating partial matches and variable tolerance for boundary differences.It was introduced as a unified framework for segmentation agreement.
- Unitising annotations: Holistic gamma combines segmentation and categorisation by optimally aligning annotators’ units and applying chance-corrected agreement.It accounts for both positional boundary discrepancies and categorical label discrepancies.
- Unitising annotations: Boundary Edit Distance measures the minimum edits needed to transform one segmentation into another, accommodating insertions, deletions, near misses, and varying granularity.It is presented as a flexible option when partial overlaps and fuzzy boundaries are common.
4. Continuous data
Continuous annotation measures consistency in numeric ratings rather than exact categorical matches. The appropriate statistic depends on whether the goal is absolute agreement, consistency, rater generalisation, or reliability of aggregated scores.
- Reliability measures: The Intraclass Correlation Coefficient estimates the proportion of rating variance attributable to differences between items rather than rater differences or random error.ICC is widely used for continuous tasks such as fluency, coherence, and emotional-intensity ratings.
- ICC variants: ICC variants differ by rater model and by whether reliability concerns a single rating or an average across k raters.The main variants include one-way random, two-way random, and two-way mixed-effects models.
- ICC variants: ICC(2,*) treats raters as randomly selected from a larger population, whereas ICC(3,*) treats the observed raters as fixed and uniquely relevant.This distinction changes the population to which reliability is intended to generalise.
- Internal consistency: Cronbach’s α measures internal consistency and is equivalent to certain ICC forms under a one-way random-effects model with the same raters assessing each item.Its interpretation depends on rater and item assumptions and does not distinguish consistency from absolute agreement.
- Agreement and association: CCC evaluates continuous ratings through both precision and accuracy, penalising mean and scale differences that ordinary correlation does not capture.It assesses closeness to perfect concordance rather than association alone.
- Ordinal judgments: Spearman’s ρ and Kendall’s τ assess whether annotators preserve item rankings without assuming equal intervals between ranks.They are suited to ordinal or relative judgments and indicate association rather than strict absolute agreement.
- Agreement and association: Pearson’s r measures linear association but cannot establish absolute agreement because perfectly correlated ratings may remain systematically offset.Correlation-based metrics should therefore be interpreted as association indicators rather than reliability measures in the strict sense.
5. Metric Selection and Interpretation
IAA metrics are not interchangeable: their interpretation depends on task structure, rater design, assumptions, label prevalence, and annotation granularity. Reliable reporting therefore requires uncertainty estimates, contextualised interpretation, and analysis of disagreement, validity, annotator incentives, expertise, and cultural perspective.
- Metric selection: Metric selection depends on data type, annotator number, chance correction, and missing-data requirements, so scores should be interpreted relative to task and data.Different metrics measure consistency, overlap, or correlation under different assumptions.
- Interpretation: Structured and segmentation metrics lack standard interpretive scales because scores depend on task setup, annotation granularity, and tolerance for partial matches.Relative comparisons are more informative than absolute thresholds for these metrics.
- Interpretation: Fixed agreement thresholds can be overly rigid for complex or subjective tasks, while reporting often omits class imbalance, sample size, or annotator expertise.The cited work calls for replication-based benchmarks and more transparent reporting.
- Uncertainty reporting: Confidence intervals quantify the uncertainty and precision of IAA estimates, enabling more meaningful comparisons across tasks or groups.Narrow intervals indicate greater precision, whereas wider intervals indicate greater variability or instability.
- Reliability and validity: High IAA demonstrates consistency but not validity, because agreement can arise from biased or oversimplified guidelines and low agreement can reflect genuine ambiguity.Reliability should therefore be reported alongside validity evidence such as ambiguous cases or external references.
- Disagreement: Disagreement can reveal ambiguity, underspecified guidelines, rater preferences, or contextual effects rather than simply indicating poor data quality.Soft labels, dispersion measures, and rater-aware models preserve or model these differences.
- Annotation conditions: Performance-based incentives may promote accuracy-oriented behaviour, but their effects depend on task difficulty and evaluation clarity, while flat-rate schemes can incentivise speed over accuracy.Ethical analyses also emphasise transparency, autonomy, and worker agency.
- Human–model comparison: Human and model evaluators should be compared against diverse human perspectives and multiple criteria because models may reproduce bias even when internally consistent.Models often match human reliability on structured tasks but underperform on nuanced or affective judgments.
6. Conclusion
IAA is most useful when treated as part of a broader methodological process rather than as a standalone number. Reliable evaluation aligns metrics with task and rater design, reports uncertainty, and examines disagreement patterns for ambiguity, bias, or inconsistency.
- Conclusion: Reliable NLP evaluation requires aligning the agreement measure with task and rater design, reporting uncertainty, and analysing disagreement patterns.These practices support more transparent and interpretable annotation and evaluation.
7. Limitations
The paper prioritises breadth in surveying commonly used IAA metrics, so it does not provide detailed mathematical or domain-specific coverage. Some specialised tasks and metrics, including multimodal, interactive, humour, and creativity-focused settings, may fall outside its scope.
- 7. Limitations: The overview prioritises breadth across commonly applied IAA metrics rather than detailed mathematical descriptions or domain-specific adaptations.Readers seeking those deeper treatments are directed to the primary literature cited by the paper.
- 7. Limitations: Certain tasks or niche IAA metrics may have been omitted despite the paper’s broad coverage.
- 7. Limitations: Multimodal annotation, interactive evaluation, and subjective phenomena such as humour or creativity may require approaches beyond those discussed.