PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability

Tim Menzner, Jochen L. Leidner

arXiv:2610.37292v1cs.CL

TL;DR

Common IAA metrics often assume one selected label per item, leaving multi-label agreement choices about joint selections and omissions. The paper presents a Hamming-based framework and shows that agreement values depend on the selected chance baseline, while offering diagnostics and reporting guidance for interpreting them.

  • Problem

    Common IAA metrics assume each item receives one selected category, whereas multi-label tasks require decisions about comparing full label sets and joint omissions.

    Source: §1 · p. 2, §5 · p. 14

  • Method

    MLA compares binary label vectors using average pairwise Hamming agreement and offers pooled-marginal, AC1, and fixed-baseline normalizations with item-, label-, and corpus-level diagnostics.

    Source: §2.2 · p. 4, §8 · p. 15

  • Results

    The pooled-marginal, AC1, and fixed 0.5 baselines can yield very different values on the same annotations, especially under extreme prevalence.

    Source: §8 · p. 15

  • Takeaways & Limitations

    Reporting raw agreement, label-wise results, the label universe, and a conditional minimum-agreement diagnostic gives a more informative account than a single coefficient.

    Source: §8 · p. 15

    The simple formulas assume complete, balanced designs with binary, non-hierarchical labels, do not handle missing annotations, and do not model dependencies among labels.

    Source: §7 · p. 14

Abstract

from arXiv · show

Reliable evaluation of inter-annotator agreement (IAA) is important for assessing the consistency of annotation processes. While established IAA metrics such as Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are commonly applied in single-label settings, multi-label tasks require additional decisions about how full label sets, joint selections, and joint omissions should be compared. In this work, we present Multi-Label Agreement (MLA), a flexible framework for evaluating agreement in multi-label annotation tasks. MLA preserves the full binary label vectors assigned to each item and uses average pairwise Hamming agreement, thereby accounting for both joint selections and joint omissions. Its pooled-marginal form is not a mathematically independent coefficient: it is a chance-disagreement-weighted average of label-wise Fleiss/Scott-type coefficients and is closely related to Krippendorff's alpha with normalized Hamming distance. The framework supports pooled-marginal, Gwet AC1, and fixed-baseline normalizations, yields item-level, label-level, and corpus-level results, and includes a conditional minimum-agreement diagnostic. Through a worked example and Monte Carlo simulations, we show how the resulting values depend on label prevalence, the declared label inventory, and the selected model of expected agreement. We also provide a Python implementation. MLA thus offers a transparent and interpretable framework for multi-label agreement analysis without claiming that one normalization is universally preferable.

1. Introduction

Inter-annotator agreement helps assess annotation reproducibility, but disagreement can reflect either annotation-process problems or meaningful differences in interpretation. The paper frames multi-label decisions as binary label vectors and makes explicit how Hamming-based comparison relates to label-wise agreement coefficients.

1. Introduction

  • IAA can reveal unclear guidelines, inconsistent category use, and systematic annotator differences, while subjective disagreement may itself provide useful information.Ambiguous guidelines and lapses of concentration can also contribute to disagreement.Source: §1 · pp. 1–2
  • Codebooks clarify classes with examples and counterexamples, and are typically revised over several rounds of pre-trial annotation.Source: §1 · p. 2
  • For multi-label tasks, each item-annotator decision is a binary label vector rather than the single category assumed by common IAA metrics.The presence and absence of each label both carry information.Source: §1 · p. 2
  • Normalized Hamming distance makes full label-set comparison additive over labels, linking vector comparison to a particular weighting of label-wise comparisons.The paper names this relationship Multi-Label Agreement (MLA) and presents it as a framework, not a mathematically independent coefficient family.Source: §1 · p. 2

2. Related Work

Prior multi-label agreement work ranges from set-distance approaches to specialized coefficients, while related research also examines disagreement distributions and diagnostics. Chance-corrected coefficients depend on the chosen baseline model, so their values answer different questions.

2.1 Past Work on Multi-Label IAA

  • Krippendorff’s α can treat a label set as a structured value using Hamming, Jaccard, or MASI distance; MASI can also be reported directly as a similarity or distance.MASI penalizes partial overlap differently from exact matches.Source: §2.1 · p. 3
  • Rosenberg and Binkowski’s Cohen’s kappa augmentation assumes one true label and is unsuitable when items can have several true labels.Its score is designed to decrease as the number of labels increases.Source: §2.1 · p. 3
  • Marchal et al. estimate chance agreement by bootstrapping over annotators’ label-set intersections, excluding labels never assigned to an item.This omits joint non-selections, and its simulated chance term may be harder to interpret.Source: §2.1 · p. 3
  • The Am coefficient compares decisions for category pairs, yielding one corpuswide score but weighting frequent co-occurrences more and scaling quadratically with label count.The passage also notes sensitivity to skewed prevalence.Source: §2.1 · p. 3
  • Related work preserves annotator-judgment distributions or supplies classification diagnostics, but those diagnostics do not themselves model chance agreement.These lines of work support reporting diagnostics alongside a single corpus-level coefficient.Source: §2.1 · p. 3

2.2 The Ongoing Chance Correction Debate

  • Chance-corrected coefficients use a model-dependent baseline; AC1 and fixed-baseline PABAK use different baselines than kappa-like pooled-marginal measures.AC1 is not a drop-in replacement for κ, and the passage recommends treating normalizations as sensitivity analyses rather than choosing the most favorable value.Source: §2.2 · pp. 3–4

3. Multi-Label Agreement as a Decomposable Construction

MLA compares annotators’ full binary label vectors with normalized Hamming agreement, counting joint selections and joint omissions. Its corpus-level coefficients depend on the chance baseline and label prevalence, and the paper relates them to label-wise coefficients and Krippendorff’s alpha.

3.1 Observed agreement

  • MLA’s observed agreement uses normalized Hamming distance, so both joint selections and joint omissions count as agreement.The construction compares label-vector components independently and does not retain dependencies or co-occurrence patterns between labels.Source: §3.1 · pp. 4–5

3.2 Alternative baseline models

  • A coefficient reaches one at perfect observed agreement; zero means observed agreement equals the selected baseline, while item-level values are corpus-relative diagnostics.The baseline models are not interchangeable because each coefficient gives zero a different interpretation.Source: §3.2 · p. 5

3.3 Exact relation to label-wise coefficients

  • The corpus-level coefficient is a weighted average of label-wise coefficients, with weights determined by the selected baseline and label prevalence.Pooled-marginal weights are 2pj(1 − pj), AC1 weights are pj^2 + (1 − pj)^2, and PABAK weights labels equally.Source: §3.3 · p. 6

3.4 Relation to Krippendorff’s alpha

  • Pooled-marginal MLA has the form of Krippendorff’s alpha with normalized Hamming distance, differing by a finite-sample convention and converging as the sample grows.Krippendorff’s framework also handles missing data and permits other distances, such as Jaccard or MASI.Source: §3.4 · pp. 6–7

3.5 Minimum feasible agreement conditional on prevalence

  • Given fixed empirical label prevalence, the minimum feasible agreement occurs when positive counts across items differ by at most one.This is a feasible-range diagnostic, not a chance model or part of observed agreement.Source: §3.5 · pp. 7–8

3.6 Interpreting Prevalence

  • Near-zero prevalence can make Hamming agreement high through joint omissions, but whether omissions are informative depends on the task and the label-wise pattern.The paper recommends inspecting label-wise prevalence and positive and negative agreement alongside a corpus-level coefficient.Source: §3.6 · p. 8
  • No normalization resolves whether joint omissions matter or which chance model is correct; the paper recommends pre-specifying a model and reporting sensitivity analyses.Natural ambiguity may also make disagreement defensible and worth preserving rather than hiding behind a reliability threshold.Source: §3.6 · p. 8

3.7 Example

  • In the worked example, removing never-selected labels leaves pooled-marginal agreement unchanged but changes AC1 and fixed-baseline summaries; neither label inventory is automatically correct.Label-wise statistics vary, illustrating why a corpus-level number should not be reported alone.Source: §3.7 · pp. 10–11, Table 2 · p. 11, Table 3 · p. 11

4. Behaviour Under Controlled Data Generators

Controlled generators show that agreement values depend on prevalence and the chosen baseline: the three normalizations respond differently across independent-draw and noisy-ground-truth conditions. The simulations are descriptive, and their compact averages do not assess sampling variance.

4. Behaviour Under Controlled Data Generators

  • The simulations are descriptive and do not establish that one coefficient is generally more valid than another.Source: §4 · pp. 11–12
  • The simulations vary label prevalence across 0.80, 0.50, and 0.20, and compare perfect agreement, ε = 0.10 flips, and independent Bernoulli draws.Source: §4 · p. 12
  • Table 4 averages 540 metric values per scenario across annotator count, item count, label count, and 20 repeats, so it does not assess sampling variance.Source: §4 · pp. 12–13, Table 4 · p. 12
  • All measures except augmented kappa reach 1.0 under perfect agreement, illustrating that coefficients should not share an interpretive scale without qualification.Source: §4 · p. 13
  • Under independent Bernoulli draws, pooled-marginal agreement stays near zero while AC1 and PABAK become positive at extreme prevalence; under noisy ground truth, pooled agreement falls, AC1 rises, and PABAK stays constant.Figure 1 contrasts the independent-draw and noisy-ground-truth generators across prevalence.Source: §4 · p. 13, Figure 1 · p. 13
  • With more than two annotators and fixed binary marginals, strongly negative agreement may be unattainable, limiting universal verbal cut-offs such as “good” or “poor.”Source: §4 · p. 13

5. Reporting Recommendations

The paper recommends making the label universe, treatment of joint omissions, chance baseline, diagnostics, and uncertainty reporting explicit. It also cautions that additive Hamming agreement may not suit dependent label structures.

5. Reporting Recommendations

  • State whether the label inventory was fixed before annotation and whether annotators were expected to consider every label; do not drop never-selected labels just because they change a preferred summary.Source: §5 · p. 14
  • Use Hamming-based agreement when joint absence is informative; when overlap among positive selections is the focus, additionally report Jaccard or MASI, preferably with chance correction when needed.Source: §5 · p. 14
  • Pre-specify the primary chance model because pooled-marginal agreement, AC1, and PABAK are not interchangeable estimates of the same quantity.Source: §5 · p. 14
  • Report N, annotators per item, k, observed agreement, label-wise prevalence and agreement, and the conditional lower bound; item-level values can flag difficult examples.Source: §5 · p. 14
  • Accompany corpus-level and important label-level estimates with design-appropriate confidence intervals, using resampling that respects item and annotator structure.Source: §5 · p. 14
  • For hierarchical, mutually exclusive, or otherwise dependent labels, use a domain-specific distance or explicit probabilistic annotator model rather than assuming additive Hamming agreement suffices.Source: §5 · p. 14

6. Implementation

The paper provides a Python package implementing the agreement framework and its summaries, normalizations, and conditional minimum-agreement calculation.

6. Implementation

  • The ml-agreement package is available from the project repository and Python Package Index.Source: §6 · p. 14
  • The package provides corpus-, item-, and label-level summaries, the three discussed baseline choices, label prevalence, and conditional minimum agreement.Source: §6 · p. 14
  • The package retains pooled marginals as its legacy k/kappa option, while the paper uses F to distinguish pooled marginals from Cohen’s annotator-specific model.Source: §6 · p. 14

7. Limitations

The analysis is limited to complete, balanced designs with binary, non-hierarchical labels, and its simple formulas do not handle missing annotations. Equal-weight additive Hamming distance also does not represent label dependencies or unequal error costs.

7. Limitations

  • The analysis covers complete, balanced annotation designs with binary, non-hierarchical labels; its simple formulas do not handle missing annotations.Coding a missing decision as 0 would treat missingness as an observed omission; pair counts and marginal denominators must instead use available values.Source: §7 · p. 14
  • Additive normalized Hamming distance weights label decisions equally and is insensitive to dependencies, so hierarchical or mutually exclusive labels and unequal false-positive/false-negative costs need another distance or model.Source: §7 · p. 14
  • The worked example has only four items, while the simulations average over annotator, item, and label counts without inferential comparisons or confidence intervals.Source: §7 · pp. 14–15
  • For tasks with genuine ambiguity, confidence judgements or distributions over interpretations may represent the task better than binary decisions alone.Source: §7 · p. 15

8. Conclusion

MLA’s pooled-marginal form is a weighted average of label-wise coefficients and is closely related to Krippendorff’s α with Hamming distance, but it does not model label dependencies. Baseline choice changes agreement values, while a conditional minimum-agreement diagnostic adds context tied to annotator count and label prevalence.

8. Conclusion

  • Pooled-marginal Hamming agreement is a chance-disagreement-weighted average of label-wise Fleiss/Scott-type coefficients and is closely related to Krippendorff’s α with Hamming distance.It scales linearly in the number of labels and counts joint omissions, but decomposes over labels and does not model their dependencies.Source: §8 · p. 15
  • Different pooled-marginal, AC1, and fixed 0.5 baselines can yield very different values on the same annotations, especially with extreme prevalence.These differences follow from the models rather than numerical instability alone; multiple coefficients support sensitivity analysis but do not replace justifying a primary estimand.Source: §8 · p. 15
  • The conditional minimum-agreement diagnostic estimates how low observed and normalized agreement can feasibly fall given annotator count and empirical label prevalences.Together with raw agreement, label-wise results, and an explicit label universe, it provides a more informative account than a single coefficient.Source: §8 · p. 15
Loading 2610.37292v1…