Quantifying Annotation Agreement in Object Detection with Optimal Transport

• Annotators may disagree on object presence, box extent, or class, and a single reference can obscure uncertainty that matters for dataset quality and model validation.Source: §1 · p. 1
• Thresholding box overlap into binary matches discards spatial detail, so annotation pairs with different disagreement structures can receive the same summary score.Source: §1 · p. 2
• The OT metric compares labeled box sets using costs for localization, label consistency, and missing or extra boxes.Source: §1 · p. 2
• The paper analyzes bounds and special cases, then evaluates OT on real and synthetically perturbed annotations across domains, finding distinctions threshold-based metrics collapse.Source: §1 · p. 2
• OT-based agreement is presented as a practical tool for assessing annotations and making model validation more sensitive to noisy or conflicting labels.Source: §1 · p. 2

2.1 Uncertainty and disagreement in object detection annotations

• This work complements noise modeling and annotation aggregation by measuring pairwise discrepancies directly to characterize annotation quality or support multi-reference validation.Source: §2.1 · p. 2

2.2 Agreement and evaluation metrics for object detection

• IoU-threshold matching is used for detection agreement, but hard thresholds discard spatial disagreement structure; richer pairwise measures can support methods comparing predictions with multiple annotations.Source: §2.2 · pp. 2–3

2.3 Optimal transport for detection evaluation

• OT matches structured sets at minimum cost and can handle unmatched elements; unlike OC-cost's prediction-to-ground-truth comparison, this work compares annotation sets using spatial, label, and unmatched-box costs.Source: §2.3 · p. 3

3.1 Optimal Transport score formulation

• The pairwise transport cost balances GIoU-based localization and label mismatch, while real boxes may match dummy boxes at cost β when no suitable counterpart exists.The formulation compares boxes and class labels without detector confidence scores.Source: §3.1 · pp. 3–4
• The OT cost lies in [0, β], so the corresponding score lies in [1 − β, 1].β is the real–dummy matching cost, constrained to β ≤ 1.Source: §3.1 · p. 5
• The chance-corrected coefficient αOT compares average same-image annotation costs with Monte Carlo estimates from cross-image pairs.αOT values of 1, 0, and below 0 indicate perfect, chance-level, and below-chance agreement, respectively.Source: §3.1 · p. 5

3.2 Special cases and theoretical boundaries

• With λ ≤ β, dummy matches occur only when annotation sets contain different numbers of boxes.The theoretical analysis studies special cases to clarify dependence on object count, spatial mismatch, and dummy-matching penalty.Source: Case: dummy matching only for unequal… · p. 5, §3.2 · p. 5

Case: dummy matching only for unequal box counts

• The theoretical special-case bounds account for both box-count mismatch and spatial mismatch, with perfectly aligned matched boxes attaining the maximum score.If β < λ, all boxes may match to dummies in the worst case, making the lower bound 1 − β.Source: Case: dummy matching only for unequal… · pp. 5–6
• When box counts differ slightly, the possible score range is large and equals λ at equal counts; with substantial mismatch, the range shrinks toward zero.As the log box-count ratio tends to either extreme, both score bounds approach 1 − β.Source: Case: dummy matching only for unequal… · p. 6, Figure 1 · p. 6
• Higher localization noise shifts real-box matching toward larger β, and theory agrees well with empirical estimates at small perturbations but diverges at higher noise.At higher noise, a perturbed box may become closer to a different object than its source box, violating the noisy-box assumption.Source: Case: dummy matching only for unequal… · p. 8
• Larger β allows higher-cost boxes to remain matched to real boxes, extending the noisy-box approximation to wider perturbations.At β = 0.2, even modest shifts can send boxes to dummies and break the assumed one-to-one noisy correspondence earlier.Source: Case: dummy matching only for unequal… · p. 8

4.1 Datasets

• ACNE04 supplies two annotation sets for direct comparison; AcneSCU adds 10 labeled lesion categories, and TexBiG adds multi-annotator document-layout data across 19 classes.ACNE04 contains 1,457 facial images; TexBiG has more than 52,000 expert-annotated instances.Source: §4.1 · pp. 8–9
• Controlled perturbations include box dropping, center shifting scaled by γ, and label swapping, with γ’s meaning varying by perturbation type.For box dropping, γ is the fraction removed per image; for label swapping, selected boxes are paired randomly and their labels exchanged within each pair.Source: §4.1 · pp. 9–10

4.2 Results and Discussion

• ACNE04 OT scores span much of the feasible range, while TexBiG scores lie near the upper bound, where spatial costs are generally smaller.The paper attributes this difference to TexBiG’s larger, easier-to-localize objects compared with acne lesions.Source: Figure 3 · p. 10, §4.2 · p. 11
• As shift noise rises, IoU-threshold metrics drop rapidly, αOT with β = 0.5 and 0.8 decreases more gradually, and β = 0.2 also drops rapidly.The smaller dummy-matching penalty at β = 0.2 makes real-box matches less preferable under modest perturbations; mIoU becomes insensitive once boxes no longer overlap.Source: §4.2 · p. 11, Figure 4 · p. 12
• Under box dropping and label swapping, agreement curves are approximately linear; OT’s costs scale with unmatched-box penalties and discrete label costs.The unmatched-box penalty is β, while the label-swap cost enters the pairwise cost; the experiments report several β values.Source: §4.2 · pp. 11–12, Figure 4 · p. 12
• Per-image Kendall’s τb is 0.998 for OT and 0.993 for mIoU, versus 0.790 for F1 and 0.790 for Krippendorff’s α, for agreement decreasing as noise increases.Source: §4.2 · p. 12
• OT and mIoU preserve expected ordering in most adjacent-noise comparisons, while threshold-based metrics often tie from ceiling or floor saturation.Reversals at the largest shifts can occur when perturbed boxes move closer to neighboring reference boxes and are rematched.Source: Figure 5 · p. 13, §4.2 · p. 13
• At 200% shift, about 19% of ACNE04 images have mIoU = 0, while OT scores span approximately 0.20–0.47 for β = 0.8.OT retains graded spatial information after overlap vanishes when β/λ > 1/2.Source: Figure 5 · p. 13, §4.2 · p. 13

5 Conclusion

• OT formulates agreement as transport between box sets, combining spatial and label information and allowing missing or extra annotations.The paper adapts an OT-based detection metric specifically to quantify annotator agreement.Source: §5 · p. 14
• The theoretical analysis derives bounds for known transport plans, characterizes possible score ranges, and studies score behavior under a noisy-box assumption as β varies.The analysis relates β and noise levels to remaining real matches, task-specific β selection, and deviations from the assumptions.Source: §5 · p. 14
• Across real and synthetically perturbed datasets, OT reflects spatial differences and distinguishes annotation patterns that IoU-threshold metrics such as F1 and Krippendorff’s α may score identically.The metric is particularly sensitive to bounding-box positions.Source: §5 · p. 14
• The authors conclude that model-performance evaluation and annotator-agreement assessment are related but not identical, and suggest OT for domains with many small or medium boxes and non-trivial domain knowledge.They cite medical imaging as an example and attribute OT’s main sensitivity to spatial localization.Source: §5 · p. 14

Supplementary Material: Quantifying Annotation Agreement in Object

• The supplementary material is titled “Detection with Optimal Transport” and lists Kirill Prokhorov and Alexandr A. Kalinin as authors.Source: Supplementary Material: Quantifying… · p. 17

1 On the evaluation of P (cij < β)

• P(c_ij < β) is the integral of the joint density of absolute noise components over the region where c(ξ, η) < β.The cost distribution’s PDF is denoted f_c, and the joint PDF is that of the absolute noise components.Source: §1 · p. 17
• The integration domain is split into three regions: an intersecting-box case and two non-intersecting cases, with boundaries derived from the GIoU expressions.The non-intersecting boundary is given by ϑ_I=0(u); the intersecting case leads to a quadratic whose relevant root defines a boundary.Source: §1 · pp. 17–18
• Region-specific limits a1, a2, and a3 are substituted into the integral, with each limit clipped or constrained to the stated feasible range.These limits correspond to the three regions used to decompose the integration domain.Source: §1 · p. 18, Figure 1 · p. 18
• When noise components are independent, their joint density factorizes, allowing the inner integrals to be expressed using the CDF of ϑ.The resulting expression integrates across the three regions using f_u and F_ϑ.Source: §1 · p. 18

2 Examples of spatial properties of the OT score

• OT distinguishes a moderate spatial shift from a deliberately displaced annotation even when F1 and Krippendorff’s α assign the same scores; mIoU remains sensitive only while boxes overlap.The displaced example has only one overlapping box pair, and OT reflects spatial costs for unmatched and poorly matched boxes.Source: §2 · pp. 18–19, Figure 2 · p. 19
• For 1/2 < ρ < 1, the OT real-match region extends beyond the region where mIoU > 0; for ρ ≤ 1/2, it lies within the overlap region or reference boxes.If ρ > 1, all real edges are eligible, producing min(M, N) real matches.Source: §2 · p. 19, Figure 3 · p. 20
• Fig. 4 presents observed versus expected distances for TexBiG.Source: Figure 4 · p. 20

3 Data-driven estimation of β using KαLOS

• τ∗ = 0.222 yields candidate β∗ = 0.222 for λ = 1, based on box-level spatial distances from 2,250 images and six annotators.The dataset included images with at least two nonempty annotation sets.Source: §3 · p. 20
• Mean same-image distance Do = 0.072 and cross-image distance De = 0.440 are well separated, supporting calibration from real multi-annotator data.Source: §3 · p. 20
• Synthetic perturbations are controlled stress tests, not models of human annotation noise; expert severity judgments would provide complementary validation.Source: §3 · p. 20

4 Additional information about datasets

• The vast majority of Acne04 and AcneSCU boxes occupy less than 1% of image area, while TexBiG has a median relative box area of approximately 7%.TexBiG also includes a substantial number of objects covering 30% or more of an image.Source: §4 · p. 21, Figure 5 · p. 21
• The main TexBiG experiments use paragraph, image, and caption classes as larger-object counterparts to the small acne-dataset objects.For each TexBiG class, examples show two annotators with the largest numbers of annotated objects.Source: §4 · p. 21
• ACNE04 examples use two annotation sets; AcneSCU covers 10 lesion categories, and TexBiG contains 19 document-layout classes.In the AcneSCU examples, colors denote lesion classes.Source: §4 · p. 21, Figure 6 · p. 22