Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

• The study measures reference sensitivity by holding system output fixed and varying the reference, using shipped gold and independent expert re-annotation from a six-language PII benchmark.The retained record includes two independent annotator labellings and expert re-annotation of a sample; both references are defensible.Source: §1 · p. 2
• +2.95 points (95% CI [+1.97, +3.87]): exchanging references changed the paired comparison between outputs and reversed the Hindi margin.The interaction survived Holm correction over all 14 reported tests.Source: §1 · p. 2
• Holding each output fixed, reference exchange moved its score by 4.95 F1 points for one annotator’s output and 2.00 for the other.The corresponding 95% CIs were [3.23, 6.27] and [0.26, 3.37].Source: §1 · p. 2
• An undocumented default usually retained Annotator A when adjudication did not fire, leaving the shipped reference identical to one annotator’s labels on 94.9% of audited prompts.The shipped reference was therefore partly copied from an output being scored.Source: §1 · p. 2
• The procedure requires no new annotation, but the reported magnitude comes from one record and human annotation passes, so transfer to model outputs remains empirical.The authors also note that a neutral aggregation rule would show less sensitivity.Source: §1 · p. 2

2 The Dataset and Its Pipeline

• 24,939 prompts across six languages retain independent annotator labels, shipped gold, and expert re-annotation, enabling reference comparisons on shared items.The reviewer re-annotation covers 10.5% of the corpus; its random selection is stated in the preparation record but could not be verified from the export. Prompts are not released, and only aggregate statistics and type labels are reported.Source: §2 · pp. 2–3

2.1 How much the annotators disagree

• Annotator agreement is 79.6 STRICT and 82.2 RELAXED pooled, ranging from 71.7 in Arabic to 87.8 in Chinese; most disagreement concerns boundaries.When offsets match, annotators assign the same type 95.6–97.6% of the time; exact-match scoring turns boundary conventions into score differences.Source: §2.1 · p. 3

2.2 Two pipeline mechanisms that shape the gold label

• When annotators disagreed and the third pass did not fire, the pipeline retained Annotator A in 98.7% of cases, creating the measured skew toward A.This silent branch occurred on 10.6% of prompts pooled and 16.5% in Arabic; where adjudication fired on disagreement, gold reproduced B and A at similar rates. Adjudication rates vary across languages, and the export does not explain why.Source: §2.2 · p. 4

2.3 The audited subset

• The audited subset contains 2,628 prompts where all four labellings coexist, allowing two defensible references to be compared on identical items.The reviewer’s labels differ from delivered gold on 22.0% of prompts; restricting analysis to this sample costs statistical power.Source: §2.3 · p. 4

3 Method

• The design treats two annotator labellings as outputs and delivered/reviewer gold as references, using paired prompt-level bootstrap and an interaction to test ranking change.Source: §3 · p. 4

3.1 Setting and matching rules

• All measurements use audited prompts carrying all four labellings; STRICT requires exact start, end, and type, while RELAXED requires exact boundaries and ignores type.Figure 1's designed pipeline adds a third pass on disagreement and reviewer re-annotation of a sample, though the third pass sometimes skipped disagreements and covered agreed prompts.Source: §3.1 · p. 4, Figure 1 · p. 5

3.2 Agreement between two labellings

• Agreement is micro-averaged F1 over span sets summed across prompts, weighting long prompts more, with no chance-corrected coefficient because boundary choices make chance agreement undefined.Source: §3.2 · p. 5

3.3 Reference sensitivity

• The reference swap measures score movement at fixed output and tests ranking change; naive estimates retain realistic circularity, while decontamination removes prompts matching either scored labelling.The decontaminated subset is small (n = 135 pooled), so the authors draw only negative conclusions from it.Source: §3.3 · p. 5

3.4 Uncertainty

• Uncertainty uses 2,000 prompt-level paired bootstrap resamples with a fixed seed, stratified by language when pooled; pooled results do not replace per-language reporting.Source: §3.4 · p. 5

4.1 The delivered gold is not neutral between annotators

• The delivered gold matched Annotator A’s spans on 83.4% of prompts and B’s on 76.1%, a +7.3-point skew; reviewer gold had a pooled skew of +0.04.The delivered-gold preference for A appeared in all six languages, while reviewer-gold skews had no consistent direction.Source: §4.1 · pp. 5–6

4.2 Exchanging the reference moves the score

• 4.95 F1 points for Annotator A and 2.00 for Annotator B were the score movements; Hindi had the largest shift at 7.55 points.Each output was held fixed while the reference changed.Source: §4.2 · p. 6

4.3 Exchanging the reference changes the comparison

• +2.95 points was the pooled reference-by-system interaction (95% CI [+1.97, +3.87]); in Hindi, the margin changed sign, but the reverse leg was not significant.The pooled interaction remained significant after Holm correction.Source: §4.3 · p. 6

4.4 How much of this is circular?

• On 94.9% of audited prompts, shipped gold matched one annotator’s labels because an undocumented default usually kept A when adjudication did not fire.The reference was therefore partly copied from outputs being compared.Source: §4.4 · p. 7

4.5 A reference-sensitivity band

• The 6.3-point reference-sensitivity band is descriptive and uncorrected for selecting the largest of 24 deltas; B’s corresponding band is 3.4 points.Several observed margins are within the 6.3-point band, while three of seven exceed the stricter 3.4-point band.Source: §4.5 · p. 7

5 What This Means for Benchmarks

• The benchmark score reflects both the system and its reference, and the reference term can exceed every between-annotator margin measured in this paper.A position-based aggregation default made shipped gold a partial copy of one compared output, rewarding alignment with its boundary conventions without showing either annotator to be more correct.Source: §5 · p. 7, §5.1 · pp. 7–8

5.2 What we recommend

• Retain and release annotation records, report a reference-sensitivity band and between-system interaction, document and test aggregation neutrality, and report per-language agreement.These practices require no new annotation; the procedure transfers, but the reported band is pipeline-specific.Source: §5.2 · p. 8

5.3 Limitations

• The measured magnitude may not transfer to model outputs: the study scores two human passes from one record, and their sensitivity may differ from independently developed systems.The passes share guidelines, vendor, and protocol; the export also lacks annotator identifiers, precluding person-level conclusions.Source: §5.3 · p. 8

6 Conclusion

• The paired interaction was +2.95 points (95% CI [+1.97, +3.87]), survived multiple-testing correction, and the Hindi margin changed sign.The pipeline usually kept the first annotator when adjudication did not fire.Source: §6 · p. 8
• A reference-sensitivity band can be computed from a retained annotation record without new annotation and should be reported alongside the score.The paper says the procedure, not the size of the effect, is what transfers across pipelines.Source: §6 · p. 8

A.1 Neutrality of each reference

• Delivered gold favors Annotator A in all six languages, whereas reviewer gold shows no consistent direction.Table 3 compares how often each reference reproduces each annotator’s span set; a neutral reference would favor neither.Source: §A.1 · p. 10

A.2 Effect of exchanging the reference, all languages

• For Annotator A, exchanging the reference raises STRICT F1 in every language, with effects varying by roughly an order of magnitude; Swedish alone is indistinguishable from zero.Table 4 gives per-language movements for both annotators and matching rules with bootstrap confidence intervals.Source: §A.2 · p. 10

A.4 Ranking survival, STRICT matching

• Table 6 reports STRICT margins under each reference, paired swings, and Holm-adjusted p-values for the 14-test family.Its swing is the delivered-gold margin minus the reviewer-gold margin, recomputed within each bootstrap resample.Source: Table 6 · p. 11, §A.4 · p. 11

A.3 Score range available from reference choice

• Annotator A’s pooled STRICT F1 ranges from 84.54 to 91.62 across available labellings, a 7.08-point spread, reaching 12.46 in Arabic.With the two curated references, the pooled spread is 4.95 points; 13 of 14 cells peak against delivered gold, except Swedish B.Source: §A.3 · p. 11, Table 5 · p. 11

A.5 Ranking survival, both matching rules

• Under both matching rules, Hindi and Portuguese swing significantly, the other four languages are indistinguishable from zero, and Hindi changes sign.Table 7 repeats the ranking comparison under STRICT and RELAXED matching.Source: §A.5 · pp. 11–12, Table 7 · p. 12

B Two scoring conventions that void a metric

• The benchmark’s type merge collapses categories annotators are known to confuse, making that confusion unmeasurable and causing the benchmark to report agreement it did not obtain.Source: B Two scoring conventions that void a… · p. 12
• Exact string search makes partial overlap nearly unreachable; across 24 reported result blocks, partial-match and exact-match metrics are numerically identical.The benchmark effectively reported three matching rules rather than four, with the issue visible only in scoring code.Source: B Two scoring conventions that void a… · p. 12

C Additional result detail

• Under delivered gold, Annotator A’s lead excludes zero in Hindi, Portuguese, Swedish, and pooled; none of those leads excludes zero after reference exchange.The Portuguese margin moves from +4.59 to +0.18, Swedish from +4.47 to +0.94, and pooled from +2.83 to −0.12.Source: C Additional result detail · p. 12
• Polish never reverses and B leads under both references; pooled and Portuguese-RELAXED sign changes are ties, not reversals.The reviewer-gold margins are −0.12 and −0.05 points, respectively.Source: C Additional result detail · p. 12
• A merge collapses 25 fine-grained PII types into 17, making recurring confusion between two types unobservable after both become TAX NUMBER.The released vocabulary differs from the one annotators worked in.Source: C Additional result detail · p. 12

D Robustness checks and the duplicate defect

• The Hindi reversal persists under all four duplicate-handling policies: dropping duplicates, keeping the first, keeping the last, or retaining every copy.The Chinese reversal under one policy was instead spurious and arose from retaining defective duplicated items.Source: D Robustness checks and the duplicate… · p. 13
• Eleven duplicated items had reviewer annotations copied from one annotator with zero overlap with the other; the 22 affected rows were excluded.They were 0.8% of the audited subset, and no comparable defect was found elsewhere in the record.Source: D Robustness checks and the duplicate… · p. 13

E Definitions in full

• Micro-averaged F1 sums over prompts rather than averaging per-prompt scores, so long prompts carry more weight; the measure is symmetric between span sets.No chance-corrected coefficient is reported because freely chosen span boundaries in continuous text provide no fixed set of items for defining agreement chances.Source: E Definitions in full · p. 13
• Confidence intervals use 2,000 prompt-level bootstrap resamples, stratified by language for pooled results, with paired resamples reused across statistics.The method accounts for dependence among spans within a prompt; per-language results are reported rather than inferred from pooled results.Source: E Definitions in full · p. 13

F Invented illustrations of boundary disagreement

• The table uses invented prompts to illustrate boundary disagreements, not corpus text or values, and is not a corpus sample.All names, numbers, and addresses are fictitious.Source: F Invented illustrations of boundary… · p. 13, Table 8 · p. 13
• Each row shows only the disputed span; the prompt’s other spans are identical under both labellings.Source: Table 8 · p. 13
• The illustrative A/B assignment makes no claim about which annotator produced either labelling.Source: Table 8 · p. 13