Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge

• Existing benchmarks often omit real-world guidelines and provide only one gold label, obscuring whether model judgments align with valid human disagreement on boundary cases.Source: §1 · p. 1
• ANT-OR uses agent-level contrastive knowledge to update decision heuristics and improve future judgments and inter-agent coordination.Source: §1 · p. 2
• ANT-OR outperforms the strongest baseline by 4.5 points in accuracy and achieves the highest alignment with human annotator distributions, measured by Fleiss’ κ, within 2 points on average.Source: §1 · p. 2
• ANT-A combines 7 datasets, detailed guidelines, and 5 independent human labels per sample to support distribution-aware evaluation using Fleiss’ κ.Source: §1 · p. 2

• Existing benchmarks commonly omit detailed task guidelines or reduce each sample to one gold or aggregated label, limiting evaluation of guideline following and human-label distributions.Source: §2 · p. 2
• Prior disagreement-modeling methods predict individual labels or distributions, but cited approaches are tied to particular tasks and do not target cross-domain, guideline-grounded settings.Source: §2 · p. 2
• Prompt optimization methods often lack fine-grained error analysis or cross-iteration knowledge retention, while instance-level error memory does not abstract structural error patterns.Source: §2 · p. 3
• ANT-OR distinguishes itself by mining structured decision rules from iterative error-success contrasts for multi-agent adaptation without model training.Source: §2 · p. 3

3 ANT-A Benchmark

• ANT-A jointly provides detailed guidelines and 5 independent human judgments per sample across a multi-source benchmark, reflecting task ambiguity and annotation subjectivity.Source: §3 · p. 3

3.1 Dataset Preparation

• Each dataset contains 1,500 domain-relevant samples from public internet sources, filtered for duplicates and low-quality instances and including clear-cut and boundary cases.Source: §3.1 · p. 3
• Guidelines are refined from real-world requirements by clarifying ambiguous definitions, adding boundary-case rules, and specifying fine-grained decision boundaries and edge cases.Source: §3.1 · p. 3

3.2 Human Annotation Process

• A pilot with all 10 annotators surfaces ambiguous or underspecified rules, which are addressed through structured Q&A and guideline revision before formal annotation.Source: §3.2 · pp. 3–4
• For formal annotation, each sample is independently labeled by 5 of 10 annotators, who cannot access one another’s labels.Source: §3.2 · pp. 3–4

3.3 Benchmark Statistics

• Fleiss’ κ varies across the 7 datasets, reflecting differences in task and domain difficulty; samples with five completely different labels are nearly 1%.Source: §3.3 · p. 4

4.1 Overview

• ANT-OR uses contrastive evidence from negative and similar positive samples to uncover criteria that distinguish confusable categories in boundary cases.Source: §4.1 · p. 4

4.2 Contrastive Knowledge Mining

• Each agent maintains positive and negative sample pools, with a prediction counted as positive when it matches at least two of five human labels.Source: §4.2 · pp. 4–5
• From an error and two semantically similar correct samples, an LLM extracts a structured dimension containing category-specific criteria and applicable scenarios.Source: §4.2 · p. 5
• Similar dimensions are fused to reduce redundancy, and only dimensions with support counts of at least 2 are retained for traceability to source samples.Source: §4.2 · p. 5

4.3 Knowledge-Augmented Reasoning

• Three annotators independently produce labels and rationales using separately optimized prompts, then a verifier compares their justifications to select the final label.Source: §4.3 · p. 5
• ANT-OR retrieves samples using semantic similarity and historical reliability, then injects both concrete examples and associated discriminative rules into prompts.Source: §4.3 · p. 5

4.4 Attribution-Guided Optimization

• When an output matches fewer than two of five human references, an attribution model analyzes the query, interaction history, and majority label to identify responsible agents and error reasons.Source: §4.4 · pp. 5–6
• Flagged-agent prompts are refined using failure reasons, a relevant positive sample, and its discriminative dimensions; updates are retained only when batch performance improves, while verified outcomes expand the knowledge base.Source: §4.4 · p. 6

5.1 Experimental Setup

• The experiments compare single-agent and multi-agent baselines, using agreement@≥2 and Fleiss’ κ; Medical Entity Extraction uses Entity-Level F1 instead of κ.ANT-OR uses three LLM-based annotators and one verifier; models are evaluated on 1,000 test samples per task.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2 Main Results

• ANT-OR achieves the best overall performance across all seven datasets, beating the strongest baseline by 4.5 points in average agreement@≥2 accuracy.The gains are consistent across medical, e-commerce, and general-domain tasks, as well as text and vision-language modalities.Source: Table 2 · p. 7, §5.2 · p. 7

5.4 Analysis

• As human agreement decreases, all methods lose accuracy; on instances with at most 2/5 agreement, ANT-OR beats MAS by nearly 8 points.SPO provides limited gains on these highly ambiguous cases.Source: Figure 4 · p. 8, §5.4 · p. 8
• ANT-OR averages 3.7 percentage points higher test performance than SPO, despite SPO’s higher training accuracy, including 98.5% versus 96.5% on BillSC.The authors interpret the test-set comparison as evidence that ANT-OR generalizes better to unseen instances.Source: Figure 5 · p. 8, §5.4 · p. 9

5.3 Ablation Study

• Removing dimension construction lowers average accuracy from 71.75 to 68.05 and average κ from 61.37 to 60.22; removing contrastive triples also reduces both averages.Without contrastive triples, average accuracy falls from 71.75 to 69.47 and average κ from 61.37 to 60.74; larger dimension-mining declines on BillSC and SocialTC suggest decision-level abstractions matter beyond instance-level corrections.Source: Table 3 · p. 7, §5.3 · p. 8
• Coupled injection raises overall accuracy from 69.53 to 71.75 and κ from 61.04 to 61.37, while retrieval gains vary by task and injected-sample count has little effect.Quality-aware similarity retrieval performs best overall and random selection performs worst; retrieved examples can add noise on rule-based tasks.Source: Table 3 · p. 7, §5.3 · p. 8
• Using two positive samples yields clear average agreement@≥2 gains over one, while a third brings only marginal improvement, supporting the reported 2:1 positive-to-negative ratio.The authors describe two positives as sufficient to stabilize decision patterns without unnecessary cost.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Analysis

• ANT-OR costs $101.43 on average versus $65.94 for SPO, alongside a 4.7-point accuracy improvement, from 67.08% to 71.75%.The additional cost comes from multi-agent contrastive mining, knowledge-base construction, and sample injection.Source: Table 5 · p. 9, §5.4 · p. 9

6 Conclusion

• ANT-A spans seven datasets across domains and modalities, with detailed guidelines and five independent human annotations per sample for evaluating inter-annotator agreement.It supports distribution-aware evaluation using Fleiss’ κ.Source: §6 · p. 9
• ANT-OR iteratively optimizes prompts using a structured knowledge base and contrastive patterns to mine discriminative dimensions and produce reusable decision rules.The framework turns collective annotation experience into guidance for annotation decisions.Source: §6 · p. 9
• Experiments show ANT-OR narrows the gap to human-level consistency and consistently outperforms single-agent and multi-agent baselines across diverse tasks.Source: §6 · p. 9

Limitations

• ANT-A covers only a subset of real-world annotation settings, leaving long-document annotation, open-ended judgment, and specialized expert workflows as areas for broader evaluation.Source: Limitations · p. 9
• Multi-agent collaboration and iterative optimization add computational overhead, motivating work on more efficient optimization and inference strategies.Source: Limitations · p. 9
• ANT-OR depends on backbone models for retrieval, reasoning, and prompt refinement; its scaling and transfer across model families and settings remain to be studied.Source: Limitations · p. 9

Ethics Statement

• Personally identifiable or sensitive information is removed or anonymized before benchmark release.Source: Ethics Statement · p. 9
• Professional annotators receive task guidelines and opportunities to ask for clarification.Source: Ethics Statement · p. 9
• ANT-OR is not intended to replace human experts in high-stakes medicine, finance, or legal decision-making.Source: Ethics Statement · p. 9
• The authors caution that LLM annotations may contain errors or biases and call for task-specific privacy, bias, and reliability assessments before deployment.Source: Ethics Statement · p. 9

Debate-based Architecture

• Debate performs poorly on guideline-governed annotation: discussions can drift, encourage groupthink, and incur costs that scale quadratically with debate rounds.Source: A Multi-Agent Architecture Comparison · p. 14

Planner-Worker Architecture

• Planner-worker underperforms single-agent baselines on all three reported tasks: -1.1% on MedEE, -3.7% on DisPA, and -0.4% on BillSC.The passage attributes the failure possibly to coordination noise: workers receive truncated context for interdependent rules, while aggregation lacks guideline-grounded adjudication; costs are 34–50% lower than full debates.Source: A Multi-Agent Architecture Comparison · p. 14

Annotator-Verifier Architecture

• Annotator-verifier design improves accuracy by 1–2% over baselines with a 3–4× cost increase, versus 10–50× for alternatives.Parallel annotators diversify perspectives, and a dedicated verifier adjudicates conflicts using guideline-grounded reasoning.Source: A Multi-Agent Architecture Comparison · p. 14

Case Study

• The appendix presents two representative Billing Scenario Classification cases to illustrate benchmark realism and ANT-OR's qualitative improvements.The task guideline is provided alongside the case tables.Source: B Case Study · p. 14

Case 1: A boundary instance with human disagreement

• A boundary instance shows annotators can disagree while following the same guideline, a difficulty a single gold label cannot fully capture.Five independent annotations make the ambiguity explicit for more realistic system evaluation.Source: B Case Study · p. 14

Case 2: Improvement after optimization

• On another difficult billing-classification example, ANT-OR gives the correct label by following the guideline more precisely than a vanilla multi-agent system driven by coarse lexical or topical cues.Source: B Case Study · p. 14

C Benchmark Construction and Annotation Details

• The appendix provides comprehensive supplementary details about ANT-A's construction, annotation process, and characteristics.Source: C Benchmark Construction and… · p. 14

C.1 Dataset Overview and Task Selection

• ANT-A’s seven tasks span modalities, domains, and scenarios, pairing real-world-derived annotation guidelines with queries collected and reannotated for those tasks.Examples include billing-category identification and image-text relevance for image search; queries are manually annotated according to the guidelines.Source: §C.1 · pp. 14–15

C.2 Annotation Guidelines and Query Characteristics

• Lower agreement in SocialTC and ImageAR reflects defensible competing labels on ambiguous semantic boundaries that single-label benchmarks collapse.The benchmark retains five independent annotations to represent uncertainty in hierarchical semantic classification, subjective social-text content, and visual identifiability.Source: §C.2 · p. 16

C.3 Annotation Process

• About 1% of instances have five completely different labels on average, suggesting disagreements mostly persist at boundary cases after guideline refinement.The three-stage process uses roughly 50 pilot samples to expose ambiguities and revise guidelines before five annotators independently label each formal sample.Source: §C.3 · p. 16, Table 9 · p. 17

C.4 Annotator Demographics and Quality Control

• Annotators hold bachelor’s degrees and at least two years of experience, with competence checked in piloting; the medical-related tasks require no professional medical expertise.Those tasks use explicit textual patterns and predefined schemas, and annotators can perform them after training on the detailed guidelines.Source: §C.4 · pp. 16–17

C.5 Qualitative Analysis of Prompt Optimization

• The billing-task prompt evolves from a simple industry-keyword rule to a two-tier rule that handles generic merchant names and ambiguous transaction content.The refined rule prioritizes explicit industry signals but assigns “Other” when merchant names are generic and transaction content is ambiguous, aiming to prevent false positives and false negatives.Source: §C.5 · p. 17

D Prompts

• The prompt suite combines base annotation and contrastive, dimension-aware examples with agents for verification, contrastive-dimension mining, and prompt refinement.Contrastive triples reveal rule intent and decision boundaries; refinement integrates new cases with existing dimensions, while optimization prompts use error analysis to improve annotator or verifier instructions.Source: D Prompts · pp. 17–18
• When aggregating annotations, the prompts compare candidate reasoning and select the strongest rationale when a three-way tie prevents majority resolution.The aggregation template also supplies discriminative dimensions and similar correct or incorrect examples as references.Source: D Prompts · pp. 24–26
• Contrastive mining derives reusable rules by comparing similar correct and incorrect cases for their surface differences, decision boundaries, intent, and applicability.It checks for an existing historical dimension before naming a new one and cautions against overly absolute logic.Source: D Prompts · pp. 27–28
• Rule refinement classifies new cases as supporting, supplementing, correcting, or extending historical rules, then tests generalizations against conflicts and counterexamples.The prompts also call for extracting task-level logic without over-speculating and tightening applicability boundaries when needed.Source: D Prompts · p. 30
• Prompt optimization prioritizes observed error samples, integrates related suggestions into existing prompt content, and checks revisions for contradictions.Learned dimensions and contrastive examples are auxiliary references rather than the main optimization basis.Source: D Prompts · pp. 31–33
• Prompt revisions should analyze each error sample, use dimensions and contrastive examples as references, integrate suggestions into existing content, and check for contradictions.The principles prioritize fixing error causes over adding patch paragraphs and recommend extending or rewriting existing prompt content.Source: D Prompts · p. 34
• Each response must provide a sample-by-sample root-cause analysis and reference-informed optimization suggestions, followed by a complete, directly usable optimized prompt.Source: D Prompts · p. 34