PaperlayerView exact revision on arXiv

Source-linked AI summary

The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs

Nitay Calderon, Roi Reichart, Rotem Dror

arXiv:2501.10970v4cs.CLcs.AIcs.HC

TL;DR

Because LLM annotations increasingly shape research while existing measures do not provide rigorous replacement decisions, the paper proposes the Alternative Annotator Test and a complementary judge-comparison measure. Evaluated across ten datasets, six LLMs, and four prompting techniques, the results show that some LLMs can replace humans, with closed-source models generally outperforming the examined open-source models.

  • Problem

    Researchers lack a standard, rigorous procedure for deciding whether LLM annotations can replace human annotations despite LLMs’ growing research role and reported biases.

  • Method

    The paper proposes the alt-test, which compares an LLM with a small group of human annotators on a modest annotated subset, alongside the average advantage probability for comparing judges.

  • Results

    Across ten datasets, at least one LLM passed the alt-test on nine datasets, while closed-source models consistently outperformed the examined open-source models.

  • Takeaways & Limitations

    The findings indicate that LLMs can sometimes serve as alternatives to recruiting additional human annotators, supporting more rigorous evaluation of such use.

  • Takeaways & Limitations

    The experiments may be affected by data contamination because some datasets could overlap with evaluated LLMs’ training data.

Abstract

from arXiv · show

The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are widely used, not only in NLP research but also in fields like medicine, psychology, and social science. Despite their role in shaping study results and insights, there is no standard or rigorous procedure to determine whether LLMs can replace human annotators. In this paper, we propose a novel statistical procedure, the Alternative Annotator Test (alt-test), that requires only a modest subset of annotated examples to justify using LLM annotations. Additionally, we introduce a versatile and interpretable measure for comparing LLM annotators and judges. To demonstrate our procedure, we curated a diverse collection of ten datasets, consisting of language and vision-language tasks, and conducted experiments with six LLMs and four prompting techniques. Our results show that LLMs can sometimes replace humans with closed-source LLMs (such as GPT-4o), outperforming the open-source LLMs we examine, and that prompting techniques yield judges of varying quality. We hope this study encourages more rigorous and reliable practices.

1 Introduction

LLMs are increasingly used for annotation and evaluation because they offer speed and scalability, but biases and expertise gaps make rigorous replacement tests necessary. The paper proposes the alt-test and average advantage probability, then evaluates them across diverse datasets, models, and prompting strategies.

  • LLM annotations offer scalability and speed, but studies report bias, inconsistent judgments, and difficulty with deep contextual or domain-specific tasks.
  • LLMs are increasingly used to annotate datasets, evaluate models, and support research beyond NLP.
  • Traditional agreement and correlation measures lack established thresholds for deciding whether an LLM can replace humans or provide a better alternative.
  • The alt-test compares an LLM with at least three human annotators on 50–100 examples to justify replacement using modest effort.
  • The average advantage probability measures how often LLM annotations are at least as good as those of randomly chosen human annotators and is versatile and interpretable.
  • Across ten datasets, six LLMs, and four prompting techniques, at least one LLM passed the alt-test on nine datasets; closed-source models generally outperformed examined open-source models.

2 Previous Work

Prior work largely improves LLM annotators through tuning and prompting or studies their alignment with humans, while not directly establishing when human replacement is justified.

  • Existing research on LLM judges commonly focuses on parameter tuning and prompting strategies to improve performance.
  • Related studies have explored personalized judges, diverse LLM panels, and multimodal extensions of LLM-as-a-judge.
  • Statistical research has corrected estimations built with LLM annotations, but this paper addresses whether LLMs can replace human annotators.
  • Many studies measure alignment between LLMs and humans while emphasizing specific limitations or biases, rather than directly justifying human replacement.

3 Method

The method treats collective human responses as an approximation to unavailable gold labels and compares LLM predictions with human annotators to assess replacement suitability.

  • The paper compares LLM predictions with human responses to determine which more closely emulates the gold-label distribution.
  • Because expert gold labels are often unavailable, the method approximates them using the collective responses of multiple annotators on a randomly sampled subset.
  • The procedure excludes each annotator in turn and estimates whether the LLM aligns better with the remaining annotators than the excluded one, or vice versa.

2. Conduct hypothesis tests to compare the probabilities and obtain p-values. 🤖

The alt-test compares an LLM’s alignment with the remaining human annotators against each excluded annotator, tests the resulting advantage probabilities with a cost-benefit penalty, and controls multiple comparisons. It aggregates rejected hypotheses into a winning rate, using ω ≥0.5 as the criterion for justifying LLM annotation.

  • Conduct hypothesis tests: Leave-one-out comparisons evaluate how closely the LLM and each excluded annotator align with the remaining annotators.The alignment score can use ACC, negative RMSE, or a task-relevant similarity metric, depending on the annotation task.
  • Conduct hypothesis tests: The test compares the LLM’s advantage probability with the excluded annotator’s probability while incorporating the cost-benefit penalty ε.The penalty reflects the higher cost and effort associated with human annotation.
  • Conduct hypothesis tests: A paired t-test assesses whether the LLM’s relative advantage is statistically significant; for n < 30, a non-parametric alternative is recommended.The procedure rejects the null hypothesis when p-value < α, typically α = 0.05.
  • Conduct hypothesis tests: The winning rate ω is the proportion of rejected hypotheses, and ω ≥0.5 indicates that the LLM wins a majority of human annotators.The alt-test then statistically justifies using the LLM’s annotations.
  • Conduct hypothesis tests: False Discovery Rate control addresses Type-I error accumulation and dependence among the multiple annotator-specific hypotheses.The recommended procedure is the Benjamini-Yekutieli procedure.
  • How to Compare LLM Judges?: The Average Advantage Probability ρ summarizes the probability that LLM annotations are at least as good as those of a randomly chosen annotator.Unlike ω, ρ spans a denser range and applies across discrete, continuous, and free-text tasks.

4 Experimental Setup

The experiments evaluate six LLMs and four prompting strategies on ten diverse datasets spanning language and vision-language tasks. The datasets vary in annotation format, annotator expertise, size, and annotator coverage, while results report both traditional alignment and proposed measures.

  • Datasets: The study uses ten datasets covering sentiment analysis, word-relation labeling, summarization evaluation, conversation comparison, prompt quality, emotional support, and two vision-language tasks.The vision-language datasets concern skin lesion examination and abstract visual reasoning.
  • Datasets: Datasets include discrete, continuous, and free-text annotations, with multiple identified annotators contributing to each item.These selection criteria support comparisons that retain individual annotator perspectives.
  • LLMs: The six candidate models are Gemini-1.5-Flash, Gemini-1.5-Pro, GPT-4o, GPT-4o-mini, Llama-3.1-7B-Instruct, and Mistral-7B-Instruct-v0.3.Llama-3.1 and Mistral-v0.3 lack results for the image datasets because they cannot process images.
  • Prompting strategies: The prompting strategies are Zero-shot, Few-shot with four demonstrations, Chain-of-Thought, and Ensemble prediction.The Ensemble strategy determines the final prediction using a majority label across outputs.

5 Results

Across datasets and annotation aspects, LLMs sometimes pass the alt-test, but success varies by task, model, and prompting strategy. Few-shot prompting generally helps, while test reliability depends on the number of instances and the average advantage probability provides a complementary comparison measure.

  • LLMs can sometimes replace humans: On nine datasets, at least one LLM with some prompting technique passed the alt-test, while no LLM passed on MT-Bench or SummEval.In Framing, CEBAB-A, CEBaB-S, and Lesion, almost all LLMs achieved ω ≥0.5; only Gemini-Flash passed on KiloGram.
  • Aspect-dependent performance: LLM test success depends on the dataset and annotation aspect, so passing the alt-test is not universal.In Lesion, models passed color-related aspects but struggled with shape-related aspects; in SummEval, most passed Coherence and Relevance but failed on two other aspects.
  • Comparing LLM judges: The average advantage probability ρ usually ranked models similarly to traditional measures, with average Kendall τ values of 0.92 for discrete tasks and 0.84 for other tasks.SummEval was an exception, showing a negative correlation that the authors discuss separately.
  • Model and prompting effects: Closed-source LLMs outperformed the examined open-source models, while in-context learning generally improved performance and chain-of-thought often reduced it.CoT led to declines in 45% of cases, and ensembling did not improve the few-shot approach without ensembling.
  • Model and prompting effects: Few-shot prompting with four demonstrations improved nearly all LLM judges, including two models that achieved ω ≥0.5 on SummEval.The authors attribute this improvement to demonstrations aligning LLM scoring distributions more closely with human distributions.
  • The number of instances: The winning rate ω depends strongly on the number of instances, whereas ρ is expectation-stable but becomes less variable with more instances.More instances increase statistical-test power and reduce the variance of ρ.
  • The number of instances: For ε = 0.2, LLMs generally began passing before 100 instances, sometimes before 50; for ε = 0.1, they typically required 100–150 instances.At least 30 instances are needed to satisfy the t-test’s normality assumption, but LGBTeen, MT-Bench, and SummEval remained failures regardless of sample size.

6 Conclusion

The paper concludes that the alt-test offers a simple way to justify LLM annotations using a small human-annotated subset. It recommends transparent, careful evaluation because suitability varies across tasks and research settings.

  • Conclusion: The alt-test uses at least three human annotators and a subset of 50 to 100 examples to justify using LLM annotations.The required subset may depend on task complexity.
  • Conclusion: The paper provides extensions for imbalanced labels, single-expert benchmarks, annotator quality scores, minority opinions, and testing whether LLMs outperform humans.These modifications are presented in Appendix D.
  • Conclusion: The authors encourage researchers to adopt the procedure and disclose annotator information and human annotations for more reliable and transparent evaluations.Publishing human annotations is intended to support reproduction and validation of results.

7 Limitations

The paper identifies contamination, weak or unreliable human comparators, low agreement, and subjectivity as important boundaries for interpreting or applying the alt-test. It also provides reporting and parameter-selection guidance to improve reliability and transparency.

  • Data contamination: Data contamination may occur when evaluated datasets overlap with LLM training data, although the study notes that most datasets are recent or less widely known.SummEval and MT-Bench are publicly available and may have been included in training data; LGBTeen is available only upon request.
  • Subjectivity: For inherently subjective tasks, disagreement may represent a spectrum of opinions rather than annotation error, motivating methods that account for minority views.The paper points to modifications of the procedure for better accounting for subjectivity and minority opinions.
  • Human annotator quality: Low human-annotator agreement makes alt-test conclusions less reliable and requires ε ≤0.1 plus more annotated instances.The procedure may be less likely to pass when disagreement reflects unclear guidelines, unqualified annotators, or task subjectivity.
  • Human annotator quality: Intentionally comparing an LLM with weak human annotators can make the LLM appear better and constitutes a potential misuse of the procedure.Researchers should report annotator expertise and IAA so readers can assess whether the comparison is appropriate.
  • Reporting: Researchers should report annotator profiles, ε rationale, sample-size rationale, human reliability, candidate ρ values, and the selected LLM’s winning rate.These practices support transparent interpretation and reproducibility of alt-test results.
  • Procedure assumptions: The procedure is less reliable with only two annotators because it lacks a consensus signal and is more sensitive to individual biases.The paper recommends at least three annotators, with more annotators improving reliability and robustness.
  • Parameter selection: Recommended ε values are 0.2 for experts, 0.15 for skilled annotators, and 0.1 for crowd-workers or low-reliability annotators.The choice should also reflect human-annotator reliability, with smaller ε when IAA is low.
  • Sample size and ε: At least 30 instances are recommended to support the t-test’s normality assumption, while 50–100 instances are described as a modest practical subset.The paper also studies ε’s effect on winning rate and identifies an effective range between 0.05 and 0.3.

B.3 Case study: SummEval

The SummEval case study shows that alt-test outcomes can differ sharply across annotation aspects and that distributional imbalance can produce unusual advantage-probability results. The paper uses this case to motivate aspect-specific testing and an imbalance adjustment.

  • Case-study result: Mistral-v3 achieves the highest ρ on SummEval despite a low Pearson correlation of 0.12.Mistral passes only the Consistency aspect, where ρ = 0.87 compared with approximately 0.45 for other LLMs.
  • Aspect-specific analysis: SummEval performance varies by aspect, demonstrating why coherence, consistency, fluency, and relevance should be tested separately.The reported pass is specific to Consistency rather than the full annotation task.
  • Label imbalance: 89% of human Consistency annotations receive score 5, and Mistral is the only LLM with a similarly skewed prediction distribution.Other LLMs predict score 5 only about 30% of the time, helping explain Mistral’s high ρ.
  • Label imbalance: Few-shot prompting helps LLMs adjust and skew their output distributions, improving alignment in this case.The case motivates an Inverse Probability Weighting adjustment for label imbalance.

C Simulations

The simulations vary annotation noise, category count, sample size, and ε to examine when the alt-test passes. They show that required sample size depends on the relative quality of the LLM and human annotators, overall noise, and task cardinality.

  • Simulation design: The simulations control human noise ηh, LLM noise ηf, number of categories K, sample size n, and ε to assess alt-test behavior.Figures report winning rate ω, average advantage probability ρ, empirical 0.9 confidence intervals, human IAA Cohen’s κ, and LLM accuracy against majority vote.
  • Practical guidance: The simulation framework can guide how many human annotations to collect based on expected LLM quality and human reliability.The authors provide simulation code for further use by researchers.
  • Simulation design: 2,500 bootstraps aggregate each simulated data point across 50 datasets and 50 resamples of items and three human annotators.The simulation uses six human annotators and 500 instances per dataset before resampling.
  • Noise dynamics: A stronger LLM requires fewer instances, and a sufficiently large advantage can allow passing even at ε = 0.When the noise gap exceeds 0.3, the LLM can pass with smaller ε values; equal-noise settings pass at ε = 0.2 with fewer than 50 instances but not for ε <0.1.
  • Noise dynamics: Higher noise requires more instances, so the alt-test discourages comparisons between low-quality LLMs and low-quality annotators.The paper interprets this as a need for greater statistical power when both annotator and LLM noise are high.
  • Human reliability: More reliable human annotators require more instances because they provide a stronger baseline against which the LLM must compare.Lower human noise corresponds to higher reliability and IAA, making it harder for the LLM to pass.
  • Number of categories: With reliable annotators, increasing the number of categories requires more instances for the LLM to pass when ηh ≤0.2 and IAA ≥0.6.The passage contrasts this with less reliable annotators, where increasing category count has a different effect.

D.1 Handling Imbalanced Labels

The paper addresses label imbalance by modifying the evaluation so overrepresented majority classes contribute less, while retaining a flexible alignment function. This helps prevent misleading assessments based on dominant labels.

  • 0.89 accuracy is achieved by an annotator always assigning ‘5’ in SummEval Consistency, illustrating how imbalance favors majority-class predictions.
  • Pearson and Spearman correlations of -0.03 and -0.04 can penalize annotators despite substantial label overlap with others.
  • An LLM that always predicts ‘5’ can pass the original test because at least 89% of labels are ties.
  • Inverse probability weighting assigns each annotation the inverse probability of its majority-vote label, denoted πy,j.
  • The weighted formulation mitigates class overrepresentation so each class contributes equally to ρf,π, with corresponding variance and effective-sample-size adjustments.
  • The alignment score can use task-specific functions, such as weighted accuracy for hate-speech labels, while the rest of the procedure remains unchanged.

E The Optimal LLM-as-a-Judge

The paper characterizes the optimal LLM-as-a-judge for a chosen alignment score. Predicting the majority vote optimizes accuracy, while predicting the annotation mean optimizes RMSE, achieving ρ = 1 in both cases.

  • The optimal judge depends on the scoring function: majority-vote predictions maximize ACC, whereas mean predictions minimize RMSE.
  • ρ = 1 is achieved by both score-specific optimal judges, for every excluded annotator.
  • For ACC, the majority vote is formed from the annotators’ labels, with ties resolved by random sampling among tied labels.
  • Excluding an annotator who disagrees with the majority leaves the majority count unchanged while reducing the competing annotator’s matching count.
  • For RMSE, the arithmetic mean uniquely minimizes the sum of squared errors over the annotator labels.

F Datasets •

The evaluation spans diverse language, vision-language, classification, ranking, rating, and free-text annotation datasets. These datasets provide varied settings for testing LLM judges against human annotations.

  • The collection includes WAX relation labeling, where annotators select one of 16 predefined relations from word associations and explanations.
  • SummEval evaluates summaries on coherence, consistency, fluency, and relevance using expert ratings from 1 to 5.
  • LGBTeen uses expert assessments of queer-youth responses across ten questions with categories Irrelevant, No, Partially, and Yes.
  • MT-Bench contains 80 multi-turn questions across eight categories, evaluated by experts for conversational and instruction-following ability.
  • Lesion, Framing, and CEBaB cover medical-image features, climate-change narrative framing, and restaurant-review aspect sentiment or star ratings.
  • 10K Prompts contains Likert-scale rankings from 1 to 5 by 13 HuggingFace community annotators.
  • KiloGram contains tangram images with short free-text descriptions, compared using SentenceTransformer cosine similarity and e5-large-v2 embeddings.

G Additional Results

Additional results report aspect-level evaluation and task-specific prompting formats. The materials define varied annotation prompts and summarize metrics for comparing LLMs with human judgments.

  • Table 5 reports Acc, Pears, winning rate ω, and average advantage probability ρ across annotation aspects in SummEval, Lesion, and LGBTeen.
  • Table 6 indicates that few-shot prompting improves alignment between LLM and human annotation distributions when human scores are highly skewed.
  • Prompts: The WAX prompt asks the model to infer a relation type from a cue, association, and explanation, returning the relation name in JSON.
  • Prompts: The LGBTeen prompt evaluates a queer teenager’s post and response using the categories Irrelevant, No, Partially, Yes, and Ignore.
  • Prompts: The MT-Bench prompt compares two model conversations beginning with the same user prompt and outputs model_a, model_b, or tie.
Loading 2501.10970v4…