Judging the Judges: A Collection of LLM-Generated Relevance Judgements

• Relevance judgments require substantial human effort and quality control, unlike the comparatively straightforward acquisition of corpora and topics.Editorial judgments are costly, crowd judgments can be noisy, and implicit feedback can carry noise, bias, and privacy concerns.Source: §1 · pp. 1–2
• LLMs are a promising fourth approach to relevance judgments, but prompt effects, bias, and evaluation circularity remain open questions [12] [20] [25] [34].The LLMJudge challenge was organized as a shared task to study LLMs as annotation tools.Source: §1 · p. 2
• The paper releases 42 pools of automatically generated relevance judgments from eight LLMJudge teams.The resource is intended to support investigation of LLM biases, ensemble effectiveness, trade-offs with human assessments, and improved automated evaluation methods.Source: §1 · p. 2
• The paper examines whether submissions maintain ranking consistency despite differing in their absolute scoring tendencies.It also evaluates approaches for assessing LLM-based relevance judgments and provides figures as baselines for future work.Source: §1 · p. 2

2 Related Work

• Prior work examines human–LLM collaboration, LLM judgment at scale, prompt sensitivity, and synthetic test collections with system-ordering performance similar to real collections.LLM-based autograding also evaluates generated content through question banks, sometimes with human supervision.Source: §2 · p. 2

2.1 Criticisms and Open Challenges

• LLM-generated relevance judgments may reflect biases conditioned by the context in which concepts appear, while bias measurement and mitigation remain open issues [36].The paper hopes the released collection will support research into addressing these biases.Source: §2.1 · p. 2
• Using the same LLM to generate relevance judgments and rank documents risks circular evaluation and can undermine judgment validity and generalizability [12] [32].Source: §2.1 · pp. 2–3
• LLM-based collection building may reduce monetary and temporal costs, but energy use, carbon emissions, and water consumption remain environmental costs [37] [38].The authors caution against relying on disposable relevance judgments.Source: §2.1 · p. 3
• Manipulating documents or runs can exploit LLM assessors; an adversarial run achieved higher effectiveness under fully automatic than manual evaluation [23, 1, 5].Examples include adding relevance-related keywords or query text to documents.Source: §2.1 · p. 3
• The authors release the collection to foster study of systematic errors and support more robust future LLM-assisted annotation.Source: §2.1 · p. 3

3 LLMJudge Resource

• The resource section introduces the challenge’s design, data construction, and evaluation metrics.Source: §3 · p. 3

3.1 LLMJudge Task

• Participants assign each query–passage pair a relevance score from 0 to 3, from irrelevant to perfectly relevant.The four categories distinguish no relevance, topical relation without an answer, partial or unclear answers, and an exact answer.Source: §3.1 · p. 3

3.2 LLMJudge Data

• The challenge dataset samples 25 queries for each of its development and test sets from the TREC 2023 Deep Learning track qrels.Each set includes 15 real queries, 5 T5-generated queries, and 5 GPT-4-generated queries; the test set is for judgment generation, while the development set may support few-shot learning or fine-tuning.Source: §3.2 · p. 3

3.3 Evaluation

• Evaluation measures both agreement between submitted and human judgments and correlation between the system rankings produced from those judgments.Cohen’s Kappa and Krippendorff’s Alpha assess label agreement; Kendall’s Tau and Spearman’s rank correlation assess system ordering.Source: §3.3 · p. 3

3.4 Publicly Available Resources

• The resource includes the dataset, sample prompt, starter materials, submitted runs, evaluation code, and detailed results on the LLMJudge webpage.The authors make these materials available to facilitate future research.Source: §3.4 · p. 4

Approach overview

• Submissions use direct or criteria-decomposed prompting, fine-tuning, multi-phase pipelines, aggregation, and classification to produce relevance judgments.The approaches include reasoning prompts, binary filtering before graded scoring, question- or nugget-based assessments, and classifiers trained on LLM outputs.Source: §4 · p. 4

Prompting methods

• Prompt-based methods range from direct scoring to criteria decomposition, semantic labels, reasoning, and multi-prompt aggregation.TREMA-4prompt assesses exactness, coverage, topicality, and contextual fit independently before a final prompt assigns the label.Source: §4 · pp. 4–6

Fine-tuning

• Fine-tuned approaches train models on relevance datasets, including TREC Deep Learning qrels and the LLMJudge development set.Prophet-setting tests three Llama-3-8B training-data settings; other submissions also test adaptation and different model sizes.Source: §4 · pp. 4–5

Aggregation and multi-phase methods

• Some submissions combine judgments by averaging prompt outputs, taking a conservative minimum, or filtering binary judgments before graded scoring.Olz-multiprompt averages nine scaled prompt outputs; William-umbrela3 takes the minimum of two approaches; RMIT-IR filters first, then averages three scores.Source: §4 · p. 5

Classification-based refinement

• Classifier-based approaches convert question, nugget, or direct-label features into final relevance labels.TREMA variants use ExtraTrees models, while Olz-halfbin uses logistic regression features from nine prompts and two model variants.Source: §4 · pp. 5–7

5.1 Descriptive Statistics

• The challenge received 42 runs from eight teams; five used fine-tuning, while 16 used proprietary and 26 used open-source LLMs.Nine organizer runs were included as baselines.Source: §5.1 · p. 7

5.2 Methodological Comparison

• Fine-tuning produced top agreement results in some runs, while TREMA-4prompts was the best non-fine-tuned method for system ranking correlation.h2oloo-fewself achieved the highest Krippendorff’s α (0.4958), and prophet-setting2 was the best-performing submission by Kendall’s τ; willia-umbrela1 achieved the best Cohen’s κ, with h2oloo-zeroshot1 second by 1.61%.Source: §5.2 · p. 7
• Methods with numerical and semantic labels tended to perform consistently across evaluation metrics, while fine-tuning and prompt design distinguished several top-performing submissions.TREMA-4prompts uses criteria decomposition; the results also report high correlations for some fine-tuned submissions.Source: §5.2 · p. 7

Overall results

• Top submissions exceeded 0.27 in Cohen’s κ and were around 0.49 in Krippendorff’s α, while their Kendall’s τ and Spearman’s ρ exceeded 0.9.h2oloo-fewself, h2oloo-zeroshot1, and willia-umbrela1 are identified among the top performers.Source: §5.3 · pp. 7–8

Agreement and ranking differences

• TREMA-nuggets and TREMA-rubric0 had Cohen’s κ below 0.1 and Krippendorff’s α below 0.2, despite some moderate ranking correlations.The paper describes these methods as having limited but specific utility in ranking-focused scenarios.Source: §5.3 · p. 8

Group trends

• NISTRetrieval runs had Kendall’s τ and Spearman’s ρ above 0.9 but lower agreement, with Cohen’s κ around 0.18 and Krippendorff’s α around 0.38.Olz-multiprompt and h2oloo-fewself are described as more comparable across metrics.Source: §5.3 · p. 8

5.4 Label Correlation (Cohen’s 𝜅)

• Across groupings, h2oloo-fewsel and h2oloo-zeroshot1 show relatively high agreement, while 4-point κ reaches only 0.0604 for TREMA-nuggets and 0.0779 for TREMA-rubric0.The reported low scores reflect limited reliability in their judgments.Source: §5.4 · p. 8
• Coarse 0|123 groupings tend to produce higher κ than the 4-point scale, but 012|3 increases variability as systems differ in consistency.willia-umbrela1 performs well on 012|3, while other systems struggle to maintain consistency.Source: §5.4 · p. 8

5.5 Cohen’s 𝜅vs. Kendall’s 𝜏

• Submitted runs cluster in a narrow range of Kendall’s τ but vary more in Cohen’s κ, indicating steadier system rankings than label agreement.Figure 2 plots Cohen’s κ against submission agreement on system order.Source: §5.5 · p. 8

5.6 Average Label Comparison

• For NDCG@10, most methods have ranking-correlation values generally above 0.85, yet their average labels vary around the human baseline of 0.90.TREMA-other and TREMA-4prompts assign higher average labels, while TREMA-rubric0 and prophet-setting4 assign lower scores.Source: §5.6 · p. 8
• Inter-method score-distribution agreement varies, and TREMA-4prompts and TREMA-other have higher average labels, underscoring the importance of calibration when aggregating synthetic judgments.Some methods may systematically overestimate or underestimate relevance scores despite high ranking agreement.Source: §5.6 · p. 8

5.7 Krippendorff’s 𝛼vs. Kendall’s 𝜏

• Higher Krippendorff’s α generally corresponds to better Kendall’s τ, but the relationship varies across models.The figure reports strong ranking consistency for prophet-setting2 and TREMA-4prompts, and weaker agreement for TREMA-rubric0 and TREMA-other.Source: §5.7 · p. 9

5.8 Binarized Cohen’s 𝜅vs. Kendall’s 𝜏

• Higher binary Cohen’s κ tends to align with stronger Kendall’s τ, though TREMA-rubric0 and TREMA-other show weak agreement despite moderate ranking correlations.prophet-setting2, TREMA-4prompts, and RMITIR-llama70B show the stronger alignment described.Source: §5.8 · p. 9

5.9 LLMJudge Resource Use Cases

• JudgeBlender has used LLMJudge as a baseline and to analyze variability and bias in ensemble-based judgment aggregation.JudgeBlender aggregates evaluations from multiple smaller models to enhance relevance-assessment robustness.Source: §5.9 · p. 9

6 Conclusion

• The resource contains 42 sets of judgments from eight international teams and supports comparisons of LLM-based assessments and prompting strategies.Source: §6 · p. 9
• Fine-tuning can lead to the highest correlation among the submitted judgment sets.Five submissions use fine-tuning.Source: §6 · p. 9
• Open-source submissions tend to be more stable, while proprietary-model submissions are affected by higher variability.Source: §6 · p. 9
• The paper provides strategies for comparing automatically generated relevance assessments to help practitioners evaluate new LLM assessors.Source: §6 · p. 9
• Future work will examine nugget-based relevance judgments and fully automatic collections with generated queries and documents.Source: §6 · p. 9