Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

1 Introduction

• Position effects, verbosity preferences, and prompt sensitivity have raised concerns about the reliability of LLM-based judgments.Prior work identifies these issues in automated evaluation (Shi et al., 2025; Li et al., 2026; Tan et al., 2025).Source: §1 · p. 1
• The controlled experiment tests whether evaluative signal can be recovered from a rubric without the candidate response or its context.This isolates the rubric’s contribution from biases arising during response evaluation.Source: §1 · p. 1
• Rubric text alone predicts judge outputs, suggesting an association between rubric formulation and evaluation outcomes.Embedding-geometry analysis suggests the rubric carries evaluative signal.Source: §1 · p. 1

2 Related Work

• Prior work identifies position effects, scoring bias, prompt framing, and other nonsemantic factors as concerns for model-based judgment reliability.The cited studies include Zheng et al. (2023), Shi et al. (2025), Chen et al. (2024), Li et al. (2026), Tan et al. (2025), and Gu et al. (2026).Source: §2 · p. 1
• In HEALTHBENCH and RESEARCHRUBRICS, rubrics are generally treated as evaluation instructions rather than potential sources of signal.The benchmarks use expert-authored medical rubrics and instance-specific research rubrics, respectively (Arora et al., 2025; Sharma et al., 2026).Source: §2 · p. 1
• Figure 1 frames the rubric probe as testing whether rubric text alone predicts judge labels beyond chance.This probe connects to shortcut-learning research on exploiting superficial cues rather than intended semantic signal (Zhou et al., 2024; Ong Ly et al., 2024).Source: §2 · p. 1, Figure 1 · p. 2

3 LLM-as-a-Judge Probe Design

• The probe tests whether rubric text predicts judge labels without access to the candidate response or input context.Figure 2 shows the classifier receiving only rubric text while excluding the evaluated answer, contextual metadata, and grounding information.Source: §3 · p. 2, Figure 2 · p. 2
• Rubric-based evaluation is intended to condition judgment, not independently determine the label; accordingly, the design treats predictability from rubric alone as a validity concern.The stated expectation is p(y | r) ≈ 0.5.Source: §3 · p. 2
• Beyond predictive performance, the study examines rubric-signal structure through generalizability studies and embedding-space geometry..Source: §3 · p. 2

4.1 Experiment Details

• HEALTHBENCH includes clinician-provided, conversation-specific rubrics and 5k conversations, while RESEARCHRUBRICS supplies instance-specific rubrics for research queries.The probe uses candidate-model outputs judged with binary labels; the complementary benchmarks span healthcare and non-medical settings.Source: §4.1 · pp. 2–3

4.2 Rubric Probe Results

• >0.5 classifier outputs, sometimes exceeding 0.8, are observed after accounting for class imbalance; a similar beyond-chance pattern appears on RESEARCHRUBRICS.The results cover HEALTHBENCH cohorts and RESEARCHRUBRICS; Figure 3 presents per-metric comparisons across datasets and candidate models.Source: Figure 3 · p. 3, §4.2 · pp. 3–4

4.1 Experiment Details

• The PubMedBERT probe uses 80/20 train–test splits, with weighted classification and balanced subsampling to address class imbalance.The classifier datasets include approximately 43k HEALTHBENCH-EVAL rubrics, 11k HEALTHBENCH-HARD rubrics, and 2.5k RESEARCHRUBRICS rubrics.Source: §4.1 · p. 3, Figure 3 · p. 3

4.3 Generalization Checks

• Robustness checks include five-fold cross-validation, bootstrap confidence intervals, alternative classifier architectures, cross-dataset transfer, and TF-IDF with logistic regression.The passages describe these checks but defer their detailed results to appendices.Source: §4.3 · p. 4

5.1 Output Perturbation

• 37.7% of valid response-reversal pairs produced the expected verdict flip, leaving a 62.3% inconsistency rate despite fixed context and rubric.The authors note that rubric ambiguity or response-construction errors may contribute, but the failures cannot be attributed solely to cross-item differences in difficulty or rubric wording.Source: Figure 4 · p. 4, §5.1 · p. 4

5.2 Rubric Perturbation

• 16.8% for Gemma and 32.2% for LLaMA are expected verdict-flip rates after rubric reversal; corresponding inconsistency rates are 83.2% and 67.8%.The conversation and candidate response remained fixed, yet judges often retained their original verdict after the criterion changed.Source: §5.2 · pp. 4–5, Figure 5 · p. 5

5.3 Semantic Analysis of Rubrics

• BERT-embedding UMAP analysis finds rubric text correlated with judge labels, with HEALTHBENCH-EVAL rubrics more separable by label than HEALTHBENCH-HARD rubrics.The authors interpret these patterns as label-correlated signal in rubric text, even without the response.Source: Figure 6 · p. 5, §5.3 · p. 5

5.4 Discussion

• The probe identifies an association between rubric text and judge outputs, not causality; the authors recommend validating rubric design, stress-testing sensitivity, auditing rationales, and using complementary metrics.They report that the signal is distributed across semantic structure rather than reducible to keyword artifacts.Source: §5.4 · p. 5

• Rubric-conditioned evaluation may encode predictive signal correlated with judge outputs, motivating benchmarks to aim to eliminate recoverable rubric priors.The authors call for closer scrutiny of rubric design in automated evaluation.Source: §6 · p. 5

• The probe isolates rubric text rather than measuring end-to-end judge behavior in full evaluation settings.The study covers HEALTHBENCH and RESEARCHRUBRICS, and its findings may not generalize to other domains, rubric styles, or languages.Source: §7 · p. 6
• A small set of probing methods and a binary formulation may understate or distort subtler rubric effects.The results are consistent with shortcut learning, but do not identify the full causal mechanism behind rubric-conditioned judgments.Source: §7 · p. 6

AI Usage Declaration

• AI assistance was used for grammatical corrections and minor rewrites.Source: AI Usage Declaration · p. 6

A.1 Models Explored

• Four candidate language models generated HEALTHBENCH responses, with research-use licensing described for Llama-3.1-8B-Instruct, Gemma-7b-it, MedGemma, and MedLLaMA.The passage lists the applicable model and repository licenses and says the use is consistent with research and experimentation.Source: §A.1 · p. 7

A.2 Computing Requirements

• Experiments used Python 3.10, Transformers 4.38.2, PyTorch 2.1, and NVIDIA A100 GPUs, with an estimated 30–35 GPU-hours and no model training or fine-tuning.The study used fixed decoding parameters across queries.Source: §A.2 · p. 7

C.1 Classifier Evaluation Protocol

• The evaluation reserves 20% of each probe cohort for held-out testing and uses five-fold cross-validation on the remaining 80%.The held-out set is not used during development; detailed results are reported in Table 2.Source: §C.1 · pp. 8–9, Table 2 · p. 9

B Rubric Classification Exploration

• Across accuracy, macro-F1, weighted-F1, ROC-AUC, and four judge models, logistic regression outperformed Naïve Bayes and the majority-class dummy baseline.The authors say rubric-derived lexical features carry discriminative signal beyond class imbalance, though performance varies by judge.Source: Figure 8 · p. 8, B Rubric Classification Exploration · p. 8

C.2 Bootstrap Confidence Intervals

• 10,000 bootstrap resamples of each held-out test set produced 95% confidence intervals, which were relatively narrow across the four judge cohorts.Intervals were reported for accuracy, precision, recall, F1, and balanced accuracy.Source: §C.2 · p. 9

C.3 Cross-Dataset Generalization

• The classifier is trained on one HEALTHBENCH probe variant and tested on the other in both transfer directions.This cross-dataset setting tests generalization to rubrics from a different HEALTHBENCH variant.Source: §C.3 · p. 9
• Figure 9 compares within-split and cross-split balanced accuracy; a lower gap indicates better generalization.Solid bars denote within-split evaluation and hatched bars denote cross-split evaluation.Source: Figure 9 · p. 9
• Performance remains above the 0.5 chance level in both cross-dataset settings.The authors conclude that the rubric-only classifier captures patterns that transfer across the two variants, rather than patterns specific to one partition or dataset distribution.Source: §C.3 · pp. 9–10

D.1 Choice of PubMedBERT

• All four classifiers perform substantially above chance, while PubMedBERT provides a modest, generally consistent improvement over BERT, RoBERTa, and DeBERTa-v3.The authors use PubMedBERT because HEALTHBENCH rubrics contain specialized biomedical terminology; the shared pattern indicates the signal is not specific to one architecture.Source: §D.1 · p. 10

D.2 Majority-Class Baseline

• The majority-class baseline has balanced accuracy of 0.50 by construction, while probe performance remains above this baseline across metrics.The majority baseline is computed separately for each response model because their judge-label distributions differ.Source: §D.2 · p. 10, Table 4 · p. 10

E.1 Output Perturbation

• 37.7% desired flips, 39.6% stickiness, and 62.3% inconsistency were observed across 493 valid paired response perturbations.Desired flips follow the intended change in rubric satisfaction; sticky outcomes retain the original verdict, while wrong-direction flips reverse it oppositely.Source: Table 5 · p. 10, §E.1 · pp. 10–11

E.2 Rubric Perturbation

• The desired verdict reversal occurred in 16.8% of Gemma pairs and 32.2% of LLaMA pairs, with rubric inconsistency rates of 83.2% and 67.8%, respectively.The experiment reverses rubric criteria while keeping each conversation and candidate response fixed.Source: §E.2 · p. 11, Table 6 · p. 11

F.1 Keywords

• Failure and negation terms such as “fail,” “pass,” and “not” occur more in label 0 rubrics, while affirmative and epistemic verbs characterize label 1 rubrics.The top class-weighted vocabularies are reported as entirely disjoint; failure-predictive language is encoded more distinctly than success-predictive language.Source: §F.1 · pp. 11–12, Figure 10 · p. 11, Figure 11 · p. 11

F.2 Word Length Analysis

• Label 0 rubrics are longer on average, but their word-count distributions overlap substantially with label 1 rubrics.The figure caption also notes many outliers in both groups.Source: Figure 12 · p. 12

F.3 BERTopic Modeling

• BERTopic finds largely disjoint top-five topic sets: label 0 rubrics cluster around failure modes and clinical edge cases, while label 1 rubrics center on affirmative clinical competencies.The reported topics include symptom-justification requirements and medication interactions for label 0, and dosage correctness and antibiotic stewardship for label 1.Source: §F.3 · p. 12, Figure 13 · p. 12

F.2 Word Length Analysis

• Label 0 rubrics are longer on average than label 1 rubrics.The word-count distributions still overlap substantially.Source: §F.2 · p. 12
• The word-count distributions for the two judge labels overlap substantially, so word count alone does not cleanly separate the classes.This qualification accompanies the comparison of average rubric length by label.Source: §F.2 · p. 12
• Both label groups contain a small number of very long rubric outliers.Source: §F.2 · p. 12