Evaluating Rubric Generation with Interventional Transfer

1 Introduction

• Instance-specific rubrics specify response criteria, but expert effort scales with the number of examples, motivating LLM-based rubric generation.The paper focuses on evaluating generated rubrics when expert-written references are available.Source: §1 · p. 1
• Semantic-overlap methods can miss differences in criterion specificity, while score comparisons depend on the response population and comparison being performed.The paper notes that implementation choices can also affect existing metrics.Source: §1 · pp. 1–2
• Interventional transfer edits responses to pass or fail target-rubric criteria, then measures the effect on scores from the other rubric.Different intervention directions and target rubrics correspond to distinct downstream uses, including optimization and degradation monitoring.Source: §1 · p. 2
• On HealthBench responses from GPT-5.6-Terra, degrading responses transferred substantially across generated and expert rubrics, while improvements transferred little.The authors conclude that, in this task and response-model setting, generated rubrics were more useful for detecting degradation than as benchmarks for improvement.Source: §1 · p. 2
• Qwen3.8-27B rubric interventions produced zero expert-score improvement, whereas Opus-5 interventions produced a small, statistically significant improvement.The study evaluated Qwen3.8-27B, Deepseek-V4-Flash, and Opus-5 rubrics against HealthBench expert rubrics using GPT-5.6-Terra responses.Source: §1 · p. 2

2 Notation and Background

• A rubric consists of binary criteria, and its score vector records which criteria a response passes or fails.The notation distinguishes expert rubrics C∗ from generated rubrics Cg.Source: §2 · p. 2
• Existing evaluation approaches include semantic matching of criteria and score correlation across responses; targeted response perturbations instead measure changes in the other rubric.The comparison illustrates semantic matching, population correlation, and an interventional-transfer example.Source: §2 · pp. 3–4, Figure 2 · p. 3
• The paper uses checklist rubrics and does not assess whether criteria should receive different importance weights.This scope reflects a focus on concept coverage rather than matching the importance assigned to concepts.Source: §2 · p. 3
• Figure 3 maps intervention direction and target rubric to downstream uses, including optimization targets and degradation flags.It also shows generated-rubric assessment of good and bad responses against expert rubrics.Source: Figure 3 · p. 4

3 Our Framework: Interventional Transfer

• Interventional transfer minimally edits responses to pass or fail selected target-rubric criteria, then measures whether scores change under a non-target rubric.The target may be a generated rubric and the non-target rubric an expert rubric.Source: §3 · p. 4
• The framework includes single-criterion interventions and whole-rubric interventions that edit responses into sets passing or failing every rubric criterion.The pass set contains responses passing every criterion; the fail set contains responses failing every criterion.Source: §3 · p. 4
• Four combinations of target rubric and intervention direction define metrics that measure criterion movement in the non-target rubric.Each quality metric counts measurement-rubric criteria that moved as a fraction of criteria that could have moved.Source: §3 · pp. 4–5
• For optimization, Q+(Cg → C∗) measures whether passing generated criteria leads a response to pass expert criteria.Its numerator counts expert criteria changing from 0 to 1, and its denominator counts expert criteria initially at 0.Source: §3 · p. 5

4.1 Data and Setup

• The primary analysis sampled 105 HealthBench queries across seven themes, used GPT-5.6-Terra responses, and evaluated rubrics from Qwen3.8-27B, Deepseek-V4-Flash, and Opus-5.The sample excluded HealthBench Hard; Gemini 3.8 Flash scored criteria using majority vote over three samples.Source: §4.1 · p. 5

4.2 Implementation of Existing Approaches

• Existing baselines cover semantic matching and score correlation, including correlations on response panels or physician responses and discrimination between physician and degraded responses.Figure 4 lays out these four evaluation views.Source: §4.2 · pp. 5–6, Figure 4 · p. 6

4.3 Case Study: Application of Existing Approaches

• Existing metrics yielded scores broadly between 0.3 and 0.75, with no significant generator differences in most pairwise tests.Exceptions were Qwen3.8-27B's lower F1 than Deepseek-V4-Flash and Opus-5, and lower discrimination than Opus-5.Source: §4.3 · p. 6

4.4.1 Rubric-Level Interventions

• Failing target criteria transfers more strongly than passing them; only Opus-5 pass interventions significantly improve expert scores, by 11%.Opus-5 is also significantly better than Qwen3.8-27B at yielding gains on the expert rubric.Source: Figure 5 · p. 7, §4.4.1 · pp. 7–8

Existing approaches to identifying or generating “good” and “bad” responses have limitations

• Physician responses were preferred over constructed “bad” responses only about 60% of the time under expert rubrics, questioning their use as a gold standard.Constructed degradation reduced at most 37% of expert criteria; the reduction was about 10% for larger models and about 20% for smaller models, with Grok 4.20 as the 37% outlier.Source: Existing approaches to identifying or… · p. 7

4.4 Case Study: Application of Our Approach

• Interventional transfer yields qualitatively different insights from prior approaches by evaluating rubric utility through rubric-level and criteria-level interventions.The framework distinguishes rubric utility for different tasks, including improvement targets versus degradation detection.Source: §4.4 · p. 7, §4.4.1 · p. 7, §4.4.2 · p. 8

4.4.2 Criteria-Level Interventions and Measures

• 82–87% of initially failing generated criteria were irrelevant, versus 47–57% of initially passing criteria; passing criteria were more redundant in every rubric.Criteria-level interventions edit one criterion at a time from the original response; cross-rubric movement is categorized as irrelevant, matched, or general, while within-rubric movement diagnoses redundancy.Source: §4.4.2 · pp. 8–9, Figure 6 · p. 9

5 Conclusion

• The framework treats a generated rubric as a suitable expert-rubric proxy when response-score changes measured by one are reflected in the other.The paper argues that evaluation should be tied to the rubric’s downstream task.Source: §5 · p. 9
• Passing generated criteria often does not improve expert-rubric scores, even when generated-rubric scores are moderately correlated with expert scores.For degradation monitoring on GPT-5.6-Terra responses, most rubric generators show some utility.Source: §5 · pp. 9–10
• For monitoring response quality changes in a given model, comparing score changes under intervention is more natural than testing whether a rubric distinguishes stronger from weaker models.The paper contrasts this with comparisons such as GPT-5.6-Terra versus GPT-4o.Source: §5 · p. 10

A.1 Prior Work

• Instance-specific rubrics give each example its own expert-written criteria when answering requires domain expertise, unlike general criteria used across a dataset.They are used in some benchmarks in expert-knowledge domains.Source: §A.1 · p. 14
• Rubric-generation work spans retrieval, self-evolving generation, checklists, preference-trained rubrics, decomposition, and criteria clustering; proposed quality measures include coverage, precision, and contribution.Examples include RubricRAG [6], Chen et al. [3], Cook et al. [5], and Shen et al. [20].Source: §A.1 · p. 14
• Rubrics are commonly used as reinforcement-learning rewards, with studies reporting gains on expert rubrics [8] [20] or standard benchmarks [11] [24].Other work mixes generated and human-written rubrics [11], uses checklist scores as rewards [24], or trains on HealthBench expert criteria [29].Source: §A.1 · p. 14
• Most rubric evaluations compare generated criteria or scores with expert rubrics [6] [10] [3], while other checks assess preference alignment, rubric coverage and injected-error detection, or preference for a physician-written response over a degraded one [6].Some work also assesses rubric failure modes [16] or whether judges apply individual criteria correctly [15].Source: §A.1 · p. 14
• A recent survey lists four papers on evaluating rubrics versus roughly 50 on using rubrics to evaluate responses; a concurrent survey reports a similar imbalance.Source: §A.1 · pp. 14–15
• Prior intervention work evaluates models by expected changes to prompts or clinical recommendations [28] [23]; this paper instead uses one rubric’s score change as the reference for another’s on edited responses.The clinical-case method specifies expected changes by hand, whereas this paper edits responses to satisfy or violate a rubric.Source: §A.1 · p. 15

B Framework

• Table A.1 covers metrics for evaluating generated rubrics against a reference rubric or human labels.Source: Table A.1 · p. 16
• Table A.2 covers metrics that evaluate a generated rubric through the responses it ranks or the judge it produces.Source: Table A.2 · p. 16
• Intrinsic semantic precision, recall, and F1 use criterion-to-criterion matching based on cosine similarity above τ = 0.7, without an LLM matcher.The metric description also places aggregation across approximately five repositories within a paper, then averages across papers.Source: B Framework · p. 17
• The listed human-agreement measures include pairwise agreement and Spearman and Pearson correlations against human scores, pooled over instance-response pairs.Dynamic rubrics measure the judge rather than the rubric; z-hat denotes the response preferred under rubric C.Source: B Framework · p. 17

C.1 Data and Setup

• Table A.3 lists model choices and implementation settings for the experimental pipeline.Source: Table A.3 · p. 18

• Most generator comparisons for existing approaches are inconclusive; the few conclusive comparisons suggest large models generate slightly better rubrics.Source: Table A.4 · p. 18
• Semantic-matching estimates of absolute rubric quality vary with whether similarity means measuring the same thing, changing one criterion changing the other, or logical entailment.Source: Figure A.1 · p. 19
• Score-correlation values change with the selected response panel, so the response population affects the metric.Source: Figure A.2 · p. 19
• RubricRAG’s degradation method has limited impact overall, with big or strong response models less affected than small or weak models.Source: Table A.5 · p. 20
• A near-perfect correlation on an existing-response population can coexist with little expert-score movement after rubric-targeted perturbation; RubricRAG [6] correlates across queries for one model, while GenRubric [3] averages correlation across models.Source: Figure A.3 · p. 20

C.3 Our Approach

• Table A.6 reports LLM-matching similarity from each generated criterion to its closest expert criterion, split by whether it initially passes or fails on GPT-5.6-Terra responses.Source: Table A.6 · p. 21
• Table A.7 reports the share of initially passing and failing generated criteria meeting or exceeding each LLM-matching similarity cutoff.Source: Table A.7 · p. 21
• Table A.8 reports paired differences between rubric generators for each interventional-transfer metric on GPT-5.6-Terra responses, with 95% paired bootstrap intervals.Source: Table A.8 · p. 21
• Figure A.4 distinguishes four transfer tasks: training target, detecting good responses, detecting bad responses, and degradation flag; the paper uses the last column.Source: Figure A.4 · p. 22

C.5 Properties of Intervention

• Failure-directed interventions require more editing per targeted-criterion word than pass-directed edits, shortening responses to 42–62% for whole-rubric and 92% for single-criterion interventions; pass-directed edits lengthen them.The table describes the intervention edit and response-length measures; the reported percentages come from the accompanying prose.Source: Table A.9 · p. 23, §C.5 · p. 23

D Algorithms

• Algorithms pool criteria for transfer metrics and categories, average baselines over examples, correlate rubric scores across an eight-model response panel, and report paired-bootstrap intervals.The response panel uses one response from each of eight models per conversation; the bootstrap uses paired resampling over examples.Source: D Algorithms · p. 24

E Prompts

• The rubric-generation prompt requires binary, atomic, independent, context-specific criteria that together cover everything a physician would care about for the response.The intervention controls aim to make minimal edits, changing only what is needed to pass or fail the target criteria while preserving the other criteria and a fluent reply.Source: E Prompts · pp. 25–27
• The intervention prompt maps each generated criterion to true criteria that change from unsatisfied to satisfied; similarity judgments instead match criteria tracking the same underlying quality, even at opposite polarity.The similarity prompt defines a match by whether knowing one criterion’s value predicts the other across plausible responses.Source: E Prompts · p. 28
• The prompt requires a 1/0 score for every rubric criterion, returned exactly once in JSON, and supplies the conversation through a `{conversation}` placeholder.Source: E Prompts · p. 29