Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training

1. Introduction

• Expert rubrics are painstaking to produce, while LLM-generated criteria can be shallow, misaligned, or insufficiently sensitive to flaws in stronger future models.Datasets with rubrics can require hundreds of physicians or bankers.Source: §1 · p. 1
• Rubricon uses Generate-and-Select, Add, and Prune to target discrimination, coverage, and headroom without human annotation or model fine-tuning.The stages can be reapplied iteratively and use any LLM.Source: §1 · p. 1
• Rubricon supplements self-critique and ensembling with observed rubric scores on example responses, addressing limitations of those approaches in this setting (Shen et al., 2026).The resulting rubrics score responses more consistently with expert-authored rubrics and improve post-training outcomes.Source: §1 · pp. 1–2

Related work

• Synthetic rubrics can omit relevant dimensions or contain generic, conflated, or misaligned criteria, and evaluation on stronger future models is underexplored.The paper notes that expert editing or validation may be part of practical use.Source: §2 · p. 2

Rubric construction strategies

• Rubric construction spans prompting, contrastive generation, feedback extraction, dimension decomposition, interactive building, response-grounded criteria, and retrieval-guided generation.RubricRAG retrieves rubrics for similar queries as few-shot examples (Dhole and Agichtein, 2026).Source: §2 · p. 2
• Rubricon starts from a single LLM-generated draft and can incorporate expert responses, notes, critiques, preferences, or rubric examples as context.Its refinement stages can also be applied to drafts produced by other approaches.Source: §2 · p. 2

Iterative and adaptive refinement

• Refinement methods include recursive decomposition and filtering, criterion aggregation, strong-response tail coverage, and verifier-guided updates with adversarial responses.The cited methods include RRD (Shen et al., 2026), RubricHub (Li et al., 2026), Chasing the Tail (Zhang et al., 2026), and SibylSense (Xu et al., 2026b).Source: §2 · p. 2

Training rubric generators

• Other work trains rubric generators using preference-ordered responses, human-preference rewards, or joint reasoning and rubric-generation supervision.OpenRubrics fine-tunes a generator and judge, while Lv et al. (2026) train a query-specific rubric generator through reinforcement learning.Source: §2 · p. 2

3. Task Formulation

• A rubric is a set of weighted binary natural-language criteria, and a verifier judges whether each response satisfies each criterion.The response receives a normalized overall score used to rank responses or as a post-training reward.Source: §3 · p. 3
• A strong rubric should discriminate response quality, cover relevant requirements and failure modes, and preserve headroom for future-model responses.Headroom means remaining sensitive to meaningful quality differences as model capabilities improve.Source: §3 · p. 3
• More criteria are not always desirable because verifier noise, computation, and human review carry costs; rubrics should avoid criteria models already pass or simple verifiers cannot reliably grade.The stated aim is the fewest criteria that meet the rubric desiderata.Source: §3 · p. 3

4. Rubricon

• The modular pipeline can use any off-the-shelf LLM, measures criterion pass rates on sample responses, and uses these signals to select, add, and remove criteria.Generate-and-Select targets discrimination, Add expands coverage, and Prune removes criteria that reward wrong qualities or produce noisy scores.Source: §4 · p. 3, §4.2 · p. 4

4.1. Aspects of Effective Rubric Criteria

• Individual criteria should be clear, self-contained, atomic, and interpretable without cross-references; each also has text, an importance label, and a category.Critical, Major, and Minor labels receive relative weights of 5, 3, and 1; categories do not affect rubric scores.Source: §4.1 · pp. 3–4

4.2. The Rubricon Pipeline

• Generate-and-Select uses five candidate drafts informed by the prompt, strong-model responses, and optional reference context, then chooses the candidate with the largest average strong–weak edge.It retains unselected candidates as a pool for the Add stage.Source: §4.2 · pp. 4–5, Figure 1 · p. 4
• The strong–weak edge is the difference in criterion pass rates between stronger- and weaker-model responses; a positive edge indicates discrimination in the intended direction.Rubricon uses edge estimates to select the candidate with the highest average edge and to filter criteria.Source: §4.2 · p. 5
• Add sequentially fills gaps from unselected drafts, checks explicit prompt requirements, and probes weak-response defects and omissions in intended high-quality responses.New criteria must be task-relevant and not already covered; sample-response scores help identify misaligned additions for pruning.Source: §4.2 · pp. 5–6
• Prune removes criteria whose estimated strong–weak edge is below −0.10 or that an LLM assessor flags as satisfiable by evasive or weak responses.The remaining criteria form the refined rubric.Source: §4.2 · p. 6
• Add and Prune can be repeated in refinement rounds, reusing the same strong- and weak-model responses while rerunning coverage checks and probes against the updated rubric.Gap fill continues to draw from the original unselected drafts.Source: §4.2 · p. 6

5.1. Assessing Rubric Quality

• Rubric quality is measured by Spearman’s rank correlation between response rankings under generated and expert-authored rubrics, averaged across tasks.The same responses are scored under both rubrics; agreement tests whether they capture similar quality distinctions, even if their numerical scores differ.Source: §5.1 · p. 6
• Selective preference accuracy measures agreement with human pairwise preferences only for response pairs whose normalized rubric scores differ by more than 0.10.The margin avoids treating small score differences that may reflect verifier variability as meaningful preferences.Source: §5.1 · p. 6

5.2. Verifier Scoring

• For rubric construction, the verifier grades five stronger-model and three weaker-model responses per task, with each response–criterion pair evaluated three times.The resulting pass rates are used to estimate the strong–weak edge; quality assessment instead uses completed rubrics and separate fixed responses, also with three judgments per pair.Source: §5.2 · p. 7

5.3. Datasets

• The evaluation spans five benchmarks across professional, consumer, clinical, and general-purpose tasks, using expert rubrics from four benchmarks and human preferences from RubricBench.The expert-rubric benchmarks are ProfBench, PRBench, ACE, and HealthBench.Source: §5.3 · p. 7

5.4. Baselines

• Rubricon is compared with prompt-based Agentic and RRDWU methods and post-trained QSRG and OpenRubrics generators; each method produces one rubric per task.The Agentic and RRD baselines receive the same task prompt, example responses, and reference material as Rubricon, while OpenRubrics and QSRG receive the task prompt alone; QSRG+Context is also tested with added responses and reference material.Source: §5.4 · pp. 7–8

5.5. Models Employed

• GPT-5.1 powers prompt-based methods in human-alignment experiments, and GPT-5 mini serves as the criterion verifier.Rubric construction uses GPT-5.1 and Llama-3.1-8B-Instruct responses as strong and weak examples, while rubric alignment is assessed on separate responses from eight models, including newer or stronger models.Source: §5.5 · p. 8

6.1. Alignment with Human Rubrics

• Rubricon achieves the highest ρ on all four expert-rubric benchmarks, with a 0.052 margin over the next-best method on ACE.Its 0.428 macro-average ρ is a 9.0% relative improvement over RRDWU.Source: §6.1 · p. 8

6.2. Alignment with Human Preference

• Rubricon’s selective preference accuracy exceeds the Agentic baseline by 0.5 percentage points, although all methods have fairly low alignment on the challenging RubricBench examples.Source: §6.2 · p. 8

6.3. Iterative Refinement Across Multiple Rounds

• The first refinement round improves macro-average ρ by 11.9% over the selected initial rubric, and the second adds 5.1% over the first.Relative to RRDWU, the improvements after one and two rounds are 3.7% and 9.0%, respectively.Source: §6.3 · p. 8

6.1. Alignment with Human Rubrics

• 9.0% higher macro-average ρ than RRDWU (0.428 vs. 0.393), and 0.5 percentage points higher selective preference accuracy than Agentic (59.8% vs. 59.3%).Rubricon has the highest observed scores across all five benchmarks.Source: Table 1 · p. 9

6.4. Performance Across Refinement Steps

• Each pipeline stage improves alignment on at least one benchmark, and most stages also improve macro-average ρ across the four benchmarks.Some Add steps slightly reduce macro-average ρ while improving individual benchmarks; pruning is empirically effective overall, suggesting coverage expansion and filtering play complementary roles.Source: §6.4 · p. 9, Table 2 · p. 9
• 4.5% higher held-out HealthBench score at the last evaluated checkpoint for Qwen3.5-9B trained with Rubricon rewards than with RRDWU rewards.Rubricon-trained performance exceeds RRDWU-trained performance at every evaluated checkpoint.Source: Figure 2 · p. 10

• The post-training experiment evaluates whether Rubricon rubrics can provide effective rewards for Qwen3.5-9B, measuring resulting models against expert-authored rubrics on held-out tasks.Source: §7 · p. 10
• Training uses 1,204 HealthBench prompts, with 500 prompts held out, and compares two-refinement-round Rubricon rubrics against RRDWU rewards under the same eight-epoch GRPO recipe.The released Qwen3.5-9B checkpoint, data split, and evaluation procedure are otherwise held fixed.Source: §7 · p. 10
• Rubricon-reward training yields higher held-out HealthBench scores than RRDWU-reward training at every evaluated checkpoint; the final-checkpoint improvement is 0.0315 (paired 95% bootstrap CI: [+0.0207, +0.0427]).The advantage persists when final-checkpoint responses are re-scored with GPT-5 mini or Claude Sonnet 5.Source: §7 · p. 11

8. Discussion

• Rubricon improves rubrics beyond basic LLM or agentic generation without additional data collection, annotation, or model fine-tuning.The pipeline selects discriminative drafts, fills coverage gaps, and removes misleading criteria.Source: §8 · p. 11
• The refinement stages use how draft rubrics score example responses, rather than relying only on LLM self-critique and ensembling.Source: §8 · p. 11
• Rubricon’s systematic selection, coverage additions, and pruning produce more effective rubrics for LLM evaluation and training.Source: §8 · p. 11

Limitations

• The experiments cover single-turn LLM responses in certain domains and English only.Whether these rubrics continue distinguishing quality beyond the most advanced models tested remains unclear.Source: Limitations · p. 11
• Rubricon relies heavily on model-derived signals, and its automatically generated rubrics are not perfect.As models improve, flaws in their responses may also become subtler.Source: Limitations · p. 11
• The authors recommend that human experts review and improve Rubricon rubrics, while noting this still saves substantial time versus crafting them from scratch.Its modular stages can be adjusted using expert feedback, and can also refine human-crafted initial rubrics.Source: Limitations · p. 11

A.1. Benchmarks for Assessing Rubric Quality

• Benchmark selection followed review of expert rubrics, and preprocessing excluded datasets or domains with rubric-quality concerns.ProfBench Chemistry was excluded for issues including vague criteria and inconsistent numerical-precision requirements.Source: §A.1 · p. 15
• Rubric-quality evaluation uses ProfBench, PRBench, ACE, and HealthBench, covering professional reasoning, consumer tasks, and health conversations.HealthBench also supports post-training experiments; generated and expert rubrics score the same responses on the four rubric benchmarks.Source: §A.1 · p. 15
• RubricBench contributes 1,147 pairwise preference examples with binary human labels, evaluated using selective preference accuracy.Examples include a task, two LLM responses, and a human preference label.Source: §A.1 · p. 15

A.2. Reference Material for Rubric Construction

• Rubricon, Agentic, RRDWU, and QSRG+Context receive prompts, example responses, and available task references; QSRG and OpenRubrics receive only task prompts.Expert rubrics used for alignment evaluation are never given to rubric generators.Source: §A.2 · p. 15

A.3. Data Preprocessing

• Preprocessing requires at least 10 expert criteria for ProfBench, PRBench, and HealthBench, but not ACE, whose rubrics average seven criteria.PRBench and HealthBench are restricted to single-turn tasks; HealthBench also requires expert-written reference responses.Source: §A.3 · p. 15

B.1. Agreement with Expert Human Graders

• On a 367-criterion subset, GPT-5 mini reaches κ=0.455 against expert labels, compared with non-expert human agreement of κ=0.281.The corresponding raw agreement rates are 72.5% for GPT-5 mini and 64.0% for non-expert humans.Source: Table 4 · p. 16
• The authors compare verifier judgments with expert annotations on more than 3,400 response-criterion pairs to isolate criterion-level grading accuracy.They test four verifier models with their prompt and the ProfBench prompt; their prompt generally yields higher agreement and F1.Source: §B.1 · p. 16
• 97.2% of GPT-5 mini’s criterion judgments agree across eight repeated runs, with mean pairwise κ=0.943.Source: §B.1 · p. 16

B.2. Post-training Results Across Verifiers

• Rubricon’s post-training advantage over RRDWU holds across all three verifier models on HealthBench.Each verifier scores the same saved responses from the two policies after eight training epochs against physician-authored rubrics.Source: §B.2 · p. 16, Table 5 · p. 17

B.3. Verifier Prompt

• The verifier prompt asks an expert judge to return YES or NO according to whether a response satisfies a criterion.The prompt supplies the response and criterion in separate fields.Source: §B.3 · p. 17

C.2. Detailed Performance Across Refinement Steps

• Second-round refinement increases macro-average ρ by 17.6% over the selected draft, while first-round gains vary and no stage leads consistently.Table 8 reports improvements on every benchmark; rows are cumulative, with each stage operating on the preceding rubric.Source: Table 8 · p. 18, §C.2 · p. 18

C.3. Rubric Size and Category Composition

• Add expands first-round rubrics by 33–60% and second-round rubrics by a further 26–41%; final rubrics remain 19–82% larger than generated baselines.Prune removes 8–20% in Round 1 and 4–15% in Round 2; rubric size is mean criteria per task.Source: Table 10 · p. 19
• Across benchmarks, category distributions differ by domain, while refinement increases the share of Reasoning criteria by 33–45% relative.HealthBench allocates a larger share to Safety, while Reasoning is particularly prominent in PRBench and ProfBench.Source: Figure 3 · p. 19, §C.3 · p. 20

C.5. Post-training details

• The shared training configuration was selected by training on original expert-authored HealthBench rubrics, not by tuning against automatically generated rubrics.Table 11 summarizes the configuration shared by RRDWU and Rubricon.Source: Table 11 · p. 20, §C.5 · p. 20
• Rubricon achieves higher macro-average alignment than merging with a 3.2× smaller rubric.Merging improves alignment over the selected rubric but uses approximately 5× its rubric size.Source: Table 12 · p. 21

D.1. Merging versus Refinement

• 0.428 vs 0.402 macro-average ρ: Rubricon exceeds merged rubrics with 32.3 rather than 103.2 criteria per task.Merging raises alignment over the selected draft, but accumulating criteria alone does not match targeted addition and pruning.Source: §D.1 · p. 21

D.2. Deduplication

• Near-duplicate criteria are difficult to remove without losing distinct checks; LLM deduplication struggled to balance the trade-off and added cost without consistent quality gains.Union-based filtering was too aggressive, while intersection-based filtering left duplicates unresolved.Source: §D.2 · p. 21

D.3. Few-shot Rubric Examples

• Few-shot examples of expert rubrics from the same dataset did not improve rubric quality, possibly because task requirements differ or the initial prompt already provides extensive guidance.The authors report testing this approach following Dhole and Agichtein (2026).Source: §D.3 · p. 21

D.4. LLM-based Autograder to identify Flaws

• Reference-response flaw detection can overfit to an expert answer’s particular details, risking penalties for valid alternative solutions.The tested approach converted uncovered flaws into additional criteria.Source: §D.4 · p. 22

E. Rubricon Refinement Examples

• 0.613 to 0.925 Spearman correlation: Add and Prune improve alignment with the physician-authored rubric in the suspected-anaphylaxis example.The examples track rubric changes and their effect on response scores and quality ranking against the expert rubric.Source: E. Rubricon Refinement Examples · pp. 22–23, Figure 4 · p. 24

STOP READING - CALL 911 RIGHT NOW

• The rural-lidocaine example’s rubric emphasizes locally actionable guidance and nuanced availability claims alongside safeguards against unsafe self-procedures.Its criteria include checking nearby facilities, acknowledging that availability varies, and avoiding invasive procedures without trained professionals.Source: STOP READING - CALL 911 RIGHT NOW · p. 26

E.1. Analysis of the Examples

• Example 1’s ρ rises from 0.613 after Generate-and-Select to 0.925 after Prune, while Example 2’s ρ falls from 0.632 to 0.531 after Add, then rises to 0.682 after Prune.For Example 1, these values are across eight assessment responses; for Example 2, Prune retains seven of Add’s ten criteria.Source: §E.1 · pp. 28–29
• In Example 1, Add reversed the initial Claude-over-Gemini ordering, but Prune removed criteria that rewarded hedging or lacked discriminative signal, restoring the physician rubric’s ordering.The final rubric distinguishes the responses through criteria on conditional positioning instructions, preparing for CPR, and advising against food or drink.Source: §E.1 · p. 28
• In Example 2, Add captured useful distinctions including topical versus injectable lidocaine, while Prune removed criteria whose scoring penalized GPT-5.2 without reflecting substantive omissions.The removed criteria contributed an unnormalized weighted score difference of 4.3 points in Gemini 3 Flash’s favor; Prune retained seven of Add’s ten criteria.Source: §E.1 · pp. 28–29

F.2.2. Prompt Coverage

• New prompt-coverage criteria target uncovered requirements only when they are prompt-grounded, discriminative, objectively gradable, and not path-dependent.For partial coverage, a new criterion should address only the missing portion and match the existing rubric’s style and specificity.Source: §F.2.2 · p. 35

F.2.3. Negative Probing

• Negative probing first generates a plausible weakest response that satisfies every current criterion while degrading untested qualities.It should remain coherent and realistic, avoid blatant errors, and be checked criterion by criterion, revising only as needed to pass.Source: §F.2.3 · pp. 35–36
• An independent expert reviewer flags specific, actionable omissions relevant to the user’s actual question, excluding out-of-scope, stylistic, or speculative issues.If the response already substantively answers the question, the reviewer may return an empty list.Source: §F.2.3 · pp. 36–38
• Flagged issues become new criteria only if they reveal a real rubric gap and support specific, binary, self-contained grading.Criteria should be outcome-grounded, not tied to one derivation path; negative and inverse tests help reject criteria that permit wrong answers or exclude valid alternatives.Source: §F.2.3 · pp. 38–40
• Proposed criteria must be concrete, relevant, uncovered, and traceable to prompt-provided, reasonably derivable, or common-knowledge content.The process skips already-covered issues and rejects vague, stylistic, out-of-scope, or ungradable flags; content without a valid source risks being fabricable.Source: §F.2.3 · pp. 39–40
• Negative probing checks new criteria against plausible incorrect and correct responses, and requires criteria to capture the underlying failure mode rather than one adversarial example.If a criterion admits an incorrect response or rejects a correct one, it should be skipped, rephrased to target the answer, or downgraded to Minor.Source: §F.2.3 · p. 40

F.2.4. Completeness Probing

• Completeness probing converts concrete, in-scope omissions not already covered by the rubric into atomic, outcome-grounded criteria that allow valid alternative solution paths.The process filters vague, stylistic, redundant, or path-specific gaps; checks whether omissions are prompt-grounded or otherwise verifiable; and rejects criteria that could reward a wrong answer or fail a correct one.Source: §F.2.4 · pp. 43–44

F.3. Prune: Shallow-Criterion Assessor

• The shallow-criterion assessor flags rubric tests that reward omission, hedging, or declaring a method instead of executing it, while preserving necessary safety and outcome-focused criteria.It distinguishes surface compliance from substantive execution and exempts safety criteria needed for the prompt, numerical or verifiable reasoning, and domain-specific outcomes.Source: §F.3 · pp. 45–46
• For each flagged criterion, propose a rewrite that rewards positive execution and is at least as specific as the original.The rewrite should state what a correct response must do or produce, rather than what it should avoid or how it should look.Source: §F.3 · p. 46

F.4. Criteria Principles

• Criteria should be specific, objective, binary, self-contained, internet-independent, and non-stacked, while also anticipating likely response flaws.Rubrics should cover meaningful requirements and supporting information; criteria may target common mistakes as well as desired content, and carry importance and category annotations.Source: §F.4 · pp. 46–50

G. Agent Rubric Generation Baseline

• The agentic baseline runs GPT-5.1 in a per-task Codex SDK Docker sandbox, with examples and reference material provided through workspace files rather than the generation prompt.The prompt contains the task only; the workspace is writable, but tool-command network access is disabled.Source: G. Agent Rubric Generation Baseline · pp. 51–52
• The baseline provides example AI responses and optional expert methodology and human-response files, then saves the final rubric as `rubric.json`.The response file contains one JSON object per line; the expert methodology and example human response are included only when available.Source: G. Agent Rubric Generation Baseline · p. 52