From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

1 INTRODUCTION

• Existing rubric methods are often guided by aggregate outcomes, so gains cannot be reliably attributed to specific design choices.Earlier generation approaches can also produce incomplete coverage and inconsistent granularity.Source: §1 · pp. 1–2
• Structural decomposition improves criterion coverage and targeted diagnosis, semantic specification alone does not ensure effective judgments, and alternative aggregation strategies add limited gains.The analysis disentangles criterion organization, criterion operationalization, and score aggregation.Source: §1 · p. 2
• ARTS-Hybrid formulates rubric construction as constrained search over hierarchical criteria, combining structured decomposition with semantic-first, response-calibrated selection.The framework adapts tree construction and refines criteria using both semantic and response-based evidence.Source: §1 · p. 2
• ARTS-Hybrid achieves competitive performance across benchmarks while producing high-quality rubrics.The paper evaluates both end-to-end performance and intrinsic rubric quality.Source: §1 · p. 2
• The benefits transfer to a standalone rubric generator, and ARTS-Hybrid rubrics provide effective reward signals for policy optimization.Source: §1 · p. 2

Automatic rubric generation and refinement

• Rubric construction has progressed from one-shot generation and contrastive criteria to offline refinement, hierarchical decomposition, and online co-evolution.Examples include RRD (Shen et al., 2026), RubricHub (Li et al., 2026), Q-World (Gao et al., 2026), and online methods (Sheng et al., 2026; Shao et al., 2025; Chen et al., 2026).Source: §2 · p. 2

Rubrics as rewards for LLM optimization

• Rubrics also serve as reward functions for open-ended tasks, with later methods adapting criteria to medical cases, policy rollouts, or training utility.Examples include Rubrics as Rewards (RaR) (Gunjal et al., 2026), InfiMed-ORBIT (Wang et al., 2025), DR-Tulu (Shao et al., 2025), and OptimSyn (Fan et al., 2026).Source: §2 · p. 3

• Despite improvements in rubric generation and refinement, how structural decomposition, criterion-level validity, and score aggregation contribute to effectiveness remains unclear.Source: §2 · p. 3

3 WHAT MAKES AN EFFECTIVE RUBRIC? A MECHANISTIC ANALYSIS

• The controlled analysis samples 1,500 instances from HealthBench and RubricBench and builds validated gold rubric trees from expert-authored criteria.Semantically related gold leaves are recursively grouped into higher-level evaluation dimensions.Source: §3 · p. 3

3.1 RQ1: HOW DOES STRUCTURAL ORGANIZATION AFFECT RUBRIC QUALITY?

• Tree-structured generation outperforms FLAT and FLAT-MULTI in criterion coverage and valid novelty across judge models.The comparison uses the same generator, and the reported gains persist when generator-call counts are matched.Source: §3.1 · pp. 3–4, Table 1 · p. 3
• PATH outperforms FLAT across all six settings, while top-down TREE routing does not outperform FLAT under either judge model.A missed internal node can exclude descendant leaves; hierarchy works better as semantic context than as pruning without recall safeguards.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 RQ2: WHAT MAKES AN INDIVIDUAL CRITERION EFFECTIVE IN PRACTICE?

• Semantic specification shifts criterion discrimination and cross-judge stability only marginally, despite explicitly requesting semantically valid, discriminative, and stable criteria.The analysis measures discrimination across responses and stability across judges.Source: §3.2 · pp. 4–5, Table 3 · p. 5
• Calibration alone achieves the highest discrimination and lowest cross-judge variance, but 17.84% of selected criteria are semantically invalid.SEMANTIC+CALIBRATION eliminates invalid selections while retaining most gains and achieves the highest average pairwise evaluation accuracy.Source: §3.2 · p. 5, Table 3 · p. 5

3.3 RQ3: DOES SCORE AGGREGATION AFFECT RUBRIC EVALUATION?

• The aggregation comparison holds rubric leaves and criterion-level judgments constant while varying global or hierarchical weighting and aggregation.FLAT+WU applies weighting globally, while TREE+WU applies it locally within the hierarchy.Source: §3.3 · p. 6
• Neither hierarchical aggregation nor WU weighting yields consistent accuracy improvements; FLAT is highest with DeepSeek-V4-Flash-GA, and FLAT+WU gains 0.27 percentage points with Llama3.1-405B-Instruct.Once criterion-level evaluation is established, aggregation adds limited gains, and rubric effectiveness appears to depend more on criterion design.Source: Table 4 · p. 5, §3.3 · p. 6

4.1 ADAPTIVE RUBRIC TREE CONSTRUCTION

• Best-first expansion prioritizes rubric nodes using task importance discounted by depth, then finalizes sufficiently atomic, independently assessable nodes as leaves.Expansion continues until no nodes remain expandable or a predefined stopping condition is reached.Source: §4.1 · pp. 6–7

4.2 SEMANTIC-FIRST RESPONSE-CALIBRATED SELECTION

• Semantic validation filters decompositions for clarity, task relevance, independent assessability, parent-requirement coverage, and non-redundancy before calibration.Only candidates that pass all semantic checks remain feasible.Source: §4.2 · p. 7
• Calibration evaluates feasible criteria on six responses with two independent judges, using response variation for discrimination and judge variation for scoring consistency.Comparisons across child-criterion response-score profiles also identify behavioral redundancy.Source: §4.2 · p. 7
• The calibration objective rewards discrimination and complementary forms of cross-judge consistency while penalizing redundant scoring behavior among child criteria.The final decomposition is selected from the semantically feasible set using this objective.Source: §4.2 · p. 7

4.3 GLOBAL VALIDATION AND WEIGHT ASSIGNMENT

• After tree search, global validation checks coverage, leaf atomicity, and semantic non-redundancy, then recovers missing requirements or restructures leaves as needed.Local validity alone does not guarantee a coherent, complete rubric.Source: §4.3 · pp. 7–8
• Table 5 reports pairwise accuracy (%) across benchmarks and highlights the best result.Source: Table 5 · p. 8
• ARTS-Hybrid assigns each leaf a normalized weight from its own importance score rather than propagating weights through the hierarchy.The generator scores each leaf’s importance from the query and criterion.Source: §4.3 · p. 8

5 EVALUATION OF ARTS-HYBRID

• The study assesses end-to-end evaluation, intrinsic rubric quality, transfer to a standalone rubric generator, and rubric-based rewards for policy optimization.Source: §5 · p. 8, §5.2 · pp. 8–9

5.1 EXPERIMENTAL SETUP

• The evaluation covers preference benchmarks across domains and difficulty levels, comparing pairwise accuracy against human preferences and using matched generator and judge models for baselines.Baselines include direct judgment, query-only and response-aware rubric generation, and structured rubric refinement.Source: §5.1 · p. 8

5.2 MAIN RESULTS

• ARTS-Hybrid achieves the highest accuracy in five of six benchmark settings, remains competitive on PreciseIF, and outperforms Q-World on five settings.Source: §5.2 · p. 8
• ARTS-Hybrid earns the highest scores across coverage, concreteness, non-redundancy, and actionability in the intrinsic rubric-quality evaluation.Source: §5.2 · p. 8, Table 6 · p. 9
• Fine-tuning Qwen3-4B on ARTS-Hybrid rubrics yields the highest JudgeBench accuracy among the tested supervision sources.This experiment tests whether offline construction benefits transfer to a standalone rubric generator.Source: §5.2 · pp. 8–9, Table 7 · p. 9
• Rubric-based rewards from ARTS-Hybrid produce the best HealthBench results in both thinking and non-thinking settings.Directly generated and expert-provided rubrics provide limited gains over the untrained base model.Source: §5.2 · p. 9, Table 8 · p. 9

6 CONCLUSION

• The conclusion synthesizes mechanistic analysis of criterion organization, criterion-level operationalization, and score aggregation with the ARTS-Hybrid approach.It reports higher-quality rubrics, strong evaluation performance across judge models, and effective reward signals for model optimization.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

• The authors release counterfactual responses, gold rubric trees, and core code for the mechanistic analysis and ARTS-Hybrid.The statement also directs implementation details and prompts to the appendices.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATASET

• The mechanistic analysis set contains 1,500 instances across six task categories, drawn from HealthBench and RubricBench.Source: §A.1 · p. 12
• The authors sample 750 instances from fixed versions of each dataset using stratified procedures that preserve HealthBench theme and rubric-complexity distributions and retain all RubricBench Safety and Instruction Following instances.Remaining RubricBench instances are sampled across Chat, STEM, and Code according to task domain, data source, and rubric-count intervals.Source: §A.1 · p. 12
• Figure 2 summarizes the analysis set’s task-category and rubric-count distributions.Source: Figure 2 · p. 12, §A.1 · p. 12

A.2 METRIC

• Coverage tracks whether generated criteria fully cover expert-annotated gold leaves, while valid novelty concerns criteria that are relevant and absent from those leaves.The appendix defines instance-level coverage and valid-novelty measures over generated criteria and gold leaves.Source: §A.2 · p. 13

A.3 COUNTERFACTUAL RESPONSE

• Counterfactual responses alter only content needed to violate targeted gold leaves, retaining an edit only when targeted states change and non-targeted states remain unchanged.Across 1,500 instances, the procedure yields 1,498 single-leaf, 1,482 within-dimension multi-leaf, and 1,409 cross-dimension multi-leaf counterfactual responses.Source: §A.3 · p. 13

A.4 SHUFFLED PATH

• Correct paths generally outperform shuffled paths across judges, seeds, and counterfactual types, indicating gains from hierarchical semantics rather than added context alone.The comparison uses the same random subsets for correct and shuffled paths; correct-path scores therefore vary across seeds and differ from full-set results.Source: §A.4 · p. 13, Table 9 · p. 14

A.5 CALIBRATION-BASED SELECTION FOR RQ2

• Calibration utility rewards response discrimination while penalizing judge variance, empirical redundancy, and unnecessary criteria; calibration follows semantic filtering and replaces its default only with a utility gain of at least 0.05.Calibration responses select candidates exclusively; discrimination, judge variance, and pairwise accuracy are assessed independently on held-out response pairs.Source: §A.5 · pp. 13–14

B.1 CALIBRATION OBJECTIVE

• Each child criterion is independently scored by two calibration judges on the same six responses, with each judge assigning the probability that a response satisfies the criterion.The score is denoted p_c,r,j and lies in [0, 1].Source: §B.1 · p. 14
• A soft-min aggregates child-criterion scores for each response, approximating the minimum so low satisfaction on any criterion lowers the candidate score.For a single-criterion candidate, the aggregated score is set directly to that criterion’s score.Source: §B.1 · p. 14
• Response discrimination measures variation in aggregated scores across responses, while cross-judge agreement measures similarity of judges’ scores on the same criterion-response pair.The objective averages agreement across child criteria.Source: §B.1 · pp. 14–15
• Judge stability captures whether judges show similar discrimination patterns across responses, complementing agreement on individual responses.It is based on each judge’s variance in aggregated scores across responses.Source: §B.1 · p. 15
• Score redundancy measures similarity between child criteria’s scoring profiles across calibration responses; stronger positive rank correlation indicates greater redundancy.For a single-child decomposition, redundancy is set to zero.Source: §B.1 · p. 15

B.2.1 RESPONSE-BASED CALIBRATION SETUP

• ARTS-Hybrid uses six fixed calibration responses per query, generated by three model families, and two independent judges score them; these responses are excluded from final evaluation.Each generator produces two responses, and each calibration judge evaluates all six.Source: §B.2.1 · pp. 15–16, Table 10 · p. 16
• Two independent evaluators assign decreasing average scores from strong to medium to weak responses, supporting the intended quality ordering of the calibration panel.Figure 3 presents the independent validation of the response quality levels.Source: Figure 3 · p. 16, §B.2.1 · p. 16

B.2.2 HYPERPARAMETERS

• Table 11 reports the tree-search settings, semantic and calibration selection thresholds, regeneration settings, and calibration-objective weights and regularization terms.The supplied passages identify the hyperparameter categories but do not state their values.Source: §B.2.2 · p. 16, Table 11 · p. 17

B.2.3 RUBRIC GENERATOR DISTILLATION

• The rubric-generator distillation experiment fine-tunes Qwen3-4B on 3,000 OpenRubrics instances, using identical training examples and hyperparameters across supervision sources.Training uses LoRA for one epoch, and GLM-4.7-Flash judges responses on JudgeBench.Source: §B.2.3 · p. 17

B.2.4 RUBRIC-AS-REWARD TRAINING DETAILS

• Policy optimization starts from Qwen3-4B and uses rubric-criterion judgments from a frozen Qwen2.5-7B-Instruct model as normalized-weight scalar rewards.The experiments use a fixed 3,000-prompt set and Dr.GRPO for three epochs; HealthBench evaluation uses Qwen2.5-7B-Instruct as judge.Source: §B.2.4 · p. 17

C ADDITIONAL RESULTS

• ARTS-Hybrid achieves the highest overall intrinsic rubric quality and leads in concreteness, non-redundancy, and actionability under an independent evaluator.The evaluation is repeated with Llama3.1-405B-Instruct.Source: C ADDITIONAL RESULTS · p. 17, Table 12 · p. 18
• Training diagnostics show ARTS-Hybrid provides informative rewards to more prompt groups than Direct-Prompt, while stable response lengths and KL losses suggest gains are not driven by longer outputs or more aggressive updates.OpenRubrics Expert has the highest informative-group rate, but ARTS-Hybrid achieves better downstream performance.Source: C ADDITIONAL RESULTS · pp. 17–18, Figure 4 · p. 18
• ARTS-Hybrid achieves the highest pairwise accuracy with both Qwen3.5-397B-A17B and Llama3.1-405B-Instruct, while one-shot rubric generation varies across judges.The authors report consistent improvements over direct judgment, particularly with Llama3.1-405B-Instruct.Source: C ADDITIONAL RESULTS · p. 18, Table 13 · p. 18
• Removing semantic validation lowers performance across all four benchmarks, while removing response-based calibration lowers performance on RM-Bench Chat, PreciseIF, and JudgeBench but leaves Focus comparable.The authors report a more consistent effect across benchmarks for semantic validation.Source: C ADDITIONAL RESULTS · pp. 18–19, Table 14 · p. 19

D PROMPTS

• The prompts section omits auxiliary prompts for formatting, retries, and other implementation-specific operations.The appendix presents core prompts used in mechanistic analysis and ARTS-Hybrid.Source: D PROMPTS · p. 19, §D.2 · p. 22

D.1 PROMPTS FOR MECHANISTIC ANALYSIS

• The mechanistic-analysis prompts cover flat and hierarchical rubric generation, recursive decomposition, criterion judgments, and semantic-versus-calibration selection.Calibration-based candidate selection uses a programmatic utility rather than an additional model call; the same judging protocol holds criterion judgments fixed when varying aggregation.Source: §D.1 · pp. 19–22

D.2 PROMPTS FOR ARTS-HYBRID

• ARTS-Hybrid prompts generate child criteria, assess semantic feasibility, and globally validate the final leaf set; admissible criteria must be concrete, verifiable, unambiguous, and task-specific.A valid but non-atomic criterion may remain an internal node but cannot be a final leaf.Source: §D.2 · pp. 22–24

E CASE STUDY OF ARTS-HYBRID RUBRIC CONSTRUCTION

• In the case study, ARTS-Hybrid separates independently assessable requirements, whereas direct generation combines two cross-text connections and adds constraints on supporting phrases and sentence form.The example concerns a response task connecting an article to Chapter One and the Prologue of Hidden Figures.Source: E CASE STUDY OF ARTS-HYBRID RUBRIC… · p. 24