Source-linked AI summary
Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation
Jiayuxuan Yang, Jie M. Zhang, Yiling Lou, Zhenpeng Chen
TL;DR
Automatically generated rubrics can struggle to capture the multiple dimensions of response quality. Mubric uses controlled response defects to refine rubrics, achieving the highest overall evaluation accuracy across 703 tasks in four domains and exceeding the strongest baseline by 7.48 percentage points.
Problem
Automatically generating rubrics that reliably capture task-specific quality requirements is challenging because response quality is multidimensional.
Source: §1 · p. 1Method
Mubric derives reusable mutation operators from real response defects and uses insufficiently penalized mutations of reference responses to refine task-specific rubrics.
Source: §1 · p. 2Results
57.27% overall evaluation accuracy, 7.48 percentage points above the strongest baseline, across 703 tasks in four representative domains.
Source: §1 · p. 2Takeaways & Limitations
The results support mutation testing as a complementary rubric-generation perspective that probes which quality requirements rubrics fail to capture.
Source: §6 · p. 10The reported overlap check establishes that none of the 703 evaluation tasks appeared in the OpenRubrics data used to construct mutation operators.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Rubric-based evaluation is widely used to assess LLM-based systems by decomposing response quality into task-specific scoring criteria. However, automatically generating rubrics that reliably capture task-specific quality requirements remains challenging. We introduce Mubric, a mutation testing-guided approach to rubric generation. Mutation testing, a classic software testing methodology, evaluates a test suite by injecting faults into programs and checking whether the tests detect them. We draw an analogy between test suites and rubrics: if a rubric captures an important quality requirement, introducing a corresponding defect into an otherwise high-quality response should reduce its score. Mubric first mines common defects from real pairs of preferred and dispreferred responses and abstracts these defects into reusable mutation operators, each specifying how to introduce a particular type of response defect. For a new task, it applies relevant operators to a reference response, checks whether the injected defects reduce response quality, and uses insufficiently penalized defects to refine the rubric. We evaluate Mubric on 703 tasks across four representative domains against six advanced rubric generation methods. Mubric achieves the highest overall evaluation accuracy, outperforming the strongest baseline by 7.48 percentage points.
1 INTRODUCTION
Mubric uses controlled response defects to test whether task-specific rubrics capture important quality requirements, then refines rubrics using defects they insufficiently penalize. Across 703 tasks in four domains, it achieves the highest overall evaluation accuracy among six recent baselines.
1 INTRODUCTION
- Rubric generation remains challenging because response quality is multidimensional and difficult to capture comprehensively.Rubrics decompose response quality into explicit, task-specific scoring criteria.Source: §1 · p. 1
- Unlike approaches drawing on task instructions, model responses, or evaluator perspectives, Mubric investigates controlled response defects as rubric-generation feedback.The paper publicly releases its scripts and data to support reproducibility and future research.Source: §1 · pp. 1–2
- Mubric abstracts defects in preferred–dispreferred response pairs into reusable mutation operators and refines task rubrics when injected defects receive little or no score reduction.The approach treats weak penalties as evidence that a corresponding quality requirement is insufficiently captured.Source: §1 · p. 2
- 57.27% overall evaluation accuracy, 7.48 percentage points above the strongest baseline, with gains broadly consistent across four domains.The evaluation covers 703 tasks and six recent rubric-generation baselines.Source: §1 · p. 2
2 RELATED WORK
Prior work treats rubric-based evaluation as a way to assess LLM responses with explicit criteria and has developed methods for generating rubrics from task descriptions, model responses, and evaluator perspectives. Mubric distinguishes itself by using controlled response defects to guide rubric generation.
2 RELATED WORK
- Rubric-based evaluation decomposes response quality into explicit criteria; examples include G-Eval [Liu et al., 2023], FLASK [Ye et al., 2024], and BiGGen [Kim et al., 2025].RubricBench [Zhou et al., 2026] and RM-Bench [Liu et al., 2025] assess whether evaluators distinguish preferred from dispreferred responses despite subtle quality differences and misleading presentation.Source: §2 · p. 2
- Rubric-generation methods derive criteria from task descriptions, candidate responses, preference contrasts, or complementary evaluator roles, while Mubric uses injected response defects as feedback.Examples include TICK [Cook et al., 2024], Dynamic [Wang & Blanco, 2026], and MRRG [Fu et al., 2026].Source: §2 · p. 2
3 METHODOLOGY
Mubric generates task-specific rubrics by testing whether they penalize controlled defects in a reference response. It builds reusable defect operators from real response pairs, then uses under-penalized mutations to guide rubric refinement.
3.1 MUBRIC: IN A NUTSHELL
- Mubric generates an initial rubric and reference response, tests the rubric against validated, task-relevant mutations, and retains refinements only when they improve defect penalization.If no applicable operator or validated mutation is available, or all defects are already sufficiently penalized, it returns the initial rubric.Source: §3.1 · p. 3, §3.3 · pp. 4–5
3.2 STAGE 1: MUTATION OPERATOR CONSTRUCTION
- Mubric extracts concrete defects by comparing preferred and dispreferred responses, then organizes recurring patterns by transformation type and affected quality aspect.The source sample contains 1,000 tasks; the resulting operators encode applicability conditions and instructions for introducing defects, yielding 201 reusable operators.Source: §3.2 · p. 4
- The abstraction groups defects that share a modification pattern across tasks, such as invalid concept application in mathematical and coding responses.The shared pattern preserves surrounding content while altering a concept application, allowing a defect observed in one task to be instantiated in another.Source: §3.2 · p. 4
3.3 STAGE 2: MUTATION-GUIDED RUBRIC GENERATION
- For each mutation, Mubric compares rubric scores on the reference and mutated responses; either an overall decrease or an item-level decrease can mark sufficient detection.The thresholds δtotal and δitem capture aggregate change and a strong penalty on one criterion; insufficient penalties are reevaluated three times with majority voting.Source: §3.3 · p. 5
- Under-penalized defects guide a refiner to clarify criteria, adjust weights, or add criteria, and Mubric keeps a revision only if retesting improves defect penalization.The revision expresses a general task requirement rather than a rule tailored to one mutated response.Source: §3.3 · p. 5
4 EVALUATION SETUP
The evaluation asks whether Mubric distinguishes preferred from dispreferred responses and how its components contribute, across 703 tasks and four domains. It compares recent rubric-generation baselines and ablations under controlled model settings.
4.1 RESEARCH QUESTIONS (RQS)
- The evaluation addresses Mubric’s effectiveness, the contributions of its components, and the mutation operators derived from real defects and selected across domains.Source: §4.1 · p. 5
4.2 DATASETS AND EVALUATION METRIC
- The 703 tasks span STEM, Code, instruction following, and Chat, using RubricBench and RM-Bench; RubricBench’s Safety subset is excluded because its policy constraints may be unspecified.The study also checks that no evaluation task overlaps with the OpenRubrics data used to construct operators; accuracy is the proportion of tasks where the rubric ranks the preferred response higher.Source: §4.2 · p. 6
4.3 BASELINE METHODS
- Baselines cover instruction decomposition, response-based criteria, and multiple evaluator perspectives; ablations remove mutation-guided refinement, mutation feedback, or reusable operators.TICK uses binary checklist questions; RLCF has candidate-response, reference-response, and combined configurations; Dynamic generates task-specific rubrics, while MRRG combines evaluator-role criteria.Source: §4.3 · p. 6
4.4 IMPLEMENTATION DETAILS
- Comparisons use GPT-5.4-mini and Gemini-2.5-flash-lite as rubric-generation backbones and judges, while Mubric’s auxiliary components use GPT-5.4.The detection thresholds are δtotal = 8 and δitem = 2; the reported sensitivity analysis finds accuracy stable across a broad range of threshold settings.Source: §4.4 · p. 7
5 RESULTS
Mubric leads the overall evaluation and performs strongly across settings; ablations support both reusable mutation operators and mutation-guided refinement. Operator patterns and selections also vary with task requirements.
5.1 RQ1: EFFECTIVENESS
- 57.27% overall accuracy gives Mubric the lead over MRRG at 49.78% by 7.48 percentage points, with the highest accuracy in eight of ten settings.Seven settings show statistically significant improvements over the strongest baseline; three show no significant difference, including two where MRRG is slightly higher.Source: §5.1 · p. 7
5.2 RQ2: ABLATION STUDY
- 51.78% overall accuracy for full Mubric exceeds Initial (47.94%), Direct-Refine (45.38%), and On-the-Fly Ops (44.24%) across every domain.The ablation used Gemini-2.5-flash-lite because of computational budget constraints.Source: §5.2 · pp. 7–8, Table 2 · p. 8
- Replacing reusable operators with task-specific operators generated on the fly lowers accuracy from 51.78% to 44.24%, a 7.54-point drop.On-the-Fly Ops underperforms Mubric in every domain.Source: §5.2 · p. 8
- Mutation-guided refinement improves accuracy over Initial by 3.84 points, while Direct-Refine without mutation feedback scores 45.38% versus Initial's 47.94%.Direct-Refine trails Mubric by 6.40 percentage points.Source: §5.2 · p. 8
5.3 RQ3: OPERATOR ANALYSIS
- The 201 reusable operators span transformation types and quality aspects, with Logical Correctness × Break Logic the largest individual category at 17 operators.Across datasets, selected operators align with task requirements: STEM emphasizes Factuality and Fabricate; Code emphasizes Logical Correctness and Break Logic; IF emphasizes Comprehension, Completeness, Distort Form, and Omit. RB-Chat and RM-Chat share dominant categories but differ in their relative frequencies.Source: §5.3 · pp. 8–9, Figure 2 · p. 8
6 CONCLUSION
The conclusion presents Mubric as a mutation-testing approach to rubric generation and reports an overall accuracy advantage, with ablations supporting both major components.
6 CONCLUSION
- Mubric constructs reusable mutation operators from real response defects and uses under-penalized mutations to guide refinement of task-specific rubrics.It applies controlled mutations to reference responses to test whether generated rubrics capture task-specific quality requirements.Source: §6 · p. 9
- Across 703 tasks spanning four domains, Mubric exceeds the strongest baseline by 7.48 percentage points.Ablation results support contributions from both reusable mutation operators and mutation-guided refinement.Source: §6 · pp. 9–10
- The conclusion proposes actively probing what rubrics fail to capture as a complementary perspective alongside extracting criteria from task descriptions or observed responses.The authors hope this perspective provides a useful basis for more reliable and systematic rubric-based LLM evaluation.Source: §6 · p. 10
AI USE STATEMENT
LLMs support Mubric's experimental pipeline and related writing tasks, while the authors retain responsibility for the research and manuscript.
AI USE STATEMENT
- LLMs are used for rubric generation and refinement, mutation-related operations, and response evaluation.These uses are described in the methodology and experimental setup.Source: AI USE STATEMENT · p. 10
- AI tools also assist with code implementation and manuscript proofreading.Source: AI USE STATEMENT · p. 10
- The authors are responsible for methodological design, experimental setup, analysis, interpretation, and the final manuscript, including AI-assisted content.Source: AI USE STATEMENT · p. 10
A RUBRIC REFINEMENT EXAMPLES
Tables 4–6 illustrate Stage 2 rubric refinement through examples involving a false factual claim in a recipe, an incorrect intermediate calculation, and code displaying the wrong page after form submission.
A RUBRIC REFINEMENT EXAMPLES
- The examples cover a false recipe claim, an incorrect intermediate calculation, and code displaying the wrong page after form submission.Each is shown as a task-specific rubric-refinement case.Source: A RUBRIC REFINEMENT EXAMPLES · p. 11, Table 4 · p. 12, Table 5 · p. 12
- Each table presents the task, response change, injected defect, and relevant rubric items before and after refinement.Transformation type describes how the response changes, while quality aspect describes the affected quality.Source: A RUBRIC REFINEMENT EXAMPLES · p. 11
- A defect counts as sufficiently detected when the reference scores at least 8 points higher overall or at least 2 points higher on any rubric item.If neither threshold is reached, the defect is not sufficiently detected.Source: A RUBRIC REFINEMENT EXAMPLES · p. 11
B THRESHOLD SENSITIVITY
Mubric uses overall-score and rubric-item thresholds to determine whether injected defects are sufficiently penalized. Accuracy remains stable across most tested settings, supporting use of the default thresholds without dataset- or model-specific tuning.
B THRESHOLD SENSITIVITY
- Mubric treats an injected defect as sufficiently penalized when either the overall score decreases by at least δtotal or an item score decreases by at least δitem.The study uses (δtotal, δitem) = (8, 2) as its default.Source: B THRESHOLD SENSITIVITY · p. 14
- The item-level threshold captures a targeted local score decrease that may be attenuated in the normalized aggregate score.Items use a 0–4 scale and the weighted aggregate is normalized to 0–100; the defaults are δitem = 2 and δtotal = 8.Source: B THRESHOLD SENSITIVITY · p. 14
- The sensitivity analysis tested seven unique settings by varying δtotal ∈ {4, 8, 12, 16} at δitem = 2 and δitem ∈ {1, 2, 3, 4} at δtotal = 8.The overlapping (8, 2) setting appears in both sweeps.Source: B THRESHOLD SENSITIVITY · p. 14
- 42.64% accuracy held for six of seven threshold settings; only (δtotal, δitem) = (8, 4) yielded 41.86%.Figure 3 plots final accuracy while varying one threshold at a time and fixing the other at its default.Source: B THRESHOLD SENSITIVITY · p. 14, Figure 3 · p. 15
- The authors use (8, 2) throughout the main experiments rather than tuning thresholds separately for datasets or LLMs.They report that effectiveness is not sensitive to a narrow choice of threshold values.Source: B THRESHOLD SENSITIVITY · p. 15