Source-linked AI summary
SedarEval: Automated Evaluation using Self-Adaptive Rubrics
Zhiyuan Fan, Weinong Wang, Xing Wu, Debing Zhang
TL;DR
LLM-as-a-judge evaluation can reduce human labor, but generic rubrics do not capture question-specific scoring criteria. The paper introduces self-adaptive rubrics, SedarEval, and a specialized evaluator LM; its evaluator reports higher consistency with human evaluation than GPT-4.
Problem
Existing LLM evaluators use question-agnostic rubrics that overlook question-specific scoring emphases, while human evaluation requires substantial resources.
Source: §1 · p. 1Method
The paper creates question-specific rubrics with scoring and penalty points, builds SedarEval, and trains a specialized evaluator LM using this paradigm.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The evaluator LM offers an automated scoring approach designed to maintain alignment with human judgment.
Source: §7 · p. 8For questions with multiple correct answers, the approach requires manually writing multiple self-adaptive rubrics.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach. We release our dataset at https://github.com/wwn1233/sedareval.
1 Introduction
The paper proposes question-specific self-adaptive rubrics to address the limits of generic criteria in LLM evaluation. It builds SedarEval and trains an evaluator LM that achieves higher agreement with human evaluation than GPT-4.
1 Introduction
- Generic, question-agnostic rubrics overlook each question’s distinct scoring emphases, while automated metrics and human grading respectively face preference-alignment and resource challenges.The paper notes that human evaluation requires substantial resources and that automated metric-based evaluation often falls short amid diverse ground truths.Source: §1 · p. 1
- Self-adaptive rubrics specify question-level scoring and penalty points, with primary and secondary criteria intended to align evaluation more closely with human judgment.The rubric considers preferred and rejected responses and assigns different values to scoring points.Source: §1 · pp. 1–2
- The authors investigate evaluator-LM training, including data-diversity and evaluation-format gaps, and use the self-adaptive rubric paradigm to train a specialized evaluator.They also report analyzing whether LLMs correctly evaluate questions they can answer correctly and propose human-AI consistency for evaluator training.Source: §1 · p. 2
- SedarEval contains 1,000 questions with detailed rubrics, and the authors manually evaluate 20 LLMs.The benchmark is described as covering eight major categories and as having questions and rubrics constructed under the self-adaptive rubric paradigm.Source: §1 · p. 2
- The specialized evaluator LM surpasses GPT-4 in agreement with human evaluations, including model- and question-level Pearson correlation, GSB, and ACC metrics.The reported metrics indicate higher consistency with human judgment; no metric values are given in these passages.Source: §1 · p. 2
2 Related Work
Prior work spans capability benchmarks, text-generation evaluation, and measures of evaluator confidence. The paper positions its contribution against question-agnostic evaluation criteria and a focus on model-level rather than individual-question alignment.
Benchmark LLMs Capabilities
- Reference-free benchmarks directly assess generated-text quality, and the authors characterize their benchmark as offering diverse test cases and a broad label distribution for comprehensive, stable model assessment.The preceding discussion notes that NLU-style benchmarks use general-exam questions across domains and may be misaligned with human preferences in real-world use.Source: §2 · pp. 2–3
Automatic NLG Evaluation
- Existing generated-text evaluation approaches, including trained scoring models, may use reference answers or general rubrics and overlook differing criteria across questions, even within one category.The paper contrasts its question-customizable rubrics with Prometheus 2, whose rubrics it describes as question-agnostic.Source: §2 · p. 3
- Traditional n-gram and embedding metrics assess lexical or semantic similarity to references and have been found to correlate relatively weakly with human preferences.The paper also notes concerns with proprietary LLM judges, including cost and potential evaluation-data leaks, motivating specialized open-source scoring models.Source: §2 · p. 3
Quantifying Evaluation Confidence
- LLM-evaluator studies commonly measure alignment using Pearson, Spearman, or Kendall-Tau correlations, but prior work has primarily examined model-level rankings or scores rather than individual-question ratings.The paper says this focus limits interpretability and utility for guiding model development and iteration.Source: §2 · p. 3
3 SedarEval Benchmark
SedarEval combines tailored rubrics with human-assessed benchmark construction and a pipeline that aggregates response scores. Its rubric design uses problem-specific criteria, penalties, and background context.
3 SedarEval Benchmark
- The benchmark construction includes comprehensive human assessment and filters out samples that do not meet its quality standards.The section introduces SedarEval as a benchmark built on the self-adaptive rubric paradigm.Source: §3 · p. 3
3.1 Self-Adaptive Rubrics
- Self-adaptive rubrics tailor criteria and weights to each problem, use penalty points for deviations, and provide background knowledge to support scoring.The paper identifies scoring points, penalty points, and background knowledge as the rubric’s three components.Source: §3.1 · pp. 3–4
3.2 Dataset Construction
- Question candidates are retained after unanimous agreement among five annotators, model-score variance screening, and pairwise review for similarity of problem-solving ideas.Three individuals jointly generate each question’s rubric.Source: §3.2 · p. 4
3.3 Evaluation Pipeline
- The evaluation pipeline generates a model response from a question, scores it with the corresponding rubric using humans or LLMs, and aggregates scores into a total.Source: §3.3 · p. 4
4 Evaluator Language Model
The evaluator LM uses self-adaptive rubrics in a direct-scoring format, with human-consistent training data and methods for generating rubrics automatically. The section describes the evaluation prompt, data filtering, and two approaches to rubric generation.
4.1 Evaluation Format
- Direct assessment scores individual responses, avoiding the greater comparison burden of pairwise evaluation while still allowing scores to be compared.The prompt includes each question’s reference answer, self-adaptive rubrics, and scoring examples.Source: §4.1 · p. 4
4.2 Human-AI Consistency
- Human-AI Consistency retains training examples only when GPT-4 scores agree with human scores, rather than selecting the best of multiple GPT-4 outputs.Human annotators provide scores without explanations; GPT-4 supplies reasoning steps for training.Source: §4.2 · pp. 4–5
4.3 Automatic Rubric Generation
- The rubric-generation model is first supervised-fine-tuned on human-labeled data, then preference-trained with base-model rubrics as rejected and human rubrics as preferred responses.The second stage uses Direct Preference Optimization to align the model with human preferences.Source: §4.3 · p. 5
- A reference-free GPT-4 approach generates both a rubric and an ideal answer, accepts the rubric when the answer matches ground truth, and iteratively refines outputs.The authors present this as another approach to automating rubric generation.Source: §4.3 · p. 5
5 Experiment
The experiments describe model training, evaluation metrics, and the response sources used to construct evaluator-LM training data. The metrics assess agreement with human judgments, scoring against rubrics, model comparisons, and generated-rubric quality.
5.1 Experimental Setting
- The evaluator LM and rubric-generation model are trained with LLaMA-3 and internal model XD1, using tensor parallelism and GPU counts scaled by model size.The setup uses 128 H100 GPUs for 7B/8B models and 512 for 70B models.Source: §5.1 · p. 5
5.2 Evaluation Metrics
- Pearson and Spearman correlations measure agreement with human evaluators, while accuracy and a weaker threshold measure adherence to rubric scores.The weaker threshold counts a result as correct when it falls within a specified range.Source: §5.2 · p. 5
- GSB compares two models by counting cases where model A is better, worse, or equivalent to model B.The metric is intended to help assess whether a model version outperforms its predecessor.Source: §5.2 · p. 6
- GPT-4 measures generated-rubric quality by matching automatically generated rubric items against the ground-truth rubric set.The evaluation draws on ACU and FactScore paradigms.Source: §5.2 · p. 6
5.3 Selected Models
- Training responses are collected from a broader range of language models and versions to better reflect real-world distributions and mitigate single-source synthetic-data bias.The selected models include systems proficient in different languages.Source: §5.3 · p. 6
6 Analysis
The analysis tests the benchmark’s robustness across question sampling, score distributions, out-of-distribution evaluation, evaluator training, and rubric components. It also compares self-adaptive rubrics with general rubrics.
6 Analysis
- The analysis examines benchmark robustness across data distribution, evaluator training phases, and evaluator training paradigms before developing an evaluator aligned with self-adaptive rubrics.The authors report limitations in current evaluator-LM training methodologies.Source: §6 · p. 6
6.1 Scaling Law for Robust Evaluation
- Model rankings become more consistent as question-set size increases, stabilizing with minor fluctuations at approximately 300 questions.The analysis compares rankings from two independent samples drawn without replacement from the 1,000-question pool.Source: §6.1 · pp. 6–7, Figure 2 · p. 7
6.2 Score Distribution Shift
- Despite score-distribution shifts between training and test data, model-output score distributions closely align with human-provided ground truth.The method uses genuinely collected model responses rather than score-balanced synthetic responses.Source: §6.2 · p. 7, Figure 3 · p. 7
6.3 Out of Distribution Evaluation
- Under self-adaptive rubrics, the evaluator performs well when tested on held-out models and on held-out questions, indicating generalization across both dimensions.Model-level evaluation holds out models; question-level evaluation holds out questions.Source: §6.3 · p. 7
6.4 Merged SFT or Continual SFT
- Mixing evaluator-format data into SFT from a pretrained model preserves general ability while supporting strong evaluation; Human-AI Consistency filtering improves both capabilities over alternative data selection.Continual SFT performs well on evaluation but significantly reduces general ability.Source: §6.4 · p. 7, Table 2 · p. 8
6.5 Ablation Study
- Adding self-adaptive rubrics substantially increases evaluator-human scoring consistency, with smaller improvements from reference answers and in-context examples.The ablation identifies self-adaptive rubrics as the primary driver of improved alignment.Source: §6.5 · p. 8, Table 3 · p. 8
6.6 Comparison with Alternative Paradigm
- The self-adaptive rubric approach significantly outperforms general-rubric methods and provides fine-grained capability evaluations aligned with human assessments.The authors describe these evaluations as practical for iterative LLM development.Source: §6.6 · p. 8
7 Conclusion
The paper introduces self-adaptive rubrics and an associated benchmark, and proposes an evaluator LM trained to follow this paradigm. The evaluator LM achieves higher consistency with human evaluation than GPT-4.
7 Conclusion
- Self-adaptive rubrics tailor scoring criteria to each question to align scoring with human judgment and reduce bias.The paper introduces a benchmark based on this paradigm and an evaluator LM that follows it.Source: §7 · p. 8
- The paper identifies training-phase data-diversity issues and introduces Human-AI Consistency to align chain-of-thought evaluation with human judgment.The evaluator LM is proposed to automate scoring under the self-adaptive-rubrics paradigm.Source: §7 · p. 8
- The evaluator LM achieves higher consistency with human evaluation than GPT-4.Source: §7 · p. 8
Limitations
The authors identify two limitations of self-adaptive rubrics: multiple-correct-answer questions require multiple manually written rubrics, and subjective work may be judged by differing standards.
Limitations
- Multiple-correct-answer questions require manually writing multiple self-adaptive rubrics.The authors note that, to their knowledge, no current work focuses on the multi-solution direction.Source: Limitations · p. 8
- For subjective work, varying definitions of quality mean each group or individual needs rubrics aligned with its own standards rather than predefined ones.The authors name creative writing and poetry as examples and describe the paradigm as flexible and interpretable.Source: Limitations · pp. 8–9
Ethical Considerations
The authors frame self-adaptive rubrics as improving interpretability and controllability, while noting that LLM hallucinations and differing subjective values constrain reliable automated evaluation.
Ethical Considerations
- Self-adaptive rubrics are proposed to enhance the interpretability and controllability of automated scoring.The authors say the approach aims to improve the credibility of LLM-generated evaluation results.Source: Ethical Considerations · p. 9
- Inherent LLM hallucinations challenge complete accuracy, so the authors recommend human review of certain judged outputs.They describe this review as a way to increase the process’s overall reliability and credibility.Source: Ethical Considerations · p. 9
- Subjective rubrics may reflect group or individual value differences, potentially leading to bias and discrepancies.The authors encourage dialogue and mutual understanding when aligning rubrics with those values and preferences.Source: Ethical Considerations · p. 9
A Training Details
The supplementary material specifies training and annotation practices, evaluator prompts, and rubric-generation procedures, including filtering to address error propagation.
A Training Details
- Training used a 2e-5 default learning rate and Adam, with batch size adjusted to data volume and machine count.For scaling experiments with n below 300, results averaged three repetitions; invoked APIs used default parameters.Source: A Training Details · p. 12
B.1 Annotator Qualifications
- Annotators were internal team members with at least a Master’s degree and received additional compensation above the standard salary.Compensation was based on the amount of data annotated.Source: §B.1 · p. 12
C.1 AutoPrompt
- The AutoPrompt asks a judge to assess a model output using the question, reference answer, and rubric, with a scoring process and final score from 0 to 5.The prompt requires justification based strictly on the rubric, without adding criteria.Source: §C.1 · p. 12
C.2 Prompt for Set Matching
- The set-matching prompt adds a point for an exact rubric match, subtracts one for an exact opposite, and leaves unrelated items unchanged.The output includes the scoring process and final score.Source: §C.2 · pp. 12–13
C.3 Prosecutor Prompt
- The Prosecutor prompt checks a generated answer against the reference answer and requests a correctness result in a fixed format..Source: §C.3 · p. 13
C.4 Refinement Prompt
- The refinement prompt asks for a new answer when the generated answer is not standard, using the scoring points and full-score answer as guidance.The prompt first asks the model to reflect on the mismatch.Source: §C.4 · p. 13
D.1 Automatic Rubric Generation
- Automatic rubric experiments compare GPT-4 generation, SFT on human-labeled rubrics, and DPO alignment; the generation procedure iterates until acceptance or its limit.A prosecutor checks the ideal answer, and GPT-4 is informed when it is incorrect.Source: §D.1 · p. 13
D.2 Error Propagating
- An inconsistent generated rubric can propagate errors and increase final-score deviation; the authors report that filtering improves overall performance.The concern is a mismatch between a generated rubric and a human-provided rubric.Source: §D.2 · p. 13
D.3 Joint Training vs. Expert Training
The experiments compare joint category training with category-specific expert training for evaluator LMs and rubric-generation models, and report better results with joint training.
D.3 Joint Training vs. Expert Training
- The authors explored whether to combine data, but the cited fragment does not identify the data sources or report the exploration’s outcome.Source: §D.3 · p. 13
- The reported advantage applies to training either evaluator LMs or rubric-generation models across categories.Source: §D.3 · p. 13
- Joint training achieved better results than expert training.Source: §D.3 · p. 13