PaperlayerView exact revision on arXiv

Source-linked AI summary

RocketEval: Efficient Automated LLM Evaluation via Grading Checklist

Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, Jianghong Ma

arXiv:2503.05142v1cs.CL

TL;DR

Human preference evaluation is costly, and powerful LLM judges raise cost, privacy, and reproducibility concerns. RocketEval has lightweight LLMs grade instance-specific checklist items and aggregates the judgments. With Gemma-2-2B, it reports 0.965 Spearman correlation with human preferences, comparable to GPT-4o at 2% of the evaluation cost.

  • Problem

    Human annotation is costly and difficult to reproduce, while powerful LLM judges also incur high costs and raise privacy and reproducibility concerns.

    Source: §1 · p. 1

  • Method

    RocketEval has a powerful LLM create an instance-specific checklist, uses lightweight LLMs to grade its items, and aggregates the scores with an unsupervised mean or supervised predictor.

    Source: §3 · p. 6

  • Results

    0.965 Spearman correlation with human preferences was achieved by RocketEval using Gemma-2-2B, comparable to GPT-4o at 2% of the evaluation cost.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RocketEval is presented as suitable for efficient, reliable, and reproducible LLM evaluation.

    Source: §1 · p. 3

Abstract

from arXiv · show

Evaluating large language models (LLMs) in diverse and challenging scenarios is essential to align them with human preferences. To mitigate the prohibitive costs associated with human evaluations, utilizing a powerful LLM as a judge has emerged as a favored approach. Nevertheless, this methodology encounters several challenges, including substantial expenses, concerns regarding privacy and security, and reproducibility. In this paper, we propose a straightforward, replicable, and accurate automated evaluation method by leveraging a lightweight LLM as the judge, named RocketEval. Initially, we identify that the performance disparity between lightweight and powerful LLMs in evaluation tasks primarily stems from their ability to conduct comprehensive analyses, which is not easily enhanced through techniques such as chain-of-thought reasoning. By reframing the evaluation task as a multi-faceted Q&A using an instance-specific checklist, we demonstrate that the limited judgment accuracy of lightweight LLMs is largely attributes to high uncertainty and positional bias. To address these challenges, we introduce an automated evaluation process grounded in checklist grading, which is designed to accommodate a variety of scenarios and questions. This process encompasses the creation of checklists, the grading of these checklists by lightweight LLMs, and the reweighting of checklist items to align with the supervised annotations. Our experiments carried out on the automated evaluation benchmarks, MT-Bench and WildBench datasets, reveal that RocketEval, when using Gemma-2-2B as the judge, achieves a high correlation (0.965) with human preferences, which is comparable to GPT-4o. Moreover, RocketEval provides a cost reduction exceeding 50-fold for large-scale evaluation and comparison scenarios. Our code is available at https://github.com/Joinn99/RocketEval-ICLR .

1 INTRODUCTION

Automated LLM evaluation is needed because human annotation is costly and difficult to reproduce, while powerful LLM judges also raise cost, privacy, and reproducibility concerns. RocketEval uses lightweight judges with instance-level checklists and reports human-aligned judgments at substantially lower evaluation cost.

1 INTRODUCTION

  • Human-vote evaluation is costly and challenging to reproduce and interpret, and powerful LLM judges bring cost, privacy, and reproducibility concerns.The introduction also notes demand for more efficient, reproducible, and interpretable evaluation.Source: §1 · pp. 1–2
  • Fine-tuned lightweight judges can be cost-effective and competitive, but fine-tuning may degrade other capabilities and impair instruction comprehension.The cited discussion also notes that evolving base models and new data can require costly iterative updates.Source: §1 · pp. 2–3
  • More powerful LLM judges tend to align better with human preferences, while CoT prompting has minimal impact on lightweight judges.High uncertainty and positional bias are identified as potential contributors to this limitation.Source: §1 · p. 3
  • 0.965 Spearman correlation with human preferences was achieved by RocketEval using Gemma-2-2B, comparable to GPT-4o at 2% of the evaluation cost.The reported cost comparison is for large-scale evaluation scenarios.Source: §1 · p. 3
  • RocketEval creates an instance-level checklist, grades its items independently with lightweight LLMs, and aggregates normalized judgments into a final score.A supervised prediction process aligns lightweight-model scores with human annotations.Source: §1 · p. 3

2 HOW LIGHTWEIGHT LLMS PERFORM AS A JUDGE?

LLM judges’ agreement with human preferences depends on model capability, and lightweight judges show uncertainty and position bias. Their own analysis adds little over direct scoring, while analysis from a powerful LLM improves evaluation performance.

2.1 SETUP

  • The benchmarks use point-wise scoring in which judges analyze a response before assigning a score from 1 to 10.MT-BENCH has 80 multi-round queries across domains; WILDBENCH has 1,024 real-world queries and uses GPT-4o as judge.Source: §2.1 · p. 4

2.2 HOW DO LIGHTWEIGHT LLM JUDGES PERFORM?

  • LLM–human agreement ranges from human–human agreement (64.8%) to below random (33.3%), and judge capability significantly affects alignment with human preferences.The evaluation uses point-wise scores converted to pairwise comparisons, retaining cases with score differences below 0.1 as ties.Source: §2.2 · p. 4

2.4 WHY DO LIGHTWEIGHT LLMS NOT GOOD AT JUDGING?

  • Qwen-2-1.5B’s checklist judgments disagree across more than 50% of three repeated samples, indicating high uncertainty.Checklist analysis is tested as independent Yes/No judgments on WILDBENCH items.Source: §2.4 · p. 5, Figure 4 · p. 5
  • Inconsistency grows as more preceding checklist questions are added, and smaller models tend to be less consistent.The Llama-3 series shows higher overall consistency than the Qwen-2 series.Source: §2.4 · p. 5, Figure 5 · p. 5
  • Lightweight judges exhibit both high uncertainty and position bias, making reliable judgments difficult.The authors propose that an efficient evaluation method should avoid these defects.Source: §2.4 · p. 6

3 METHODOLOGY

RocketEval uses instance-specific checklists to guide lightweight LLM judges, then combines checklist judgments into response scores. Its scoring can use an unsupervised mean or a supervised predictor adjusted for annotation distribution.

3 METHODOLOGY

  • RocketEval creates a checklist per query with a powerful LLM, has lightweight judges assess its items, and derives final scores from those judgments.The checklist is intended to be query-relevant, discriminative across responses, and composed of independent questions; 5–10 items are created per instance and reused across responses.Source: §3 · p. 6, §3.1 · p. 6

3.2 CHECKLIST GRADING

  • Judges assess checklist questions independently, while conditional normalized scores account for Yes/No probabilities so uncertain judgments have less impact.Independent queries reduce interference between checklist-item judgments; prefix caching can reduce their computational cost.Source: §3.2 · p. 6

3.3 SCORE PREDICTION

  • The unsupervised score is the arithmetic mean of normalized checklist-item scores, while supervised prediction fits a model to checklist judgments and annotations.The paper uses an Extremely Randomized Tree estimator for supervised prediction when annotations are available.Source: §3.3 · p. 7
  • Because annotations may not distinguish query difficulty well, RocketEval adjusts the supervised predictor’s influence using their distribution and an ideal uniform score distribution.The method defines the maximum KL divergence from the ideal distribution as ε and combines the supervised and unsupervised scores using α_r.Source: §3.3 · p. 7

4 EXPERIMENTS

Experiments compare RocketEval with human judgments and other evaluation approaches, estimate evaluation costs, and examine checklist content and reweighting. The reported results cover ranking correlation, instance-level agreement, and large-scale evaluation costs.

4.1 HUMAN AGREEMENT ON THE EVALUATION

  • Latest 7–12B judges exceed 64% agreement with MT-BENCH human judgments, and RocketEval raises smaller judges above 60%, outperforming GPT-4.The fixed checklist shows no notable improvement and may degrade performance; the authors attribute the gains to query-tailored checklist context.Source: §4.1 · p. 8, Table 2 · p. 8
  • 0.986 Spearman correlation with CHATBOT ARENA ELO RATING is reported for Mistral-Nemo, while Gemma-2-2B reaches 0.965 on WILDBENCH rankings.The paper reports that Mistral-Nemo surpasses GPT-4o and Gemma-2-2B surpasses Qwen2-72B and Prometheus-7B-v2.0; supervised RocketEval has higher correlation than its unsupervised version.Source: §4.1 · p. 8, Table 3 · p. 9

4.2 EVALUATION COST ESTIMATION

  • On WILDBENCH, conducting 1,000 tests costs $3,400 with GPT-4o as evaluator versus $71 with Llama-3-8B, which achieves higher correlation with human preferences.Checklist generation is a one-time process, and its cost becomes increasingly significant as the number of tests escalates.Source: §4.2 · p. 9

4.3 QUALITATIVE ANALYSIS

  • Checklist predicates such as “Include” and “Provide” reflect demands for comprehensive answers, while “Calculate” and “Specify” point to quantitative and advisory tasks.The predicate distribution is described as reflecting the original questions’ demands.Source: §4.3 · pp. 9–10, Figure 7 · p. 10
  • Reweighting assigns item 1’s radial-stress calculation a 0.296 weight and item 3’s inner-radius calculation 0.069, reflecting their different importance.The paper describes radial stress as critical to subsequent calculations and inner radius as simpler and less prone to error.Source: §4.3 · p. 10, Figure 8 · p. 10

5 CONCLUSION

RocketEval uses instance-specific checklists with lightweight LLMs to evaluate responses. The conclusion reports alignment with human preferences and lower cost for large-scale evaluation.

5 CONCLUSION

  • RocketEval reframes evaluation as multi-faceted Q&A using instance-specific checklists to address uncertainty and positional bias in lightweight LLM judgments.The framework uses lightweight LLMs and is presented as efficient, low-cost, interpretable, and aligned with human preferences.Source: §5 · p. 10
  • 0.965 Spearman correlation with human preferences is reported for Gemma-2-2B, comparable to GPT-4o.Source: §5 · p. 10
  • The reported cost reduction makes RocketEval feasible for large-scale evaluation and comparison scenarios.Source: §5 · p. 10

A.1 PROMPTS USED IN ROCKETEVAL

The appendix presents prompts for creating instance-level checklists and for judging responses. It also includes variants for different benchmark and scoring settings.

A.1 PROMPTS USED IN ROCKETEVAL

  • The checklist-creation prompt requests concise binary questions that include necessary content and cover key evaluation factors without duplicates or vagueness.Source: §A.1 · p. 15
  • Prompt variants use reference answers for MT-BENCH, reuse the WILDBENCH prompt for other pairwise benchmarks, and adjust direct-score output to 0–9.Source: §A.1 · p. 16
  • The response-judgment prompts provide the query, conversation history, response, and checklist, then request a score or a Yes/No answer.Source: §A.1 · pp. 16, 17, 19

A.2 DATASETS AND BASELINES

The added baselines include GPT-4o and Prometheus-2. Prometheus-2 is prompted to assess responses against a rubric and return feedback with a score.

A.2 DATASETS AND BASELINES

  • GPT-4o is added as a judge baseline, using publicly available code for WILDBENCH and WILDBENCH scoring templates for consistency.For WILDBENCH, the baseline judgments come from GPT-4o results provided by Lin et al. (2025).Source: §A.2 · p. 19
  • Prometheus-2 uses WILDBENCH rubrics and a fixed criterion emphasizing high-quality, correct, relevant, helpful responses focused on the query.The baseline is Prometheus-7B-v2.0, a fine-tuned judge model that introduces custom scoring criteria and rubrics.Source: §A.2 · p. 19
  • Its prompt supplies an instruction, response, score-5 reference answer, and rubric, then requests rubric-based feedback and an integer score from 1 to 5.The output is formatted as feedback followed by a [RESULT] score.Source: §A.2 · p. 19
  • The score rubric ranges from a very poor, nonsensical response at 1 to a perfect, helpful response at 5, with intermediate scores describing graded quality.The middle levels include fair responses with issues and good-enough responses that could be improved.Source: §A.2 · pp. 19–20

A.3 DETAILS OF EXPERIMENTS ON LIST-LEVEL CORRELATION

The list-level correlation experiments add ALPACAEVAL and ARENA-HARD to the benchmark set, with dataset statistics reported in Table 5.

A.3 DETAILS OF EXPERIMENTS ON LIST-LEVEL CORRELATION

  • ALPACAEVAL and ARENA-HARD are added to the benchmarks to assess RocketEval more comprehensively.Table 5 reports statistics for all benchmarks.Source: §A.3 · p. 20
  • Table 5 is identified as the location of the statistics for all benchmark datasets.The supplied passage does not state the table's individual values.Source: Table 5 · p. 20, §A.3 · p. 20

A.3.1 EXPERIMENTAL SETUP

The setup revises model selection to avoid overlapping 95% confidence intervals and accounts for response availability and Elo ratings on other benchmarks.

A.3.1 EXPERIMENTAL SETUP

  • For the CHATBOT ARENA ELO RATING–WILDBENCH correlation, selected models are revised to avoid overlap in their 95% confidence intervals.The authors note that overlap among the previously selected models could compromise correlation reliability.Source: §A.3.1 · p. 20
  • For test models on other benchmarks, selection considers whether responses are available in the official released data and the models’ Elo ratings.Source: §A.3.1 · p. 20
  • The Elo ratings of the selected models are presented in Figure 9.Source: §A.3.1 · p. 20

A.3.2 RESULTS

RocketEval produces more distinguishable score distributions than chain-of-thought grading by lightweight judges, and its scores and Elo ratings follow trends closer to human preferences on ARENA-HARD.

A.3.2 RESULTS

  • The Elo procedure converts response scores into pairwise wins, treating score differences below 0.1 as ties, then fits ratings from match outcomes.It uses Bradley–Terry maximum-likelihood estimation and bootstrap confidence intervals.Source: §A.3.2 · pp. 20–22
  • Figure 10 shows RocketEval scores from lightweight judges are more distinguishable and closer to GPT-4o’s distribution than scores graded with CoT.The authors attribute the difference to checklist context guiding differential judgments; supervised scoring also reduces score deviation toward the GPT-4o-like distribution.Source: §A.3.2 · p. 20, Figure 10 · p. 21
  • The models with large deviations from human judgments on ARENA-HARD belong to the same Llama series, suggesting potential bias toward response patterns.Source: §A.3.2 · p. 22
  • On ARENA-HARD, scores and Elo ratings from LLM judges follow similar trends and align more closely with human preferences, as Figure 11 illustrates.The analysis combines matches from three judges and evaluates 50 test models after exclusions.Source: §A.3.2 · p. 22, Figure 11 · p. 23

A.4 ABLATION STUDY

The ablation findings favor conditional normalized scoring and independent checklist-item judgments for lightweight judging. The checklist analysis shows how task-specific subjects, predicates, and item reweighting tailor evaluation across task types.

A.4 ABLATION STUDY

  • Conditional normalized scoring consistently improves LLM judges, especially smaller ones; setting αr to 1 makes the predictor decisive, and limited annotations may weaken instance scoring despite stronger list-level alignment.The weight factor mitigates biased annotations, supporting both instance-level agreement and list-level ranking correlation.Source: §A.4 · pp. 23–24
  • Removing independent item judgments causes only a nonsignificant performance drop, while multi-round judging prevents batch inference; the authors retain independent judgments as their preferred choice.The passage suggests position bias is further alleviated during score prediction.Source: §A.4 · p. 24

A.5 CHECKLIST ANALYSIS

  • Checklist subjects combine task-specific terms such as code, essay, strategy, and calculation with shared terms such as response and explanation across task types.The analysis covers Math & Data Analysis, Coding & Debugging, Creative Tasks, Information/Advice Seeking, and Planning & Reasoning.Source: §A.5 · p. 24
  • Task-specific predicates, such as “handle” for coding and “calculate” for math, complement shared “explain” and “describe” predicates used across question types.Source: §A.5 · pp. 24–25
  • Reweighting foregrounds critical code-structure steps and key reasoning, including inferring DKA from symptoms, with examples also spanning creative and advice-seeking tasks.The coding and planning examples are shown in Figures 14 and 17, respectively.Source: §A.5 · p. 26, Figure 14 · p. 26, Figure 17 · p. 27
Loading 2503.05142v1…