Source-linked AI summary
LLMEval: A Preliminary Study on How to Evaluate Large Language Models
Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, Xuanjing Huang
TL;DR
LLM evaluation research has focused more on what to test and where than on how to score and rank evaluations. This paper compares criteria, human and GPT-4 evaluation methods, scoring, and ranking, finding that informativeness and accuracy distinguish models while onsite star scoring is the strongest manual approach reported.
Problem
Prior LLM evaluation research has discussed tasks and knowledge domains more extensively than evaluation criteria, grading, ranking, and annotator choices.
Source: Introduction · p. 2Method
The study compares evaluation criteria, manual annotator types, GPT-4 assessment, scoring methods, and ranking systems using the LLMEval dataset.
Source: Introduction · p. 2Results
Informativeness and accuracy distinguish existing LLMs, and onsite star scoring is reported as the best manual method for accuracy, consistency, and alignment with LLM evaluators.
Source: Discussion · p. 7Takeaways & Limitations
The authors prioritize informativeness and accuracy for future evaluations and describe automated evaluation as a complementary approach to onsite star scoring.
Source: Discussion · p. 7Evaluation differences between automated and manual methods are most noticeable on subjective questions, where the lack of a standard answer makes assessment challenging.
Source: Discussion · p. 7
Abstract
from arXiv · showhide
Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval .
Introduction
The introduction frames “how to evaluate” as an underexplored question and compares criteria, annotation choices, and ranking systems. It reports findings on which criteria distinguish models, how annotators compare, and the stability of Elo ratings.
Introduction
- Traditional generative-model metrics capture only one or a few aspects, while prior evaluation research has discussed “what” and “where” more than “how.”Here, “how” covers criteria, grading, ranking, and annotator types for manual evaluation.Source: Introduction · pp. 1–2
- The study compares five criteria, manual and GPT-4 evaluation, annotator types, scoring methods, and ranking systems.Manual annotators include onsite, crowd-sourcing, and public participants; scoring includes separate ratings and pairwise competition.Source: Introduction · p. 2
- Existing LLMs show notable harmlessness, while informativeness and accuracy are the differentiating criteria; onsite evaluation is more accurate and consistent, and Elo is unstable.Elo results vary with match sequence and are sensitive to annotation noise.Source: Introduction · p. 2
- The authors introduce LLMEval, evaluate 20 models, and report 243,337 manual annotations and 57,511 automatic evaluation results.They present their conclusions as insights for future LLM evaluation.Source: Introduction · p. 2
Design
The design operationalizes LLM evaluation through five criteria, multiple human and GPT-4 annotation settings, and two ranking systems for pairwise results. The compared ranking systems award either relative-skill-adjusted Elo ratings or points for match performance.
Design
- The design section sets out criteria selection, annotation-method design, and ranking-system choice.These choices structure the evaluation comparisons that follow.Source: Design · p. 2
Criteria
- The authors refine the 3H principles into five human-preference criteria: accuracy, fluency, informativeness, logical coherence, and harmlessness.The criteria are used to assess LLM responses from the perspective of human preference.Source: Criteria · p. 2
Annotation Method
- Five annotation settings combine star scoring or pairwise comparison with onsite, crowd-sourced, public, or GPT-4 evaluation.Onsite annotators rate answers from one to three stars; crowd-sourced and public annotators compare paired answers, while GPT-4 is used for star and pairwise evaluation. Tasks are double-blind and randomly assigned.Source: Annotation Method · pp. 2–3
Ranking System
- For pairwise results, the study compares Elo, which adjusts ratings based on opponents and outcomes, with a points system based on absolute match performance.Elo uses K=32 in the experiment; the points system disregards opponent skill and adds each match's score to prior points.Source: Ranking System · p. 3
Experiments
The experiments introduce two datasets and define accuracy and consistency measures for comparing annotation methods. The metrics use a multi-annotator ground truth and repeated tasks, with low-volume user annotations excluded from quality assessment.
Experiments
- The experiments section introduces the datasets and metrics used to evaluate annotation methods.Source: Experiments · p. 3
Dataset
- LLMEval-1 covers 17 question types, while LLMEval-2 targets 12 academic subjects with both objective and subjective questions.The LLMEval-1 tasks include classification, code, conversation, factual questions, math, reasoning, and generation tasks; LLMEval-2 was developed to investigate specialized-domain performance.Source: Dataset · p. 3
Metrics
- Accuracy uses the average score from multiple annotators as ground truth; an annotation is correct when its score differs from that ground truth by less than σ.Consistency is measured using repeated tasks, which make up approximately 2% of evaluations; the rate is the proportion of identical annotations. User annotations with fewer than five results are excluded from quality assessment.Source: Metrics · p. 4
Results
The results distinguish model performance most on accuracy and informativeness, and report variation across annotators, scoring setups, and ranking methods. Onsite annotation performs best on accuracy and consistency, while Elo rankings fluctuate with long runs and reordered matches.
Comparison of Criteria
- Accuracy and informativeness are the most distinguishing criteria; top- and bottom-ranked models differ by 0.853 in informativeness and 0.776 in accuracy.All tested LLMs performed well on harmlessness.Source: Figure 1 · p. 4, Comparison of Criteria · p. 4
Comparison of Annotation Methods
- Onsite annotators have the best accuracy and consistency; their star-scoring accuracy averages 0.892, while public annotators show the lowest accuracy and consistency.The onsite minimum accuracy is 0.825; public evaluation’s minimum accuracy is 0, after excluding annotators with fewer than five evaluations.Source: Comparison of Annotation Methods · p. 4, Figure 3 · p. 5
- GPT-4 and manual ranks align more closely under star scoring (Spearman’s ρ = 0.949) than under pairwise comparison (ρ = 0.902).The star-scoring correlation also exceeds that between manual star scoring and pairwise comparison.Source: Comparison of Annotation Methods · pp. 4–5, Table 2 · p. 5
- When responses differ in length by more than 300 characters, GPT-4 selects the longer response 78.8% of the time, versus 51.4% for human annotators.This comparison reports a stronger preference for longer responses by GPT-4 than by human evaluators.Source: Table 3 · p. 5, Comparison of Annotation Methods · p. 5
- Accuracy-score differences exceeding 2 points occur for 12.98% of objective questions and 37.05% of subjective questions.The study reports greater manual–GPT-4 discrepancy on subjective questions in LLMEval-2.Source: Comparison of Annotation Methods · p. 5, Table 4 · p. 5, Table 5 · p. 6
- Without answer hints, annotators’ scores are 9.79% higher on average; the authors suggest hints help identify factual errors.The study compared scores with and without hints for questions with determined answers.Source: Comparison of Annotation Methods · pp. 5–6, Figure 4 · p. 6
Comparison of Ranking Systems
- Elo rankings continue to fluctuate significantly after 100,000 comparison rounds, and a few recent losses can affect the final ranking.Figure 5 illustrates rank and score variation across rounds 100,000–100,100.Source: Figure 5 · p. 6, Comparison of Ranking Systems · p. 6
- With the same 10,000 annotations shuffled into different orders, GPT-4’s Elo rank fluctuates by 1 to 3 places.The study reports order sensitivity even when the annotation results are unchanged.Source: Figure 6 · p. 6, Comparison of Ranking Systems · p. 6
Details
The paper details question-set construction, response generation, and annotation for LLMEval-1, then describes LLMEval-2’s domain-focused design and evaluation.
Details
- LLMEval-1’s 453 questions across 17 tasks were contributed by 20 college students, each providing 15 to 25 questions.Hints were provided for factual, coding, and math-solving questions.Source: Details · p. 6
- 5,436 responses and 29,898 pairs were collected for LLMEval-1 from 12 models, using the first response per question and a new conversation for each question.Testing ran from May 1 to May 8, 2023, and all questions and answers were in Chinese.Source: Details · pp. 6–7
- Each LLMEval-1 response received star scores from three onsite annotators, while each pair received at least three crowd-sourcing or public judgments, alongside GPT-4 evaluation.GPT-4 scoring and pairwise comparisons consumed 33 million tokens.Source: Details · p. 7
- LLMEval-2 comprised 480 questions from 12 disciplines, collected by 12 college students recruited from distinct disciplines between June 24 and July 10, 2023.The study evaluated 20 open-source and commercial models.Source: Details · p. 7
- LLMEval-2 used correctness and explanation correctness for objective items (maximum 5 points), and accuracy, fluency, informativeness, and logicality for subjective items (maximum 14 points); harmlessness was excluded.Correctness and accuracy received higher proportions of the total score.Source: Details · p. 7
Related Works
Prior LLM evaluation work includes automated benchmarks and manual comparisons, while this paper examines scoring, annotator types, and ranking systems alongside these approaches.
Related Works
- Earlier n-gram and embedding-based methods often evaluated limited task sets and did not assess overall capability, while recent work broadened evaluation through automatic and manual approaches.Examples include BLEU, ROUGE, WMD, and MoverScore(Zhao et al. 2019).Source: Related Works · p. 7
- Benchmarks such as HELM(Liang et al. 2022), MMLU(Hendrycks et al. 2021), C-Eval(Huang et al. 2023), and AGIEval(Zhong et al. 2023) use combined datasets, multiple-choice questions, or cloze tasks, but do not assess fluency, coherence, and harmlessness simultaneously.LLM-based evaluators can disagree with manual evaluations and are susceptible to response position and length.Source: Related Works · p. 7
- Manual evaluation is expensive but essential; this study compares GPT-4 and human results and examines scoring methods, annotator types, and ranking systems.Prior examples include AlpacaFarm(Dubois et al. 2023) and Chatbot Arena(Zheng et al. 2023).Source: Related Works · p. 7
Discussion
The discussion identifies informativeness and accuracy as distinguishing criteria, favors onsite star scoring for manual evaluation, and notes greater manual–automated differences on subjective questions.
Discussion
- Informativeness and accuracy are identified as the most distinguishing criteria for evaluating LLMs and are prioritized for future evaluations.Source: Discussion · p. 7
- Onsite star scoring is reported as the optimal manual method for accuracy, consistency, and alignment between human and LLM evaluators; automated evaluation is described as a complementary approach.The authors note that automated evaluation can cover many tasks in a short time and exhibits reasonable alignment with humans.Source: Discussion · p. 7
- Differences between automated and manual evaluations are most noticeable on subjective questions, where the absence of a standard answer makes evaluation challenging.Source: Discussion · p. 7
Dataset
The datasets span cognitive-psychology-oriented tasks and academic subjects, with question sets incorporating different task types and subject domains.
Dataset
- LLMEval-1 covers 17 cognitive-psychology-oriented task types, including factual and open questions, translation, retrieval, coding, math, reasoning, comprehension, and generation tasks.Source: Dataset · p. 8
- Figure 7 presents the distribution of tasks in LLMEval-1.Source: Figure 7 · p. 8
- LLMEval-2 includes questions from 12 academic subjects, with both objective and subjective questions for each subject; Figure 8 presents their distribution.Source: Dataset · p. 8, Figure 8 · p. 9
Mathematical Proof of Elo Rating Instability
The analysis finds Elo ratings unstable under noisy, sequence-dependent judgments, while presenting the Points system as a stable, sequence-independent alternative for extensive evaluations.
Mathematical Proof of Elo Rating Instability
- The analysis assumes Model A starts at 1500 and always beats Model B, with judge accuracy p, then uses a first-order Taylor approximation that may be imprecise.The approximation disregards higher-order terms and is presented as a conceptual framework for linear fitting.Source: Mathematical Proof of Elo Rating… · p. 9
- At 200,000 evaluation points, even a small share of noisy samples produced substantial Elo fluctuations, which persisted after 100,000 rounds.The analysis uses a hypothetical 70% manual-evaluation accuracy and an initial rating of 1500.Source: Mathematical Proof of Elo Rating… · p. 9
- Elo ratings vary with match order despite identical win-loss counts, and instability worsens at lower evaluator accuracy.The analysis also identifies sequence dependence in large-scale evaluations and settings with many annotations.Source: Mathematical Proof of Elo Rating… · p. 9
- The Points system sums individual performance scores independently of opponent skill, making its rankings sequence-independent and more stable for extensive evaluations.The paper presents it as a viable alternative for scenarios involving extensive annotations.Source: Mathematical Proof of Elo Rating… · pp. 9–10
Implementation Details
The implementation combines several annotation formats with star scoring and pairwise judgments, alongside GPT-4 evaluations using task-specific prompts.
Implementation Details
- Onsite annotators rated answers on five criteria using one to three stars, with cash incentives and a consistency-linked reward deduction.The criteria were accuracy, informativeness, logical coherence, fluency, and harmlessness.Source: Implementation Details · p. 10
- Crowd-sourcing annotators compared paired responses side by side and chose the left or right answer, a tie, or that both were bad.The pairs were randomly presented, and the option setting followed Zheng et al. (2023).Source: Implementation Details · p. 10
- For LLMEval-1, GPT-4 automatic evaluation used prompt templates for star scoring and pairwise comparison.The prompts are shown in Figures 9 and 10.Source: Figure 10 · p. 10, Implementation Details · p. 11
- LLMEval-2 uses only star scoring for evaluation and uses different templates for subjective versus objective questions because some criteria are irrelevant; Figures 11 and 12 show these templates.Data samples are displayed in Figure 17.Source: Implementation Details · p. 11
- The reported outputs include LLMEval-1 star-scoring and pairwise results, plus LLMEval-2 rankings and objective and subjective scores.These results appear in Tables 6–10.Source: Table 6 · p. 14, Table 7 · p. 15, Table 8 · p. 15