PaperlayerView exact revision on arXiv

Source-linked AI summary

Are LLM-based Evaluators Confusing NLG Quality Criteria?

Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, Xiaojun Wan

arXiv:2402.12055v2cs.CL

TL;DR

LLM evaluators may confuse distinct NLG quality aspects, raising questions about whether they reliably follow the requested criteria. The paper builds a hierarchical system for common aspects and tests models with targeted perturbations checked by human annotators; results show persistent confusion and motivate further research.

  • Problem

    LLM evaluations can correlate better with human judgments on a different aspect than the requested one, raising questions about reliability when aspects should be evaluated separately.

    Source: §1 · p. 1

  • Method

    The authors organize 11 common aspects into a hierarchical classification system and use 18 aspect-targeted perturbation attacks, checked by human annotators, to test LLM evaluation behavior.

    Source: §1 · p. 2

  • Results

    Perturbation tests reveal significant deviations from expectations, including persistent aspect confusion and limited effects from different criterion-detail levels, with exceptions for uncommon terms.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The reported confusion, including in GPT-4, warrants further attention and in-depth research on LLM-based evaluation.

    Source: §1 · p. 2, §8 · p. 9

    The classification system and attacks mainly cover commonly used aspects in closed-end text generation, excluding strongly subjective aspects such as interestingness.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Some prior work has shown that LLMs perform well in NLG evaluation for different tasks. However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability. For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves. So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved. Inspired by behavioral testing, we elaborately design 18 types of aspect-targeted perturbation attacks for fine-grained analysis of the evaluation behaviors of different LLMs. We also conduct human annotations beyond the guidance of the classification system to validate the impact of the perturbations. Our experimental results reveal confusion issues inherent in LLMs, as well as other noteworthy phenomena, and necessitate further research and improvements for LLM-based evaluation.

1 Introduction

LLM evaluations can confuse distinct NLG quality aspects, raising questions about their reliability. The paper clarifies commonly used criteria and uses aspect-targeted tests to examine how models respond to them.

1 Introduction

  • LLM scores can correlate more strongly with human judgments on a different aspect than the requested one, while model scores converge more across aspects than human scores.The authors report these patterns as evidence that LLMs confuse aspects.Source: §1 · p. 1
  • The paper collects criteria from prior work and organizes commonly used aspects into a clear hierarchical classification system.The motivation is that existing criteria can have inconsistent conceptualizations or ambiguous expressions.Source: §1 · p. 2
  • The authors frame distinguishing aspects and examining criteria descriptions as a contribution that bridges human and LLM-based evaluation.They describe this as the first exploration of these capabilities and impacts.Source: §1 · p. 2
  • The study proposes 18 aspect-targeted perturbation attacks, verified by human annotators, to test fine-grained LLM evaluation behavior across 11 common aspects.The criteria include descriptions at different levels of detail.Source: §1 · p. 2
  • The authors report confusion across aspects even for GPT-4, motivating further attention and research on LLM-based evaluation.They also release related resources to facilitate future work.Source: §1 · p. 2

2 Preliminary Study

A preliminary study of GPT-3.5 on SummEval found signs of confusion among evaluation aspects. Analyze-rate prompting with multiple samples and temperature 1 performed best among the tested forms, while few-shot evaluation worsened performance.

2 Preliminary Study

  • The preliminary experiments use GPT-3.5 on the SummEval summarization dataset and compare evaluation forms introduced in prior work.The study also adds few-shot methods to prior zero-shot evaluations.Source: §2 · pp. 2–3
  • Analyze-rate prompting with multiple samples and temperature 1 achieved the best performance, while few-shot evaluation led to worse performance.The authors use these settings in subsequent experiments.Source: §2 · p. 3
  • GPT-3.5 evaluations can correlate more strongly with human judgments on another aspect than the current aspect, and model scores are more correlated across aspects than human scores.The authors also found cases where GPT-3.5 incorporated assessments of other aspects into its outputs; similar issues were observed in GPT-4 and Prometheus.Source: §2 · p. 3

3 Methodology

The methodology tests whether LLMs respond selectively to aspect-targeted changes in text quality. It defines expected score changes and varies both perturbations and criterion descriptions.

3.1 Classification System for Aspects

  • Criteria are organized around inclusive and non-inclusive relationships, with each perturbation expected to affect its target criteria and criteria whose scopes include it.The remaining criteria are treated as unaffected by that perturbation.Source: §3 · p. 3

3 Methodology

  • The test prompts models to score original and perturbed texts, expecting significant changes for affected criteria and zero impact for unaffected criteria.The distinction between affected and unaffected criteria is informed by the classification system and human annotations.Source: §3 · pp. 3, 5

3.2 Perturbation Attacks

  • The 18 perturbations target fundamental aspects and are designed to affect those aspects and their ancestors while limiting effects on unrelated aspects.Researchers manually created demonstrations and prompted GPT-4 in 10-shot settings to generate perturbations, then sampled and manually inspected the outputs.Source: Table 2 · p. 4, §3.2 · p. 5

3.3 Different Descriptions of Criteria

  • The study varies criterion-description detail and compares five description types, while also collecting criteria used in prior NLG evaluation studies.The types are default, simplified, detailed, term, and list.Source: §3.3 · p. 5, Table 3 · p. 5

4 Data and Test Settings

The study tests three NLG task groups using GPT-3.5, GPT-4, and Prometheus, and uses human judgments to assess perturbation effects and criteria.

Data and tasks

  • The experiments cover summarization, paraphrase, and table-to-text generation, using 1,000 sampled items, each subjected to 18 perturbations.Summarization includes news and dialogue.Source: §4 · p. 6

LLMs in tests

  • The tested evaluators are proprietary GPT-3.5 and GPT-4, plus the open-source Prometheus model fine-tuned on Llama-2-Chat-13B.The paper notes that Prometheus was among the few released specialized open-source evaluators supporting specified criteria.Source: §4 · p. 6

Human judgment

  • Forty annotators judged sampled data four times per item to validate perturbation effects beyond the classification system.Detailed definitions achieved the highest match rate, 94.4%, and more detailed definitions generally aligned better with expected judgments and higher annotation consistency.Source: §4 · p. 6

5 Experiments

Experiments examine GPT-3.5’s response to targeted perturbations, invariance, and definition changes, then compare it with GPT-4 and Prometheus. Results show aspect confusion across models, with different strengths and weaknesses.

5 Experiments

  • The experiments use detailed definitions to align evaluation expectations with human judgments and compare scores before and after perturbations across impactful and nonimpactful attacks.Table 4 reports GPT-3.5 score variances for the listed criteria.Source: §5 · p. 6, Table 4 · p. 7

5.1 Directional Expectation Test

  • Coherence and Informativeness perturbations changed scores by less than 0.2, while most Grammaticality and Non-contradiction perturbations and Non-hallucination complements decreased scores by more than 2.Human judgments indicated that Coherence and Informativeness perturbations should affect their respective aspects; repetition was often ignored despite explicit Fluency instructions.Source: §5.1 · p. 6

5.2 Invariance Test

  • Grammaticality and especially Non-contradiction perturbations altered criteria that should remain unaffected; Non-contradiction effects were nearly indistinguishable across criteria, unlike Non-hallucination effects.The results suggest GPT-3.5 is highly sensitive to conflicts between target and source content but more restrained with unverifiable information.Source: §5.2 · p. 7

5.3 Different Definition Types

  • GPT-3.5’s evaluations were highly correlated across five criterion-description types, suggesting it relied mainly on criterion terminology; less detailed descriptions were associated with more similar cross-aspect evaluations.Exhaustive descriptions made score variations in the affected situation more pronounced, but changes in the invariant situation were erratic; Figure 4 presents score distributions and correlation matrices.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 Different LLMs

  • GPT-4 performs better than GPT-3.5 on directional expectations but worse on invariance, while Prometheus performs worst on both tests and largely fails to distinguish aspects.GPT-4 is especially sensitive to grammatical and conflict-related perturbations in the invariance test.Source: §5.4 · p. 8

6 Discussions

The discussion finds that LLMs retain aspect-confusion behaviors despite changes to criteria and attempted interventions. Ambiguous and inconsistently conceptualized descriptions affect human judgments and, more weakly, LLM evaluations.

6 Discussions

  • Grammaticality and contradiction sensitivity persists with detailed, minimal, or empty criteria, suggesting it may reflect LLMs’ default evaluation behavior.Detailed definitions do affect aspects with less common terms, such as non-hallucination.Source: §6 · p. 8
  • Explicit instructions, staged evaluation, self-checking, and added background knowledge yield no generally significant improvement in LLM evaluation reliability.The authors describe explicit criterion-only instructions as producing only slight, unstable gains, and conclude the issues remain stubborn and challenging.Source: §6 · p. 8
  • Ambiguous expressions affect human and LLM evaluations similarly to less informative criteria descriptions designed by the authors.Inconsistent conceptualizations alter human judgments on related perturbations, with a similar but weaker effect in LLM evaluation.Source: §6 · pp. 8–9

7 Related Works

Related work examines diagnostic tests of NLG metrics, biases and limitations in LLM evaluators, and the specification or improvement of quality criteria. The cited studies do not address the paper’s focus on distinctions among evaluation aspects and varying criterion expressions.

Diagnostic Tests for NLG Evaluation Metrics

  • Diagnostic tests use synthetic perturbations to expose metric divergence from human judgments and blind spots, or target specific tasks and aspects.Prior LLM-evaluator tests examined dialogue robustness and judging closed-ended responses without references, but not aspect distinctions or varying expressions.Source: §7 · p. 9

Analyzing the limitations of LLM-based evaluators

  • Prior analyses identify order effects, preferences for longer or self-generated responses, system- and aspect-dependent performance, and possible non-Latin-language bias.Failure modes in evaluator-generated explanations were also identified, but were not directly usable for aspect-specific evaluation.Source: §7 · p. 9

NLG Quality Criteria

  • Work on quality criteria includes classification for comparability, mapping descriptions to normalized criteria, and LLM drafting or review of criteria.LLMs have been used to suggest disambiguation, merging, and splitting of user-defined criteria.Source: §7 · p. 9
  • Other studies seek to improve aspect-specific evaluation through chain-of-thought methods and instruction tuning.Source: §7 · p. 9

8 Conclusions

The authors use classification-guided perturbation tests and human judgments to assess LLMs’ aspect-specific NLG evaluation. The findings point to unreliable distinctions among aspects and a need for further analysis and improvement.

8 Conclusions

  • LLM performance in perturbation tests deviates from expectations, with both unawareness and oversensitivity for some aspects.Source: §8 · p. 9
  • Different levels of criterion detail almost never change LLM evaluation behavior, except for criteria using uncommon terms such as non-hallucination.Source: §8 · p. 9
  • Oversensitivity remains with empty criteria, suggesting it may be inherent in LLMs rather than caused by criterion-description problems.Source: §8 · p. 9
  • Instructions to include or exclude specific problems do not have obvious effects on the reported confusion issues.Source: §8 · p. 9
  • The authors conclude that LLM-based evaluation is not reliable across aspects and warrants in-depth analysis and effective improvement methods.Source: §8 · p. 9

Limitations

The study’s scope and resources constrain the settings to which its conclusions can be applied and may make replication costly.

Limitations

  • The classification system and attacks target commonly used aspects in closed-end text generation, excluding subjective aspects such as interestingness.Source: Limitations · p. 9
  • Experiments cover limited domains and task types, use English data, and generally evaluate texts of a few hundred words.The authors cannot guarantee the same conclusions for long texts, other languages, or special-domain data.Source: Limitations · p. 9
  • Extensive use of GPT-3.5 and GPT-4 APIs incurs significant costs that may discourage replication.Source: Limitations · p. 9

A Details for Preliminary Study

The preliminary study found that LLMs’ evaluation scores can align more strongly with human judgments on another aspect than on the specified aspect. It also reports serious aspect confusion across models and describes how the perturbation attacks were constructed.

A Details for Preliminary Study

B Details for Perturbation Attacks

  • Four perturbation types used rules, while the remaining 14 were generated by GPT-4 in 10-shot settings using 140 manually written and checked demonstrations.The authors also provide GPT-4 instructions and rule-based construction code in released resources.Source: B Details for Perturbation Attacks · pp. 12–13

A Details for Preliminary Study

  • LLM scores for one aspect could correlate more strongly with human judgments on a different aspect, while cross-aspect score correlations were higher than corresponding human correlations.The study highlights fluency and relevance as an example of the cross-aspect mismatch.Source: Table 5 · p. 13

C.1 Datasets

The appendix describes the datasets, models, and human annotation used to study perturbation attacks, then compares the attacks with non-LLM metrics. References were prepared for four NLG tasks, and the human study used paired comparisons across criteria and perturbations.

C.1 Datasets

  • GPT-4 generated or improved references for news summarization, paraphrase generation, and table-to-text; original human-written SAMSum references were used for dialogue summarization.The authors note that some original references are incoherent or include information absent from source news.Source: §C.1 · p. 13
  • Across four common NLG tasks, the authors report consistent evaluation issues and use task-agnostic perturbations and aspects.For each reference, they generated 18 perturbations, producing 19,000 samples and 1.52 million evaluations across 80 criteria.Source: §C.1 · p. 13

C.2 LLMs

  • The study tested GPT-3.5 Turbo (1106), GPT-4 Turbo (1106), and Prometheus, using zero-shot prompts and 1–5 ratings.The reported settings use temperature 1.0 and 10 samples.Source: §C.2 · p. 20

C.3 Human annotation

  • Forty professional annotators compared perturbed and unperturbed text pairs against quality criteria using four response options; each test sample was annotated by four people.The study used 80 descriptions across 11 quality criteria and collected 23,040 annotations; consistency excludes the uncertain option, while match rate measures whether judgments matched expected perturbation effects.Source: §C.3 · p. 20

D.1 Comparison with non-LLM Evaluation Metrics

  • Because most non-LLM metrics cannot accept customized aspect criteria, the comparison tested only whether scores declined for overall quality; rule-based metrics, especially BLEU, were generally more sensitive than model-based metrics.The comparison was against GPT-4 and was limited to the directional expectation test.Source: §D.1 · pp. 20–21

D.2 Other Results

The appendix presents additional correlations across aspect definitions and aspects, then organizes perturbation results by definition type or evaluation aspect. It also reports tests using criteria with descriptions or terms removed.

D.2 Other Results

  • The correlation figures compare GPT-3.5 scores across five aspect-definition types and across different aspects within each definition type.The figures include comparisons for detailed, default, list, simplified, and term descriptions.Source: §D.2 · p. 22, Figure 8 · p. 22, Figure 9 · p. 24
  • The perturbation figures organize score variances either by description type to compare aspects or by evaluation aspect to compare description types.The appendix includes results for all three LLMs and the named aspects, including overall quality, readability, and informativeness.Source: §D.2 · p. 22, Figure 14 · p. 25, Figure 19 · p. 30
  • Additional attacks test criteria retaining only descriptions or only terms, including a single-word “Aspect” criterion and an empty criterion.Source: §D.2 · p. 22, Figure 30 · p. 41, Figure 31 · p. 41
Loading 2402.12055v2…