PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models

Yen-Ting Lin, Yun-Nung Chen

arXiv:2305.13711v1cs.CLcs.AI

TL;DR

Open-domain conversation evaluation often relies on human annotations, ground-truth responses, or multiple LLM prompts. LLM-EVAL uses a unified schema in a single prompt and model call; across varied benchmarks, it consistently outperforms most baselines and state-of-the-art evaluation methods. Its performance nevertheless depends on the underlying LLM, which may misinterpret instructions and yield inaccurate scores.

  • Problem

    Many existing conversation-evaluation methods rely on human annotations, ground-truth responses, or multiple LLM inferences, which can be expensive and time-consuming.

    Source: §1 · pp. 1–2

  • Method

    LLM-EVAL uses one prompt with a unified multi-dimensional evaluation schema to obtain scores in a single model call.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across varied benchmarks, LLM-EVAL consistently outperforms most baselines and state-of-the-art evaluation methods in correlation with human judgments.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The paper presents LLM-EVAL as a robust and versatile evaluator that adapts to different scoring ranges and evaluation scenarios.

    Source: §1 · p. 2

    Performance depends heavily on the underlying LLM, whose biases or misinterpretation of the schema and prompt may produce inaccurate evaluation scores.

    Source: Limitations · p. 8

Abstract

from arXiv · show

We propose LLM-Eval, a unified multi-dimensional automatic evaluation method for open-domain conversations with large language models (LLMs). Existing evaluation methods often rely on human annotations, ground-truth responses, or multiple LLM prompts, which can be expensive and time-consuming. To address these issues, we design a single prompt-based evaluation method that leverages a unified evaluation schema to cover multiple dimensions of conversation quality in a single model call. We extensively evaluate the performance of LLM-Eval on various benchmark datasets, demonstrating its effectiveness, efficiency, and adaptability compared to state-of-the-art evaluation methods. Our analysis also highlights the importance of choosing suitable LLMs and decoding strategies for accurate evaluation results. LLM-Eval offers a versatile and robust solution for evaluating open-domain conversation systems, streamlining the evaluation process and providing consistent performance across diverse scenarios.

1 Introduction

LLM-EVAL addresses the cost and limits of existing open-domain conversation evaluation with a single-prompt, multi-dimensional approach. The paper reports stronger correlation with human judgments across varied benchmarks and emphasizes choosing suitable LLMs and decoding strategies.

1 Introduction

  • Existing approaches may require human annotations, ground-truth responses, or multiple LLM inferences, which can limit efficiency and adaptability.The introduction also notes that traditional automatic metrics such as BLEU and ROUGE do not capture conversational nuance adequately.Source: §1 · pp. 1–2
  • Across varied benchmarks, LLM-EVAL consistently outperforms most baselines and state-of-the-art methods in correlation with human judgments.The paper also describes the method as robust and adaptable to different scoring ranges and evaluation scenarios.Source: §1 · p. 2
  • LLM-EVAL uses one prompt and a unified schema to evaluate multiple dimensions of conversation quality in a single model call.The prompt defines the task and criteria and can combine dialogue context, an optional reference, and the generated response.Source: §1 · p. 2
  • The paper analyzes how different LLMs and decoding methods affect evaluation performance, underscoring their importance for accurate scores.The introduction identifies choosing suitable models and decoding strategies as important to evaluation accuracy.Source: §1 · p. 2

2 Related Work

Related work covers multi-dimensional dialogue metrics, unsupervised evaluation, and LLM-based evaluators. LLM-EVAL is positioned as a single-prompt alternative that produces multi-dimensional scores in one model call.

Multi-Dimensional Metrics

  • Multi-dimensional metrics including USR, GRADE, and FlowScore assess dialogue qualities through multiple models or specialized representations, whereas LLM-EVAL uses one prompt and a unified schema.USR measures qualities such as fluency and relevance, GRADE models topic transitions, and FlowScore uses information flow in dialogue history.Source: §2 · p. 2

Unsupervised Metrics

  • Unsupervised metrics such as DEB and FED avoid human annotations, while the paper describes LLM-EVAL as more effective and adaptable than existing unsupervised methods.DEB uses an NSP objective with relevant and adversarial responses; FED uses response-embedding and language-model-probability features.Source: §2 · p. 2

Large Language Models for Evaluation

  • GPTScore and G-EVAL use multiple prompts or complex scoring functions, while LLM-EVAL returns multi-dimensional evaluation scores in a single model call.G-EVAL combines chain-of-thought prompting with form filling, and its GPT-4 version is reported to correlate highly with human judgments on summarization.Source: §2 · pp. 2–3

3 Methodology

LLM-EVAL combines a unified natural-language evaluation schema with a single prompt to obtain multiple dimension scores efficiently. The prompt supplies dialogue inputs and the schema to an LLM, which returns structured scores.

3 Methodology

  • LLM-EVAL’s prompt combines the evaluation task, desired criteria, and a unified multi-dimensional scoring schema, avoiding numerous inferences or intricate scoring functions.The schema defines dimensions such as content, grammar, relevance, and appropriateness, along with score structure and ranges.Source: §3 · p. 3
  • The evaluation prompt includes dialogue context, an optional reference, the target response, and the schema before being sent to the LLM.The prompt template explicitly labels context, reference, and dialogue response.Source: §3 · p. 3
  • A single LLM call directly returns a score for each evaluation dimension, streamlining multi-dimensional evaluation.The example output contains separate appropriateness, content, grammar, and relevance scores.Source: §3 · p. 3

4 Experiments

LLM-EVAL is tested across diverse dialogue benchmarks, with and without human references, using two score ranges and comparisons against existing metrics. It performs strongly across settings, including on DSTC10 and in reference-free evaluation.

4.1 Datasets and Benchmarks

  • The benchmark suite includes datasets with human annotations and sometimes references; for data without references, only reference-free metrics are evaluated.The datasets span varied dialogue systems and evaluation dimensions.Source: §4.1 · p. 3, Overall Scores without Human Reference · p. 4

4.2 LLM-EVAL Configurations

  • LLM-EVAL offers 0-5 scores with one decimal place or integer scores from 0 to 100, while keeping the evaluation schema the same.The 0-5 range is closer to a common 1-5 Likert scale; 0-100 provides a finer-grained scale.Source: §4.2 · p. 4

4.3 Baseline Evaluation Metrics

  • The baselines include traditional metrics and learned or LLM-based approaches, such as BLEU, ROUGE, USR, GRADE, FlowScore, and GPTScore.The listed methods use varied mechanisms, including dialogue graphs, multiple models, and multiple prompts.Source: §4.3 · pp. 4–5

4.4 Results of DSTC10 Hidden Set

  • 0.378 average Spearman correlation was the highest among methods without human reference, achieved by LLM-EVAL’s 0-5 configuration on DSTC10.The evaluation spans multiple dimensions and datasets.Source: §4.4 · p. 6
  • On DSTC10, both score ranges performed competitively, with 0-5 slightly outperforming 0-100 with and without human references.The reported comparison covers multiple dimensions and datasets.Source: §4.4 · p. 6

4.5 Overall Scores with Human Reference

  • For human-reference overall scores, the evaluation uses different labeled dimensions by dataset but reports the Overall score for TopicalChat-USR and PersonaChat-USR.Relevance is used for DailyDialog-GRADE, ConvAI2-GRADE, and EmpatheticDialogue-GRADE; DSTC6 uses Overall.Source: §4.5 · p. 6
  • LLM-EVAL 0-100 achieved the highest average correlation across datasets and outperformed most baselines with human references.The comparison uses Pearson and Spearman correlations between human ratings and automatic metrics.Source: Table 2 · p. 5, §4.5 · p. 6

4.6 Overall Scores without Human Reference

  • Both score-range configurations performed strongly across reference-free datasets, maintaining competitiveness with state-of-the-art methods.The authors characterize the method as adaptable and robust in reference-free settings.Source: Table 3 · p. 6, §4.6 · p. 7

5 Analysis

The analysis finds that dialogue-optimized LLMs generally perform better as evaluators than GPT-3.5, while Claude-instant remains a possible resource-conscious option. Greedy decoding generally performs better than nucleus sampling across evaluation dimensions.

5.1 Different LLMs

  • Dialogue-optimized Claude and ChatGPT generally outperform GPT-3.5 across evaluation dimensions, while Claude-instant remains a possible option for resource-limited settings.Claude slightly outperforms ChatGPT in some configurations, and smaller Claude-instant is less competitive than its larger counterpart but performs reasonably in some cases.Source: §5.1 · pp. 7–8

5.2 Decoding Methods

  • Greedy decoding generally performs better than nucleus sampling across evaluation dimensions; the authors suggest its deterministic output may be more coherent and focused.The authors consider greedy decoding more suitable for LLM-EVAL, while describing the explanation as a possible reason for the performance difference.Source: §5.2 · p. 8

6 Conclusion

LLM-EVAL uses a single prompt and unified schema to assess multiple dimensions of open-domain conversations. Experiments show it outperforms most baselines and state-of-the-art methods; future work will explore LLM feedback and LLM-in-the-loop evaluation.

6 Conclusion

  • LLM-EVAL combines one prompt with a unified schema covering dimensions such as content, grammar, relevance, and appropriateness.The approach is intended to streamline evaluation and eliminate the need for multiple prompts.Source: §6 · p. 8
  • Experiments on various datasets show LLM-EVAL consistently outperforms most baselines and state-of-the-art evaluation methods.The conclusion describes the method as effective and efficient.Source: §6 · p. 8
  • Future work will explore reinforcement learning from LLM feedback and LLM-in-the-loop evaluation as alternatives to human-in-the-loop methods.The authors intend to investigate these approaches across dialogue-system evaluation scenarios.Source: §6 · p. 8

Limitations

LLM-EVAL has limitations tied to its underlying models, scoring format, and prompt and schema design. The authors suggest alternative prompts, refined schemas, and broader evaluation coverage as future directions.

Limitations

  • The method depends heavily on its underlying LLMs, whose biases or unexpected outputs can produce inaccurate scores if they misinterpret the schema or prompt.Source: Limitations · p. 8
  • Dialogue-optimized LLMs perform better, but choosing them may restrict applicability to particular tasks or dialogue systems.Source: Limitations · p. 8
  • Single-number scores for each dimension may miss subtleties in human judgments, especially for subjective qualities such as engagement, creativity, or humor.Source: Limitations · p. 8
  • Effectiveness depends on clear prompts and evaluation schemas, whose creation may require domain expertise and knowledge of LLM behavior, challenging non-experts.Source: Limitations · p. 8
  • Future work could explore alternative prompt designs, refine schemas, and extend the method to more evaluation dimensions and dialogue-system types.Source: Limitations · p. 9

Ethics Statement

The ethics discussion identifies potential bias in the LLMs used for evaluation and recommends caution in interpreting scores, alongside future debiasing or schema-based approaches.

Ethics Statement

  • Biases learned from training data may be reflected and perpetuated in LLM-EVAL scores, potentially affecting future dialogue-system development by encouraging biased behavior.Source: Ethics Statement · p. 9
  • Researchers and developers should consider potential model biases when interpreting LLM-EVAL results.Source: Ethics Statement · p. 9
  • Future work could explore debiasing language models or evaluation schemas that actively account for bias.Source: Ethics Statement · p. 9

A Prompt Templates

The appendix provides prompt templates illustrating the evaluation schema and reference-based, reference-free, and dialogue-level evaluation formats.

A Prompt Templates

  • The prompt templates exemplify natural-language task and criteria instructions alongside format instructions for score structure and ranges.Source: A Prompt Templates · p. 11

A.1 Evaluation Schema

  • The evaluation schema defines the task and desired criteria across content, grammar, relevance, and appropriateness.Source: §A.1 · p. 11
  • The appendix illustrates a well-formatted JSON instance and contrasts it with an object that does not conform to the schema.Source: §A.1 · p. 11
  • The example output schema requires JSON with integer scores from 0 to 100 for content, grammar, relevance, and appropriateness.Source: §A.1 · p. 11

A.2 Reference-based Turn-level Evaluation

  • Reference-based turn-level prompts include dialogue context, a human reference, the target response, and the schema, with scores on a continuous scale.Source: §A.2 · p. 11

A.3 Reference-free Turn-level Evaluation

  • Reference-free turn-level prompts include the dialogue context and target response, but no human reference, along with the schema and a continuous scoring scale.Source: §A.3 · p. 12

A.4 Dialogue-level Evaluation

  • Dialogue-level prompts evaluate the entire dialogue rather than individual turns, include the schema, and use a continuous scoring scale.Source: §A.4 · p. 12
Loading 2305.13711v1…