PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram

arXiv:2309.07462v2cs.CL

TL;DR

Evaluating LLMs across languages is challenging, and scaling assessments by native speakers is costly. The authors test GPT-4 as a multilingual evaluator against human judgments across tasks, languages, and metrics. GPT-4 evaluators show a bias toward higher scores and may perform worse in low-resource and non-Latin-script languages, motivating calibration with human judgments in each language.

  • Problem

    Evaluating LLMs is challenging because standard benchmarks may not reflect real-world applications, while scaling multilingual assessments by native speakers is difficult.

    Source: §1 · pp. 1–2

  • Method

    The study compares GPT-4 multilingual evaluator judgments with over 20K human judgments across three tasks, eight languages, and five dimensions, and tests prompting strategies.

    Source: §1 · p. 2

  • Results

    GPT-4 evaluators show a bias toward higher scores, and may perform worse on low-resource and non-Latin-script languages.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors recommend calibrating multilingual LLM evaluations with human-labeled judgments in each language before deployment.

    Source: §5 · p. 9

    Because output quality was uncontrolled and most generated texts received positive human ratings, the source of high Percentage Agreement remains unclear.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top $20$, remains inadequate due to existing benchmarks and metrics limitations. Employing LLMs as evaluators to rank or score other models' outputs emerges as a viable solution, addressing the constraints tied to human annotators and established benchmarks. In this study, we explore the potential of LLM-based evaluators, specifically GPT-4 in enhancing multilingual evaluation by calibrating them against $20$K human judgments across three text-generation tasks, five metrics, and eight languages. Our analysis reveals a bias in GPT4-based evaluators towards higher scores, underscoring the necessity of calibration with native speaker judgments, especially in low-resource and non-Latin script languages, to ensure accurate evaluation of LLM performance across diverse languages.

1 Introduction

Multilingual evaluation is difficult to scale because benchmarks and automated metrics have limited coverage, while native-speaker judgments are costly and difficult to access. This work tests GPT-4 as a multilingual evaluator, calibrating its judgments against human ratings and proposing a framework for evaluation across tasks, metrics, and languages.

1 Introduction

  • LLM performance has been reported to degrade on under-resourced languages and languages using non-Latin scripts, motivating multilingual evaluation beyond English.The cited benchmarking work is Ahuja et al. (2023a,b).Source: §1 · p. 1
  • Multilingual evaluation is difficult to scale because benchmarks underrepresent some language families and real-world tasks, while metrics may not suit complex language features.Native-speaker evaluation is considered the gold standard, but cost, turnaround time, and access constraints make it difficult to scale.Source: §1 · pp. 1–2
  • LLM evaluators could reduce the cost and access barriers of native-speaker assessment, but their use requires caution because they can perform poorly even in some high-resource languages.Misleading evaluation results may further widen the digital divide.Source: §1 · p. 2
  • The study calibrates GPT-4 judgments against over 20K human judgments across three tasks, eight languages, and five dimensions, and evaluates multilingual prompting strategies.It also proposes a framework for evaluating LLM evaluators across tasks, metrics, and languages.Source: §1 · p. 2

2 Related Work

Prior work uses LLMs both to score generated text against human references and to judge output quality or safety directly. Existing studies report promising evaluation performance alongside biases, while multilingual calibration remains largely unstudied.

2 Related Work

  • LLM evaluators can score outputs against human references or judge quality and safety directly using task descriptions, rubrics, and sometimes examples.This work focuses on direct evaluation without a human output for comparison.Source: §2 · p. 2
  • Prior studies report LLM evaluation results across annotation, summarization, NLG, translation, and coherence, including strong correlations with human judgments in some settings.Reported findings also include performance that varies with summary quality and strong translation-evaluation benchmark performance.Source: §2 · pp. 2–3
  • Studies identify evaluator biases involving answer length, grammatical errors, model self-preference, and answer position, and call for task-specific validation.Other work proposes wider or deeper evaluators and multiple-agent consensus approaches.Source: §2 · p. 3
  • Prior calibration studies focused on English; the authors describe their work as the first, to their knowledge, to address calibration in multilingual evaluation.Source: §2 · p. 3

3 Experimental Setup

The study evaluates GPT-4 on three multilingual text-generation tasks using human-judged outputs and several evaluation dimensions. It compares prompting approaches and examines calibration against human judgments, including consistency and sensitivity ablations.

3 Experimental Setup

  • The evaluation covers Open Prompt, Continue Writing, and Summarize, assessing linguistic acceptability, output content quality, task quality, problematic content, and hallucinations.The tasks generate a document from a concise prompt, continue between text inputs, or condense a document of at least 500 words.Source: §3 · pp. 3–4

3.1 Human Evaluation Setup

  • Three annotators judged each output; they received task instructions and metric descriptions with score ranges and task-specific examples.The annotators were recruited through an external services company, with locale- and experience-adjusted pay.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.2 Data Statistics

  • The full human-evaluation dataset contains over 7,300 data points, while the prompting-variation subset contains over 2,700; each point has three annotations.Table 1 is described as reporting dataset statistics across tasks and languages, but its cell values are not provided here.Source: §3.1.2 · p. 4, Table 1 · p. 4

3.2 LLM-based Evaluators

  • GPT4-32K serves as the evaluator at temperature 0, except in ablations.The model was accessed through Azure.Source: §3.2 · p. 4

3.2.1 Prompts

  • Prompts use the guidance toolkit and English instructions, with task and metric descriptions kept close to the annotators’ wording; formatting and JSON schemas improved output reliability.Prompts include SYS, USER, and ASSISTANT components; unstructured instructions sometimes produced formatting errors or omitted expected outputs.Source: §3.2.1 · p. 4

3.3 Prompting Variations

  • Prompting variations compare separate per-metric calls, a compound call for all metrics, and a detailed-instruction version; experiments are generally zero-shot unless specified.The detailed metric instructions were based on annotator instructions and manually modified.Source: §3.3 · p. 5

3.4 Calibration with Human Judgments

  • Calibration analysis compares GPT-4 with the majority score of three annotators using percentage agreement and examines score-class distributions across tasks, metrics, and languages.The study also reports annotator agreement using percentage agreement, weighted F1, and Fleiss’ Kappa; it cautions that skewed class distributions can make Kappa misleading.Source: §3.4 · p. 5

3.5 Ablation Experiments

  • Ablations test repeated-prompt consistency, few-shot examples, linguistic-acceptability sensitivity to shuffled words, and temperature variation on the smaller dataset.Few-shot examples are fixed and drawn from held-out human judgments, with up to two examples per class and a minimum of two and maximum of six per task, language, and metric.Source: §3.5 · pp. 5–6

4 Results

GPT-4’s agreement with human annotators varies by language, metric, task, and prompt, while its scores often skew high when annotators disagree. Additional experiments find limited gains from few-shot examples and lower agreement at higher temperatures.

4.4 Temperature Variation

  • At temperature 0, GPT-4 returned the same score and justification in all five tries; higher temperatures reduce agreement and make evaluations more stochastic and less reproducible.The reported temperature comparison covers 0, 0.3, 0.7, and 1.0.Source: §4.2.1 · p. 7, §4.4 · pp. 7–8

4.2.2 Few-shot Prompting

  • Few-shot in-context examples produced no significant change in percentage agreement, suggesting they may not significantly help LLM-based evaluators.This finding aligns with Min et al. (2022).Source: §4.2.2 · p. 7

4.3 Sensitivity Analysis

  • Word-order perturbations affected linguistic-acceptability scores most in Summarization, while Chinese and Japanese showed little or no sensitivity on the specified tasks.For Open Prompt, Chinese and Japanese showed no sensitivity; the authors offer flexible word order as one possible explanation, alongside genuinely lower evaluator sensitivity.Source: §4.3 · p. 7

4.1 Percentage Agreement

  • GPT’s agreement with annotators is lower than human-human agreement for Japanese and Czech, and for Open Prompt and Continue Writing; compound prompting agrees less than single prompting.Agreement is also lower for the Continue Writing task and the OCQ metric; annotator agreement is relatively low for TQ, while agreement is very high for problematic-content metrics.Source: §4.1 · pp. 6–7, Figure 5 · p. 8
  • Figures 5a, 5b and 5c compare percentage agreement (PA) between LLM evaluators using detailed versus simpler instructions; PA drops slightly for all metrics with detailed instructions.Source: §4.1 · p. 7, Figure 5 · p. 8

4.2 Class Distribution

  • GPT-4 usually assigns the highest score (2), even when annotators disagree; detailed instructions may slightly reduce this bias despite slightly lowering percentage agreement.For non-English languages, annotators partially or do not agree around 15% of the time on average; in 26% of disagreement cases, the evaluator almost never assigns scores of 0 or 1 for lower-resource or non-Latin-script languages.Source: §4.2 · p. 7, Figure 6 · p. 8, Figure 7 · p. 8

5 Discussion

GPT-based evaluators show relatively high consistency for non-English languages at temperature 0, yet tend to assign positive scores when human judgments differ. The authors report prompting trade-offs and recommend cautious use with human calibration in each language.

5 Discussion

  • At temperature 0, GPT-based evaluators show relatively high consistency for non-English languages, but this does not remove positive-score bias.Source: §5 · pp. 8–9
  • Evaluators show fair sensitivity to linguistic-acceptability input variations; Figure 9 tracks cases where perturbation lowers GPT scores.Source: §5 · p. 9, Figure 9 · p. 9
  • Higher scores persist when human annotators disagree, with this bias especially evident for Chinese, Japanese, and Czech.Source: §5 · p. 9
  • Evaluating one metric at a time performs better than evaluating all metrics together, at the cost of multiple LLM calls.Source: §5 · p. 9
  • Few-shot examples do not improve performance, and more detailed instructions do not eliminate higher-score bias.Source: §5 · p. 9
  • The study uses only GPT-4 evaluators and does not conduct extensive prompt tuning.Smaller evaluators and better prompting approaches are identified as directions for future work.Source: §5 · p. 9
  • The authors recommend cautious use and calibration against human judgments in each language before deployment, as evaluators may perform worse on low-resource and non-Latin-script languages.Source: §5 · p. 9

6 Limitations

The study's human-rated outputs were not controlled for quality and were mostly rated positively, making the source of high agreement unclear. The in-house annotations also cannot be released, limiting reproducibility.

6 Limitations

  • Because output quality was uncontrolled and most generated texts received positive human ratings, the source of high Percentage Agreement remains unclear.Source: §6 · p. 10
  • Future work calls for balanced human judgments and datasets with broad language coverage, multiple annotators per item, and clear instructions.The authors also identify evaluator personas representing diverse human perspectives as a research direction.Source: §6 · p. 10
  • Restrictions prevent release of the in-house annotated dataset, limiting reproducibility.Source: §6 · p. 10

7 Ethical Considerations

The ethical considerations use Bender and Friedman’s framework and describe the dataset, annotation, and instruction practices, while cautioning against using calibration methods to replace human judgments.

7 Ethical Considerations

  • The authors frame their ethical discussion using the framework by Bender and Friedman (2018).They state their intent is to highlight the gap between human and LLM judgments and urge caution about replacing human judgments with LLM judgments.Source: §7 · p. 10
  • The calibration dataset is in-house, not publicly released, and was not designed as a balanced dataset.The authors note that problematic content may still be possible even though their ratings found none.Source: §7 · p. 10
  • Annotators were recruited through an external company, but their demographic information is unavailable.Pay was adjusted based on annotator location and expertise.Source: §7 · p. 10
  • Annotators received general task instructions, detailed metric descriptions, and English examples.Source: §7 · p. 10
  • The authors say calibration methods could be misused to replace human judgments, but that is not their stated intent.Source: §7 · p. 10

A.1 Prompts for Simple Instructions

The appendix presents the task description and the prompts used for simple instructions across four evaluation metrics.

A.1 Prompts for Simple Instructions

A.2 Prompts for Detailed Instructions

The appendix shows the complex-instruction prompts for the evaluation metrics.

A.2 Prompts for Detailed Instructions

  • Figures 15–18 show complex-instruction prompts for various metrics.Source: §A.2 · p. 13

A.3 Fleiss’ Kappa

The appendix reports Fleiss’ Kappa table scope and shows class distributions, temperature-variation agreement, and few-shot agreement across the evaluated languages, tasks, and metrics.

A.3 Fleiss’ Kappa

  • Table 3 reports Fleiss’ Kappa values for annotator combinations on the full dataset.The values are aggregated by language, task, and metric; GPT4_SD denotes GPT4_single_detailed.Source: §A.3 · p. 13, Table 3 · p. 15

A.4 Class distribution for Metrics with 3 classes

A.5 Class distribution for Metrics with 2 classes

A.6 Temperature Variations

  • Figure 23 presents Percentage Agreement across temperature variations, aggregated over language, task, and metric on the small dataset.Source: §A.6 · p. 13, Figure 23 · p. 20

A.7 few-shot Results

  • Figure 24 presents Percentage Agreement for few-shot prompting, aggregated over language, task, and metric on the small dataset.Source: §A.7 · p. 13, Figure 24 · p. 20

Related papers

Loading 2309.07462v2…