PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, Seunghyeok Hong

arXiv:2409.11239v2cs.CL

TL;DR

Automated evaluators are widely used, but their reliability beyond familiar English evaluation settings is not well established. This paper introduces KUDGE, a bilingual meta-evaluation benchmark with original, corrupted-response, and challenging-reasoning tests. Evaluation ability transfers to Korean, yet evaluators often miss factual and cultural errors and struggle with complex reasoning in both languages.

  • Problem

    The broader capabilities of automated evaluators remain insufficiently understood across non-English prompts, factual verification, and complex reasoning.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper introduces KUDGE, a bilingual Korean–English meta-evaluation dataset covering standard judgments, corrupted responses, and challenging reasoning prompts.

    Source: §1 · pp. 1–2

  • Results

    English evaluation ability predicts Korean evaluation performance, but evaluators fail to detect many false or culturally misrepresentative responses and struggle with complex reasoning in both languages.

    Source: §1 · p. 2, §9 · p. 10

  • Takeaways & Limitations

    Automated evaluators can transfer evaluation ability across languages but are not consistently reliable factual verifiers or judges of complex reasoning.

    Source: §1 · p. 2, §9 · p. 10

    Fine-tuned evaluator rankings can diverge between KUDGE and REWARDBENCH, possibly because training-data curation encourages overfitting to English evaluations.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

LLM-as-a-Judge and reward models are widely used alternatives of multiple-choice questions or human annotators for large language model (LLM) evaluation. Their efficacy shines in evaluating long-form responses, serving a critical role as evaluators of leaderboards and as proxies to align LLMs via reinforcement learning. However, despite their popularity, their effectiveness in diverse contexts, such as non-English prompts, factual verification, or challenging questions, remains unexplored. In this paper, we conduct a comprehensive analysis of automated evaluators, reporting several key findings on their behavior. First, we discover that English evaluation capabilities significantly influence language-specific evaluation capabilities, often more than the language proficiency itself, enabling evaluators trained in English to easily transfer their skills to other languages. Second, we identify critical shortcomings, where LLMs fail to detect and penalize errors, such as factual inaccuracies, cultural misrepresentations, and the presence of unwanted language. Finally, we find that state-of-the-art evaluators struggle with challenging prompts, in either English or Korean, underscoring their limitations in assessing or generating complex reasoning questions. We release the dataset and codes used.

1 INTRODUCTION

This paper examines when automated evaluators work across languages and where they fail, using a bilingual benchmark and corrupted responses. It finds transferable evaluation ability but weaknesses in factual and cultural error detection and complex reasoning.

  • The paper investigates automated evaluators’ effectiveness across languages, factual verification, cultural representation, and challenging prompts.Source: §1 · pp. 1–2
  • KUDGE evaluates Korean and English responses through pointwise and pairwise judgments, while its Challenge subset targets STEM questions requiring complex reasoning.Source: §1 · p. 1
  • Human-corrupted responses containing cultural misrepresentations or false information test whether judges detect knowledge-related errors.Source: §1 · p. 2
  • English evaluation performance predicts Korean evaluation ability more strongly than Korean language benchmarks, suggesting that evaluation capability is substantially language-agnostic.Source: §1 · p. 2
  • Automated evaluators fail to reliably penalize false information and cultural misrepresentations and struggle with complex reasoning questions in both English and Korean.Source: §1 · p. 2
  • The work releases KUDGE, a bilingual meta-evaluation dataset with a challenge subset for evaluating English and Korean models.Source: §1 · p. 2

2 RELATED WORKS

The related work frames automated evaluators as scalable alternatives to human and lexical-overlap evaluation, while noting that their reliability across languages and difficult tasks remains insufficiently examined.

  • BLEU and ROUGE measure lexical overlap but cannot reliably assess semantically equivalent outputs with different syntax, making human evaluation desirable yet costly.Source: §2 · p. 2

Automated Evaluators

  • LLM-as-a-Judge models generate rubric-based analyses and scores, whereas reward models use classification heads to output continuous scores efficiently for alignment systems.Source: §2 · p. 2

Meta-Evaluation

  • Meta-evaluation benchmarks measure how closely automated evaluators mirror human judgments and assess their reliability as proxies for annotators.Source: §2 · p. 2
  • Existing meta-evaluation has not fully examined automated evaluators in non-English, factual-verification, and complex-reasoning settings.Source: §2 · pp. 2–3

3 KUDGE: A BILINGUAL BENCHMARK FOR AUTOMATED JUDGES

KUDGE is a bilingual benchmark built from culturally grounded Korean questions, human annotations, paired evaluation formats, and bilingual reasoning challenges. Its quality controls combine annotator agreement analysis with filtering of substantial disagreements.

3 KUDGE: A BILINGUAL BENCHMARK FOR AUTOMATED JUDGES

  • KUDGE provides Korean-focused original evaluations and English–Korean Challenge evaluations for assessing automated judges.Source: §3 · p. 3

3.1 DATASET CREATION

  • The dataset uses 90 handcrafted instructions spanning nine Korean knowledge categories and seven reasoning skills, with rubrics and gold-standard responses.Source: §3.1 · p. 3
  • Fifteen annotators rate model responses from 1 to 5 using reference answers and rubrics, with each instance evaluated independently by two annotators.Source: §3.1 · p. 3

3.2 QUALITY ANALYSIS

  • Annotations are identical or within one point in 83.85% of cases, while disagreements exceeding two points occur in 8.36% and are excluded from averaging.Source: §3.2 · pp. 3–4

3.3 POINTWISE & PAIRWISE SUBSETS

  • The Original subset contains 2506 pointwise and 818 pairwise instances, converting high- and low-scored responses into chosen and rejected pairs.Source: §3.3 · p. 4

3.4 CHALLENGE SUBSET

  • The Challenge subset covers English and Korean STEM questions at easy and hard difficulty, using MMLU and GPQA questions with correct and incorrect chain-of-thought responses.Source: §3.4 · p. 4

4 EXPERIMENTAL SETUP

The experiments standardize generative judge outputs, reuse an established reward-model implementation, and evaluate 20 models on KUDGE. The setup includes multilingual benchmark accounting and both proprietary and open-weight model families.

4.1 PROMPT CONFIGURATION

  • LLM-as-a-Judge models generate analyses followed by standardized pointwise or pairwise decisions, with up to three retries for formatting failures.Source: §4.1 · p. 4
  • Reward models are evaluated using the codebase provided by Lambert et al. (2024).Source: §4.1 · p. 4

4.2 EVALUATED MODELS

  • The study evaluates 20 LLMs, including proprietary models, open-source families, and bilingual EXAONE-3.0-7.8B-Instruct.Source: §4.2 · pp. 4–5
  • Table 3 reports KUDGE ORIGINAL results using off-by-0.5 pointwise accuracy, with random-guessing baselines of 30.86% for pointwise and 50% for pairwise.Source: Table 3 · p. 5

5 EVALUATION RESULTS

On KUDGE ORIGINAL, proprietary models lead overall, but pointwise agreement with human judgments remains limited. Aggregating models improves accuracy only marginally and remains inferior to GPT-4o alone.

5 EVALUATION RESULTS

  • GPT-4o leads overall performance at 74.51, while Claude-3.5-Sonnet scores 72.79 and open models Llama-3.1-405B and Mistral-Large-Instruct score 73.15 and 73.19.Source: §5 · p. 5
  • Pointwise Pearson correlation remains below 0.6 for most models, indicating only moderate agreement with human evaluations.Source: §5 · p. 5
  • Larger models can underperform peers, with Qwen1.5-72B-Chat trailing Mixtral-8x22B-Instruct by 21.28% on KUDGE and Command-R-Plus trailing by 12.31%.Source: §5 · p. 5
  • GPT-4o, Claude-3.5-Sonnet, Llama-3.1-405B, and Mistral-Large-Instruct show the lowest score-margin variances, ranging from 0.06 to 0.08.Source: §5 · p. 6

5.1 EVALUATION WITH MAJORITY VOTING

  • Ensembling sampled models yields minimal gains and still underperforms GPT-4o alone, as increasing VIF from 10.02 to 16.91 indicates limited diversity.Source: §5.1 · p. 6, Figure 3 · p. 6

6 ARE JUDGING CAPABILITIES TRANSFERRED TO NEW LANGUAGES?

KUDGE results indicate that English evaluation ability transfers to Korean more strongly than Korean-language proficiency, although transfer introduces biases and fails on some error types.

6.1 CORRELATION WITH DIFFERENT FEATURES

  • REWARDBENCH predicts Korean KUDGE performance better than Korean-language benchmarks, indicating that English evaluation ability transfers across languages.Source: §6.1 · p. 7, Figure 4 · p. 7
  • After adjusting for model size, Korean capability features show poor significance, whereas size-adjusted REWARDBENCH remains a stronger KUDGE predictor.Source: Table 4 · p. 6, §6.1 · p. 7

6.2 EVALUATION WITH FINE-TUNED LLMS

  • Prometheus2 improves accuracy over Mistral by 24.1% and 24.7% across model sizes, but favors longer English strings.Source: §6.2 · p. 7, Table 5 · p. 7
  • English-only reward models transfer effectively to Korean pairwise evaluation, with FsfairX-RM-8B reaching 90.22 versus Claude-3.5-Sonnet at 89.11.Source: §6.2 · p. 8, Table 6 · p. 8
  • The inverse KUDGE–REWARDBENCH rankings are based on only four reward models, so broader generalization requires future study.Source: §6.2 · p. 8

6.3 QUALITATIVE ANALYSIS ON BIASES

  • Models show significantly different score distributions on responses with unwanted characters or incomplete answers, with EXAONE-3.0-7.8B-Instruct showing the largest shift.Source: §6.3 · pp. 8–9, Figure 5 · p. 8

7 CAN LLMS JUDGE RESPONSES WITH FALSE INFORMATION?

The paper tests automated evaluators on human-verified factual corruptions and finds that subtle factual errors remain difficult to detect, although pairwise comparison can help.

  • Pointwise judges struggle to identify factual errors, especially subtle word- and sentence-level corruptions, while paragraph-level changes are easier to detect.Source: §7 · p. 9, Table 7 · p. 9
  • In pairwise evaluation, reward models score below the 50% random baseline, whereas GPT-4o and Claude-3.5-Sonnet detect correct factual errors 26 and 32 times.Source: §7 · p. 9, Table 8 · p. 9
  • Pairwise comparison may improve factual-error detection, but nearly identical response pairs are uncommon and judges can generate fabricated reasons.Source: §7 · p. 9

8 CAN LLMS EVALUATE CHALLENGING PROMPTS?

Automated evaluators perform worse as prompts become more difficult, revealing substantial limitations on complex reasoning questions.

  • No model exceeds 70% accuracy on the harder GPQA subset, despite a 50% random-guessing baseline.Source: §8 · p. 10
  • All models perform better on simpler MMLU questions, while reward models underperform on harder prompts and may be constrained by their smaller sizes.Source: §8 · p. 10

9 CONCLUSION

KUDGE evaluates automated evaluators in Korean and English, revealing comparable cross-language performance alongside weaknesses in factual and cultural error detection and complex reasoning. Its supporting K2-EVAL study also documents human-annotation variability and a modest relationship between response length and score.

9 CONCLUSION

  • KUDGE finds LLM-as-a-Judges and reward models perform comparably in Korean, while struggling with factual or cultural errors and complex reasoning tasks.The dataset evaluates meta-evaluation capability across Korean and English and challenges the assumption that native Korean proficiency is essential for high-quality evaluation.Source: §9 · p. 10

A.1 K2-EVAL

  • K2-EVAL combines 90 Korean prompts across nine knowledge areas and seven reasoning skills, with GPT-4-generated answers reviewed by two authors and task-specific rubrics.The dataset measures response effectiveness, cultural accuracy, and Korean linguistic features such as honorifics.Source: §A.1 · p. 15, Table 10 · p. 15
  • Higher human scores show a slight association with longer responses, but not to a concerning extent.Response length is measured in characters against average human-annotated scores.Source: §A.1 · p. 15, Figure 6 · p. 16

A.2 HUMAN ANNOTATION

  • Human annotators completely agree on 52.2% of cases, differ by one point in 30.7%, and disagree by more than two points in 17.1%.The study averages scores when differences are at most one point and excludes larger disagreements from analysis.Source: §A.2 · pp. 15–16

A.3 BENCHMARK RESULTS

The benchmark evaluates 31 language models with human judgments and standardized pointwise or pairwise prompts. It also examines common evaluator errors and uses manually corrupted responses to probe factual-error detection.

A.3 BENCHMARK RESULTS

  • HyperCLOVA X ranks highest by human evaluation, followed by GPT-4 and Command-R-Plus, while smaller Korean-finetuned models outperform larger general models.The comparison includes EEVE-Korean-Instruct-10.8B and KULLM3 outperforming Mixtral-8x22B-Instruct and Qwen-1.5-72B-Chat, highlighting the role of localized tuning.Source: §A.3 · p. 16, Figure 8 · p. 17

B EVALUATION PROMPTS

  • The evaluation prompts require either choosing the better response or assigning a 1-to-5 score with rubric-based feedback in fluent Korean.Pairwise prompts emphasize helpfulness, relevance, accuracy, depth, creativity, detail, and avoiding position or length bias.Source: B EVALUATION PROMPTS · pp. 17–18

C EXAMPLES FROM QUALITATIVE ANALYSIS

D FURTHER DETAILS ON FACTUAL PERTURBATIONS

E ADDITIONAL RESULTS

Additional analyses extend the benchmark with a Two-Sample Kolmogorov-Smirnov test and examples of factual-error detection and response corruptions. The figures define the tested error categories and significance notation.

E ADDITIONAL RESULTS

  • Figure 15 reports Two-Sample Kolmogorov-Smirnov Test outcomes for models not covered in the preceding Section 6.3 results.The test uses significance markers of ** for p < 0.05 and *** for p < 0.01, with unwanted characters and incomplete answers abbreviated as U.C and I.A.Source: E ADDITIONAL RESULTS · p. 19, Figure 15 · p. 23
  • Figures 11–14 provide examples of Claude-3.5-Sonnet factual-error feedback and word- or sentence-level response corruptions, with translations for accessibility.The figures distinguish word-level and sentence-level perturbations and annotate corruption details.Source: Figure 11 · p. 20, Figure 12 · p. 20, Figure 13 · p. 21

Related papers

Loading 2409.11239v2…