PaperlayerView exact revision on arXiv

Source-linked AI summary

PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, Sunayana Sitaram

arXiv:2406.15053v2cs.CL

TL;DR

Multilingual evaluation is constrained by benchmark contamination and limited linguistic and cultural coverage. This study evaluates Indic-language models with human and LLM judgments, finding evaluator agreement varies by task and cultural context and identifying bias in LLM judging.

  • Problem

    Multilingual benchmarks are few, may be contaminated, and often translate English benchmarks in ways that lose linguistic and cultural context.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates 30 Indic and multilingual models using human and LLM judgments on independently created prompts across 10 Indic languages, with pairwise and direct-assessment tasks.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    GPT-4o performs best on Indic languages, Llama-3 models show promise, and LLM–human agreement is lower for culturally nuanced evaluations, with evidence of GPT evaluator self-bias.

    Source: §5 · p. 9

  • Takeaways & Limitations

    LLM evaluators capture general trends, but the authors conclude that multilingual, multi-cultural evaluation requires a human-in-the-loop or hybrid system.

    Source: §5 · p. 9

    The study covers only 10 Indic languages, and its prompt set is limited; under the pairwise design, scaling human evaluation to hundreds of prompts is intractable.

    Source: §6 · p. 10

Abstract

from arXiv · show

Evaluation of multilingual Large Language Models (LLMs) is challenging due to a variety of factors -- the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and the lack of local, cultural nuances in translated benchmarks. In this work, we study human and LLM-based evaluation in a multilingual, multi-cultural setting. We evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations and find that models such as GPT-4o and Llama-3 70B consistently perform best for most Indic languages. We build leaderboards for two evaluation settings - pairwise comparison and direct assessment and analyze the agreement between humans and LLMs. We find that humans and LLMs agree fairly well in the pairwise setting but the agreement drops for direct assessment evaluation especially for languages such as Bengali and Odia. We also check for various biases in human and LLM-based evaluation and find evidence of self-bias in the GPT-based evaluator. Our work presents a significant step towards scaling up multilingual evaluation of LLMs.

1 Introduction

The paper motivates multilingual evaluation as a response to contaminated benchmarks and limited linguistic and cultural coverage. It introduces large-scale human and LLM evaluation of models on independently created prompts across Indic languages.

1 Introduction

  • Popular benchmarks may be unsuitable for fair evaluation because models may have encountered them during training, while human evaluation is costly and time-consuming.The paper notes that contamination may occur during pre-training, fine-tuning, and evaluation, motivating dynamic benchmarking.Source: §1 · p. 1
  • Multilingual benchmarks are scarce, cover few languages, may be contaminated, and often translate English benchmarks in ways that lose linguistic and cultural context.The paper notes that multilingual evaluation includes under-resourced languages and languages using non-Latin scripts.Source: §1 · pp. 1–2
  • The study conducts 90K human evaluations across 10 Indic languages, comparing 30 Indic and multilingual models on culturally nuanced prompts using pairwise comparison and direct assessment.Human annotators are recruited through KARYA, which employs workers across Indian states, with a focus on rural and marginalized communities.Source: §1 · p. 2
  • The paper also uses LLM evaluators, creates human- and LLM-based leaderboards, and analyzes evaluator agreement, trends, and biases.The LLM-based evaluations include the same evaluation tasks as the human evaluations.Source: §1 · p. 2

2 Related Work

Prior work includes multilingual and Indic-language benchmarks, human evaluation of LLMs, and LLM-based automatic evaluation. The cited studies span benchmark construction, evaluator agreement and bias, and leaderboard creation.

Multilingual Evaluation Benchmarks

  • Multilingual benchmarks include broad evaluations of open and proprietary models, a generative test set for five languages, and NLU suites such as XGLUE, XTREME, and XTREME-R.These works assess models across available multilingual benchmarks and languages.Source: §2 · p. 2

Indic Evaluation Benchmarks

  • Indic benchmarks cover NLU, generation, machine translation, and diverse generation tasks, with releases spanning between five and 22 languages.The cited work includes IndicGLUE, IndicXTREME, IndicNL-Gsuite, IN22, and IndicGen-Bench.Source: §2 · p. 2

Human Evaluation and LLM-based Automatic Evaluations

  • Prior human-evaluation studies build leaderboards and culturally nuanced prompts, while LLM-evaluation studies examine multilingual judging, agreement, and bias.Other cited applications use GPT-4 as an evaluator alongside humans for MTBench and ChatbotArena.Source: §2 · pp. 2–3

3 Methodology

The methodology combines native-speaker prompt curation, model responses, human and LLM judgments, and agreement analysis. It evaluates pairwise preferences, direct-assessment metrics, and preliminary safety on Indic-language data.

3 Methodology

  • The evaluation pipeline curates prompts, generates model responses, obtains human and LLM judgments in two settings, then builds leaderboards and analyzes evaluator agreement.The two settings are pairwise comparison and direct assessment.Source: §3 · p. 3

3.1 Prompt Curation

  • Native speakers created 20 prompts per language across health, finance, and culturally nuanced topics; cultural prompts were independently written rather than translated.The evaluation covers Hindi, Tamil, Telugu, Malayalam, Kannada, Marathi, Odia, Bengali, Gujarati, and Punjabi.Source: §3.1 · p. 3

3.2 Model Selection

  • The model set includes popular Indic models and their Llama-2, Mistral, and Gemma base families, plus Llama-3 8B and 70B models.The base-family instruct models were included to assess gains from fine-tuning with Indic data; the authors caution that open models and API systems are not entirely fair to compare.Source: §3.2 · p. 3

3.3 Evaluation Setup

  • For open-ended question answering, the study uses LLM-evaluated pairwise battles and direct-assessment metrics, totaling 30,330 datapoints: 21,690 battles and 8,640 direct-assessment datapoints.Three human annotations per datapoint yield 90,990 annotations.Source: §3.3 · p. 3, Table 2 · p. 4

3.3.1 Pairwise comparison

  • Pairwise comparisons use Elo ratings to convert preferences into model scores, with maximum-likelihood Elo used for rankings because it is unaffected by comparison order.The setup includes flipped duplicate pairings for 10% of original pairings to check consistency.Source: §3.3.1 · pp. 3–4

3.3.2 Direct Assessment

  • Direct assessment rates linguistic acceptability, task quality, and hallucination, then ranks models by average weighted scores from 0 to 5.Human annotators can mark gibberish responses, which receive the lowest score; LLM scoring uses a separate call for each metric.Source: §3.3.2 · p. 4

3.3.3 Safety Evaluation

  • The preliminary safety evaluation uses Hindi prompts designed to elicit toxic responses, scores model completions with an LLM, and checks for toxic words by exact match.Humans were not used for this evaluation due to ethical concerns.Source: §3.3.3 · pp. 4–5

3.4 Inter-Annotator Agreement

  • Three human annotators’ agreement is measured with Percentage Agreement and Fleiss Kappa, while human–LLM alignment uses those metrics and leaderboard correlation uses Kendall’s Tau.Majority vote is used for human judgments in the evaluation setup.Source: §3.3 · p. 3, §3.4 · p. 5

4 Results

GPT-4o and Llama-3 70B lead many Indic-language model rankings, but agreement between human and LLM evaluators varies by task and language. The analyses also identify evaluator biases and differences in annotator experience.

4.1 Leaderboard Analysis

  • GPT-4o and Llama-3 70B rank at the top across many languages, while the LLM evaluator rates GPT-4 higher than humans, showing evidence of self-bias.Indic models occupy the middle rankings, with SamwaadLLM performing best among them; the LLM evaluator also rates GPT-4 very highly in Direct Assessment.Source: §4.1 · p. 5, Figure 2 · p. 6, Figure 3 · p. 6

4.2 RTP-LX Safety Analysis

  • For Hindi, GPT-3.5-Turbo produced the fewest problematic completions (∼10%), while Gemma models and fine-tuned variants performed worst; Toxicity-200 misses variants beyond listed stems.The authors caution that the automated safety evaluations may contain biases and leave human evaluation for future work.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 Agreement between LLM and Humans

  • Pairwise human–LLM agreement is κ=0.49 and leaderboard τ=0.76, versus τ=0.65 for Direct Assessment, where agreement is especially low for Bengali and Odia.Human–human pairwise κ averages 0.54; the LLM evaluator agrees comparatively less on culturally nuanced prompts, and Bengali is lowest for both leaderboard types.Source: §4.3 · pp. 6–7, Table 4 · p. 7, Figure 5 · p. 7

4.4.1 Position Bias

  • Both humans and the LLM evaluator remain over 90% consistent when pairwise options are flipped, indicating low position bias.The study randomly flipped options in 10% of pairwise comparisons and measured the fraction of duplicate pairs whose verdict remained unchanged.Source: §4.4.1 · p. 7, Figure 6 · p. 8

4.4.2 Option Distribution

  • LLMs choose fewer ties and select a response in 87% of battles where humans marked both answers hallucinated, versus 53% for humans; they also miss hallucinations in Direct Assessment.LLMs give higher Linguistic Acceptability and Task Quality scores, while humans better distinguish those metrics.Source: §4.4.2 · p. 8

4.4.3 Verbosity Bias

  • Both evaluators favor longer answers, with LLM win fraction near 0.6 for 40–100-word differences; bias declines above 100 words and is slightly stronger for humans.The authors hypothesize that very long responses may contain gibberish, given the 300-word response limit and models’ multilingual limitations.Source: §4.4.3 · p. 8, Figure 9 · p. 8

4.4.4 Self-Bias

  • GPT-4’s average rank rises most under the GPT evaluator—1.4 places among 11 selected models—indicating evaluator self-bias.The authors use Elo rankings because win rates would not account for the GPT evaluator giving fewer ties than humans.Source: §4.4.4 · pp. 8–9

4.5 Human Feedback

  • Annotators found pairwise evaluation easier and less time-consuming; Direct Assessment required online hallucination checks, while grammatical errors made Odia and Tamil responses harder to review.Some annotators found the hallucination concept tricky, and most had not previously worked with LLM responses.Source: §4.5 · p. 9

5 Discussion

The discussion compares Indic model performance and human–LLM agreement across evaluation formats, then reports evaluator biases and the limits of cultural-context evaluation.

RQ1

  • GPT-4o performs best on Indic languages, while newer medium-sized Llama-3 models show potential and smaller Indic models outperform the open-source models they were trained on.The authors identify Llama-3 as a promising starting point for fine-tuning language-specific Indic models.Source: §5 · p. 9

RQ2

  • LLM evaluators agree fairly well with humans in pairwise comparisons, but agreement is low for Marathi, Bengali, and Punjabi and very low across languages in direct assessment, especially Bengali and Odia.Native annotators reported that direct assessment was harder; manually reviewed examples showed human preferences for friendly formatting, colloquial language, and examples.Source: §5 · p. 9
  • LLM evaluators agree less with humans on culturally nuanced responses, although their Kendall τ scores capture higher-level trends; the authors recommend human-in-the-loop or hybrid evaluation.They suggest the weaker agreement reflects insufficient cultural context in the evaluators.Source: §5 · p. 9

RQ3

  • LLM evaluators selected a response when both were hallucinated in 87% of cases, versus 53% for humans; they were also over-optimistic, slightly favored longer responses, and showed GPT self-bias.The study found no position bias from option flipping; LLM evaluators also gave higher direct-assessment scores and fewer pairwise ties.Source: §5 · p. 9

6 Limitations

The study’s scope is bounded by language, prompt, and model coverage, with each constrained by availability or evaluation-design limits.

Language Coverage

  • The study covers 10 Indic languages, and language-specific model availability determined which languages were selected.The authors hope to cover additional Indic languages in future iterations.Source: §6 · p. 10

Prompt Diversity

  • The prompt set is limited; evaluating every model against every other model makes scaling human pairwise battles to hundreds of prompts intractable.The authors plan fewer battles per prompt and more prompts sourced from native speakers.Source: §6 · p. 10

Model Coverage

  • Model coverage is limited to models the authors knew about or could access during the study.They plan to include more models as they become available.Source: §6 · p. 10

7 Ethics Statement

The ethics statement documents oversight and annotation practices, while the accompanying appendices describe rating methods, evaluated models, prompts, and evaluator setups.

7 Ethics Statement

  • Ethics procedures included IRB and KARYA approval, native-language annotators with no other demographics collected, paid Rs. 10 per approximately five-minute datapoint, trained guidelines, and no human safety/toxicity annotators.KARYA pays workers several times India’s minimum wage.Source: §7 · p. 10

A.1 Standard Elo

  • Standard Elo updates a player’s rating after each game using the difference between expected and actual outcomes scaled by K; larger K gives more weight to recent games.The passage introduces the probability of player A winning using the ratings of players A and B.Source: §A.1 · p. 15

A.2 MLE Elo

  • For fixed model weights, the Bradley–Terry approach treats pairwise win rates as fixed and uses logistic regression to calculate maximum-likelihood Elo ratings.Unlike standard Elo, it assumes performance does not change with battle order.Source: §A.2 · p. 15

B Model Details

  • The study evaluates 20 Indic and 10 multilingual models, classified as Indic, open-source, or proprietary according to language fine-tuning and access.Indic models are fine-tuned specifically for Indian languages; proprietary models are closed-source with API access and undisclosed model or training data.Source: B Model Details · p. 15

C Prompt-Response Generation

  • Models receive a system instruction and query without few-shot examples, with responses limited to 300 words and truncated when needed for smartphone evaluation.Open-source prompt templates follow model cards where available, otherwise the default Llama2 prompt is used.Source: C Prompt-Response Generation · p. 15

D LLM Evaluator Setup

  • GPT-4-32k serves as the LLM evaluator, with separate prompts specified for each evaluation type.The passage directs readers to the detailed prompts below.Source: D LLM Evaluator Setup · p. 15

D.1 Pairwise Evaluation

  • In pairwise evaluation, the LLM evaluator sees the query and two model responses, then chooses the better response or a tie and provides justification.Source: §D.1 · p. 15

D.2 Direct Assessment

  • Direct assessment gives the LLM evaluator a query-response pair and rubric, with separate calls scoring hallucinations, task quality, and linguistic acceptability.Source: §D.2 · p. 15

D.3 RTP-LX Safety Evaluation

The preliminary safety evaluation uses LLM evaluators to score Hindi model outputs for problematic content on the RTP-LX dataset.

D.3 RTP-LX Safety Evaluation

  • The preliminary safety study evaluates Hindi models using the RTP-LX Hindi dataset, with LLM evaluators only.Source: §D.3 · p. 15
  • The study reuses the direct-assessment instruction prompt to evaluate model outputs for problematic content.Source: §D.3 · p. 15
  • Problematic content is scored using the rubric referenced in Figure 15.Source: §D.3 · p. 15

E Human Evaluation Setup

The human evaluation setup uses KARYA annotators for pairwise comparisons and direct assessments, while the leaderboards report three ranking approaches across the 10 languages.

E Human Evaluation Setup

  • KARYA annotators complete both evaluation tasks after training and screening, but are not assigned safety evaluations because of ethical concerns.Evaluations are conducted on smartphones.Source: E Human Evaluation Setup · p. 16

E.1 Pairwise

  • In pairwise evaluation, annotators compare two responses to one prompt using guidelines and a Hindi app interface.The instructions and screenshots are shown in Figures 16 and 17.Source: §E.1 · p. 16

E.2 Direct Assessment

  • In direct assessment, a gibberish flag assigns the lowest score; otherwise, annotators rate three metrics for the query-response pair.The guidelines and Hindi app interface are shown in Figures 18 and 19.Source: §E.2 · p. 16

F Leaderboards

  • The leaderboard analysis uses 100 bootstrap samples of upsampled data to estimate confidence intervals because datapoint counts are lower.The evaluated-model details are listed in Table 6, including language-coverage exceptions.Source: F Leaderboards · p. 16, Table 6 · p. 17

F.1 MLE Elo Leaderboards

  • MLE Elo leaderboards cover all 10 languages in Tables 7–16.The supplied passages identify the tables but do not provide their ranking values.Source: §F.1 · p. 17

F.3 Direct Assessment Leaderboards

  • Direct Assessment leaderboards cover all 10 languages in Tables 27–36.The supplied passages identify the tables but do not provide their ranking values.Source: §F.3 · p. 17

G Percentage Agreement

Percentage Agreement (PA) provides a raw-interpretable measure of human and LLM evaluator agreement, though it does not account for class imbalance. Agreement patterns vary by evaluation setting and language.

Percentage Agreement

Pairwise Battles

  • 70% is the average agreement among humans in pairwise battles; human-average and LLM evaluator accuracy is similar but slightly lower.Both evaluators agree more on non-cultural queries.Source: G Percentage Agreement · p. 17
  • Pairwise human and LLM evaluators agree less on Marathi, Punjabi, and Bengali.Source: G Percentage Agreement · p. 17

Direct Assessment

  • In direct assessment, humans agree slightly more with one another than in pairwise evaluation, while human-LLM agreement declines most on culturally nuanced queries.The pattern is similar across prompt types.Source: G Percentage Agreement · p. 17
  • Odia and Bengali have the lowest direct-assessment agreement between humans and the LLM evaluator.Source: G Percentage Agreement · p. 17

H Self-Bias

The self-bias analysis compares GPT-evaluator Elo ranks with human ranks across languages. GPT variants show the largest rank increases, while some other models fall.

H Self-Bias

  • In the detailed self-bias analysis, Table 38 reports the average change in Elo leaderboard ranks across languages when evaluated by the GPT-4 evaluator, in comparison with evaluation by humans.Source: H Self-Bias · p. 17
  • GPT variants have the largest rank increases under the GPT evaluator, while Aryabhatta variants and Llama-3 70B have large drops.Gemma also shows an increase.Source: H Self-Bias · p. 18

Pairwise Evaluation

The supplied materials describe how annotators compare responses and rate direct-assessment criteria, and identify language-wise agreement and rank-comparison visuals. They do not provide leaderboard values or state outcomes for those visuals.

Pairwise Evaluation

Direct Assessment

  • Direct assessment rates Linguistic Acceptability on a 0–2 scale, from gibberish or poor structure to perfect language even if the answer is incorrect.Source: Direct Assessment · p. 22
  • Figure 20 breaks down PA scores by language for pairwise and direct-assessment evaluations; compare agreement across languages and settings.Table 37 reports average PA correlations between humans and Human-LLM across both evaluations and prompt types.Source: Figure 20 · p. 33, Table 37 · p. 33
  • Table 38 compares average Elo-rank changes across languages for the GPT evaluator against humans; compare the rank shifts by model.Source: Table 38 · p. 33

Related papers

Loading 2406.15053v2…