PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro

arXiv:2610.31172v1cs.CLcs.AIcs.LG

TL;DR

Whether prompting strategies transfer across judge architectures and language resource levels remains unresolved: prompt-sensitivity studies are largely English-centric, while multilingual judge studies fix the prompt. We benchmark 9 strategies across 4 open-weight judges, 27 languages, and 4 tasks, finding that prompting effects vary by judge and language. The results show that strategy choice should be validated for the intended judge, task, and language.

  • Problem

    Whether prompting-strategy effects depend on the judge and language resource level remains untested across open-weight judges and distinct strategies.

    Source: §1 · p. 2

  • Method

    We benchmark 9 prompting strategies across 4 open-weight judges, 27 languages, and 4 tasks over 13 datasets using gold-reference outputs.

    Source: §1 · p. 2

  • Results

    Prompting effects vary across judges: the judge–strategy interaction exceeds the strategy main effect, rankings show limited agreement, and target-language prompting has opposing effects across models.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Prompt design should be treated as part of judge configuration and validated for the intended judge, task, and language.

    Source: §6 · p. 8

    Gold-reference evaluation introduces ceiling effects in QA and reference-based MT, limiting strategy discrimination for those tasks.

    Source: Limitations · p. 8

Abstract

from arXiv · show

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

1 Introduction

Prior work leaves open whether prompting-strategy effects transfer across open-weight judges and language resource levels. This study benchmarks strategies across judges, languages, and tasks, finding that effects depend on the judge and can vary across languages.

1 Introduction

  • Reference-based and learned metrics require target-language human references, which are unavailable for most languages and costly to produce at scale.LLM judges apply rubric criteria directly rather than scoring against a reference, extending evaluation to languages without annotated evaluation data.Source: §1 · p. 1
  • Prior studies leave untested whether prompt-strategy effects extend to open-weight judges, lower-resource languages, and distinct prompting strategies.Existing prompt-sensitivity work mainly studies English template changes, while multilingual judge studies do not vary the prompting strategy.Source: §1 · pp. 1–2
  • The benchmark covers 9 prompting strategies, 4 open-weight judges, 27 languages, 4 tasks, and 13 datasets, yielding 1.07M pointwise scores.It scores gold-reference outputs with task-specific rubrics while holding generation quality fixed.Source: §1 · p. 2
  • Judge identity explains η2 = 0.087 of score variance versus η2 = 0.012 for prompting strategy, and the judge × strategy interaction exceeds the strategy main effect.The results indicate that strategy effectiveness is conditional on the judge rather than intrinsic to the prompt.Source: §1 · p. 2
  • Target-language prompting raises Gemma 3’s mean score by 0.83 but lowers Aya Expanse’s by 0.32 on identical inputs.Strategy-ranking agreement falls from ρ = 0.46 on high-resource machine translation to ρ = 0.04 on the low-resource tier, and is negative on summarisation.Source: §1 · p. 2
  • Across evaluations on SSA-MTE, SummEval, and WMT MQM, Role Prompting attains the highest aggregate correlation, reaching ρ = 0.483 across these three evaluation settings.Source: §1 · p. 2

2 Related Work

Related work studies LLM-judge reliability, prompt sensitivity, and multilingual evaluation, but does not jointly vary judge architecture, prompting strategy, and language resource level. This paper examines those factors together.

2 Related Work

  • Earlier LLM-as-a-judge research examines reliability factors including task complexity, reference quality, prompt design, and candidate order (Gu et al., 2024a).Fu and Liu (2025a) report mean Fleiss’ κ of approximately 0.3 across five judges and 25 languages, with lower agreement in low-resource languages.Source: §2 · p. 2
  • Prompting strategies adapted for judging include rubric-conditioned scoring, persona conditioning, and translate-then-evaluate pipelines.Calibration-example prompting has also been used for translation quality estimation.Source: §2 · p. 2
  • The Judge Sensitivity Score measures decision changes under semantically equivalent rephrasing; its evaluation covered 13 judges across four tasks, found scale did not predict stability, and was English-only.Related Sclar et al. (2024) work found formatting perturbations to a fixed prompt substantially shifted task-solving LLM performance, also in English.Source: §2 · p. 3
  • Prior multilingual judge studies report lower agreement in low-resource languages, while MM-Eval and M-Prometheus fix the prompting protocol.The related work identifies no prior study crossing prompting strategy with judge architecture and language resource level simultaneously.Source: §2 · p. 3

3 Methodology

The methodology benchmarks prompting strategies across judges, languages, tasks, and datasets using shared inputs and task-specific rubrics. It also evaluates alignment with human judgments and analyzes score variation, strategy effects, and inter-judge agreement.

3.1 Experimental Design

  • The benchmark compares four open-weight judges and nine prompting strategies on shared inputs and task-specific rubrics, with human-written references except for SSA-MTE.Human-score alignment is evaluated separately using system outputs from SSA-MTE, SummEval, and WMT MQM.Source: §3.1 · p. 3

3.2 Tasks and Datasets

  • The evaluation covers machine translation, summarisation, question answering, and mathematical reasoning across 27 languages grouped by resource level.Unless otherwise specified, up to 200 instances are randomly sampled per dataset–language split using seed 42; Hausa and Swahili are the only languages represented across all four tasks.Source: §3.2 · p. 3

3.3 Prompting Strategies

  • Nine prompting strategies vary calibration examples, explicit reasoning, evaluator roles, and prompt language while sharing the same rubric and five-point scale.Language-handling strategies include English instructions for target-language candidates, translated instructions, and translating candidates into English before evaluation.Source: §3.3 · pp. 3–4

3.4 Evaluation Protocol

  • Judges assign holistic scores from 1 to 5 and task-specific criterion scores; the protocol also measures agreement with human annotations on three datasets.Passage grounding is assessed where QA datasets provide a supporting passage, while mathematical-reasoning rubrics assess the final answer, derivation validity, and solution completeness.Source: §3.4 · p. 4, Table 3 · p. 4

3.5 Judge Models and Implementation Details

  • The study uses Aya Expanse 32B, Gemma 3 27B, Qwen 2.5 32B, and Qwen 2.5 72B-AWQ, producing 1,068,528 valid scores.Self-consistency uses three sampled responses, selecting the modal holistic score or their arithmetic mean when all three differ; other strategies use temperature 0.Source: §3.5 · p. 5

3.6 Statistical Analysis

  • Analyses quantify cross-language variation and dispersion, compare strategies by task, measure judge agreement, and estimate judge, strategy, and language effects.The analysis uses Spearman’s ρ for strategy-ranking agreement, Pearson’s r for strategy-level mean scores, and Type-II eta squared; pairwise strategy comparisons are descriptive because ceiling effects compress some score ranges.Source: §3.6 · p. 5

4 Results

Across tasks, judge identity and its interaction with prompting strategy explain more score variance than strategy alone. Strategy rankings and language effects vary across judges and resource tiers, while human-alignment evidence is limited for non-English tasks beyond MT.

4.1 Variance Decomposition

  • η² = 0.087 for judge identity and η² = 0.012 for strategy; their interaction (η² = 0.056) also exceeds the strategy main effect.All effects are significant at p < 0.001; judge identity explains approximately seven times as much variance as strategy.Source: §4.1 · p. 5

4.2 Strategy Effects Across Judges

  • ρ = 0.46 on high-resource MT but ρ = 0.04 on low-resource MT; strategy rankings also show negative agreement in three of four Summarization tiers.Summarization values range from −0.30 to −0.02, while QA ranges from −0.08 to +0.02.Source: §4.2 · p. 6, Table 5 · p. 6

4.3 Strategy Effects Across Languages

  • Cross-lingual MT score ranges widen from 0.13 in high-resource languages to 1.60 in low-resource languages, while strategy means differ by task.CoT has the highest mean on Summarization, QA, and Math; Monolingual is lowest on MT and Summarization, and Prometheus on QA and Math.Source: §4.3 · p. 6, Figure 1 · p. 6, Table 6 · p. 7
  • ∆ = +0.83 for Gemma and −0.32 for Aya under Monolingual prompting; Aya’s largest decreases are Yoruba (−2.16), Hausa (−1.56), and Swahili (−1.20).Monolingual’s overall English–low-resource difference is +0.009, while CoT reaches +0.206 overall and +0.542 on Math; excluding SSA-MTE, most MT differences are within 0.11 of zero.Source: §4.3 · p. 6, Figure 2 · p. 7

4.4 Agreement with Human Judgments

  • ρ = 0.22–0.46 for all judge–strategy pairs against human annotations; Role Prompting has the highest cross-dataset mean (ρ = 0.483).SSA-MTE is the only source of non-English human annotations, so Summarization, QA, and Math are not validated against human judgments in the low-resource languages studied.Source: §4.4 · p. 7, Table 8 · p. 7

5 Discussion

The discussion argues that prompting effects depend on both judge and target language, so strategies tuned for one judge may not transfer. It also highlights language-level instability and a trade-off between consistency and human alignment.

5 Discussion

  • Judge × strategy interactions exceed the strategy effect, and the judge’s effect on scores also depends on the target language.The discussion notes an exception on Summarization, where judges exhibit different strategy preferences.Source: §5 · p. 7
  • CoT has the lowest cross-lingual variability (mean CV = 0.036), but does not consistently achieve the highest human alignment.The discussion characterizes the strongest strategy effects as occurring in QA and Summarization (η² = 0.059 and 0.057, respectively).Source: §5 · p. 8
  • Target-language prompting raises Gemma’s mean score by 0.83 but lowers Aya’s by 0.32, with decreases reaching 2.16 on Yoruba.The authors suggest this architecture-dependent pattern may reflect different responses to target-language prompts, but do not isolate the underlying cause.Source: §5 · p. 8
  • Inter-model strategy-rank agreement falls from ρ = 0.46 on high-resource MT to ρ = 0.04 on low-resource MT, while cross-lingual ranges widen as resources decrease.The discussion states that these effects concern strategy rankings as well as absolute scores.Source: §5 · p. 8
  • Cross-lingual has the most balanced overall English–low-resource difference (∆ = −0.130), while CoT has the largest English bias in Math (∆ = +0.542).Monolingual’s near-zero overall difference (+0.009) hides positive and negative per-language biases.Source: §5 · p. 8

6 Conclusion

Across four open-weight judges, 27 languages, and four generation tasks, prompting effects varied by judge; the judge–strategy interaction exceeded the strategy main effect. Strategy stability and agreement with human judgments did not align, so prompts should be validated for the intended judge, task, and language.

6 Conclusion

  • The evaluation covered nine prompting strategies, four open-weight judges, 27 languages, and four generation tasks.Source: §6 · p. 8
  • Prompting effects varied across judges: the judge–strategy interaction exceeded the strategy main effect, judges had limited agreement on strategy rankings, and target-language prompting effects reversed direction across models.Source: §6 · p. 8
  • The strategy with the lowest cross-language dispersion differed from the strategy with the highest aggregate correlation with human judgments, so score stability did not imply evaluation validity.Source: §6 · p. 8
  • Prompt design should be treated as part of the judge configuration and validated for the intended judge, task, and language.Source: §6 · p. 8

Limitations

Gold-reference evaluation limits strategy discrimination in QA and reference-based MT, and the study's validation and model coverage are constrained. Prompt verification and language distribution also qualify comparisons across languages and resource tiers.

Limitations

  • Gold-reference evaluation introduces ceiling effects that limit strategy discrimination in QA and reference-based MT; real-output confirmation uses AfroBench but covers only a subset of judges, tasks, and strategies.Source: Limitations · p. 8
  • Native-speaker verification covered only a subset of languages, so part of the observed Monolingual effect may reflect prompt quality rather than judge behavior alone.Source: Limitations · pp. 8–9
  • The language distribution is weighted toward low-resource languages, warranting caution in cross-tier comparisons; the two-way ANOVA omits language interactions, which are analyzed separately with one-way ANOVAs.Source: Limitations · p. 9
  • SummEval and WMT MQM are English-only, leaving non-English summarization and reasoning untested; findings from four open-source judges may not generalize to closed-source models, tested only exploratorily.Source: Limitations · p. 9
  • The authors caution against excluding low-resource languages and recommend validating judge and strategy choices per language before deployment, particularly when evaluation affects model selection or content moderation.Source: Ethical considerations · p. 9

A Additional Methodology Details

This appendix provides additional methodological details, including task-specific scoring rubrics, prompt examples, and per-dataset language codes. It also describes how several prompting strategies were operationalized, including language-specific instructions, multi-step evaluation, role conditioning, and repeated scoring.

B Task-Specific Rubric Definitions

  • Every strategy uses task-specific rubric criteria, scores each dimension independently from 1 to 5, and adds a holistic overall score for cross-task comparison.The prompt appendix lists templates for nine strategies; Self-Consistency is limited to question answering and mathematical reasoning.Source: B Task-Specific Rubric Definitions · p. 11, C Prompt Examples · p. 14

Machine Translation

  • Machine-translation scoring uses MQM adequacy, fluency, and terminology dimensions, covering meaning preservation, target-language naturalness, and correct, consistent terms.Adequacy ranges from preserved meaning to unrelated translation; fluency ranges from native-like to incomprehensible.Source: Machine Translation · pp. 11–12, Dimension 3: Terminology · p. 12

A Additional Methodology Details

Summarization

  • Summarization uses SEAHORSE criteria for comprehensibility, source attribution, coverage of main ideas, and conciseness.The rubric evaluates clarity, whether claims are traceable to the source, whether key points are captured, and whether length avoids redundancy or critical omissions.Source: Summarization · pp. 12–13, Dimension 4: Conciseness · p. 13

QA with Passage

Mathematical Reasoning

C Prompt Examples

  • The prompt appendix provides examples for Zero-Shot, Few-Shot Anchored, Prometheus, and Chain-of-Thought evaluation, including calibration examples for Few-Shot and stepwise reasoning instructions for Chain-of-Thought.Examples generally use English–Swahili machine translation from FLORES-200; the Monolingual Prompting example uses Catalan.Source: C Prompt Examples · p. 14, Zero-shot · p. 14, Few-Shot Anchored · p. 14

Chain-of-Thought

  • The Chain-of-Thought prompt asks the model to reason step by step before providing its overall score.The prompt also includes the model translation as the text to evaluate.Source: Chain-of-Thought · p. 14

Cross-Lingual

  • The Cross-Lingual prompt applies the English evaluation criteria to Swahili content and explicitly says not to penalize outputs for being non-English.It requests dimension scores and an overall score from 1 to 5 with a brief justification.Source: Cross-Lingual · p. 15

Monolingual

  • The Monolingual prompt is written in Catalan, with equivalent verified templates also used for French, Spanish, Catalan, and Persian.The Catalan template requests a score from 1 to 5 and a brief justification.Source: Monolingual · p. 15

Translate-then-Evaluate

  • Translate-then-Evaluate asks the judge to mentally translate the Swahili output into English, state its understanding, then evaluate that understanding against the rubric.The prompt asks the judge to show both steps before reporting the final score.Source: Translate-then-Evaluate · p. 15

Role Prompting

  • The Role-Based prompt casts the judge as a professional translator and linguistic evaluator trained in MQM and specializing in English–Swahili assessment.It requests rubric-dimension scores and an overall score from 1 to 5 with a brief justification.Source: Role Prompting · p. 15

Self-Consistency

  • Self-Consistency runs the Chain-of-Thought prompt three times at temperature 0.7, using majority vote or the mean if all scores differ, only for question answering and mathematical reasoning.Each run identifies task requirements, scores and explains rubric dimensions, notes strengths and weaknesses, and assigns an overall score.Source: Self-Consistency · p. 15

E Per-Model Strategy × Dataset Heatmaps

Figures 3–6 provide per-model breakdowns of mean score and cross-language range by strategy and resource tier across four tasks. Table 10 reports cross-language CV for MT and summarization, with selected resource tiers only.

E Per-Model Strategy × Dataset Heatmaps

  • Figures 3–6 break down mean score and cross-language range by strategy and resource tier for each model across four tasks, extending the MT and summarization summary in Figure 1.Source: E Per-Model Strategy × Dataset Heatmaps · p. 15
  • Table 10 reports cross-language CV (σ/µ of per-language means) for MT and summarization; it includes only high and low resource tiers because mid and low-mid each have two languages.Source: Table 10 · p. 16

G Rubric Dimension Analysis

Figures 8–10 provide rubric-level evidence, identifying MT Terminology and Summarization Main Ideas as especially sensitive dimensions. The appendix extends the variance-decomposition and English–low-resource gap tables.

G Rubric Dimension Analysis

H Full Variance Decomposition Table

I Full English–Low-Resource Gap Table

J Ceiling Effect Confirmation: Gold vs. AfroBench

To test whether compressed gold-reference scores reflect a true ceiling effect, the study compares a subset of judges on MT and Summarization gold references and AfroBench real-system outputs. Every judge scores AfroBench outputs lower and with greater dispersion, indicating that the compression reflects the evaluation design rather than an inherent inability to discriminate quality.

J Ceiling Effect Confirmation: Gold vs. AfroBench

  • The comparison uses a subset of judges on MT and Summarization real-system outputs from AfroBench, matched across five strategies.The strategies are Zero-Shot, CoT, Role Prompting, Monolingual, and Monolingual-CoT.Source: J Ceiling Effect Confirmation: Gold… · p. 16
  • Every judge has lower mean scores and higher standard deviations on AfroBench outputs than on gold references.The comparison pools MT and Summarization and confirms the gold-reference ceiling effect.Source: J Ceiling Effect Confirmation: Gold… · p. 16
  • The gold-reference score compression reflects the evaluation design, not an inherent inability of judges to discriminate quality.Source: J Ceiling Effect Confirmation: Gold… · p. 16

K External Validity — SummEval and WMT MQM

External validation finds broadly consistent judge–human or judge–expert correlations across strategies and judges on two English datasets, complementing rather than replacing multilingual validation. Across tasks, the supplied figures also show resource- and judge-dependent patterns alongside ceiling effects on gold-reference evaluations.

K External Validity — SummEval and WMT MQM

  • Correlations are broadly consistent across strategies and judges on SummEval and WMT MQM, with neither dataset showing a catastrophic strategy failure.Both external datasets are English-only, complementing rather than replacing multilingual validation against SSA-MTE.Source: K External Validity — SummEval and WMT… · p. 16, Figure 11 · p. 23, Figure 12 · p. 23
  • MT high-resource panels show near-uniform scores, while low-resource cross-language ranges are 1.41–2.04 regardless of strategy.The figure attributes the high-resource pattern to a reference-based MT ceiling effect and the range pattern to a resource-level effect.Source: Figure 3 · p. 17

L Use of Generative AI

K External Validity — SummEval and WMT MQM

  • Summarization range is highest for Aya 32B under Monolingual across resource tiers, while CoT and Role show consistently low range.The figure also indicates that strategy effects on summarization scoring are model-dependent.Source: Figure 4 · p. 18

L Use of generative AI

  • QA panels show near-uniform scores and low ranges, consistent with a gold-reference ceiling effect.Source: Figure 5 · p. 19
  • Gemma 27B shows lower Math Zero-Shot scores with a visible CoT rescue effect, while Monolingual has higher low-resource range.The Math mid-resource panel is empty because AfriMGSM covers only low and low-mid languages.Source: Figure 6 · p. 20
  • Gemma 27B systematically disagrees with other judges on Summarization strategy rankings.The figure links this pattern to Gemma 27B’s divergent zero-shot scoring scale.Source: Figure 7 · p. 21

L Use of Generative AI

  • MT Terminology and Summarization Main Ideas are the most strategy- and resource-sensitive rubric dimensions, respectively.For MT, Terminology means are 3.93–4.59 for low-resource versus 4.97–5.00 for high-resource languages, while Adequacy and Fluency remain stable; Summarization Main Ideas falls to 2.99–3.07 under CoT and Prometheus versus 3.48–3.80 for Zero-Shot across tiers.Source: Figure 8 · p. 22, Figure 9 · p. 22, Figure 10 · p. 23

Related papers

Loading 2610.31172v1…