Source-linked AI summary
Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages
Isaac Chung, Linda Freienthal
TL;DR
The paper asks whether multilingual LLM evaluation can distinguish model differences from measurement instability. It holds generation conditions constant across Estonian, Finnish, and Hungarian, finding stable surface and semantic metrics but unstable pragmatic rankings, with implications for calibration before deployment.
Problem
Evaluation in morphologically rich, underrepresented languages lacks reliable evidence about whether automatic and model-based conversational scoring transfers across languages.
Method
The study generates semantically aligned synthetic customer-support dialogues with identical parameters across Estonian, Finnish, and Hungarian, then compares automatic metrics, LLM-judge rankings, and targeted human references.
Results
Surface and semantic metrics remain stable, but Coherence shows τ ≈0 with systematic rank inversions, while the pattern persists across judge models.
Takeaways & Limitations
Cross-lingual judge transfer is more dependable for surface assessment than discourse coherence, motivating language-specific calibration before deployment.
Takeaways & Limitations
The findings are limited to synthetic customer-support dialogues in three related Finno-Ugric languages and may not generalize to other models, domains, or distant languages.
Abstract
from arXiv · showhide
Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability. We investigate evaluation reliability by holding generation conditions constant while varying target language. Using synthetic customer-support dialogues generated with identical parameters across Estonian, Finnish, and Hungarian, we test whether automatic metrics and LLM-as-a-judge scoring produce stable model rankings across these morphologically rich, related Finno-Ugric languages. With a small set of Estonian native speaker annotations as a reference point, we find systematic ranking instabilities: surface-level metrics (lexical diversity, surface and semantic similarity) maintain cross-language stability, but pragmatic judgments (coherence, instruction-following) exhibit rank inversions and near-zero correlations. Because generation is controlled, these inconsistencies reflect how judge scoring behaves differently across languages rather than true model differences. This controlled design provides a diagnostic probe: evaluation methods that fail to maintain stability under identical generation conditions signal transfer failure before deployment. Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages, motivating language-specific calibration against targeted human baselines. We release our controlled generation protocol, synthetic data, and evaluation framework to enable replication across language families at https://github.com/isaac-chung/cross-lingual-stability-judges.
1 Introduction
The paper addresses unreliable evaluation in morphologically rich, underrepresented languages by isolating judge behavior from content variation through controlled multilingual generation. It tests whether automatic and LLM-judge assessments preserve model rankings when only the target language changes.
- Reliable evaluation is especially difficult in morphologically rich, underrepresented languages because human supervision is limited.
- Existing Finno-Ugric benchmarks extend language coverage but largely retain high-resource practices and poorly understood conversational scoring assumptions.
- The study generates dialogues with identical parameters in Estonian, Finnish, and Hungarian to probe whether rankings destabilize when only language varies.
- Prior multilingual judge studies report systematic cross-language inconsistency, while the sources of instability remain unclear.
- A two-stage design first checks surface comparability, then tests cross-language stability of LLM-as-a-judge pragmatic assessments.
2 Methods
The study constructs aligned synthetic customer-support dialogues, obtains a small Estonian human reference set, and evaluates generation and judge stability with complementary metrics.
- Three native Estonian speakers annotate 100 dialogues for coherence and fluency as a human reference point.Inter-annotator agreement is fair to moderate: κ = .385 for coherence and κ = .321 for fluency.
- Generation consistency is checked with lexical, repetition, and semantic metrics before judge-based ranking stability is tested.The design treats divergence after comparable surface properties as evidence of evaluation-transfer instability.
- The automatic judge scores Grammar, Readability, Coherence, Fluency, and Label Recovery Accuracy on 100 conversations per model and language.LRA assesses instruction-following and semantic consistency by recovering generation parameters from dialogue content.
- Per-language model rankings are compared with Kendall τ using bootstrap confidence intervals, while rank inversions are tested by permutation.The analysis recognizes that transferred evaluation reflects both generator output quality and judge scoring logic.
3 Results
Results separate stable semantic and surface comparisons from unstable discourse-level judgments. Under controlled generation, coherence and related pragmatic assessments produce cross-language ranking failures that are not explained by prompt language or one judge model.
- Automatic metrics show language-linked lexical variation, with Estonian generally more diverse and less repetitive than Finnish and Hungarian.These patterns are attributed to morphological complexity differences rather than generation-quality variance.
- Semantic similarity remains .89-.94 across all models and languages, despite systematic surface differences.
- LLM-as-a-judge scores remain relatively stable for Grammar and Readability but vary substantially for Coherence, Fluency, and LRA across languages and models.Estonian judge scores align imperfectly with human judgments, while Finnish and Hungarian expose greater instability.
- Surface metrics achieve τ ≥.70 with 1–3 rank inversions per pair, whereas Coherence reaches τ = −.06 for et–hu and τ = −.17 for fi–hu.The et–hu comparison has significant inversions (p = .02).
- English Coherence scores show ceiling effects near 2.98–3.00, preventing meaningful ranking comparisons with English.The analysis therefore focuses on Finno-Ugric pairs where score variance supports comparison.
- Native-language prompts produce nearly identical scores to English prompts, and six judge models differ by less than ∆< 0.02 across categories.These sensitivity and ablation results indicate that the instability is neither prompt-language-specific nor confined to one judge.
4 Discussion and Outlook
The paper frames controlled stability as a validity gate for multilingual judge deployment. Surface assessments transfer more reliably than discourse coherence, which requires language-specific calibration and targeted human checks.
- Grammar, readability, and fluency support cross-linguistic comparison with τ ≥0.70 across Finno-Ugric pairs, whereas coherence has τ ≈0.
- The proposed workflow verifies generation consistency, collects a small target-language expert sample, tests judge-human alignment, and calibrates weak correlations.The suggested expert sample is N ∼100.
Limitations
The study’s controlled synthetic setting limits how broadly its ranking-instability findings can be generalized, and surface metrics cannot exclude all discourse-level quality differences.
- Synthetic dialogues may be stylistically homogeneous and unlike real customer-support data, so validation on natural scenarios is still needed.
- Surface-level ranking stability makes judge transfer failure the more likely explanation for Coherence instability, but unmeasured discourse-quality differences remain possible.
- The analysis covers customer-support dialogues in Estonian, Finnish, and Hungarian, limiting transfer to other models, domains, languages, and discourse dimensions.
- Politeness conventions and language-specific grammatical patterns remain unexplored beyond the study’s focus on discourse coherence.
A Automatic Metrics
The study uses automatic metrics and controlled synthetic dialogue generation to compare lexical, repetition, semantic, and generation properties across languages and models.
- TTR and MATTR measure lexical diversity, while Self-BLEU detects formulaic repetition at conversation and speaker-response levels.TTR is unique words divided by total words; MATTR uses sliding 100-token windows, and Self-BLEU uses 4-gram BLEU with smoothing.
- Intra Model Conversation Similarity uses multilingual sentence-embedding cosine similarity to measure how different conversations are within each model.Lower scores indicate more template-like similarity; higher scores indicate greater conversation diversity.
- All languages use morphological lemmatization and language-specific stopword filtering for metric calculation.EstNLTK is used for Estonian, while Stanza is used for Finnish and Hungarian.
- Llama3.1-8B has lower lexical diversity than Mixtral-8x7B-Inst., while Command-R has the lowest agent-side Self-BLEU scores for Estonian and Finnish.Llama3.1-8B TTR is .42-.48 and MATTR .45-.49; Mixtral-8x7B-Inst. TTR and MATTR are .70-.80; Command-R agent-side Self-BLEU is .10-.11 for et/fi.
- Synthetic dialogues are generated under identical parameter distributions and generation models across languages to isolate evaluation behavior from content variation.Parameters cover industry, problem type, channel, agent experience, agent type, and conversation length.
- 10K conversations are generated per language for Estonian, Finnish, Hungarian, and English, yielding 40K dialogues overall.
C.2 Feedback and Examples
Human feedback on Estonian dialogues identifies fluency and coherence problems, while the evaluation setup provides instructions for judging linguistic and pragmatic dimensions.
- Three annotators’ average fluency score is 2.108±.696 on a 0–3 scale, with feedback describing rough, non-native-sounding expressions.
- The average logical-coherence reference score is .842±.367 on a binary scale.
- Reported coherence failures include inconsistent names or product amounts, illogical customer issues, and hallucinated words that reduce understandability.
- The LLM-as-a-judge prompt evaluates linguistic and pragmatic dimensions, including Label Recovery Accuracy as a diagnostic of instruction-following and semantic consistency.
D.2 English Results
The supplied passage identifies Table 8 as reporting LLM-as-a-judge results on the English dialogues.
- Table 8 reports LLM-as-a-judge results on the English dialogues.
D.3 LRA Full Results
The appendix reports label-recovery evaluation results for generated dialogues, including qualitative feedback on Estonian examples and comparative English evaluation results.
- D.3 LRA Full Results: Label Recovery Accuracy measures the judge’s ability to extract generation parameters from dialogue content across language-specific categories.Figure 3 covers Estonian, Finnish, Hungarian, and English; the supplied passage does not report category-level values.
- D.3 LRA Full Results: Most labeler feedback on Estonian examples flags uncommon expressions and overly formal tone, which lowered fluency scores.
- D.3 LRA Full Results: Table 8 reports LLM-as-a-judge evaluation of generated English dialogues and bolds the best score for each metric.
E Cross-language ranking stability
Cross-language ranking stability differs by evaluation dimension: surface-oriented metrics are comparatively stable, whereas pragmatic judgments and LRA show inversions, especially in comparisons involving Estonian.
- E Cross-language ranking stability: For n = 6 models, 0 inversions means perfect ranking preservation and 15 means a perfect reversal.
- E Cross-language ranking stability: τ typically ≥0.5 for Grammar, Readability, and Fluency, with non-significant inversion counts across languages.
- E Cross-language ranking stability: Coherence shows attenuated or negative agreement for Estonian-involving pairs, while Finnish–Hungarian agreement remains stable.
- E Cross-language ranking stability: LRA has significant inversions for et–en (7, p = 0.02), et–fi (9, p = 0.01), et–hu (6, p = 0.03), fi–hu (7, p = 0.02), and hu–en (7, p = 0.02).
- E Cross-language ranking stability: Coherence has inversions for et–en (5, p = 0.05), et–fi (5, p = 0.05), and et–hu (5, p = 0.04).
- E Cross-language ranking stability: Native-language and English meta-prompts produce nearly identical scores, with maximum variance below 0.05 for every model and dimension.
G Appendix: Judge Model Ablation Study
The judge-model ablation finds highly similar label-recovery behavior across six judges, with moderate inter-judge agreement and a shared difficulty hierarchy across categories.
- G Appendix: Judge Model Ablation Study: Six judges evaluated the same six models on Finnish dialogue label recovery using a common judge prompt.
- G Appendix: Judge Model Ablation Study: Δ< 0.02 across categories, with all judges showing near-identical performance patterns.
- G Appendix: Judge Model Ablation Study: Mean Spearman correlation was 0.66 across judge comparisons, indicating moderate-to-substantial agreement alongside meaningful judgment variance.
- G Appendix: Judge Model Ablation Study: All judges struggled with Industry and Problem while performing better on Channel and Agent classifications.
- G Appendix: Judge Model Ablation Study: GPT-5.1-high matched default reasoning, and Qwen3-32B, Llama-4-Maverick, and GPToss-120B were comparable to proprietary models.
- G Appendix: Judge Model Ablation Study: These ablation results support using GPT-5-mini as the judge in the main experiments.