PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

arXiv:2608.22432v1cs.CLcs.AI

TL;DR

Multilingual LLM judges can reverse evaluator-backbone rankings across prompt languages, motivating calibration without expensive human annotations. The paper uses double-centering to estimate and remove the language-backbone interaction, with formal guarantees including unbiasedness under task-language effects. CBC improves held-out cross-task consistency and external agreement with human gold preferences, while its strongest claims remain bounded by observed-language coverage and two-way-model assumptions.

  • Problem

    Multilingual judge rankings vary across prompt languages, with 7 of 15 backbone pairs showing significant rank reversal and no universal winner across the eight-language benchmark.

    Source: §1 · p. 2

  • Method

    CBC double-centers shared multi-backbone score matrices to estimate and subtract the language-backbone interaction without human labels.

    Source: §4 · p. 5

  • Results

    CBC raises held-out cross-task rank consistency τ from 0.650 to 0.902 and raises M-RewardBench agreement with public human gold preferences from 68.7% to 76.6%.

    Source: §8 · p. 13

  • Takeaways & Limitations

    The paper supports CBC as a label-free post-hoc calibrator for multilingual LLM judges when a shared evaluator ranking across languages is desired.

    Source: §4 · p. 5, §8 · p. 13

    CBC does not solve zero-shot transfer to unseen languages and cannot separate genuine three-way task-language-backbone effects from the estimated interaction.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmark, the top-ranked backbone alternates across English, Arabic, Chinese, Hindi, Japanese, Spanish, Turkish, and Swahili, and 7 of 15 backbone pairs show statistically significant pairwise rank reversal. We treat this as a measurement problem. The multilingual judge score decomposes additively into task difficulty, backbone skill, and a language-backbone interaction term, the last of which is recoverable without human labels by double-centering the cell-mean score matrix. We make this estimator (\textbf{Consensus-Based Calibration}, CBC) explicit, give an $O(1/\sqrt{n})$ finite-sample concentration bound with variance constant $(1-\tfrac{1}{m})(1-\tfrac{1}{k})$, and show that it is unbiased even when task-language interactions are present. Across 7{,}920 judge runs (6 backbones, 8 languages, 55 tasks, 3 frameworks), CBC raises held-out cross-task rank consistency $τ$ from 0.650 to 0.902 and agrees with the held-out additive-model oracle in 100\% of per-language decisions versus 68.5\% raw; these are consistency diagnostics, not human-grounded correctness measures. On a separately collected M-RewardBench panel (7 languages, 1{,}500 items per language, 10{,}500 language-item instances, 5 evaluators), panel agreement with the public human gold preferences rises from 68.7\% to 76.6\% (gain 7.9 percentage points, 95\% CI $[6.0, 9.9]$), our strongest external evidence of downstream usefulness. The estimator is the standard two-way ANOVA interaction-recovery operation under sum-to-zero contrasts; our contribution is its application as a label-free post-hoc calibrator for multilingual LLM judges, an explicit finite-sample concentration bound, and an unbiasedness result that holds even under task-language misspecification.

1 Introduction

Multilingual judge rankings can reverse across prompt languages, creating an unresolved calibration problem without expensive human annotations. The paper applies double-centering to recover the language-backbone interaction and evaluates CBC as a label-free calibrator.

Research gap

  • Existing approaches use human judgments, ensembles, or pairwise models, but none directly targets the additive language-backbone interaction in multi-evaluator multilingual benchmarks.Hada et al. calibrate against 20,000 human judgments, while Fu and Liu propose an ensemble.Source: §1 · p. 2

Approach

  • CBC double-centers the multi-evaluator score matrix to estimate the language-backbone interaction under a standard two-way additive layout.The model includes task difficulty, backbone skill, language-backbone interaction, task-language effects, and noise.Source: §1 · p. 2

Empirical motivation

  • 7 of 15 backbone pairs exhibit statistically significant pairwise rank reversal, and the top-ranked backbone alternates across all 8 languages.The benchmark contains 7,920 judge runs across 6 backbones, 55 tasks, and 3 frameworks.Source: §1 · p. 2

Guarantees

  • The estimator requires no human labels and is accompanied by an O(1/√n) concentration bound and an unbiasedness result under task-language interactions.The explicit variance constant is (1−1/m)(1−1/k), and the unbiasedness result concerns task-language effects that do not depend on the backbone.Source: §1 · p. 2

Scope

  • CBC is intended for language-invariant evaluator rankings, while language-specific specialization and shared language-wide shifts remain outside its correction target.Shared language-level shifts require an external anchor, and removing meaningful specialization can be inappropriate.Source: §1 · p. 3

2 Related Work

The paper builds on multilingual judge evaluation, label-free aggregation, two-way ANOVA, and agentic code-evaluation research. Its formal novelty is applying standard interaction recovery to label-free multilingual judge calibration and extending the guarantees beyond correctly specified two-way models.

Multilingual LLM-judge evaluation

  • Prior multilingual judge studies report score inflation, weak cross-lingual consistency, and linguistic or cultural bias in benchmark construction.Fu and Liu report Fleiss’ κ≈0.3 across 25 languages.Source: §2 · p. 3

Label-free calibration and aggregation

  • Existing label-free methods estimate judge reliability, model pairwise discrimination, or address position bias, but do not directly recover the additive language-backbone interaction.The related work includes Bradley–Terry–Luce extensions, Dawid–Skene methods, learning-from-crowds, and Item Response Theory.Source: §2 · p. 3

Two-way layouts

  • Double-centering under sum-to-zero contrasts is the standard two-way ANOVA interaction-recovery operation.Classical references assume correct model specification for their standard inference procedures.Source: §2 · p. 3

Novelty

  • The paper adds a finite-sample uniform bound and unbiasedness under an unmodeled task-language interaction to that standard operation.These properties are identified as non-standard relative to the cited classical ANOVA treatment.Source: §2 · p. 3

3 Score Model and Identifiability

The score model separates task difficulty, backbone skill, language-backbone interaction, task-language effects, and noise. Under balanced shared-task data and sum-to-zero normalization, the target interaction is identifiable from population cell means, but genuine three-way effects are not separable.

3.1 Setup and Notation

  • The judge score S(t, ℓ, b) is modeled as task difficulty plus backbone skill, language-backbone interaction, task-language effect, and mean-zero noise.Framework effects cancel when they do not depend jointly on language and backbone.Source: §3.1 · p. 4
  • CBC targets β(ℓ, b) when deployment requires a shared evaluator ranking across languages, not when language-specific evaluator capability is itself the objective.In the latter setting, CBC is best treated as a sensitivity analysis.Source: §3.1 · p. 4

3.2 Rank Reversal

  • Pairwise rank reversal occurs when two per-language backbone gaps have an opposite-sign product at least as strong as −δ.The criterion rules out universal dominance between the two backbones.Source: §3.2 · p. 4
  • 7 of 15 backbone pairs satisfy the reversal condition, and the empirical top-ranked backbone alternates across the eight languages.The observed data therefore contain no universal winner.Source: §3.2 · p. 4

3.3 Identifiability Without Human Labels

  • Under a complete balanced panel, mean-zero noise, and sum-to-zero normalization, β(ℓ, b) is uniquely identifiable from population language-backbone cell means.The absolute task and backbone main-effect levels are not separately identified, but that indeterminacy does not affect β.Source: §3.3 · p. 5
  • A genuine task-language-backbone interaction lies outside the two-way model and cannot be separated from β by CBC.The estimated interaction can therefore mix evaluation variation with backbone-specific task adaptation.Source: §3.3 · p. 5

4 Consensus-Based Calibration

CBC estimates and subtracts the language-backbone interaction using double-centering, requiring no human annotations. It has explicit concentration and consistency guarantees, while remaining unbiased under task-language interactions but vulnerable to genuine three-way misspecification.

Estimator

  • CBC estimates β̂(ℓ, b) by subtracting the backbone and language marginal means from each cell mean and adding the grand mean.This is the constructive double-centering estimator under sum-to-zero normalization.Source: §4.1 · p. 6
  • CBC calibrates scores as Ŝ(t, ℓ, b) = S(t, ℓ, b) − β̂(ℓ, b) and requires zero human annotations.It uses multi-backbone evaluation data already collected by labs.Source: §4.1 · p. 6

Finite-sample guarantees

  • For m=6, k=8, and n=55, the simultaneous high-probability radius is about 10.0 points at ε=0.05.This numerical radius is conditional on independent homoskedastic Gaussian noise and is not distribution-free.Source: §4.2 · p. 6

Convergence and consistency

  • Without Gaussian tails, CBC remains consistent under independent tasks and finite first absolute moments as n→∞.The O(1/√n) rate is the standard parametric rate for the sample-mean construction.Source: §4.2 · p. 6

Model misspecification

  • CBC is unbiased for β(ℓ, b) even when task-language interactions are present because those effects cancel in double-centering.This robustness holds because the task-language term does not depend on the backbone.Source: §4.3 · p. 6
  • The robustness result does not cover genuine three-way task-language-backbone interactions, and simplified-model error bars may then fail to apply unchanged.Backbone-specific task adaptation can be absorbed into the estimated language-backbone interaction.Source: §4.3 · p. 7

5 Experiments

Across internal and external evaluations, multilingual judge rankings vary by language, while CBC improves held-out rank consistency and model-based deployment decisions. External M-RewardBench validation also shows higher agreement with public human preferences, with important scope and interpretation limits.

Empirical characterization

  • 7 of 15 backbone pairs exhibit statistically significant pairwise rank reversal, and the empirical top-ranked backbone alternates across all 8 languages.The reversals remain significant after Benjamini–Hochberg correction at FDR = 0.05.Source: §5.2 · pp. 7–8

Model fit and limitations

  • The two-way model explains substantial variance, with R2=0.691 for the simplified fit versus 0.705 for the full model including task-language interactions.Task-language effects contribute approximately ΔR2≈0.015.Source: §5.2 · p. 8

Calibration results

  • CBC raises held-out cross-task rank consistency τ from 0.650 under Raw to 0.902 on the eight-language benchmark.The evaluation uses task-level bootstrap resampling and out-of-bag held-out tasks; the full-fit τ=1.000 alignment is only an in-sample sanity check.Source: §5.1 · p. 7, §5.3 · p. 8, Table 2 · p. 9
  • CBC significantly outperforms ComBat-EB, with τ_CBC−τ_ComBat positive in all 1,000 paired-bootstrap replicates (p<0.002).The comparator and controls are evaluated as continuous post-hoc methods on the same observed-language benchmark.Source: §5.3 · p. 9

Decision-level backbone selection

  • CBC agrees with the held-out additive-model oracle winner in 100% of language-decisions, compared with 68.5% for Raw.The oracle is an unattainable model-based reference, not a human-grounded correctness measure.Source: §5.4 · pp. 9–10, Table 3 · p. 10

External validation on M-RewardBench

  • 76.6% of CBC-aligned panel decisions agree with M-RewardBench human gold preferences, versus 68.7% for Raw, a gain of 7.9 points.The 95% CI for the gain is [6.0, 9.9]; this is supportive aggregate panel-level evidence rather than a per-language diagnosis.Source: §5.5 · p. 11

Model fit and limitations

  • These results do not establish universality across domains, languages, or evaluator families.The internal and external panels use different tasks, rubrics, evaluator routes, and overlapping language sets.Source: §5.5 · p. 11

6 Analysis

The paper corroborates language-conditioned evaluator differences with conditional mutual information and identifies which backbone pairs drive the observed reversals.

Methodological interpretation

  • CBC estimates the language-backbone interaction by double-centering the multi-evaluator score matrix without requiring human labels.The targeted interaction is the non-redundant signal represented by the conditional mutual information analysis.Source: §6 · p. 13

Empirical analysis

  • 0.178 nats of conditional mutual information separates Language from Score given Backbone and Task, with permutation interval [0.169, 0.187] and one-sided p < 0.002.The estimate averages KSG-style nearest-neighbor estimates over 330 task–backbone slices and uses 500 shuffles per slice.Source: §6 · p. 13
  • The seven verified pairwise reversals are exactly the cross-family pairs among GPT-4o, GPT-5.4, Sonnet, and DeepSeek.Qwen is Pareto-dominated in this panel, while Gemini is top- or second-ranked in every language and produces no sign change.Source: §6 · p. 13

7 Discussion

The paper frames rank reversal as a practical obstacle to finding a language-neutral backbone and presents CBC as a label-free correction when language-invariant ranking is desired.

Theory meets practice

  • Language-conditioned pairwise reversals make searching for a language-neutral backbone unreliable.The paper connects this observation to the need to estimate the language-backbone interaction from existing evaluator scores.Source: §7 · p. 13
  • CBC removes the language-backbone interaction using data already collected from multiple evaluator backbones scored on shared multilingual items.Its scope is continuous pointwise scores and the two-way language×backbone structure, unlike categorical-label reliability models such as Dawid–Skene.Source: §7 · p. 13

Cost and applicability

  • CBC requires zero human annotations, compared with 20,000 native-speaker annotations used by Hada et al.Reported compute costs are $54.93 for the three-language extension and $195.3 for the M-RewardBench panel.Source: §7 · p. 13
  • The same two-way decomposition and calibrator apply to settings with continuous multilingual judge scores beyond agentic code evaluation.The passage names RLHF reward modeling and safety classification as examples.Source: §7 · p. 13

8 Conclusion

The paper concludes that CBC can recover and remove observed language-backbone interactions, while its claims remain bounded by model misspecification, observed-language coverage, benchmark design, and uncertainty.

Conclusion

  • CBC raises held-out cross-task rank consistency τ from 0.650 to 0.902 and agrees with the held-out additive-model oracle in 100% versus 68.5% raw.These are consistency and model-reference diagnostics rather than human-grounded correctness measures.Source: §8 · p. 13
  • Agreement with public M-RewardBench human gold preferences rises from 68.7% to 76.6%, a gain of 7.9 percentage points with 95% CI [6.0, 9.9].The paper describes this as supportive aggregate evidence and its strongest external evidence of downstream usefulness.Source: §8 · p. 13
  • CBC is the standard two-way ANOVA interaction-recovery operation applied as a label-free multilingual LLM-judge calibrator.The paper’s additional contributions are an explicit finite-sample concentration bound and unbiasedness under task-language misspecification.Source: §8 · p. 14

Limitations

  • Genuine three-way task-language-backbone effects can make the estimated residual mix stable evaluator interaction with task-conditional performance heterogeneity.In that regime, simplified-model error bars need not carry over unchanged, so the estimate is interpreted at benchmark level rather than as universal model-family behavior.Source: Limitations · p. 14
  • CBC’s strongest results concern observed languages; LOLO agreement improves only from 0.649 to 0.740 under heuristic extrapolation.The paper does not claim that CBC solves zero-shot transfer to unseen languages.Source: Limitations · p. 14
  • The closed-form estimator and guarantees assume a complete, balanced panel with every shared item scored in every language-backbone cell.Naive double-centering need not identify the interaction with partial or unbalanced coverage.Source: Limitations · p. 14
  • CBC does not correct a language effect shared uniformly across backbones; addressing such a shift requires an external human or trusted-reference anchor.The small human-anchor experiment is downstream validation, not part of the CBC estimator.Source: Limitations · p. 14
  • The internal benchmark contains 55 DevAI tasks from one agentic code-evaluation family, while the external panel focuses on one preference-style benchmark family.Together the panels broaden evidence but do not establish universality across domains, languages, or evaluator families.Source: Limitations · p. 15

A Proof Details and Identifiability Scope

Double-centering exactly recovers the centered language-backbone interaction under the two-way model, while identifiability is limited to observed cells and excludes separable three-way effects. CBC remains strong with small backbone subsets, though larger subsets reduce variability.

Identifiability

  • Double-centering exactly recovers the centered interaction matrix and guarantees uniqueness under the stated zero-sum constraints.Source: §A.1 · p. 20

Identifiability scope

  • No human gold labels are needed for observed language-backbone cells, but unseen cells and three-way task-language-backbone effects are not identified.Source: §A.2 · p. 20

Concentration

  • CBC’s estimation error is a linear combination of centered cell-average noise terms, enabling the Gaussian concentration proof.Source: §A.3 · pp. 20–21

Consistency

  • The consistency proof uses task-level independence and finite first moments so cell means converge in probability, with linear transformations preserving convergence.Source: §A.4 · p. 21

Backbone ablation

  • With m=2 backbones, mean rank consistency is 0.901, while standard deviation is 0.206; larger subsets mainly reduce variability.Source: §B.1 · p. 22

B.2 Ablation: Number of Tasks (𝑛)

The task-count ablation shows that CBC improves and stabilizes as more tasks are available, consistent with its O(1/√n) convergence behavior. At n=55, estimation error is substantially lower than at n=10.

Task-count ablation

  • CBC’s convergence bound scales as O(1/√n), motivating empirical evaluation across task-panel sizes from 10 to 55.Source: §B.2 · p. 22
  • 2.01 at n=10 falls to 0.67 at n=55 for mean absolute interaction-estimation error, consistent with the convergence story.Source: §B.2 · p. 22

Practical power check

  • At about 52 tasks, the simultaneous high-probability radius falls below half the largest observed interaction magnitude; by n=30, mean pairwise τ is 0.853.Source: §B.2 · p. 22

Requirement-type decomposition

  • CBC improves rank stability for operational requirements from τ=0.770 to τ=0.789 and semantic requirements from τ=0.737 to τ=0.840.Source: §B.3 · p. 23

C External Validation Details

The external validation constructs CBC-ready evaluator tables from multilingual preference margins and compares CBC with a judge-aware Bradley–Terry–Luce baseline. CBC’s external diagnostic is reported alongside panel-construction and implementation boundaries.

Collection pipeline

Comparator

Diagnostics

  • An in-sample full-fit CBC diagnostic reaches τ=1.000, but the paper treats it as a sanity check rather than an independent generalization estimate.Source: C External Validation Details · p. 24

Panel limitations

  • The completed validation panel excludes Qwen3 because its provider run produced incomplete chosen/rejected outputs and did not reach a clean terminal state.Source: C External Validation Details · p. 24

Variant analysis

  • The weighted CBC variant matches uniform CBC at τ=0.902 on the observed-language bootstrap/OOB metric and is omitted as a separate main-paper method.Source: §D.2 · p. 25

D.3 Appendix Note on Dawid–Skene EM

The Dawid–Skene EM ablation treats language-backbone pairs as annotators after binarizing outcomes, but its bootstrap summary is highly unstable for this continuous score-matrix setting.

Method

  • Dawid–Skene EM binarizes requirement outcomes and treats each language-backbone pair as an annotator.Source: §D.3 · p. 26

Result

  • τ=0.446 with 95% CI [−0.042, 1.000] indicates a highly unstable bootstrap summary spanning nearly the full possible outcome range.Source: §D.3 · p. 26

Interpretation

  • The paper does not treat Dawid–Skene EM as a useful main-table baseline for this continuous score-matrix benchmark.Source: §D.3 · p. 26
Loading 2608.22432v1…