A Systematic Evaluation of Cross-Lingual Consistency Enhancement Methods in Multilingual Language Models

Motivation and approach

• Existing methods are difficult to compare directly because they have been evaluated with different models, datasets, language sets, prompts, and protocols.The paper addresses this gap with shared splits, prompts, candidate spaces, and metrics across multilingual models and benchmarks.Source: §1 · pp. 1–2

Motivation

• Consistency enhancement must preserve legitimate cultural variation because culturally or geographically grounded questions can warrant different answers across languages.An overly aggressive objective may suppress those distinctions or shift non-English behavior toward an English-dominant answer distribution.Source: §1 · pp. 1–2

Main findings

• Post-training methods are substantially more reliable than inference-time intervention, with direct distribution alignment providing the most stable gains across model–dataset setups.Culturally diverse knowledge is largely preserved in closed-form evaluation, although open-ended generation shows small accuracy drops for non-English responses.Source: §1 · p. 2

Cross-domain transfer

• Cross-domain CLC improvements generalize weakly unless source and target tasks share similar answer formats.The paper cautions that in-domain gains do not establish broader multilingual knowledge consistency.Source: §1 · pp. 1–2

Culturally diverse knowledge

• CLC enhancement does not systematically degrade culturally diverse QA in controlled closed-form evaluation, but open-ended generation shows occasional accuracy reductions, particularly for non-English responses.The introduction also reports clear separability between culture-independent and culturally diverse prompts.Source: §1 · p. 2

Cross-lingual knowledge inconsistency

• Multilingual models can rank candidate answers differently across languages, and RANKC compares full candidate rankings rather than only top-1 agreement.This captures cases where top answers agree but plausible alternatives rank differently, as well as cases where top answers disagree despite similar rankings.Source: §2 · p. 2

Consistency enhancement methods

• Prior CLC methods span inference-time intervention, English-pivot preference alignment, multilingual self-alignment, and direct consistency optimization, but differing evaluations impede direct comparison.The paper addresses this gap through a unified evaluation of representative methods.Source: §2 · p. 2

Culturally diverse knowledge

• Culturally or geographically grounded questions may legitimately require different answers, yet whether CLC enhancement harms culturally diverse knowledge remains unclear.GEOMLAMA and BLEND provide benchmarks for examining this underexplored question.Source: §2 · p. 2

3.1 Benchmarks and Framework

• Five benchmarks cover factual probing, academic and professional multiple-choice knowledge, commonsense reasoning, geographically grounded questions, and open-ended cultural QA.Parallel question versions stay in the same split; MMMLU also serves as a cross-domain target for BMLAMA- or XCSQA-trained models.Source: §3.1 · p. 3

3.2 Evaluated CLC Method Families

• The comparison includes inference-time representation intervention, English-pivot preference alignment, multilingual self-alignment, and direct consistency optimization.The methods are compared under a common protocol rather than proposed as a new training objective.Source: §3.2 · pp. 3–4

3.3 Training Setups

• Bilingual training pairs English with one target language, while joint-language training uses all available language versions of each question.All versions of an item remain in the same split to prevent language-version leakage; intervention methods fit weights without updating model parameters.Source: §3.3 · p. 4

3.4 Evaluation Setups

• Closed-form QA uses likelihood-ranked candidates for accuracy and RANKC for consistency, while BLEND uses normalized exact matching and semantic similarity as a consistency proxy.The base models provide accuracy and consistency baselines; Table 3 reports results and method-induced changes averaged over English–target language pairs.Source: §3.4 · p. 4, Table 3 · p. 5

4.1 In-Domain Evaluation

• Post-training methods were more reliable than INCLINE, whose effects were negligible on MMMLU and XCSQA and negative on BMLAMA across all three models.For Aya on BMLAMA, RANKC and accuracy fell by 15.67 and 15.99 points, respectively.Source: §4.1 · p. 5
• DCO improved RANKC in every model–dataset setting under both bilingual English-pivot and joint-language training, providing the most stable in-domain gains.The consistency gains were generally accompanied by higher task accuracy.Source: §4.1 · p. 5
• EN-Align was effective on BMLAMA but brittle on multiple-choice tasks, where bilingual Qwen2.5 training reduced both RANKC and accuracy on MMMLU and XCSQA.The passages suggest English-centered supervision works better with semantically meaningful answer completions than abstract option labels.Source: §4.1 · pp. 5–6

4.2 Cross-Domain Generalization

• DCO raised RANKC by +6.20 for Qwen2.5, +7.15 for Gemma3, and +3.96 for Aya after XCSQA→MMMLU, whereas BMLAMA→MMMLU yielded little useful transfer.XCSQA and MMMLU share multiple-choice option-letter completions, while transfer remained weaker than in-domain.Source: Figure 1 · p. 6, §4.2 · pp. 6–7

4.1 In-Domain Evaluation

• CALM produced smaller, mostly positive gains, but majority-vote completions were often only around 50% correct, weakening preference supervision.Noisy preferences can limit improvements in both RANKC and answer accuracy.Source: §4.1 · p. 6, Table 4 · p. 6

4.2 Cross-Domain Generalization

• CLC gains learned on one dataset did not automatically generalize across knowledge domains, even when source and target tasks shared response formats.The discussion emphasizes matching alignment data to intended use and using diverse training coverage when robust transfer is needed.Source: §4.2 · p. 7

5.1 Evaluation on GEOMLAMA

• On full GEOMLAMA, DCO usually raised RANKC modestly and EN-Align stayed near neutral, while neither post-training method consistently reduced accuracy.DCO’s RANKC changes ranged from +0.70 to +3.98 in most settings, with one decrease of −0.69; EN-Align’s exception was +4.77 for Qwen2.5 after BMLAMA training.Source: §5.1 · pp. 7–8
• For culturally diverse GEOMLAMA questions, BMLAMA-post-trained Qwen2.5 showed no systematic accuracy reduction; the largest decrease was 1.50 from a baseline of 56.00.EN-Align improved Persian accuracy by 4.17 for the English–Persian pair.Source: Table 5 · p. 8, §5.1 · p. 8
• INCLINE consistently reduced GEOMLAMA accuracy across source datasets and models, with BMLAMA-based drops of −14.62, −11.15, and −13.22 for Qwen2.5, Gemma3, and Aya.The authors attribute these drops more plausibly to brittle intervention than to over-alignment.Source: §5.1 · p. 8

5.2 Separability of Culture-Independent and Culture-Diverse Queries

• Culture-independent and culture-diverse queries were linearly separable using both lexical features and evaluated models’ hidden states, offering a plausible explanation for weaker transfer to culturally diverse questions.The subsets came from the same dataset and covered the same broad categories; the reported probe results use concatenated features and representative views.Source: §5.2 · pp. 8–9, Figure 3 · p. 9

5.3 Extended Evaluation on BLEND

• On BLEND, DCO improved embedding cosine similarity by 0.23–5.54 across datasets and models, while BERTScore increased in all but one setting.For Aya post-trained on BMLAMA, BERTScore decreased by 0.13 despite a 0.23 increase in cosine similarity.Source: Table 7 · p. 9, §5.3 · p. 9
• BLEND accuracy was mixed, with declines more frequent in non-English languages and the largest drop, 4.28, for Gemma3 post-trained on XCSQA.The occasional reductions, especially for culturally grounded non-English responses, suggest a trade-off less visible in closed-form QA.Source: §5.3 · pp. 9–10, Table 8 · p. 10

• Post-training methods are more reliable than inference-time interventions, with direct distribution alignment the most robust overall and cross-domain generalization limited.Strong in-domain gains do not necessarily translate into broader multilingual consistency.Source: §6 · p. 10
• On GEOMLAMA, post-training does not systematically degrade culturally diverse accuracy, while BLEND shows occasional accuracy reductions, especially for non-English responses.The authors suggest prompt separability as a plausible explanation for limited interference; accuracy risks are noted for contexts less represented in English-centric data.Source: §6 · p. 10
• The study calls for evaluating CLC enhancement beyond in-domain consistency, including cross-domain robustness and culturally grounded settings.Future work should improve transfer across tasks while preserving warranted culture-specific behavior.Source: §6 · p. 10

Limitations

• On BLEND, semantic similarity is a practical, scalable proxy for response-level alignment but may not fully capture culturally grounded differences across languages.Source: Limitations · p. 10
• The cultural-knowledge analysis uses GEOMLAMA and BLEND; broader benchmarks and larger-scale human annotation would help assess real-world multilingual systems.Source: Limitations · p. 10
• Representation-level separation does not establish explicit cultural encoding, since input-level differences may contribute and require controlled interventions or causal analyses to disentangle.Potential confounders include entity distributions, translation styles, prompt templates, and language-specific lexical cues.Source: Limitations · p. 10

A.1 Statistic of Evaluation Datasets

• The evaluation datasets span 32 languages, as detailed in Table 9.Source: Table 9 · p. 15
• Table 10 provides representative evaluation examples in English, with parallel versions used for culture-independent or culture-diverse evaluation.Source: Table 10 · p. 15, §A.1 · p. 15

A.2 Definition of RankC: Ranking-based Crosslingual Consistency

• RankC compares candidate rankings from parallel prompts via weighted top-j overlap, emphasizing agreement among highly ranked candidates.Source: §A.2 · p. 15

B.1 Full Results of CLC Enhancement Methods

• Table 11 reports joint-language post-training results on BMLAMA, MMMLU, and XCSQA.Source: §B.1 · p. 15, Table 11 · p. 16
• The table labels consistency as C and accuracy as A.Source: Table 11 · p. 16
• Baseline rows report absolute scores, post-training rows report changes relative to their baselines, and EN-A abbreviates EN-Align.Source: Table 11 · p. 16

B.2 Full Results of Cross-domain Generalization

• Table 12 reports cross-domain transfer results on MMMLU after post-training on BMLAMA or XCSQA.The caption identifies the evaluation metric and defines C as consistency and A as accuracy.Source: Table 12 · p. 17
• Baseline rows give absolute scores, while method rows give absolute changes relative to the corresponding baseline.Source: Table 12 · p. 17
• EN-A abbreviates EN-Align.Source: Table 12 · p. 17

B.3 Additional Linear-Probing Results

• All combination strategies show a consistent separability pattern between culture-independent and culture-diverse query pairs.Source: Figure 4 · p. 17
• The pattern appears across TF-IDF, Word2Vec, and middle- and final-layer hidden-state features.Source: Figure 4 · p. 17
• Figure 4 evaluates four combination strategies on English–Hindi query pairs using Qwen2.5.Source: Figure 4 · p. 17