Source-linked AI summary
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
TL;DR
Fair crosslingual evaluation remains difficult because existing downstream tasks and intrinsic metrics may not support meaningful comparisons. The paper tests these approaches with controlled monolingual models and broader monolingual and multilingual models. It finds that normalized metrics can be biased by representation choices, while sentence-level NLL over semantically equivalent sequences yields more meaningful comparisons.
Problem
Crosslingual evaluation lacks sufficient empirical evidence showing whether commonly used tasks and metrics produce meaningful, fair conclusions across languages.
Method
The paper systematically compares six intrinsic metrics using controlled monolingual models trained on parallel data, then extends the analysis to larger monolingual and multilingual models.
Results
Sentence-level NLL satisfies the paper’s fairness criteria, whereas normalized metrics introduce biases associated with tokenization, encoding, and orthographic differences.
Takeaways & Limitations
Meaning-based sentence-level NLL provides a more consistent basis for crosslingual comparison than metrics normalized by surface-level units.
Takeaways & Limitations
The framework relies on high-quality parallel sentences, which limits applicability because many languages lack reliable parallel data.
Abstract
from arXiv · showhide
Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs. We further discuss challenges in achieving comparable downstream evaluation across languages. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences. In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.
1 Introduction
Crosslingual evaluation is difficult because metrics can produce different conclusions across typologically diverse languages. The paper proposes criteria for fair metrics and finds sentence-level NLL over semantically equivalent content most suitable for comparison.
- Different metrics can yield substantially different conclusions about crosslingual model performance.
- Downstream comparisons remain difficult because conclusions are task-dependent and language-specific benchmarks are unavailable for many languages.
- A fair metric should avoid tokenization, encoding, and orthographic artifacts, remain stable across equivalent forms, and generalize across model types and scales.
- Five out of six intrinsic metrics violate at least one fairness criterion, whereas sentence-level NLL satisfies all three.
2 Background & Related Work
Prior work uses several probability-based metrics for comparing equivalent content across languages, including raw NLL and normalization by tokens, bytes, or characters. These normalizations can reflect representational differences rather than language-modeling quality.
- Tokenization: Different tokenization strategies produce different segment counts for equivalent content, affecting training dynamics and evaluation metrics.
- Orthography and Encoding: Different scripts encode equivalent content with different byte and character counts, creating byte and length premiums that can distort normalized metrics.
- Intrinsic evaluations compare model probabilities on held-out parallel texts, while downstream evaluations assess diverse linguistic competencies but are harder to compare fairly.
- Researchers normalize NLL by language-specific characters or bytes, aggregate it without normalization, or use unit-normalized metrics such as perplexity, BPB, and BPC.
3 Existing Intrinsic Metrics & Their Potential Confounds
Probability-based metrics normalize sequence likelihood over tokens, bytes, or characters, but these units vary across tokenizers and writing systems. Sentence-level NLL instead sums surprisal over semantically equivalent content without surface-unit normalization, making it the most meaningful basis among the metrics considered.
- Byte- and character-normalized metrics: BPB removes tokenizer dependence within a language but can favor scripts requiring more UTF-8 bytes when comparing languages.The same semantic content may require different byte counts across scripts, changing the denominator without changing assigned probabilities.
- Byte- and character-normalized metrics: BPC mitigates tokenization bias but retains orthographic bias because languages can express equivalent content with different character counts.The paper gives Chinese translations as an example of shorter character sequences than English, producing a length premium under character normalization.
- Rank-based metrics: MRR is probability-agnostic, but larger vocabularies raise its possible-rank range and tend to produce lower scores regardless of actual model performance.Therefore, MRR requires constant vocabulary size for fair comparison.
- Token-normalized metrics: Perplexity and Token NLL normalize sequence likelihood by token count, so tokenization differences can change crosslingual rankings.Perplexity is the average predictive cost per token; Token NLL shares perplexity’s tokenization sensitivity.
- Sentence-level NLL: Sent-NLL sums token surprisals over any parallel-text unit conveying equivalent meaning, without normalizing by token count.Because it measures total surprisal for semantically equivalent content, segmentation does not directly affect the sequence-level score.
- Summary: Only Sent-NLL over semantically equivalent content is judged suitable for both cross-model and cross-lingual comparison among the reviewed metrics.BPB and BPC are suitable for cross-tokenizer comparison within the same language, while PPL and Token-NLL are not suitable across languages or tokenizers.
4 Experiment 1: All metrics except for Sent-NLL are biased in different ways
The experiment isolates metric behavior with monolingual models trained on parallel data under varied vocabulary sizes and evaluates six metrics across parallel datasets. Crosslingual rankings expose distinct normalization biases, while Sent-NLL alone avoids significant correlation with byte, character, and token sequence length.
- Experiment Setup: 50 monolingual GPT2-small models span 10 languages and five vocabulary sizes, isolating language and tokenizer effects in controlled comparisons.The models are trained on each language’s portion of a parallel corpus, yielding 50 models in total.
- Evaluation: Six intrinsic metrics are evaluated on FLORES-200, an in-domain 40k-sentence test split, and 1,000 PUD sentences.GPT2-medium models trained on three languages with two vocabulary sizes test whether the findings hold at larger scale.
- Results: Figure 2 shows strikingly different crosslingual rankings across metrics, attributable to identifiable biases in each metric’s normalization unit.The figure ranks all 50 language-vocabulary combinations within each metric, with rank 1 indicating best performance.
- Observed Ranking Patterns: BPB and BPC consistently rank Chinese last because Chinese has shorter character and byte sequences.This pattern matches the predicted influence of character and byte counts on rankings.
- Observed Ranking Patterns: MRR, PPL, and Token NLL rankings are dominated by vocabulary size, with smaller vocabularies yielding better rankings regardless of language.For PPL and Token NLL, smaller vocabularies produce more tokens, and token averaging can make higher CTC appear better even when modeling quality is unchanged.
- Correlation Analysis: ρ>0.8 and p<0.01 characterize correlations between potential confounds and rankings for all metrics except Sent-NLL.Sent-NLL shows no significant correlation with sequence length measured in bytes, characters, or tokens.
5 Experiment 2: Sent-NLL is more robust for translation alternatives
The experiment tests whether Sent-NLL remains stable across alternative translations and sample sizes. Sent-NLL yields consistent translation-based rankings, while several normalized metrics vary with translation and vocabulary size; rankings stabilize at around 500 samples.
- Experiment setup: The experiment re-evaluates English–German translation pairs with two German references and compares Sent-NLL across five data sources.The sources are EN, DE1, DE2, DElowest, and DEhighest, using the same English- and German-trained models as Experiment 1.
- Sampling stability: The sampling follow-up varies the number of parallel-sentence samples from 1 to the full dataset and repeats each size with 15 random seeds.Figure 3 reports the sampling and replication results by sample-size row, with 95% confidence intervals across seeds.
- Translation robustness: MRR, PPL, and Token NLL vary monotonically by vocabulary size and produce different German rankings across translation variants, especially at vocabulary sizes ≥33k.These metrics are therefore more sensitive to surface-level differences in the translation alternatives.
- Translation robustness: Sent-NLL produces consistent language rankings across translation variants, with English scoring lower than all four German datasets.BPB and BPC are also consistent across variants, but they produce different rankings.
- Sampling stability: Rankings are unstable for sample sizes from 1–100 but become stable once the sample size reaches around 500.All other experiments in the paper use sample sizes greater than 500.
6 Experiment 3: Sent-NLL generalizes to multilingual language models
The paper tests whether metric biases extend from controlled models to multilingual LLMs. Across BLOOM models, BPB/BPC rankings are distorted by script and PPL/Token-NLL rankings by tokenization, while the identified biases are model-agnostic.
- Results: The identified metric biases are mathematical and model-agnostic because they arise from probability normalization rather than any particular model.
- Experiment setup: The multilingual evaluation examines nine LLMs on Arabic, English, Russian, and Chinese from FLORES-200.Models include BLOOM, mGPT, XGLM, Teuken, Qwen, and Llama, grouped by training-data transparency and multilingual usage.
- Experiment setup: For models with documented training data, expected rankings are derived from training-data proportions; for other models, the reported dominant language is checked against metric rankings.The setup assumes that more training data leads to better performance.
- Results: BPB and BPC rank Chinese worst in BLOOM despite Chinese comprising the second-largest portion of its training data, reflecting script-related distortion.Figure 4 compares expected rankings with actual rankings under six metrics for BLOOM-7B and BLOOM-1B.
- Results: PPL and Token-NLL rank Russian best despite its limited BLOOM training data because unseen languages are over-segmented into more tokens.This illustrates tokenization-driven distortion in multilingual-model rankings.
7 Considerations for Downstream Tasks
Downstream evaluations are difficult to compare across languages because translation quality, cultural specificity, item balance, annotation, and linguistic structure vary. The paper discusses conditions under which translated and language-specific benchmarks can still support more meaningful comparisons.
- Translated Benchmarks: Translated benchmarks can support crosslingual comparison when translation quality is sufficiently high, but unvalidated machine translation may introduce artifacts and noise.Professional human translations such as MMMLU and FLORES are less likely to suffer these issues.
- Culture-specific tasks: Localized benchmarks measure cultural knowledge as well as target-language performance, so equal scores across languages do not imply comparable performance.Their questions are not difficulty-matched or otherwise controlled across languages.
- Language-specific tasks: Language-specific benchmarks may be unreliable across languages because parallel tasks can differ in item quantity and quality.MultiBLiMP contains 7 Gujarati items versus 4615 Old Russian items, and its source treebanks vary in annotation schema.
- Language-specific tasks: Crosslinguistic differences in pronoun dropping, grammatical person, and negative polarity shape which agreement tests can be used.These differences affected the design of CLAMS comparisons across languages.
- Language-specific tasks: Subject-verb agreement is inherently different across languages, ranging from no agreement in Chinese and Tagalog to richer number, gender, and verb-class distinctions elsewhere.
8 Discussion & Conclusion
Fair crosslingual evaluation matters because metric choices shape how multilingual-model results are interpreted and resources are allocated. The paper argues that Sent-NLL is the least biased common intrinsic option, while acknowledging residual confounds and desirable dataset sensitivity.
- Discussion & Conclusion: Fair crosslingual evaluation is necessary because evaluation choices determine how results are interpreted and resources are allocated.
- Discussion & Conclusion: The paper provides systematic empirical evidence that normalized metrics can be unreliable across languages, addressing earlier observations based mainly on theory or intuition.The evidence spans multiple metrics, languages, and tokenizer settings.
- Discussion & Conclusion: Sent-NLL is not entirely free from confounds and assumes that languages encode roughly comparable amounts of information.Remaining sensitivity may reflect intrinsic information density rather than arbitrary engineering choices, so Sent-NLL is not a perfect standalone performance measure.
- Discussion & Conclusion: Sent-NLL rankings vary more across datasets than normalized metrics, whose apparent stability is largely determined by script, tokenization granularity, and vocabulary size.Sent-NLL is sensitive to test-data content, making dataset-dependent variability expected and desirable.
Limitations
The evaluation is limited by the language coverage, translation resources, and availability of high-quality parallel sentences. Additional data and bilingual pairs would strengthen generalizability, while supplementary results report similar patterns across datasets.
- The evaluation covers only 10 languages because parallel data are limited, constraining generalizability.The authors note that correlations for the other metrics remain significant at this sample size.
- Paraphrase experiments test only English–German pairs because translation resources are limited.More bilingual pairs with multiple translation references would strengthen the conclusions.
- The framework relies on high-quality parallel sentences, which many languages still lack.This limits its applicability despite relatively broad coverage in datasets such as FLORES.
- Correlation tests across three datasets find significant confounding correlations for every metric except Sent-NLL.The crosslingual rankings and correlation tests are reported separately for Parallel-10 and PUD, with similar patterns to the main text.
C.2 Dataset-specific results for monolingual GPT2-medium
The scaled-setting results examine language-model rankings across datasets, while the surrounding discussion highlights sensitivity to paraphrase choice and surface-level metric behavior. The section also documents the experimental corpora and training setup.
- C.2 Dataset-specific results for monolingual GPT2-medium: Scaling experiment results across different datasets are reported in Figures 5 to 7.These figures cover FLORES, PUD, and Parallel-3 scaled settings.
- D Paraphrases encode different amounts of information: Paraphrases can encode different amounts of information, so their probabilities need not be more similar than those of translations.The examples distinguish “Do you know how to play chess?” from “Can you play chess?” because the latter is more context-sensitive.
- D Paraphrases encode different amounts of information: Paraphrases permit greater variation because they are not constrained by crosslinguistic correspondence, adding a less controlled source of variation.The authors therefore do not treat paraphrase inconsistencies as necessarily undermining crosslingual evaluation.
F Multilingual Experiment
The multilingual experiment reports raw and ranking results for multilingual models across evaluation metrics and languages. Its supplied materials identify figures, tables, and language properties used for the comparison.
- F Multilingual Experiment: Crosslingual language rankings for Parallel-10 and PUD are shown in Figures 8 and 9.
- F Multilingual Experiment: Figures 10–12 show Spearman rank correlations between intrinsic metrics and potential confounding factors in FLORES, Parallel10, and PUD.Perplexity is scaled by log10 in these figures.
- F Multilingual Experiment: Results for mGPT and XGLM-4.5B are presented through expected-versus-actual rankings across six metrics.
- F Multilingual Experiment: Raw results for all evaluated multilingual models are provided in Figure 14.