Source-linked AI summary
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza
TL;DR
Formal multilingual linguistic competence is underassessed, especially beyond the few high-resource languages covered by existing targeted benchmarks. MultiBLiMP 1.0 automatically creates more than 128,000 subject-verb agreement minimal pairs across 101 languages and evaluates 42 LLMs, finding that competence tracks model size and training-data frequency and can deteriorate after post-training.
Problem
Formal linguistic competence remains underassessed in multilingual LLMs, while targeted syntactic benchmarks largely cover English and other high-resource languages.
Method
MultiBLiMP automatically creates subject-finite-verb and subject-participle agreement minimal pairs using Universal Dependencies and UniMorph.
Results
Linguistic competence is strongly driven by model size and language frequency in training data, is acquired during pre-training, and can deteriorate after post-training.
Takeaways & Limitations
The benchmark supports multilingual LLM evaluation and crosslingual study of linguistic construction learnability at substantially expanded scale.
Takeaways & Limitations
The procedure is constrained by the size, diversity, and annotation quality of Universal Dependencies and UniMorph, while manual evaluation is cost-prohibitive at benchmark scale.
Abstract
from arXiv · showhide
We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.
1 Introduction
MultiBLiMP addresses the limited multilingual coverage of formal linguistic evaluation by automatically creating subject-verb agreement minimal pairs across 101 languages. Evaluations of 42 LLMs show that linguistic competence depends on model size and training-data frequency and may deteriorate after post-training.
- Motivation: Existing multilingual LLM evaluations mostly measure functional competence, leaving formal knowledge of linguistic rules and regularities comparatively underassessed.Formal competence is commonly approximated with perplexity or generative tasks, rather than targeted syntactic evaluation.
- Motivation: Targeted syntactic benchmarks compare grammatical and ungrammatical minimal pairs that differ in one syntactic aspect, but mostly exist only for English and other high-resource languages.A formally competent model should assign higher probability to the grammatical sentence.
- Contribution: MultiBLiMP automatically creates subject-finite-verb and subject-participle agreement pairs for number, person, and gender using Universal Dependencies and UniMorph.The pipeline is designed to scale to additional linguistic phenomena.
- Contribution: More than 128,000 minimal pairs cover 101 languages and support evaluation of 42 LLMs.The benchmark applies subject-finite-verb and subject-participle agreement as its initial use case.
- Findings: Model linguistic competence is strongly driven by model size and language frequency in training data, is acquired during pre-training, and can deteriorate after post-training.These findings are reported from the evaluation of 42 language models.
- Significance: MultiBLiMP also enables large-scale crosslingual study of construction learnability, with applications to LLM interpretability and quantitative typology.Its unified framework operates at a larger scale than previously possible.
2 Background
Prior work uses syntactic minimal pairs and related methods to evaluate formal language ability, but coverage and scalability remain limited. MultiBLiMP complements language-specific depth with language-agnostic wide coverage for multilingual evaluation and typology.
- Targeted Syntactic Evaluation: Syntactic minimal-pair benchmarks test whether models prefer grammatical over minimally altered ungrammatical sentences, thereby probing acquired knowledge of specific phenomena.This approach has been used to study agreement difficulty involving distance and intervening attractors.
- Targeted Syntactic Evaluation: Parser-based evaluation measures the well-formedness of generated text, whereas minimal pairs measure intrinsic syntactic abilities on controlled stimuli.The parser-based approach requires a well-performing parser in each target language.
- Monolingual Non-English Benchmarks: Existing non-English minimal-pair benchmarks use annotated sentences, templates, translations, or language-specific resources.These methods have produced benchmarks for languages including Chinese, Japanese, and Russian.
- Monolingual Non-English Benchmarks: MultiBLiMP extends UD-based perturbation to a language-agnostic procedure for multilingual minimal-pair creation.It uses UD to identify syntactic relations and extends that approach across languages.
- Multilingual Benchmarks: Prior multilingual benchmarks covered only a small set of languages, including English, German, Hebrew, Italian, and Russian.CLAMS created agreement pairs by having native speakers translate sentences where applicable, while related work used UD treebanks.
- Multilingual Benchmarks: MultiBLiMP prioritizes wide coverage over the depth of language-specific benchmarks, supporting multilingual LLM evaluation and quantitative typological studies.The benchmark is also positioned as complementary to research on shared syntactic representations and multilingual interpretability.
3 Linguistic Resources
MultiBLiMP relies on Universal Dependencies for multilingual syntactic and morphological annotations and UniMorph for universal morphological features and inflectional mappings. These resources support language-independent pair creation, while preprocessing addresses feature compatibility and script differences.
- Universal Dependencies: Universal Dependencies is a multilingual treebank collection with 296 treebanks across 168 languages and rich syntactic and morphological annotations.MultiBLiMP 1.0 uses UD version 2.15.
- Universal Dependencies: The pipeline excludes spoken, historical, and certain genre-specific treebanks, while generally retaining low-resource treebanks unless exclusion is necessary.Very small high-resource treebanks may also be excluded when they risk adding little benefit or annotation inconsistency.
- UniMorph: UniMorph is a multilingual collection of word-level morphological features such as NUMBER, MOOD, and GENDER for nouns, verbs, and adjectives.It covers 183 languages, including 81 also covered by UD.
- UniMorph: UniMorph entries pair lemmas and word forms with morphological features, enabling efficient disambiguation and inflection creation.Its universal feature inventory makes it suitable for language-independent inflection mechanisms.
- UniMorph: UniMorph features may be implicit, so preprocessing aligns feature values with UD conventions and transliterates languages using different scripts.An unspecified feature value indicates that a form covers all values of that feature.
- UD Features: UD feature annotations are extended to 142 languages with annotations by extracting lemma-form-feature triplets and filtering potential errors using frequency.The procedure assumes each lemma-feature tuple maps to a single form.
4 Pipeline
The pipeline extracts agreement candidates from dependency-parsed data, validates language-specific agreement statistically, creates violations through morphological re-inflection, and balances the resulting minimal pairs. It also conditions validation on word order and leaves finer-grained agreement conditions for future work.
- Pipeline stages: The four-stage pipeline extracts candidates, validates agreement, re-inflects a word to create violations, and balances each language’s dataset.Candidate extraction uses dependency parses; validation uses UD collocation statistics; re-inflection creates the ungrammatical counterpart; balancing controls morphological features and pair counts.
- Candidate extraction: Candidate extraction filters dependency edges, part-of-speech tags, and morphological features for subject-verb and subject-participle agreement.Finite verbs or auxiliaries support subject-verb candidates, while participles require VERBFORM=PART; several dependency configurations and conjoined subjects are excluded.
- Inflection: Minimal pairs change an agreement feature and re-inflect one word, then retain the pair only when the perturbed sentence is ungrammatical.This filter handles cases such as optionally marked Turkish verbal plurality and prevents unchanged inflections from being treated as violations.
- Agreement validation: Agreement validation tests whether contrasting feature values produce grammatical violations, using noun-verb co-occurrence conditioned on subject-verb word order.The procedure expects high co-occurrence for matching feature values and low co-occurrence for contrasting values, and conditions Equation 1 on binary word order.
- Agreement validation: Dutch shows significant agreement for both singular and plural, whereas Turkish agreement is limited by optional verbal plurality.The authors note that a language can have strict agreement in only one direction and leave finer-grained agreement conditions for future work.
- Dataset balancing: The sampling probability Q is the inverse of the joint feature probability P(x), assuming feature independence, to balance sentence-level properties.The procedure balances subject and verb forms, subject-verb distance, attractor congruence, lexical diversity, and sentence difficulty.
5 MultiBLiMP 1.0
MultiBLiMP 1.0 applies the pipeline to two agreement phenomena across number, person, and gender, producing a large multilingual benchmark. It covers 128,321 minimal pairs in 101 languages, but has a strong Indo-European bias that motivates broader future coverage.
- Benchmark scale: 128,321 minimal pairs span 101 languages across subject-verb and subject-participle agreement for number, person, and gender.The unbalanced pipeline output contains 1.4 million pairs before balancing.
- Coverage: Subject-verb number agreement is the most common condition, covering 90 languages, while subject-participle person agreement covers only 3.The benchmark further subdivides conditions by inflected feature change and subject-verb order.
- Coverage limitations: MultiBLiMP 1.0 has a strong Indo-European bias caused by UD over-representation and agreement types that are more prevalent in Indo-European languages.Future iterations aim to broaden coverage by focusing on a more diverse set of phenomena.
6 LLM Evaluation Setup
The evaluation measures sentence-level grammaticality by comparing model probabilities for grammatical and minimally different ungrammatical sentences across 42 multilingual LLMs and controlled monolingual models.
- Metrics: Sentence-level accuracy counts minimal pairs where the model assigns higher probability to the grammatical sentence.The evaluation uses the same metric for both subject-verb word orders.
- Metrics: Judgment certainty is measured as the log probability difference between the grammatical and ungrammatical sentences.
- Models: The study evaluates 42 LLMs in both base, post-pretraining versions and chat, post-training versions.The evaluated families include Llama3, Aya-expanse, Gemma3, Qwen3, OLMo2, and EuroLLM.
- Models: Goldfish supplies 500M-parameter monolingual models trained on equal data amounts, controlling language-frequency differences unavailable for pretrained multilingual LLMs.Models trained on 1GB are used, or full models for languages with less than 1GB available data.
- Training-data analysis: Training-language frequency is estimated from language frequencies in a 3.9T-token Common Crawl split.This estimate approximates the language distribution of web-scraped data used in many LLM training corpora.
7 Experimental Results
Across agreement phenomena and language-resource groups, larger base models perform best overall, while performance rises with language frequency and post-training is associated with lower accuracy.
- General performance: 90.2% average accuracy is achieved by both Llama3-70B-base and Gemma3-27B-base, the best overall models.Llama performs better on low-resource languages, while Gemma performs better on mid- and high-resource languages.
- General performance: Performance increases with language frequency for all models, indicating that training-data disparity explains much of the variance in linguistic ability.
- General performance: Goldfish models are significantly better on 14 of 101 languages, compared with 2 for Llama3-70B and 3 for Gemma3-27B.Goldfish is excluded from the per-category ranking because it is evaluated on a subset of languages.
- Model properties: R^2 = 0.936 for the regression predicting average accuracy from model size, family, and post-training status.
- Model properties: β = 4.30 for model size, whereas post-training has β = −3.29, showing positive and negative performance impacts respectively.The corresponding test statistics are t = 15.88 and t = −6.47.
- Cross-model comparisons: 86.9% to 90.2% is the average-accuracy increase from Llama3-8B to Llama3-70B, with the larger model significantly better for 48 languages.No language improves by more than 10%, while almost all languages improve to some degree.
- Cross-model comparisons: Goldfish significantly outperforms Llama3-8B on 39 of 70 languages, while Llama3-8B performs better on 7.The authors leave open whether this reflects data frequency, data quality, or shared multilingual representations.
- Judgment factors: Agreement type, sentence perplexity, subword differences, attractors, and language frequency predict confidence in grammaticality judgments.Person agreement raises score differences, while higher perplexity, unfavorable subword deltas, and incongruent attractors reduce them.
8 Discussion and Conclusion
The authors use MultiBLiMP to discuss training implications, the continuing value of annotated linguistic resources, and future expansion beyond the current agreement-focused coverage.
- Training implications: Fine-tuning is surmised to be less effective than pre-training for boosting specific focus languages.This conclusion is based on the comparison between Aya and EuroLLM.
- Training implications: Goldfish significantly outperform models as large as 70B on 14 of 101 languages, many of them low-resource.The authors connect this pattern to the small share of some languages in multilingual training mixtures.
- Training implications: The authors suggest targeted regional language-model initiatives may better support communities underserved by current NLP technology.
- Annotated resources: Universal Dependencies and UniMorph remain useful for constructing multilingual benchmarks at scale.MultiBLiMP 1.0 was possible because of the linguistic knowledge captured by these large, collaboratively created resources.
- Future work: Future work will expand MultiBLiMP to more constructions, more diverse phenomena, and a broader language set.The benchmark is also intended to support learnability studies across typologically diverse languages.
Limitations
MultiBLiMP’s automated minimal-pair construction is constrained by its underlying resources, the difficulty of extending the method, and limited manual evaluation across many languages.
- The procedure depends on the size, diversity, and annotation quality of Universal Dependencies and UniMorph.
- Extending the procedure to more complex phenomena may be challenging because violations must be defined through morphological inflections.
- Manual evaluation is largely cost-prohibitive because the benchmark covers many languages, so evaluation is limited to spot checks.
- The benchmark is treated as a continuous effort, with language-expert feedback informing adjustments to individual languages.
A.2 Minimal Pair Sample
The sample illustrates MultiBLiMP minimal pairs across diverse languages, with each pair changing a targeted agreement-related form while preserving the surrounding sentence.
- The examples span Nhengatu, Gheg Albanian, Wolof, Low German, Faroese, and Latin.
- Each sentence presents a grammatical form alongside a minimally altered alternative marked with an asterisk.
- The sample includes subject-participle and subject-verb number or gender agreement cases, as indicated by the SV-P, SV-N, and SV-G labels.
- Figure 8 provides a sample of sentences, while the full dataset is available through the MultiBLiMP Hugging Face repository.
- The benchmark’s language coverage is summarized by the distribution of language families and languages present in MultiBLiMP 1.0.
B.2 Significantly Best Model per Language
The table reports language-level accuracy results and associated significance values for models identified as significantly best on particular languages.
- Additional reported accuracies include 99.2 for Albanian, 91.3 for Buriat, 73.7 for Erzya, and 97.0 for Wolof.
- Accuracy values for the listed languages range from 68.1 on Church Slavonic to 99.6 on Faroese.
- The results include 99.2 accuracy for Bulgarian, 80.4 for Gheg Albanian, and 98.8 for Polish.
B.3 Language-specific Results
Language-specific results are organized by model and agreement phenomenon, showing generally high accuracies alongside substantial variation across languages and tasks.
- The reported model comparison includes Llama3, Aya, Gemma3, OLMo2, Qwen3, EuroLLM, and Goldfish across multiple parameter scales and instruction-tuned variants.
- Table 5 splits MultiBLiMP accuracy scores by language and language model, marking the best-performing model per language in boldface.
- The tables separate subject-participle number, gender, and person from subject-verb number and gender evaluations.
- Subject-Participle: Subject-participle results include 73.7 accuracy for Erzya and 97.9 for Galician.
- Llama3-70B reports 77.1 accuracy for Gothic and 81.0 for Old Russian in the listed language-specific results.
- Subject-Verb Person: Subject-verb person accuracy ranges from 89.6 for Buriat to 100.0 for Faroese among the listed languages.