PaperlayerView exact revision on arXiv

Source-linked AI summary

BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items

Arzu Burcu Güven, Anna Rogers, Leonie Weissweiler, Joakim Nivre, Rob van der Goot

arXiv:2610.13607v1cs.CL

TL;DR

NPI evaluation in language models has been mostly limited to English, despite NPIs’ semantic and structural demands. BLiMPPI builds multilingual NPI minimal pairs from treebanks and evaluates models across languages. N-gram solvability is the strongest predictor of model difficulty, and multilingual models perform worst on lower-resource languages.

  • Problem

    In-depth NPI evaluation of language models has been confined to English, although NPIs test the intersection of meaning and structure.

    Source: §1 · p. 1

  • Method

    BLiMPPI builds naturalistic NPI minimal pairs by discovering candidates in UD treebanks and evaluating licensing environments across 33 languages.

    Source: §1 · p. 1, §4.2 · p. 4, §4.1 · p. 4

  • Results

    N-gram solvability is the most predictive factor across target languages, model sizes, and training regimes, while models perform worst on lower-resource languages.

    Source: §5 · p. 6, §7 · p. 8

  • Takeaways & Limitations

    N-gram baselines are crucial for assessing linguistic-benchmark difficulty and should be standard for this line of work.

    Source: §7 · p. 8

    The benchmark excludes conditional and comparison licensers because such sentences do not lend themselves to minimal pairs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Negative polarity items (NPIs), such as "anymore" and "ever", are grammatical only in select environments, predominantly in the context of negation. Because their grammaticality depends on both syntax and semantics, they have a rich, multilingual literature in theoretical linguistics, yet work on NPIs in language models is almost entirely confined to English. We introduce BLiMPPI, a multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, built by a pipeline that discovers NPIs in Universal Dependencies (UD) treebanks, labels their licensing environments, and perturbs these into minimal pairs. Evaluating 12 multilingual and 2 monolingual models alongside n-gram baselines, we find that multilingual models struggle most with lower-resource languages and with pairs that are not n-gram solvable. We further examine the effects of NPI typology, language family, licensing variability, and NPI-licenser distance. Data: https://github.com/arzuburcuguven/blimppi-data

1 Introduction

NPIs combine semantic and structural demands, yet model evaluations have rarely studied them multilingually. BLiMPPI addresses this gap with naturalistic minimal pairs and analyses of multilingual and monolingual models.

1 Introduction

  • NPIs test the intersection of meaning and structure because semantic context determines grammaticality, while violations resemble structural violations.NPIs have also ranked among the hardest phenomena in syntactic benchmarks, and children may lack adult-like comprehension as late as ages 11–12.Source: §1 · p. 1
  • Multilingual model benchmarks had focused mainly on agreement, while in-depth NPI evaluations were confined to English.Source: §1 · p. 1
  • The benchmark asks how multilingual models handle a rare, challenging phenomenon and whether they learn it evenly across typologically diverse languages.Source: §1 · p. 1
  • BLiMPPI uses perturbed natural sentences for NPI minimal pairs and analyzes 14 language models across 8 families.The authors also compare seven monolingual models’ checkpoints across typologies.Source: §1 · pp. 1–2

2 Background: NPIs

NPIs require licensing environments, with distributions varying by item and language. The section introduces licensing types, strength distinctions, and negative concord as background for analyzing model difficulty.

2 Background: NPIs

  • Licensers make NPIs grammatical and include negation, near-negative expressions, and non-negative contexts such as questions, comparatives, conditionals, superlatives, only, and modals.The section also includes n-series items and negative predicates among licensers.Source: §2 · p. 2
  • Some NPIs have item-specific licensing preferences; for example, either prefers negative contexts, while ever can also occur in questions.Either cannot be licensed in questions.Source: §2 · p. 2
  • NPIs are rare and can have theoretically unbounded dependencies, with intervening clauses not disrupting grammaticality.Source: §2 · p. 2
  • NPI strength varies from superstrong items restricted to negative contexts, through strong items in near-negative contexts, to weak items across wider licensing contexts.Source: §2 · p. 2
  • Negative Concord Items require strict collocation with negation in many Indo-European, Turkic, Uralic, and Semitic languages.The paper treats NCIs as a subgroup of NPIs and includes them in one polarity-item benchmark.Source: §2 · p. 2
  • Conditional, question, and comparative environments rarely license NPIs in negative-concord languages.Source: §2 · p. 2

3 Related Work

Earlier NPI evaluations found the phenomenon challenging but were limited in scope. BLiMPPI complements them with naturalistic multilingual pairs, targeting a phenomenon rarer than those in broad multilingual benchmarks.

3 Related Work

  • Prior NPI benchmarks reported NPIs among their hardest probes, but BLiMP focused on ever and non-English benchmarks covered Chinese and Russian.Marvin and Linzen used templatic pairs, while BLiMP contained 7,000 pairs across seven NPI test sets.Source: §3 · pp. 2–3
  • Prior studies found that licensing variability hampers performance, with strong NPIs learned better than weak NPIs.A probe study also generalized from observed licensing contexts to held-out environment types and NPIs.Source: §3 · p. 3
  • Unlike existing NPI benchmarks, BLiMPPI is multilingual and naturalistic rather than templatic, and it targets a rarer phenomenon than MultiBLiMP’s agreement benchmark.MultiBLiMP covers 101 languages and over 128,000 minimal pairs.Source: §3 · p. 3

4 Benchmark Creation

BLiMPPI draws on filtered UD treebanks and complementary English and Dutch resources to cover 33 languages. Its pipeline identifies NPI candidates and licensing environments, creates minimal pairs by perturbing natural sentences, and checks and revises the resulting contrasts.

4.1 Data sources

  • The benchmark starts from UD v2.18 treebanks with at least 100k words, stable and symmetrical negation, recoverable negation, and a living language; the Dutch NPI Lexicon brings coverage to 33 languages.Source: §4.1 · pp. 3–4

4.2 Minimal Pair Creation Pipeline

  • Where linguistic inventories are unavailable, discovery tests frequent determiner, pronoun, and adverb lemmas for higher occurrence in licensing than non-licensing environments, using a language-specific cutoff.Cutoffs are chosen to retain negative indefinites; higher thresholds work well for negative-concord languages, while polysemy can make co-occurrence statistics misleading elsewhere.Source: §4.2 · p. 4
  • Syntactic filters detect licensers, and language- and licenser-specific head-walking rules track long-distance scope; overlapping-scope multiple licensers are discarded.For example, the pipeline can climb from an NPI’s clause to a matrix clause across ccomp to detect matrix negation.Source: §4.2 · p. 4, Figure 1 · p. 4
  • Modification functions create minimal pairs by reinflecting verbs, inverting yes/no questions, deleting licensers, or swapping them with positive counterparts.Language-specific adjustments are made where needed, such as restoring contracted forms in Romanian.Source: §4.2 · p. 5
  • Reviewers flagged 61 of 428 audited pairs (9.6%) as lacking the intended contrast, after which flagged items were corrected or discarded and systematic problems triggered regeneration.Seven languages had native-speaker or linguist consultation; other checks used parallel UD benchmarks, machine translation, or audits across 17 further languages.Source: §4.2 · p. 5

5 Results

Across the benchmark, Gemma and Apertus outperform the other evaluated model families, with Gemma-3-27B the best model. Scores cluster more by model family than size, and multilingual models perform worst on several mid- to low-resource languages.

5 Results

  • Gemma and Apertus outperform the other evaluated model families, with Gemma-3-27B the best model; 8B and 14B Llama and Qwen variants perform on par with 100MB Goldfish models.Table 3 reports accuracy by language and model over 12,866 minimal pairs in 33 languages.Source: §5 · p. 5, Table 3 · p. 6
  • Scores generally cluster by model family rather than model size.The authors suggest training-data distribution may underlie the pattern, but note that reporting on training-data distribution is subpar.Source: §5 · p. 6
  • Multilingual models perform worst on Icelandic (77), Norwegian Nynorsk (83), Slovak (84), and Norwegian Bokmål (87), all described as mid- to low-resource.All multilingual models exceed 98% on English, although the English subset is not easy: the n-gram scores 62.3 and Goldfish 86/93.Source: §5 · p. 6

6 Analysis

The analysis compares NPI benchmark performance across linguistic and data properties, model families, and learning checkpoints. N-gram solvability is the strongest predictor of model errors, while results also vary with language resources, NPI-licenser distance, and licensing variability.

Typology

  • Baselines score 2.8–3.9 points higher on negative-concord languages, but multilingual models show only a 0.3-point, directionally inconsistent difference.Only Apertus scores lower on NPI languages at both tested sizes: −3.5 for 8B and −1.3 for 70B.Source: §6 · p. 6

Language Family

  • Germanic languages are the most challenging family group: 7 of 12 multilingual models and Goldfish models score lowest there.Romance and Indo-Aryan groups have the highest overall scores, though each has fewer members than other groups.Source: §6 · p. 6

Distance

  • Accuracy decreases as NPI–licenser token distance increases for every evaluated model; the negative correlation weakens with model size.N-gram models also show a significant −0.62 correlation with distance.Source: Figure 2 · p. 7, §6 · p. 7

Licensing Entropy

  • Licensing entropy is significantly negatively correlated with accuracy for larger Goldfish and Gemma-3 27B; its high-entropy tail is dominated by Dutch and English.The authors hypothesize that training-data biases override this finer-grained linguistic pattern.Source: Figure 2 · p. 7, §6 · p. 8

When NPIs are “learned”

  • ParalleLM accuracy stabilizes by the fifth checkpoint in every tested language except Turkish, despite repeated training on the same 20M words.Turkish rises above chance only at checkpoint four; the authors suggest its negation-suffix tokenization, rather than typology, likely explains the delay.Source: Figure 2 · p. 7, §6 · p. 8

N-gram Solvability

  • Multilingual-model accuracy averages 94.5 on n-gram-solvable pairs and 87.7 on unsolvable pairs, with the gap shrinking as model size increases.The difference falls from 13.1 points for Gemma-1B to 3.1 for Apertus-70B and 2.8 for Gemma-27B.Source: §6 · p. 7

Common Crawl Frequency

  • Scores track Common Crawl frequencies significantly for Aya, Llama, and Qwen, while Gemma and Apertus report balanced multilingual training data.All models reach ceiling on high-resource languages; only monolingual and balanced multilingual models do so on low-resource ones.Source: §6 · p. 7

Comparison with MultiBLiMP

  • Models score 4.7 points lower on BLiMPPI than MultiBLiMP on average (89.7 vs. 94.4), with the largest gaps for lower-scoring models.The comparison covers 28 shared languages and six models; the two benchmarks test different phenomena.Source: §6 · p. 8

7 Conclusion

BLiMPPI provides multilingual minimal-pair evaluation alongside metrics for analyzing model behavior. Across the evaluation, n-gram solvability best predicts errors, while balanced multilingual pretraining and linguistic factors offer complementary insights.

Conclusion

  • N-gram solvability is the most predictive checked factor across target languages, model sizes, and training regimes, while linguistic factors such as distance and licensing entropy add interpretability.The conclusion also reports that models prioritizing multilingual balance in pretraining perform best.Source: §7 · p. 8

Limitations

The benchmark’s coverage and quality are constrained by UD availability, uneven resources, omitted licensing environments, and incomplete training-data information.

Limited coverage and quality checks

  • BLiMPPI covers languages well represented in UD and is not fully quality-checked by native speakers; language-specific configurations are manually developed with linguists and native speakers.The authors describe this as a trade-off between high coverage from automatically extracted data and quality.Source: Limitations · p. 9

Resource imbalance

  • Germanic languages have more resources; English and Dutch receive additional examples, while most other languages rely on automatic discovery and descriptive literature.The authors report no comparable resource for most remaining languages.Source: Limitations · p. 9

Licenser coverage

  • Conditional and comparison licensers are not covered because their sentences do not lend themselves to minimal pairs.The authors note that this prevents examining whether licensing paradigms reinforce one another across languages.Source: Limitations · p. 9

Training data

  • Training-data distribution information is available only for Apertus and Goldfish, and only for non-English languages, limiting analysis of the data volume needed to capture polarity phenomena.Source: Limitations · p. 9

Ethics Statement

The study assesses linguistic functionality across languages and reports that models struggle most with lower-resource languages. Its appendices document benchmark exclusions, language-specific licensing decisions, and checks on pair construction.

Ethics Statement

  • The study aims to assess LLM linguistic functionality across 33 languages; it uses existing linguistic resources and reports no new data-sharing risks.The authors say the sources do not, to their knowledge, contain personally identifiable information or toxic content.Source: Ethics Statement · p. 9

A Excluded Treebanks

  • Table 5 lists UD treebanks above 100,000 tokens that were excluded, along with the Section 4.1 criterion each failed.The appendix identifies the excluded treebanks and the relevant criterion, but these passages do not specify the criteria themselves.Source: A Excluded Treebanks · p. 12, Table 5 · p. 13

B Consulted Work

C Licensing Scope

  • Licensing scope is encoded by licenser and language: head-walking can cross some complement clauses and verb-phrase coordination, but not specified islands or clausal coordination.Configurations set how many head-walking steps are allowed before a licenser is discarded.Source: C Licensing Scope · p. 13

C.1 Special Cases

  • For severe asymmetrical negation, the benchmark uses NPI-to-PPI substitutions for Finnish and Lithuanian but excludes other such languages when expert collaboration was unavailable.For genitive-of-negation languages, pairs exclude verbs whose object shifts from accusative to genitive under negation.Source: §C.1 · pp. 13–14

D Swap vs. Drop

  • Across seven languages containing both swap and drop items, Δ was positive or near zero for 5 of 14 evaluated transformer models, and authors conclude token differences do not distort linguistic evaluation.Source: D Swap vs. Drop · p. 14

E Example Pairs

F Multiple NPIs

  • Table 8 reports the share of pairs containing multiple NPIs and multilingual-model accuracy on single- versus multiple-NPI pairs.The reported accuracy is averaged over 12 multilingual models.Source: F Multiple NPIs · p. 14

G Human Audit

The human audit identified two English-specific issues in pair construction, while the evaluation included recent open-weight multilingual models and monolingual Goldfish models.

G Human Audit

  • The audit surfaced one item-level and one systematic problem in English; the passages describe the free-choice reading and the without/with swap issue.The latter swap can make a sentence ungrammatical for a reason unrelated to the polarity item when without governs a gerund.Source: G Human Audit · pp. 14, 17
  • Pairs built on English anyone were removed because their apparently unlicensed member can have a grammatical free-choice reading.The audit found a second systematic English issue with swapping without for with: the swap was restricted to cases where without governs a nominal phrase.Source: G Human Audit · pp. 14, 17

H Transformer-based LMs

  • The evaluation includes five recent open-weight model series and Goldfish monolingual models trained on equal-sized data using GPT-2 architecture.The listed series are Llama-3.1, Aya, Gemma-3, Qwen3, and Apertus.Source: H Transformer-based LMs · p. 17

I n-grams

The word-level n-gram baseline averages 63.2 accuracy, with substantial variation across languages and data-based explanations for several high and low scores. The appendices also describe distance and cross-benchmark comparisons.

I n-grams

  • 63.2: the n-gram models’ overall average accuracy, compared with a 50.0% random baseline, indicates the task is challenging for n-gram models.Most languages fall in the 50–65 range.Source: I n-grams · p. 17
  • 92.3 for Hungarian, 89.1 for Lithuanian, and 89.5 for Finnish are the highest n-gram scores; Turkish 40.7, Persian 34.1, and Norwegian-Bokmål 32.6 are the lowest.The high Lithuanian and Finnish scores are associated with their NPI-to-PPI substitution strategy; bound negation challenges word-level n-grams in Turkish and Persian.Source: I n-grams · pp. 17–18
  • High OOV rates in Hungarian and Serbian reflect small sets (n=13 and 24), where a few proper-noun mismatches can raise the rate; compounds contribute in Danish and German.The passage also notes possible genre mismatch and UD’s underscore tokenization for Urdu compounds.Source: I n-grams · p. 18
  • OOV tokens are shared across the acceptable and unacceptable members of each pair, so OOV mass inflates both scores equally and cannot account for the reported accuracies.Source: I n-grams · p. 18

J Distance

  • Figure 4 presents the distribution of NPI–licenser distances in tokens by language; the text also identifies the per-language mean distance.Source: J Distance · p. 18

K MultiBLiMP Comparison

  • The appendix compares BLiMPPI accuracies with published MultiBLiMP scores for shared models and languages.Table 13 covers 28 shared languages and describes per-language accuracy and pair counts.Source: K MultiBLiMP Comparison · p. 18

L Turkish Negation and Tokenization

Turkish is a difficult case for the evaluated models, and the authors connect its performance to how negation is segmented. Related tables separate model accuracy by the 5-gram baseline’s decisions and compare BLiMPPI with MultiBLiMP.

L Turkish Negation and Tokenization

  • 62.7: Turkish is the hardest language for the 100MB Goldfish model; ParalleLM rises above chance only at its fourth checkpoint.Source: L Turkish Negation and Tokenization · p. 19
  • 76.6% of 1,532 extracted Turkish NPI occurrences are under negation, while Turkish performs worse than Czech and Persian, which also use bound negation.The authors hypothesize that Turkish -mA is difficult to segment because it is medial and followed by TAME suffixes; ParalleLM scores 75.5 on Turkish with less data.Source: L Turkish Negation and Tokenization · p. 19
  • Tables 11 and 12 report accuracy on pairs the 5-gram baseline gets right and wrong, respectively; Table 13 compares BLiMPPI and MultiBLiMP across 28 shared languages.The two split tables contain 8,456 and 4,356 pairs across 33 languages; Table 13 covers six models reported in MultiBLiMP.Source: Table 11 · p. 20, Table 12 · p. 21, Table 13 · p. 22
Loading 2610.13607v1…