Source-linked AI summary
Linguini: A benchmark for language-agnostic linguistic reasoning
Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Pontus Stenetorp, Mikel Artetxe, Marta R. Costa-jussà
TL;DR
Linguini addresses the difficulty of measuring linguistic reasoning independently of language-specific knowledge by benchmarking context-contained puzzles from mostly extremely low-resource languages. It evaluates open and proprietary models across 894 questions and 160 problems, finding a gap favoring proprietary models and evidence that performance relies mainly on presented context.
Problem
Linguistic reasoning evaluations are often tied to language-specific knowledge, while saturated benchmarks provide diminishing returns for distinguishing models.
Method
Linguini uses 894 International Linguistic Olympiad questions grouped into 160 problems across 75 mostly extremely low-resource languages, evaluated from zero-shot to five-shot with same-language examples excluded.
Results
Proprietary models outperform open models, and context, rather than prior language knowledge, appears central to solving the benchmark.
Takeaways & Limitations
Linguini provides a compact benchmark for assessing linguistic reasoning without relying on a substrate of existing language-specific knowledge.
Takeaways & Limitations
Long in-context-learning evaluation is limited to a few languages because publicly available grammar books are difficult to find.
Abstract
from arXiv · showhide
We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Olympiad corpus. To attain high accuracy on this benchmark, models don't need previous knowledge of the tested language, as all the information needed to solve the linguistic puzzle is presented in the context. We find that, while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models, with the best-performing proprietary model at 24.05% and the best-performing open model at 8.84%.
1 Introduction
Linguini addresses saturated benchmarks and the difficulty of separating linguistic reasoning from language-specific knowledge by evaluating models on context-contained linguistic puzzles.
- Saturated benchmarks such as MMLU provide diminishing returns for quantifying differences between increasingly capable models.State-of-the-art MMLU performance rose from 60% in December 2021 to 90% in December 2023.
- Linguistic-skill evaluation is often confounded by models’ comprehensive knowledge of a particular language, especially English.
- Linguini evaluates meta-linguistic awareness and deductive reasoning rather than pre-existing proficiency in the tested language.Its problems are extracted from International Linguistic Olympiad tasks designed to be solved through linguistic concepts presented in the problem.
- Evaluations find a noticeable performance gap between open and proprietary models, while context ablations indicate that models mainly rely on the presented context.The experiments also use form and content ablations to examine contextual information and possible training-set contamination.
2 Related Work
Prior work has evaluated reasoning and linguistic reasoning through mathematical, symbolic, translation-oriented, and manually constructed tasks, but linguistic-reasoning coverage remains limited.
- Reasoning benchmarks have also examined arithmetic, natural-language mathematics, and symbolic reasoning in multilingual and multimodal settings.The cited work includes Qin et al. (2023), Frieder et al. (2023), and Bang et al. (2023).
- Existing linguistic-reasoning datasets include Rosetta Stone-style translation tasks and manually created examples intended to reduce data leakage.PuzzLing Machines presents seven patterns for machine translation, while Chi et al. replicate the approach with manually created examples.
3 Benchmarking linguistic reasoning
Linguini is built from International Linguistic Olympiad problems so models can solve diverse low-resource-language tasks from the supplied context rather than prior knowledge of the language.
- 3.2 Selecting problems for our benchmark: The dataset construction aims to leave models with no information about the task language outside the supplied context.
- 3.1 IOL: The benchmark draws on IOL problems designed for students to apply linguistic skills to unfamiliar languages under the Rosetta Stone paradigm.The IOL corpus includes questions, correct answers, some explanations, and total marks in multiple PDF formats.
- 3.2 Selecting problems for our benchmark: Linguini contains 894 questions grouped into 160 problems across 75 mostly extremely low-resource languages.The selection excludes categories appearing only once, image-based questions, and questions whose response is only an explanation.
- 3.2 Selecting problems for our benchmark: Its questions cover sequence transduction, fill-in-blanks, and digit/text number transliteration.These categories require transforming sequences, applying morphophonological derivation rules, and analyzing morphology and morpheme order.
- Sequence transduction: Sequence-transduction problems may require reasoning about voicing, coarticulation, and voiced–voiceless consonant substitutions.
- 3.2 Selecting problems for our benchmark: A Terenâ example requires combining fortition, deletion of an incompatible nasal, and aperture-matched epenthetic vowels.The reasoning selects i after a closed vowel and e after a mid vowel.
4 Experiments
Experiments evaluate open and proprietary language models from zero-shot to few-shot settings using context-rich prompts, leave-one-out examples, and exact-match accuracy with chrF as a softer metric.
- 4 Experiments: Evaluation ranges from zero-shot to five-shot prompting while excluding same-language examples to reduce language contamination.Leave-one-out cross-validation maximizes the number of in-context candidates for each task.
- Setup and Models: Prompts combine an instruction, solvable linguistic context, and the problem, with task scores averaged from item answers.The study evaluates major open models and commercially available state-of-the-art models.
- Evaluation: Exact-match accuracy is the primary metric, supplemented by chrF when exact-match performance is nearly null.A low chrF score indicates extremely low performance and possible failure to understand the task domain.
5 Results and Discussion
Results show that model performance depends strongly on the supplied context: removing it causes steep drops, while changing scripts generally preserves reasoning ability. Accuracy is not broadly correlated with language resourcefulness, and textbook context can support modeling linguistic phenomena despite orthographic and OCR noise.
- Main results: Several proprietary models outperform the best open model on Linguini, although in-context examples have mixed effects across models.Table 1 reports exact-match results for 0–5 in-context examples; examples may clarify answer format but can also introduce unrelated language tokens.
- 5.1 No-Context Prompting: Steep performance drops without the benchmark context indicate that models rely heavily on the information supplied in the prompt.The no-context ablation is used as a proxy for possible training-data contamination because unseen task languages should yield near-null performance without necessary context.
- 5.2 Character-wise substitution: For 13 of 16 problems, at least one non-Latin script matches or exceeds Latin-script performance, supporting context-based rather than language-specific reasoning.The 16 selected problems were transcribed into Cyrillic, Greek, Georgian, and Armenian using character or bi-character substitution; disparities may reflect tokenization or transliteration limitations.
- 5.3 Language resourcefulness and accuracy: Accuracy follows a broadly uniform trend across speaker counts and Google-search frequency, suggesting weak correlation with language resourcefulness or likely training-set inclusion.Very high-resource languages such as cat, eus, kat, and tur are notable exceptions because their institutional status makes training-set inclusion likely.
- 5.4 One-Book Prompting: A single in-context textbook can support modeling linguistic phenomena, even when orthographic variation and imperfect OCR introduce noise.For Apurinã, including the full OCR text increases performance from 0% to 16.67%.
6 Conclusions
Linguini is presented as a compact benchmark for assessing linguistic reasoning without relying on existing language-specific knowledge, and results reveal gaps between model types and low contamination likelihood.
- Linguini provides a compact benchmark for assessing linguistic reasoning without relying on existing language-specific knowledge.
- The experiments find a considerable gap between open-source and proprietary language models in linguistic reasoning.
- Subsequent experiments indicate a very low likelihood of dataset contamination in the analyzed models.
A Limitations, further work and broader impact
The paper identifies limits in evaluating long in-context linguistic reasoning and in using Linguini for chain-of-thought experiments, while describing broader potential impact on support for world languages.
- Evaluation of long in-context learning for linguistic reasoning is limited to a few languages because publicly available grammar books are difficult to find.
- The dataset lacks curated explanations for each problem that could support chain-of-thought experiments and improve models’ linguistic reasoning skills.
- The benchmark aims to quantify the root of multilingualism, with potential implications for language-model support of the majority of world languages.
B Languages of Linguini
This section presents information about Linguini’s languages and their characteristics, including language codes, speaker counts, search-result counts, families, and scripts.
- Table 5 is identified as an overview of languages and their characteristics.
- The language overview records language codes, names, speaker counts, search-result counts, families, and scripts.
- The listed languages span families including Trans-New Guinea, Abkhaz-Adyghe, Tungusic, Niger-Congo, Australian, Nilo-Saharan, Yukaghir, and Algic.
C Models
This section provides overview tables for the models evaluated and the grammar books used in the benchmark’s supporting materials.
- Table 6 provides an overview of the large language models.
- The section’s materials distinguish model coverage from grammar-book coverage through separate overview tables.
- Table 7 provides an overview of grammar books.
E chrF Results
Table 8 reports chrF results for Linguini across settings with 0–5 ICEs. The surrounding notes define a search-result measure and a billion-parameter scale.
- The accompanying notes define a search-result count for the exact string “<Language name> language” using the Google Search API.
- The notes also identify a billion-parameter scale and reference Eberhard et al. (2020).
- Table 8 reports chrF results for Linguini with 0–5 ICEs.