modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models

• Evaluating multilingual few-shot reasoning is difficult because even low-resource languages may appear in large training corpora, confounding generalization with memorization.Source: §1 · p. 1
• MODELING is a benchmark of Rosetta stone puzzles requiring models to infer rules from translated sentence pairs and apply them to new expressions.The puzzles test word meanings, morphology, and word order through compositional generalization.Source: §1 · p. 1, Figure 1 · p. 1
• The benchmark covers diverse low-resource languages and phenomena including noun/adjective order and possessive syntax.Source: §1 · p. 1
• All problems were written specifically for this work, reducing the risk that benchmark performance reflects memorization of publicly available puzzles.The authors report evidence that popular LLMs do not have data leakage on the benchmark.Source: §1 · p. 2

• Linguistics Olympiad-style puzzles originated as educational tools for introducing linguistic reasoning concepts and are proposed here for evaluating few-shot linguistic reasoning in AI systems.Source: §2 · p. 2
• PuzzLing Machines uses online Linguistics Olympiad puzzles, creating a risk that language models encountered the evaluation items during training.There is also relatively little literature evaluating language models on linguistics puzzles.Source: §2 · p. 2
• BIG-bench includes a constructed-language task, but its synthetic languages do not fully represent natural-language nuance or irregularity.Source: §2 · p. 2
• MODELING addresses these limitations with newly written problems in natural, extremely low-resource languages and fine-grained categorizations by linguistic pattern and difficulty.The resulting dataset is intended to support closer analysis of linguistic reasoning with fewer confounding factors.Source: §2 · p. 2

3.1 Task Overview

• 48 puzzles based on 19 extremely low-resource languages yield 272 questions spanning noun-adjective order, word order, possession, and semantics.Problems were test-solved and rated for difficulty by two International Linguistics Olympiad medalists.Source: §3.1 · p. 2

3.2 Problem Design

• The four task types test adjective-noun ordering, subject-verb-object ordering, possessive morphology, and mappings between target-language compounds and English meanings.Source: §3.1 · p. 2
• The dataset contains 48 problems and 272 questions, with its problem types summarized in Table 1.Source: Table 1 · p. 2
• Possession problems cover marking strategies, possessor placement, and possible distinctions between inherent and non-inherent possession.Source: §3.2 · p. 3
• Semantics problems require aligning target-language expressions with English meanings, including cases where one source-language meaning corresponds to a compound expression.Source: §3.2 · p. 3

3.3 Data leakage

• When evaluated without target/reference pairs, all models achieved 0% accuracy, suggesting that the extremely low-resource languages successfully avoid detectable data leakage.The diagnostic treats nonzero accuracy without examples as evidence that prior language knowledge may be present.Source: §3.3 · p. 3

• The study evaluates open-source models from 7B to 70B-scale families alongside GPT-3.5 and GPT-4o.Source: §4 · p. 3
• Models are tested with minimal, hand-tuned, basic chain-of-thought, and full chain-of-thought prompts.The full chain-of-thought prompt demonstrates step-by-step reasoning with a Spanish linguistics puzzle.Source: §4 · p. 3
• Exact-match accuracy is used because each puzzle has one correct solution and BLEU-like metrics poorly distinguish word-order and subtle morphological differences.Source: §4 · p. 3

5 Results and Analysis

• Strong performance comes from GPT-4 and the largest open-source models, while Gemma, Alpaca, and Llama-2 achieve accuracies closer to 0.Accuracies are roughly similar across prompting approaches.Source: §5 · p. 3
• Model difficulty generally follows human difficulty ratings, leaving room to scale MODELING with harder problems.Even the hardest current problems are relatively easy by Linguistics Olympiad standards.Source: §5 · pp. 3–4
• Possessive and semantic problems are harder for models than noun/adjective ordering and basic word order, matching human difficulty ratings of 5.0 and 4.54 versus 2.61 and 3.18.Source: §5 · p. 4

Impact of Orthography on Model Reasoning

• GPT-4 gains 3.1% absolute accuracy on diacritic-free problems under the Minimal Prompt, with performance converging as prompts become more complex.The authors suggest this may reflect the prevalence of English or English-influenced text without diacritics in online training data.Source: Impact of Orthography on Model Reasoning · p. 4

• MODELING is designed to evaluate LLMs’ capacity to reason analytically in unseen foreign languages.Source: §6 · p. 4
• Using very rare languages and extending to more challenging cases gives MODELING strong potential as a durable approach for evaluating reasoning.Source: §6 · pp. 4–5

• Because the authors are not native speakers, sentence accuracy is not guaranteed, so the dataset should not be used as linguistic data.The authors distinguish this limitation from the benchmark’s intended pattern-recognition evaluation.Source: Limitations · p. 5
• Proprietary pretraining data prevents complete certainty that benchmark examples are absent, despite newly written problems and zero-shot evidence against leakage.Source: Limitations · p. 5
• Hand selection and data availability may leave implicit language-selection biases, so results should not be interpreted as covering global linguistic diversity.Source: Limitations · p. 5

A.1 Canary String

• Table 4 presents sample orthographic conversions used in the benchmark materials.Source: Table 4 · p. 8

B Prompts

• The minimal prompt is one of four prompting styles illustrated for MODELING.Source: Figure 5 · p. 8, B Prompts · p. 8

C Data sources

• The dataset is intended solely as an evaluation metric and includes a canary string to discourage model-training use.Source: C Data sources · p. 8

A.2 Difficulty

• Table 3 reports the dataset’s difficulty-level distribution as jointly rated by two International Linguistics Olympiad medalists.Source: Table 3 · p. 8
• Figures 6–8 distinguish hand-tuned, basic chain-of-thought, and full chain-of-thought prompts.Source: Figure 6 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9
• Table 5 presents the problem data sources.Source: Table 5 · p. 10