PaperlayerView exact revision on arXiv

Source-linked AI summary

The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning

Eduardo Sánchez, Rita Berrada, Dan-Mircea Mirea, Sara Rajaee, Alexander Piperski, Ana Meta Dolinar, Boris Iomdin, Andrey Nikulin, Mariya Shmatova, Marzieh Fadaee, Julia Kreutzer

arXiv:2608.18011v1cs.CL

TL;DR

Most reasoning benchmarks give models the rules, leaving limited evidence about whether they can discover unfamiliar systems first. The IOL-AI Challenge evaluates this ability on unseen linguistics problems with expert jury grading and finds proprietary models reaching medal level while resource-constrained and open systems lag.

  • Problem

    Most reasoning benchmarks provide formal rules, leaving limited evidence about whether systems can infer unfamiliar structures before solving problems.

  • Method

    The IOL-AI Challenge tests models on genuinely unseen IOL 2026 problems under compute restrictions, using automatic metrics and official IOL jury grading.

  • Results

    Proprietary models reached medal level, while resource-constrained and open systems lagged; Claude Opus 4.8 scored above the gold-medal cutoff.

  • Takeaways & Limitations

    Unseen linguistic-olympiad problems provide a benchmark of reasoning that requires inferring linguistic systems rather than relying on memorized solutions.

  • Takeaways & Limitations

    DeepSeek-R1-32B received a 32,768-token generation budget versus 64,000 for three peers, making within-tier scores not strictly comparable and likely understated.

Abstract

from arXiv · show

Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the system before reasoning within it. We present the IOL-AI Challenge, an open-science competition run on the unseen problems of the International Linguistics Olympiad (IOL) 2026 Individual Contest, evaluated both automatically and, for the first time, by members of the official IOL Jury under the same rubrics applied to human contestants. The challenge drew 731 submissions from 46 teams under a strict compute budget (one T4, 30 mins). We additionally benchmark 15 unconstrained frontier and open models, with Claude Opus 4.8 earning a jury score equivalent to a gold medal, while both resource-constrained systems we submitted for jury grading scored in the range of the bottom 5% of contestants. Capability was not determined by scale: 14B submissions outperform models twice their size, and gains come from decoding and output-handling rather than model capacity. We also found that automatic metrics rank systems exactly as the jury does, but compress the scale, upscoring weak systems by ~13 points and understating strong ones. Our analysis shows that while frontier models might have prior knowledge about some of the problem languages, it does not significantly help them solve the linguistic reasoning tasks, leaving linguistic reasoning as a strong benchmarking proxy for generalizable reasoning skills.

1 Introduction

The paper frames linguistic puzzles as a test of generalizable reasoning because they require discovering rules from unfamiliar, limited data rather than applying predefined formal systems. It advances prior evaluations through an open challenge and official IOL jury assessment, finding proprietary models have caught up while open and resource-constrained models still lag.

  • Motivation: Mathematics and coding dominate reasoning evaluations, despite relying on formal structures, unambiguous answers, and automatically verifiable feedback.The paper contrasts these settings with real-world problems that may lack well-specified rules.
  • Motivation: Linguistic puzzles test whether solvers can identify regularities, form hypotheses, and test them against evidence in unfamiliar domains.They involve disentangling phenomena in an unfamiliar language from a limited corpus.
  • Benchmark: 46 countries participated in the 2026 IOL edition, whose problems require analyzing unfamiliar languages, deducing rules that explain the data, and using those rules to solve further tasks.The IOL is presented as the largest linguistic olympiad and an established source of such problems.
  • Contributions: The study extends prior benchmarking with an open-science challenge and expert-level evaluations from the official IOL jury instead of relying solely on shallow automatic matching.The jury evaluations enable expert-level judgments of linguistic reasoning.
  • Results: Proprietary models have caught up on linguistic reasoning, whereas open models lag substantially and resource-constrained systems mostly fail, leaving generalizable reasoning an open problem.This finding contrasts with prior work reporting consistent underperformance by frontier reasoning models.

2 Related Work

Prior reasoning benchmarks emphasize mathematics, theorem proving, code, science, and multidisciplinary tasks, while linguistic reasoning has spawned dedicated benchmarks from Linguistic Olympiad problems. Recent methods improve linguistic reasoning modestly, but advanced LLMs still lag behind math and commonsense reasoning benchmarks.

  • Reasoning Capabilities: Reasoning benchmarks increasingly evaluate large models across mathematics, theorem proving, code generation, science, and multidisciplinary tasks.
  • Reasoning Capabilities: Verification-and-refinement substantially improved state-of-the-art model performance on IMO problems, reaching a gold-medal score.The approach benefits from mathematics because individual logical steps have easily verifiable correctness.
  • Linguistic Reasoning Benchmarks: Linguistic reasoning benchmarks include PuzzLing Machines, LingOly, Linguini, IOLBENCH, and LINGOLY-TOO, drawing on Linguistic Olympiad problems and low-resource languages.
  • Advances in Linguistic Reasoning with AI: Recent interventions improve linguistic reasoning through multi-turn step-by-step prompting, analogical exemplars, and test-time scaling, but gains remain limited.Benchmark performance can increase by a few points while staying well below math and commonsense reasoning benchmarks.

3 Linguistic Reasoning at IOL

IOL linguistic problems are designed as self-sufficient reasoning tasks, requiring solvers to infer linguistic systems without prior knowledge. The IOL 2026 Individual Contest comprised five diverse problems solved individually within six hours and graded through jury-developed criteria.

  • Self-sufficiency: Linguistic problems must be solvable without prior knowledge of foreign languages or advanced linguistic concepts, with low-resource languages reducing familiarity advantages.Contestants are not expected to have prior linguistics training.
  • Problem types: Common problem structures include Rosetta stone problems based on translation or transcription and chaos-and-order problems based on matching assignments.Problems may also be classified by linguistic subdiscipline, including phonology, morphology, syntax, semantics, and writing systems.
  • IOL 2026 problem set: The IOL 2026 Individual Contest included five six-hour problems spanning phonetics and orthography, lexical semantics, syntax, kinship terminology, and verb morphology.The problems used Central Alaskan Yup’ik, Yélî Dnye, Iquito, Sakurabiat, and Komnzo, respectively.
  • Evaluation: Each problem is graded using a jury-developed scheme that separately allocates points to assignments and theoretical explanations of the rules.Submissions are independently graded by at least two jurors, with discrepancies resolved through discussion until consensus.

4 The IOL-AI Challenge 2026

The IOL-AI Challenge evaluated systems on unseen linguistic reasoning tasks from the five-problem IOL 2026 Individual Contest under compute and information constraints. It combined public development scoring, private test evaluation, automatic metrics, and jury-based human evaluation modeled on IOL grading.

  • Challenge setup: The challenge used a truly unseen test set and a compute-restricted environment in an open-science competition preceding the IOL.Submissions were accepted for one month, until July 26, 2026, when the on-site IOL competition began.
  • Task: The dataset comprised 5 IOL problems containing 14 sub-assignments across Translation, Fill-in Blanks, and Match Letters tasks.Translation mapped Solverese to an unseen extremely low-resource language; the other tasks required completing forms or matching phrases to correspondences.
  • Task: 8 development tasks supported public automatic scoring, while 6 balanced test tasks were withheld for a private leaderboard revealed after the competition.One task requiring only an explanation was excluded from automatic scoring.
  • Evaluation: Automatic evaluation combined ChrF and exact match through their geometric mean to balance partial-match overestimation against formatting and morphological underestimation.ChrF can overestimate answer quality through partial matches, whereas exact match can underestimate it because of formatting or minor morphological variations.
  • Evaluation: 15 existing reasoning models were benchmarked alongside resource-constrained challenge submissions, and five systems were selected for anonymized IOL-jury human evaluation.At least 2 graders evaluated each problem using a procedure similar to contestant grading, including discrepancy resolution through discussion.

5 Results

Results show that strong performance came from decoding and output-handling choices rather than model scale, while problem difficulty varied sharply by language and direction. Jury rankings matched automatic rankings but exposed qualitative weaknesses in model explanations, including verbosity, hallucination, underspecified rules, undergeneralization, and overfitting.

  • Submission results: 731 submissions came from 46 teams, and the best submission more than doubled the same-model baseline from GM 9.40 to 19.79.Nine of the top ten submissions used a quantized 14B Qwen model, the largest model meeting the compute constraint.
  • Leading approaches: Leading submissions lowered the repetition penalty to 1.0, omitted chain-of-thought prompting, and centered on greedy decoding with temperature-0.5 retries when sampling.These choices formed a common recipe among the strongest entries.
  • Model comparison: 14B submissions outperformed larger models, while Gemma4-31B-It scored 23.2 and its ensemble gained another 10 points, showing performance was not strictly determined by scale.The passage also reports a 19-point gap between Claude-Opus-4.8 and the following proprietary models, and a 7-point gap before open large models.
  • Problem difficulty: Iquito was hardest for submissions, with both tasks at 0% exact match, whereas Sakurabiat was easiest and nearly every team solved its short closed items.Yup’ik and Yélî Dnye followed Iquito in difficulty, while Komnzo showed a strong directional asymmetry.
  • Frontier-model results: Frontier models scored higher on every problem but ranked them differently: Yélî Dnye reached about 20% geomean, Iquito moved to the middle, and Sakurabiat remained easiest at about 55%.Komnzo’s English→Komnzo direction stayed low at about 28.5%, while only proprietary models scored non-zero on Yélî Dnye→English.
  • Jury evaluation: IOL jurors preserved the automatic metric’s model ranking; Opus 4.8 matched the lowest gold-medal score of 70.0, while both challenge models ranked in the bottom 5%.Gemini 3.6 matched the lowest silver-medal score of 58.4, and all remaining models fell below the honorable-mention threshold of 35.3.
  • Qualitative analysis: Models often produced verbose traces, hallucinated data, hedged underspecified rules, missed broad patterns, and overfit with convoluted explanations compared with human solutions.The strongest models still achieved medal-level scores, but humans more often built compact analyses across the whole problem while models reasoned assignment by assignment.

6 Conclusion

The IOL-AI Challenge evaluates models on unreleased linguistic-olympiad problems using expert jury rubrics, testing system discovery rather than benchmark recall. Inference-time methods enabled medal-level performance, while automatic metrics preserved rankings but distorted score magnitudes.

  • Evaluation setting: The challenge evaluates models on genuinely unseen linguistic-olympiad problems graded by the expert jury that assesses human contestants.Unreleased problems and solutions prevent benchmark leakage by construction.
  • Evaluation setting: Models must infer each linguistic system from supplied data, while knowledge of the problem languages does not determine success.The languages may have public documentation, but knowledge probing found no decisive relationship with performance.
  • Performance: The winning submission more than doubled the strongest baseline using the same 14B model, with gains attributed to decoding and answer handling rather than capacity.The result demonstrates a large inference-time effect under tight resource constraints.
  • Performance: Claude Opus 4.8 reached medal level and scored above this year’s gold-medal threshold, overturning earlier findings that frontier models uniformly underperformed.The medal-level result was achieved under expert grading rather than string matching.
  • Scoring: ρ = 1.00 and r = 0.99: automatic metrics rank systems exactly as the jury does but compress scores, inflating weak systems by ∼13 points.Strong systems are under-credited because correct prose analyses receive no automatic credit.

Limitations … C Baselines

The paper identifies limitations in answer formatting, evaluation breadth, ensemble interpretation, generation budgets, competition duration, and submission independence. It also describes accessibility measures, technical adaptations, and baseline implementations for the challenge.

  • Limitations: Models, including frontier systems, often failed the required list-based answer format, requiring post-hoc format-fixing rules to prevent artifacts from perturbing automatic rankings.The authors suggest future linguistic-reasoning benchmarks should reconsider prompting styles and output requirements.
  • Limitations: The IOL 2026 evaluation prioritizes depth over breadth, resulting in few test samples, low statistical power, and potentially high variance under stochastic decoding.Small automatic-score differences should therefore be interpreted cautiously, especially for long reasoning traces on out-of-domain tasks.
  • Limitations: The BoN Open ensemble is an oracle-selected upper bound rather than a deployable system, with most selected answers supplied by Gemma4-31B-It.Its advantage is therefore only partly attributable to model diversity.
  • Limitations: 32,768 tokens was allocated to DeepSeek-R1-32B versus 64,000 for the other three mid-size models, making within-tier scores incomparable and likely understating its performance.The unequal budgets are reported for the comparisons in Table 11.
  • Limitations: A one-month competition window may have limited submissions and prevented participants from improving their systems further.A longer duration might have increased both participation and submission refinement.
  • Limitations: Public HuggingFace repositories allowed participants to inspect or copy submissions, creating a loophole despite a ban on post-submission teaming and producing many similar systems.This may obscure credit for foundational work behind top-scoring submissions.
  • A Accessibility and Engagement: The organizers lowered participation barriers with an online submission session, a demo notebook, comparable T4 Colab compute, and resources on linguistic reasoning.The competition was deliberately held outside an academic venue and promoted through social media and open-science communities.
  • B Technical Details: The competition framework was adapted to route evaluation through HF Jobs, automatically archive repositories for reproducibility and human evaluation, and publicly release modifications.Baseline examples included Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct with 4-bit bitsandbytes quantization, and Qwen2.5-14B-Instruct-AWQ with pre-quantized AWQ weights and a reasoning prompt.

D Extended Model Benchmarking

The extended benchmark evaluates strong, provider-diverse models beyond the competition’s resource constraints, using development-set prompt tuning and answer postprocessing before selecting models for jury evaluation. Outputs are also formatted for jury readability, with automatic results reported before and after custom answer parsing.

  • Model selection and evaluation: Strong models outside the competition’s resource constraints are benchmarked with priority given to high Artificial Analysis Intelligence Index scores and provider diversity.The evaluation hyperparameters are provided in Table 11.
  • Model selection and evaluation: Linguini is used as a development set to tune task and explanation prompts, while the IOL 2026 test set selects models for human evaluation.The study also optimizes explanation formatting for jury readability and provides prompts with and without explanations.
  • Automatic evaluation: Automatic scores are reported for the extended model list before and after custom answer parsing.The BoN Open (oracle) ensemble builds directly on custom answer-parsed answers; EM, GM, and LFMR are defined as exact match, geometric mean, and line format match rate.
  • Automatic evaluation: Per-problem BoN Open (oracle) scores use a 0–100 scale, making them directly comparable to jury scores expressed as percentages of the 20 available points per problem.Models selected for jury evaluation are marked with ⋆ in Table 9.

E The Challenge of Format Following … G Grading of AI Solutions by the IOL Jury

The challenge treated format following as a major source of performance variation, favoring answer-only prompting and parsing over chain-of-thought. AI solutions were evaluated remotely by anonymized IOL jury members using the official grading schemes and summed scores out of 100.

  • E The Challenge of Format Following: Automatic scoring compares answers by position, so a missing or shifted line can zero out an otherwise correct row.Format errors can therefore turn partial correctness into a fully incorrect scored row.
  • E The Challenge of Format Following: 34 teams used regex answer extraction, while many added per-task format prompts or retry loops.Format following was a major focus of the open challenge.
  • E The Challenge of Format Following: 33 teams tried chain-of-thought, but it hurt on average and none of the top 6 retained it.Reasoning text made extraction harder and consumed wall-clock time; podium systems used answers-only prompts and parsers.
  • E The Challenge of Format Following: Parsing never lowers frontier-model scores and can raise GM by about 14 points, including GLM-5.2 from 30.5 to 45.0 and Claude Opus from 61.0 to 75.1.Light postprocessing repairs superficial nonconformities, but cannot fix empty answers.
  • F.1 Benchmarking Task Prompts: The out-of-the-box frontier/open-model benchmark requires explanations, but Linguini checks found no substantial performance impact for top-scoring human-evaluation models.The IOL prompt was slightly modified to allow JSON answers.
  • F.2 Baseline System Prompts: The four organizer baselines use a chat format with a separate system prompt, with {context} and {query} filled per test item.These baseline prompts differ from the single task prompts used above.
  • G Grading of AI Solutions by the IOL Jury: 1–3 jury members from each IOL grading group remotely graded AI solutions during the week after the event.This remote process differed from in-person IOL grading because of event time constraints.
  • G Grading of AI Solutions by the IOL Jury: Each anonymized AI paper was graded individually using the same criterion-level schemes as IOL, with discrepancies compared across jurors.After discrepancies were resolved, scores for the 5 problems were summed into one total score out of 100 per submission.

H Submissions

Most submissions were individual, back-loaded, and successfully scored, while leading pipelines relied on minimal prompting and repeated sampling. Model choice was highly concentrated on Qwen2.5-14B-Instruct-AWQ, which fit the competition’s T4 and time constraints.

  • Submission Statistics: 590 (80.7%) submissions ran successfully and received a score, while 141 failed across four identified failure families.Submissions were back-loaded, with 43% arriving in the final 48 hours; only four teams of two were formed.
  • Leading Pipelines: The seven highest-ranked entries reused the minimal system prompt verbatim and generally reduced repetition penalties.Sampling entries used up to 8 or 24 retries at temperature 0.5, accepting alternatives only when multiple samples agreed.
  • Submission Model: Nine of the top ten entries ran a 14B Qwen model, including eight using Qwen2.5-14B-Instruct-AWQ and one using Qwen3-14B.Across 42 teams with scored final submissions, 29 selected Qwen2.5-14B-Instruct-AWQ, the largest model fitting the T4 within the time limit.

I Evaluation Hyper-parameters

Frontier and open models were benchmarked using configurations drawn from model-card recommendations, with reasoning effort and token limits set as high as possible.

  • I Evaluation Hyper-parameters: Evaluation used model-card-recommended temperature and top-p settings, while maximizing reasoning effort and token limits.These configurations are reported in Table 11 for frontier and open-model benchmarking.

J Linguini Scores · K Knowledge Probing Details

The Linguini evaluation reports automatic benchmark results alongside documented evaluation settings and format-compliance caveats. Knowledge probing converts each problem’s language context into human-verified multiple-choice tasks covering contextual information and lexical items.

  • J Linguini Scores: Table 12 reports automatic evaluation results for the Linguini benchmark.The benchmark is attributed to Sánchez et al. (2025).
  • J Linguini Scores: The evaluation uses self-hosted models served with vLLM, with “Max Tokens” denoting generation budget rather than context-window size.These settings are summarized in Table 11.
  • J Linguini Scores: GLM 4.7 Flash requires repetition_penalty=1.05 to avoid repetition loops and otherwise exhibits low format-compliance and near-zero scores.The caveat is marked with † in the evaluation hyper-parameters.
  • K Knowledge Probing Details: Each problem’s language context is analyzed with an LLM to extract contextual information or lexical items from the problem language.The extracted information is used to create probing tasks.
  • K Knowledge Probing Details: 33 questions concern language context, while 40 concern lexic.All tasks and answer options were verified by a human annotator.
  • K Knowledge Probing Details: Human annotators edited alternative response options to prevent the correct answer from becoming too obvious.The annotators verified both tasks and answer options.
  • K Knowledge Probing Details: The probing tasks use 4-way multiple-choice questions over language-context meta-facts and attested lexical forms, with scrambled or other-language distractors.These task formats and distractor types are illustrated in Table 13.
Loading 2608.18011v1…