PaperlayerView exact revision on arXiv

Source-linked AI summary

NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap

Dasol Choi, Joonyong Park, Daegon Yu, Soo Yong Kim, Youngsook Song, Seunghyeok Hong

arXiv:2608.04397v1cs.CL

TL;DR

Because multilingual evaluations can collapse distinct Korean failures into one performance gap, this paper introduces NOLLI, a behaviorally calibrated English–Korean puzzle benchmark spanning matched, jamo-adapted, and Korean-specific tasks. It finds negligible matched-translation effects but sharper writing-system gaps, with Korean Cipher gaps reaching 68.7 pp while Cryptarithmetic over the same jamo shows no systematic penalty.

  • Problem

    Multilingual evaluations can collapse language-specific failures into a single performance gap, leaving unclear what Korean underperformance reflects.

    Source: §1 · p. 1

  • Method

    NOLLI uses a procedurally generated, behaviorally calibrated English–Korean puzzle suite spanning matched translations, jamo adaptations, and Korean-specific tasks.

    Source: §1 · p. 1

  • Results

    68.7 pp is the maximum Korean Cipher gap; matched translations show negligible effects, Cryptarithmetic over the same jamo shows no systematic penalty, and Korean-only rule deficits vary while Kinship deficits are positive.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Script-adaptation and Korean-only contrasts can diagnose candidate writing-system and Korean-specific bottlenecks, but they are not causal comparisons.

    Source: §1 · p. 1

    The sub-syllabic mechanism remains correlational; without per-model jamo-merging measures or an English character-level control, Korean-specific bottlenecks cannot be distinguished from general subword-manipulation weakness.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution, and scored deterministically. Rather than equating harder with bigger, we calibrate difficulty behaviorally, tuning each generator until a fixed reference model lands in target accuracy bands. Its three-level design combines matched direct translations, script adaptations over Hangul jamo (sub-syllabic letters), and Korean-only tasks grounded in Korean culture or orthography. We evaluate 15 frontier, open-weight, and Korean-developed models; among the 12 above a 3% overall-accuracy floor, matched English-Korean accuracy is statistically equivalent within a +/- 10 pp margin (TOST), suggesting little cost from presentation language alone. Writing-system-intensive tasks show sharper gaps: Korean Cipher falls behind English by up to 68.7 pp, whereas Cryptarithmetic over the same jamo shows no systematic penalty, and Jamo Composition accuracy predicts Korean Cipher accuracy. These contrasts are diagnostic rather than causal, consistent with difficulty in multi-step sub-syllabic execution. Korean-only tasks separate rule-application deficits, which vary in sign, from a Kinship deficit positive in all 12. Finally, a salient size measure fails to grow from Easy to Hard in 7 of 15 types, making structural size an unreliable proxy for empirical difficulty.

1 Introduction

NOLLI is a procedurally generated English–Korean puzzle benchmark designed to distinguish presentation-language effects from candidate writing-system and Korean-specific bottlenecks. Its results show little matched-language cost but sharper gaps on writing-system-intensive tasks and Korean-specific tasks.

1 Introduction

  • NOLLI contains 15 puzzle types and 25 tasks, with seed-regenerable instances verified to have unique solutions and difficulty calibrated to target accuracy bands.The benchmark provides deterministic exact-match evaluation and generators, instances, and evaluators; its Easy-to-Hard ordering transfers broadly across models of varying capability.Source: §1 · pp. 1–2
  • NOLLI separates matched direct translations from Hangul-jamo script adaptations and Korean-only tasks, treating only the first comparison as an estimate of presentation-language effects.Script-adaptation and Korean-only contrasts localize candidate bottlenecks but are diagnostic rather than causal; Figure 1 illustrates the levels with Sudoku, Cipher, and Kinship.Source: §1 · p. 1, Figure 1 · p. 2
  • Among 12 models above a 3% overall-accuracy floor, direct-translation gaps span −2.7 to +5.0 pp and are equivalent within ±10 pp; Korean Cipher gaps reach 68.7 pp.Cryptarithmetic over the same jamo shows no systematic penalty, while Jamo Composition predicts Korean Cipher accuracy (R2 = 0.91, n=12 models), consistent with multi-step sub-syllabic execution difficulty.Source: §1 · p. 1
  • Korean-only tasks show rule-application deficits that vary in sign and a positive, non-monotonic Kinship deficit; both deficits persist in at least some frontier models.Source: §1 · p. 2
  • Difficulty arises through size scaling, constraint density, distractor density, and procedural depth, making structural size alone an unreliable proxy for empirical difficulty.Source: §1 · p. 2

2 Related Work

NOLLI addresses gaps in static or structurally sized reasoning benchmarks with difficulty calibrated to measured model performance and tasks spanning translations, script adaptations, and Korean-only items.

2 Related Work

  • Procedural benchmarks mitigate contamination and saturation risks of static GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), but usually define difficulty structurally rather than behaviorally.Examples include SAT and program-verified puzzles (Wei et al., 2025; Zhu et al., 2025).Source: §2 · p. 2
  • IRT estimates response-based difficulty (Vania et al., 2021) post hoc, while Multilingual Reasoning Gym (Dobler et al., 2026) offers 14-language tasks but not script-adapted or language-exclusive tasks.Its parallel tasks are verifiable and have adjustable difficulty.Source: §2 · p. 2
  • Prior multilingual evaluations favor static translations or knowledge-oriented items, while Korean multi-step benchmarks remain fixed in size and difficulty.Character-level weaknesses are commonly linked to subword tokenization, but prior work rarely distinguishes sub-syllabic representation from multi-step manipulation over jamo.Source: §2 · p. 2
  • NOLLI combines accuracy-calibrated generators with matched translations, script adaptations, and Korean-only tasks to estimate presentation-language effects and localize candidate bottlenecks without additive causal claims.The design targets writing-system and Korean-specific bottlenecks while avoiding treating them as additive causal effects.Source: §2 · pp. 2–3

3 Benchmark Design

NOLLI organizes 15 puzzle types into matched translations, Hangul-jamo script adaptations, and Korean-only tasks, with procedural generation and deterministic scoring. Difficulty is calibrated by reference-model accuracy rather than assumed structural size.

3.1 Terminology and Scale

  • The benchmark has 15 puzzle types: 10 available in both English and Korean and five Korean-only, yielding 25 language-specific tasks with 100 instances in each of three tiers.A puzzle type is a problem family with a shared generator and scoring rule; a task is a type in one language.Source: §3.1 · p. 3

3.2 A Three-Level Cross-Lingual Design

  • Eight types use distribution-matched English and Korean translations, while two Hangul-jamo adaptations and five Korean-only tasks provide diagnostic, not causal, contrasts.Direct translations share generation algorithms and parameter settings with author-written language-specific templates; adaptations are not translation-equivalent, and Korean-only comparisons hold presentation language fixed but remain confounded by task family and procedure. The adaptations are Cipher, which decomposes, indexes, and recomposes jamo, and Cryptarithmetic, which maps jamo as opaque symbols; Korean-only tasks are Kinship, Saju, Time, Korean Units, and orthographic Jamo Composition.Source: §3.2 · p. 3

3.3 Task Suite

  • Across five reasoning paradigms, the suite uses narrow answer formats scored by exact match and includes task-specific safeguards against shortcuts.Korean Units provides a randomized conversion table to test table-based arithmetic, while Yacht Dice requires a globally optimal category assignment rather than greedy allocation.Source: §3.3 · p. 3

3.4 Procedural Generation and Verification

  • Every generated instance has a verified unique solution and deterministic exact-match evaluation, with no LLM judge used in scoring.Candidates without unique answers are rejected, and released items include solver-generated reference traces; evaluators normalize and canonicalize outputs before comparison with the gold answer.Source: §3.4 · pp. 3–4

3.5 Difficulty Calibration by Target Accuracy Bands

  • Easy, Medium, and Hard are defined by a fixed reference model’s accuracy, with generator parameters tuned to target bands and at least 10 pp separation between adjacent tiers.Gemini 3 Flash at medium reasoning effort was calibrated to 75 ± 10% (Easy), 50 ± 10% (Medium), and 25 ± 10% (Hard), using 100 instances per tier. Direct-translation settings were calibrated on English and mirrored to Korean, while script adaptations were calibrated independently by language and Korean-only tasks in Korean.Source: §3.5 · p. 4, Figure 2 · p. 4
  • 72 of 75 task–tier combinations met their target accuracy bands; all 51 directly calibrated tiers were in band and separated by at least 10 pp.The three misses were mirrored Korean tiers, each 2–7 pp above its upper boundary: SAT Puzzle Easy 87%, Yacht Dice Hard 38%, and Number Baseball Hard 42%. Script-adaptation reference accuracies averaged 48.7% EN vs. 55.0% KO for Cryptarithmetic and 51.0% EN vs. 54.7% KO for Cipher.Source: §3.5 · pp. 4–5
  • In a held-out regeneration check, all eight language-specific task pairs preserved Easy > Medium > Hard ordering, and 20 of 24 task–tier cells stayed within target bands.The four exceptions missed their bands by at most 7 pp.Source: §3.5 · p. 5

4 Experimental Setup

The experiments evaluate 15 models across frontier API, international open-weight, and Korean-developed groups using HRET and model-supported inference configurations. Difficulty ordering transfers across models: accuracy decreases from Easy to Hard for most non-floor model–task pairs.

Models

  • The evaluation covers 15 models in three groups: frontier API, international open-weight, and Korean-developed models.Examples include Claude Opus 4.8, Qwen3.5-397B-A17B, and EXAONE-4.0-32B.Source: §4 · p. 5

Implementation Details

  • All models are evaluated with HRET, using native APIs for proprietary models and vLLM or OpenRouter for open-weight models.The output budget is 32,768 tokens, reduced to 14,336 for Mi:dm-2.0 and EXAONE-3.5, the two 32k-context models.Source: §4 · p. 5

Models

  • Accuracy decreases from Easy to Hard for 255 of 300 non-floor model–task pairs (85%; 94% with ties).The difficulty bands are anchored to gemini-3-flash, which threads them by construction.Source: Figure 3 · p. 6

5 Results and Analysis

Across 12 non-floor models, calibrated difficulty tiers transfer broadly, and matched English–Korean performance is equivalent within the benchmark’s ±10 pp margin. Larger gaps appear in Korean Cipher and Kinship, while the results show that structural size alone is an unreliable proxy for empirical difficulty.

5.1 Overall Performance

  • GPT-5.5 (84.9%), Opus 4.8 (78.3%), and Gemini 3.1 (73.6%) lead macro accuracy; accuracy decreases Easy-to-Hard for 255 of 300 model–task pairs (85%).The tier ordering transfers broadly among the 12 non-floor models, reaching 94% when ties are included.Source: §5.1 · p. 6

5.2 Cross-Lingual Diagnostic Analysis

  • English–Korean gaps range from −2.7 to +5.0 pp across 12 non-floor models, and Holm-corrected TOSTs support equivalence within the ±10 pp margin.On three tasks with item-level correspondence, paired TOST reproduces equivalence; nominal McNemar differences for four models are all ≤4.8 pp and within the equivalence margin.Source: §5.2 · p. 6
  • 74.3% EN vs. 5.7% KO (+68.7 pp) is DeepSeek-V4-Flash’s largest Cipher gap, while Cryptarithmetic shows no systematic Korean penalty.Every non-frontier model with nontrivial English Cipher accuracy loses at least 21 pp; four score 21.3–55.7% in English but below 1% in Korean. Jamo Composition accuracy predicts Korean Cipher accuracy (R2 = 0.91; partial r = 0.87, p < .01), consistent with—but not proving—difficulty in multi-step sub-syllabic execution.Source: §5.2 · pp. 6–7
  • −11.6 pp is GPT-5.5’s rule-application surplus, while the Kinship deficit is positive in all 12 non-floor models and does not narrow with overall benchmark performance.Rule-application deficits vary in sign; the Kinship deficit has no evidence of narrowing (r=0.16, p=.62, n=12). GPT-5.5 has a +36.7 pp Kinship deficit, compared with +4.7 pp for Gemini 3.1; these are descriptive contrasts, not isolated cultural-knowledge effects.Source: §5.2 · p. 7

5.3 Where Difficulty Comes From

  • 7 of 15 puzzle types show no growth in a salient size measure from Easy to Hard, although empirical difficulty rises.A salient size measure grows in 8 types; in five others difficulty rises without monotonic size growth, and in two a salient length measure decreases.Source: Table 3 · p. 7, §5.3 · p. 7

5.3.1 Taxonomy of Escalation Levers

  • Four non-exclusive difficulty levers are size scaling, constraint and information structure, distractor density, and algorithmic or task-type depth.Examples include increasing variable counts, withholding clues, adding irrelevant columns, and shifting Saju or Time toward unassisted calendar arithmetic.Source: §5.3.1 · p. 8

5.3.2 Counterexamples: Input Size Is Not Sufficient

  • Kinship, Cipher, Saju, and Time become harder despite shorter or unchanged salient lengths, as distractors, algorithmic indirection, or task-mixture shifts alter task demands.Kinship’s chain length falls from 4–5 to 2–3 hops as dialogue grows from 15 to 112 utterances; Cipher plaintexts shrink from 20–24 to 6–10 characters while adding transposition. Saju accuracy falls from 79% to 23%, and Time from 77% to 19%, as tasks shift toward unassisted conversions. These contrasts motivate calibrating difficulty against measured model performance.Source: §5.3.2 · p. 8

5.4 Error Analysis: How Cipher Fails

  • Cipher failures from models strong at Jamo Composition retain most plaintext jamo, while weak models remain near the unrelated-text baseline, linking sub-syllabic competence to both success and failure patterns.Similarity correlates with Jamo Composition accuracy (r=0.75, p < .01); high scorers’ errors are not clean single-step slips, as re-segmentation and constant index shifts yield zero matches.Source: §5.4 · p. 8

6 Conclusion

NOLLI is a calibrated English–Korean puzzle benchmark whose results point to writing-system and Korean-specific challenges rather than a general cost from presentation language. Its difficulty ordering transfers across models even though a salient size measure often fails to increase with difficulty.

6 Conclusion

  • NOLLI provides 25 English–Korean tasks with verified unique solutions and calibrated difficulty tiers.Source: §6 · p. 8
  • Matched translations show negligible language effects, but Korean Cipher drops by up to 68.7 pp while same-jamo Cryptarithmetic shows none; Jamo Composition predicts Cipher accuracy.These contrasts are diagnostic rather than causal and are consistent with a multi-step sub-syllabic execution bottleneck.Source: §6 · pp. 8–9
  • Korean-specific tasks show rule-application deficits that vary in sign and a positive, non-monotonic Kinship deficit, with both deficits persisting in some frontier models.Source: §6 · p. 9
  • Difficulty ordering transfers across models, while a salient size measure fails to grow from Easy to Hard in 7 of 15 types.Source: §6 · p. 9

Limitations

The study’s conclusions are limited by calibration and template assumptions, correlational evidence about jamo processing, and confounded Korean-only contrasts. They apply to exact-match puzzle solving in one language pair under specified configurations, with equivalence defined relative to a ±10 pp margin.

Calibration and regeneration

  • Difficulty tiers use one reference model at one reasoning-effort setting, and full-suite calibration stability remains unverified.A held-out check on six task types preserves tier ordering and most band memberships; Cipher and Cryptarithmetic are calibrated independently by language.Source: Limitations · p. 9

Template equivalence

  • Presentation-language estimates assume semantic equivalence between author-written English and Korean templates, which template-level validation cannot establish.Validation cannot exclude differences in phrasing or interpretation.Source: Limitations · p. 9

Sub-syllabic mechanism

  • The Jamo Composition–Cipher relationship across 12 models is correlational and cannot distinguish a Korean-specific bottleneck from general subword-manipulation weakness.Both tasks operate on jamo; the study does not measure per-model jamo merging or include an English character-level control.Source: Limitations · p. 9

Korean-only contrasts

  • Korean-only task contrasts are diagnostic rather than causal because tasks differ from the Direct-KO baseline in family, answer space, context length, and procedural demands.Kinship is the only cultural-lexicon task, limiting generalization to Korean cultural knowledge more broadly.Source: Limitations · p. 9

Scope

  • Conclusions concern exact-match puzzle solving in one language pair under Appendix F configurations, without partial credit; ±10 pp equivalence does not imply equal accuracy.Equivalence is relative to the study’s ±10 pp margin.Source: Limitations · p. 9

A Task Specifications … A.3 Korean-Only Tasks

The appendix specifies the tasks, answer formats, difficulty levers, and representative instances for all 15 puzzle types. It distinguishes direct English-Korean translations from Hangul-jamo adaptations and Korean-only tasks.

A Task Specifications

  • The appendix defines each puzzle type’s task, answer format, difficulty levers, and representative instance; per-tier lever settings and induced accuracies are reported in Table 4 and Appendix C.Released instances are seed-regenerable; the appendix notes that examples may abbreviate inputs and omit evaluation-time prompts.Source: A Task Specifications · p. 11

A.1 Direct Translations

  • All eight direct-translation types use the same algorithm and parameters in English and Korean, with prompt language as the only difference.They include table filtering and aggregation (Array Formula), event propagation (Causal DAG), Latin-square inequalities (Inequality), and mine deduction (Minesweeper).Source: §A.1 · pp. 11–12
  • The other direct-translation tasks test multi-clue number deduction, Boolean satisfiability, Sudoku completion with queried-cell extraction, and optimal Yacht Dice scoring.Their specifications describe distinct levers including digit count and hints, variables and clauses, Sudoku givens, and queried rounds and score-gap bands.Source: §A.1 · p. 13

A.2 Script Adaptations

  • Cipher and Cryptarithmetic exist in both languages, but their Korean forms adapt the tasks to Hangul’s compositional jamo rather than translating them directly.Cipher applies specified operations to Roman letters or jamo indices; Korean instances can require extracting a keyword and chaining jamo-level transformations.Source: §A.2 · pp. 13–14
  • Cryptarithmetic maps letters or jamo to digits under independent group mappings and arithmetic constraints, with operand length and minimum carries as difficulty levers.The operand count is fixed at four in both languages; Korean syllable values concatenate the digits for their jamo components.Source: §A.2 · p. 14

A.3 Korean-Only Tasks

  • Korean-only tasks cover kinship-term reference resolution, traditional hour-pillar derivation, and relative-date arithmetic that may require computing the sexagenary day cycle.Kinship difficulty levers include distractor utterances and relation-chain length; Time varies the share of instances requiring sexagenary-cycle computation.Source: §A.3 · pp. 14–15
  • The remaining Korean-only tasks test Hangul jamo decomposition and recomposition, and multi-step conversion arithmetic using a self-contained table of traditional Korean units.Jamo Composition shifts initial consonants while preserving medial and final components; Korean Units uses supplied conversions to scale, sign, and sum quantities rather than requiring recall of real-world unit values.Source: §A.3 · p. 15

B Construction of Direct-Translation Tasks · C Calibration Details

Direct-translation tasks use matched English and Korean renderings of language-independent generators, while calibration places all directly calibrated tiers within target bands. A cross-model TOST found no systematic accuracy difference beyond the prespecified ±10 pp margin.

B Construction of Direct-Translation Tasks

  • The eight direct-translation types use language-independent generators and author-written English and Korean templates, with no machine-translation service or LLM during generation.Linguistic validation was conducted at the template level because each language-specific template is reused across instances.Source: B Construction of Direct-Translation… · p. 15
  • Language conditions either render the same latent instance for three item-paired types or sample independently from the same generator and parameter configuration, matching all eight in distribution.Source: B Construction of Direct-Translation… · p. 15
  • The cross-model TOST found no systematic accuracy difference exceeding the prespecified ±10 pp margin, supporting no substantial difficulty shift but not replacing linguistic validation.Source: B Construction of Direct-Translation… · p. 15

C Calibration Details

  • All 51 directly calibrated tiers fall within their target bands, with adjacent tiers separated by at least 10 pp.Table 5 reports reference-model accuracy for all 75 task–tier combinations.Source: C Calibration Details · p. 15
  • Of the 24 mirrored Korean tiers, three exceed their target bands by 2–7 pp because they inherit English-calibrated configurations without re-tuning.Per-tier generator parameters are listed in Table 4.Source: C Calibration Details · p. 15

D Held-Out Regeneration Check

A held-out regeneration check found that calibrated difficulty largely persisted across fresh seeds for six representative puzzle types, though the results do not establish full-suite stability.

D Held-Out Regeneration Check

  • The check covered six representative puzzle types across eight language-specific tasks, so full-suite stability remains untested.Types included Cipher and Cryptarithmetic in both languages, English Array Formula and Causal DAG, and Korean Jamo Composition and Time.Source: D Held-Out Regeneration Check · pp. 15–16
  • In the representative subsample, all 16 adjacent-tier comparisons had Easy–Medium and Medium–Hard gaps of at least 10 percentage points, preserving Easy > Medium > Hard ordering.The evaluated cells had n = 50 each; full-suite stability remains untested.Source: D Held-Out Regeneration Check · p. 16
  • Twenty of 24 task–tier cells (83%) remained within target bands; the four exceptions missed their nearest boundary by only 1–7 pp.Given 50 instances per cell, the modest shifts were plausibly attributable to sampling variation.Source: D Held-Out Regeneration Check · p. 16

E Equivalence Testing Details · F Model Configurations

All 12 non-floor models meet the ±10 pp equivalence criterion for matched English-Korean accuracy, including under paired testing on three shared-seed tasks. Four models show statistically detectable differences, but each remains within the equivalence margin; the configurations section summarizes inference-model groupings and identifies the calibration reference.

E Equivalence Testing Details

  • The analysis applies two one-sided tests (TOST) for independent samples using a ±10 pp equivalence margin.The margin matches the tolerance of the difficulty bands.Source: E Equivalence Testing Details · p. 16
  • All 12 non-floor models are equivalent within ±10 pp; the maximum Holm-corrected p-value is 3.1×10−4.The equivalence tests use two independent samples.Source: E Equivalence Testing Details · p. 16
  • Paired TOST on Causal DAG, Inequality, and Minesweeper (900 paired items per model) reproduces equivalence for all 12 models, with maximum Holm-corrected p = 2.5 × 10−3.These three tasks use identical random seeds across Korean and English items.Source: E Equivalence Testing Details · p. 16
  • The reported comparisons are statistically distinguishable (all p < .05), yet each effect is at most 4.8 percentage points and falls well within the equivalence margin, indicating practical equivalence.Source: E Equivalence Testing Details · p. 16

F Model Configurations

  • Inference configurations are summarized by model category, with Gemini 3 Flash listed separately as the calibration reference only.Source: F Model Configurations · p. 16

G Full Per-Task Results

The section reports per-task accuracy across the benchmark and shows that presentation-language equivalence holds for all 12 non-floor models, despite some detectable point differences.

G Full Per-Task Results

  • TOST found equivalence within the ±10 pp margin for all 12 non-floor models across eight Direct tasks (max p=3.1×10−4) and three seed-paired tasks (max p=2.5×10−3).The p-values are Holm-corrected across the 12 non-floor models.Source: Table 7 · p. 17
  • Paired McNemar tests detected point differences for four models, but every effect was ≤4.8 pp and therefore within the ±10 pp equivalence margin.Source: Table 7 · p. 17
  • Table 9 reports accuracy for every task, language, difficulty tier, and evaluated model, including an Opus 4.8 Kinship tier inversion (59/54/77).Its macro average is the per-task macro average over all 25 task-languages.Source: G Full Per-Task Results · p. 16, Table 9 · p. 18

Related papers

Loading 2608.04397v1…