PaperlayerView exact revision on arXiv

Source-linked AI summary

NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap

Dasol Choi, Joonyong Park, Daegon Yu, Soo Yong Kim, Youngsook Song, Seunghyeok Hong

arXiv:2608.04397v1cs.CL

TL;DR

NOLLI addresses what multilingual models fail at when Korean performance lags, beyond an undifferentiated language gap. It uses calibrated English–Korean puzzles spanning translations, jamo adaptations, and Korean-specific tasks, finding negligible presentation-language effects but sharper writing-system-specific gaps, including up to 68.7 pp on Korean Cipher.

  • Problem

    Multilingual evaluations often collapse language-specific failures into one performance gap, leaving unclear what models fail at when they perform worse in Korean.

  • Method

    NOLLI uses procedurally generated, behaviorally calibrated English–Korean puzzles across matched translations, jamo adaptations, and Korean-specific tasks.

  • Results

    Matched translations show negligible presentation-language effects, while writing-system demands produce gaps up to 68.7 pp on Korean Cipher and no systematic penalty on jamo Cryptarithmetic.

  • Takeaways & Limitations

    The contrasts diagnose candidate sub-syllabic execution and Korean-specific bottlenecks rather than establishing causal explanations for the observed gaps.

  • Takeaways & Limitations

    Difficulty tiers rely on one reference model and reasoning setting, so absolute difficulty-band membership may not transfer and full-suite stability remains unverified.

Abstract

from arXiv · show

We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution, and scored deterministically. Rather than equating harder with bigger, we calibrate difficulty behaviorally, tuning each generator until a fixed reference model lands in target accuracy bands. Its three-level design combines matched direct translations, script adaptations over Hangul jamo (sub-syllabic letters), and Korean-only tasks grounded in Korean culture or orthography. We evaluate 15 frontier, open-weight, and Korean-developed models; among the 12 above a 3% overall-accuracy floor, matched English-Korean accuracy is statistically equivalent within a +/- 10 pp margin (TOST), suggesting little cost from presentation language alone. Writing-system-intensive tasks show sharper gaps: Korean Cipher falls behind English by up to 68.7 pp, whereas Cryptarithmetic over the same jamo shows no systematic penalty, and Jamo Composition accuracy predicts Korean Cipher accuracy. These contrasts are diagnostic rather than causal, consistent with difficulty in multi-step sub-syllabic execution. Korean-only tasks separate rule-application deficits, which vary in sign, from a Kinship deficit positive in all 12. Finally, a salient size measure fails to grow from Easy to Hard in 7 of 15 types, making structural size an unreliable proxy for empirical difficulty.

1 Introduction

NOLLI is a procedurally generated English–Korean puzzle benchmark designed to diagnose whether Korean performance gaps arise from presentation language, writing-system processing, or Korean-specific reasoning. Its calibrated, deterministic design reveals little matched-language cost but sharper and task-specific writing-system and Korean-only deficits.

  • Benchmark design: NOLLI contains 15 puzzle types across 25 tasks and 7,500 items, with seed-regenerable instances, verified unique solutions, and deterministic scoring.Its generators are calibrated to target accuracy bands on a fixed reference model rather than defining difficulty through structural size alone.
  • Diagnostic design: Its three-level spectrum separates matched direct translations, Hangul-jamo script adaptations, and Korean-only tasks grounded in Korean culture or orthography.Only direct translations estimate presentation-language effects; the other comparisons localize candidate writing-system and Korean-specific bottlenecks.
  • Main findings: −2.7 to +5.0 pp direct-translation gaps were equivalent within ±10 pp under Holm-corrected TOST, indicating little presentation-language cost among 12 models above the 3% floor.The evaluated models included frontier, open-weight, and Korean-developed systems.
  • Main findings: 68.7 pp Korean Cipher gaps contrasted with no systematic Cryptarithmetic penalty over the same jamo, while Jamo Composition predicted Korean Cipher accuracy.The contrast is diagnostic of task-specific writing-system-intensive difficulty rather than a uniform Korean penalty.
  • Main findings: Korean-only tasks yielded rule-application deficits varying in sign and a positive, non-monotonic Kinship deficit, with both sub-syllabic and Kinship deficits persisting in some frontier models.NOLLI’s difficulty analysis attributes performance variation to size scaling, constraint density, distractor density, and procedural depth, making structural size alone unreliable.

2 Related Work

Prior benchmarks face contamination, saturation, static difficulty, or limited multilingual and Korean coverage. NOLLI addresses these gaps by calibrating generators to measured model accuracy while separating presentation-language, writing-system, and Korean-specific factors.

  • Static reasoning benchmarks risk contamination and saturation, while procedural benchmarks mitigate these problems but generally define difficulty structurally rather than behaviorally.Examples include GSM8K and MATH, alongside procedurally generated, SAT, and program-verified puzzle benchmarks.
  • Item Response Theory estimates difficulty from model responses, but typically operates post hoc rather than calibrating benchmark generators.
  • Multilingual reasoning evaluations largely use static translations or knowledge-oriented items, while Korean multi-step benchmarks remain fixed in size and difficulty.Character-level weaknesses are commonly linked to subword tokenization, including Korean tokenization studied at the jamo level.
  • NOLLI combines accuracy-calibrated generators with matched translations, script adaptations, and Korean-only tasks to estimate presentation-language effects and localize candidate bottlenecks.The design avoids treating writing-system and Korean-specific bottlenecks as additive causal effects.

3 Benchmark Design

NOLLI organizes 15 puzzle types into 25 language-specific tasks spanning direct translations, Hangul-jamo script adaptations, and Korean-only cultural or orthographic tasks. Its procedurally generated instances use deterministic verification and scoring, while behavioral calibration targets reference-model accuracy bands for Easy, Medium, and Hard tiers and supports regeneration.

  • Benchmark organization: NOLLI contains 15 puzzle types and 25 tasks, with 10 types in both English and Korean and 5 Korean-only types, each tiered into Easy, Medium, and Hard.Each tier contains 100 generated instances.
  • Cross-lingual categories: The benchmark separates 8 direct-translation types from 2 Hangul-jamo script adaptations and 5 Korean-only tasks, making only direct translations suitable for matched language comparisons.Script adaptations and Korean-only tasks differ in content or procedure and are diagnostic rather than causal contrasts.
  • Reasoning coverage: The 15 types span five reasoning paradigms: constraint satisfaction, algorithmic execution, causal and temporal reasoning, combinatorial optimization, and deductive inference.The suite includes tasks such as Sudoku, Cipher, Saju, Yacht Dice, and Kinship across these paradigms.
  • Difficulty calibration: Calibration defines Easy, Medium, and Hard behaviorally by reference-model accuracy, targeting 75 ± 10%, 50 ± 10%, and 25 ± 10% over n=100 instances per tier.Gemini 3 Flash at medium reasoning effort served as the reference model, with generator parameters tuned iteratively for each task.
  • Calibration outcome: 72 of 75 task–tier combinations fell inside their target bands, while all 51 directly calibrated tiers were in band and adjacent tiers differed by at least 10 pp.The three misses were mirrored Korean tiers of direct-translation tasks and exceeded upper-band boundaries by 2–7 pp.
  • Regeneration: In regeneration testing, all eight language-specific task pairs preserved Easy > Medium > Hard ordering, and 20 of 24 cells (83%) remained within target bands.The four exceptions missed by at most 7 pp, showing that calibrated configurations support new-instance sampling without repeating full calibration.

4 Experimental Setup

The experiments evaluate 15 models spanning frontier APIs, international open-weight systems, and Korean-developed models under standardized HRET-based inference settings. Difficulty ordering generally transfers across models, with accuracy declining from Easy to Hard for most non-floor model–task pairs.

  • Models: 15 models are evaluated across three groups: frontier API, international open-weight, and Korean-developed models.The listed systems include Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro preview, multiple Llama, Qwen, Gemma, DeepSeek, and Korean-developed models.
  • Implementation Details: All models are evaluated with HRET, using native APIs for proprietary systems and vLLM or OpenRouter for open-weight systems.The study follows recommended sampling configurations and uses medium reasoning effort where available.
  • Implementation Details: 32,768 tokens is the standard output budget, reduced to 14,336 for Mi:dm-2.0 and EXAONE-3.5 because they use 32k-context models.Full inference configurations and exceptions are reported in Appendix F.
  • Difficulty Calibration: 85% of 300 non-floor model–task pairs show accuracy decreasing from Easy to Hard, rising to 94% when ties are included.The difficulty bands are anchored to gemini-3-flash, whose target-band performance is constructed by design.

5 Results and Analysis

Across 7,500 items, calibrated difficulty transfers broadly across models, while matched English–Korean performance is equivalent within ±10 pp. Diagnostic contrasts reveal severe Korean Cipher gaps, persistent Kinship deficits, and structural size measures that often fail to track empirical difficulty.

  • Overall Results: 85% of model–task pairs decrease from Easy to Hard, or 94% including ties, across 12 non-floor models.The tier ordering transfers beyond the reference model despite differing absolute accuracy levels across developer groups.
  • Presentation Language: English–Korean gaps on matched direct-translation tasks range from −2.7 to +5.0 pp, with TOST supporting equivalence within ±10 pp.The comparison covers eight puzzle types and 12 non-floor models under identical generator parameters, conditional on semantic template equivalence.
  • Writing System: DeepSeek-V4-Flash shows the largest Korean Cipher gap: 74.3% EN versus 5.7% KO, a +68.7 pp difference.Every non-frontier model with nontrivial English accuracy loses at least 21 pp, while only GPT-5.5 closes the gap among frontier models (+1.7 pp).
  • Writing System: Cryptarithmetic shows no systematic Korean penalty, while Jamo Composition predicts Korean Cipher accuracy with R^2 = 0.91 and partial r = 0.87.The contrast is consistent with difficulty in multi-step sub-syllabic execution, although the correlation does not establish the mechanism.
  • Korean-Specific Knowledge: The rule-application deficit varies in sign, whereas the Kinship deficit remains positive in all 12 non-floor models and does not narrow with benchmark performance.Kinship ordering reverses within the frontier group: Gemini 3.1 scores 69.0%, Opus 4.8 63.3%, and GPT-5.5 45.7%.
  • Difficulty Calibration: A salient size measure grows from Easy to Hard in only 8 of 15 puzzle types, while Kinship and Cipher become harder despite smaller structural inputs.Kinship accuracy falls from 77% to 31% as dialogue distractors rise from 15 to 112 utterances; Cipher accuracy drops 52 pp despite shorter plaintexts.

6 Conclusion

NOLLI is a 25-task English–Korean benchmark with verified unique solutions and calibrated difficulty tiers. Its conclusion finds negligible presentation-language effects but persistent deficits tied to sub-syllabic execution and Korean-specific reasoning.

  • NOLLI provides 25 tasks with verified unique solutions and calibrated difficulty tiers.
  • Up to 68.7 pp, Korean Cipher drops relative to English, while Cryptarithmetic over the same jamo shows none; Jamo Composition predicts Korean Cipher accuracy.
  • Korean-specific tasks show rule-application deficits that vary in sign and a positive, non-monotonic Kinship deficit, with both persisting in some frontier models.
  • 7 of 15 types lack growth in a salient size measure from Easy to Hard, although difficulty ordering transfers across models.

Limitations

The study’s conclusions are bounded by calibration choices, assumed template equivalence, correlational evidence for the jamo mechanism, task-confounded Korean-only contrasts, and a narrow exact-match evaluation scope.

  • Calibration and regeneration: Difficulty tiers rely on one reference model and reasoning-effort setting, and full-suite stability remains unverified.A six-task held-out check preserves tier ordering and most band memberships, but absolute bands may not transfer across models.
  • Template equivalence: Presentation-language estimates assume semantic equivalence between author-written English and Korean templates, which template validation cannot establish.Phrasing or interpretation differences may remain.
  • Sub-syllabic mechanism: The Jamo Composition–Cipher relationship across 12 models is correlational and cannot isolate a Korean-specific bottleneck.Both tasks use jamo; per-model jamo-merging measures and an English character-level control are absent.
  • Korean-only contrasts: Korean-only deficits are diagnostic rather than causal because those tasks differ from Direct-KO in family, answer space, context length, and procedural demands.Kinship is the only cultural-lexicon task, limiting generalization to Korean cultural knowledge.
  • Scope: Conclusions cover exact-match puzzle solving for one language pair under Appendix F configurations, without partial credit.The ±10 pp equivalence margin is relative and does not imply equal accuracy across languages.

A Task Specifications … A.3 Korean-Only Tasks

The appendix defines 15 procedurally generated puzzle types through task statements, answer formats, difficulty levers, and representative instances. They span faithful English-Korean translations, Hangul-jamo script adaptations, and Korean-only tasks involving kinship, calendars, orthography, and unit conversion.

  • A Task Specifications: The appendix specifies each puzzle type’s task statement, answer format, difficulty levers, and a representative released instance.Per-tier lever values and induced accuracies are provided elsewhere in the appendix.
  • A.1 Direct Translations: Eight types use faithful English-Korean translations generated by identical algorithms and parameters, differing only in prompt language.Examples include structured-data aggregation, causal propagation, Latin-square inequalities, Minesweeper, number deduction, SAT, Sudoku, and Yacht Dice.
  • A.1 Direct Translations: The direct-translation tasks cover multi-step arithmetic, graph dependency tracking, constraint satisfaction, local deduction, feedback integration, and combinatorial rule application.Difficulty levers vary by task, including table size, event count and density, grid size, mine density, digit count, variables, clauses, givens, queried rounds, and score-gap bands.
  • A.2 Script Adaptations: Two types have English and Korean variants adapted to Hangul’s compositional jamo structure rather than translated equivalently.Cipher operates over Roman-alphabet letters in English and jamo indices in Korean, while Cryptarithmetic uses alphabet or jamo groups with independent digit tables.
  • A.2 Script Adaptations: Korean Cipher requires character-level transformations over decomposed initial, medial, and final jamo, including keyword extraction and a specified operation stack.The illustrated stack is CHO_SHIFT -> JUNG_SUB -> REVERSE -> CHO_SHIFT -> JUNG_SUB -> CHO_SHIFT.
  • A.3 Korean-Only Tasks: Korean-only tasks test Korean kinship-term reasoning, sexagenary-calendar rules, Korean date arithmetic, Hangul orthographic manipulation, and table-based traditional-unit conversion.Their answers range from kinship labels and stem–branch pairs to Gregorian dates, transformed strings, and integer base-unit totals.
  • A.3 Korean-Only Tasks: Jamo Composition is a purely orthographic task that decomposes Hangul syllables, cyclically shifts initial consonants, and recomposes the syllables.Its difficulty levers are the number of syllables and final-consonant complexity.
  • A.3 Korean-Only Tasks: Korean Units requires multi-step conversion arithmetic using only a randomized, self-contained table of traditional Korean units, with scaling, signs, and accumulation.The answer is a single integer in the base unit.

B Construction of Direct-Translation Tasks · C Calibration Details

Direct-translation tasks use matched, author-rendered English and Korean conditions that preserve the same generators, constraints, and scoring logic. Calibration places all directly calibrated tiers within target bands, while TOST finds no systematic cross-language accuracy difference beyond ±10 pp.

  • B Construction of Direct-Translation Tasks: Eight direct-translation types use language-independent procedural generators with fixed author-written English and Korean rendering templates.Generation invokes neither machine translation services nor LLMs.
  • B Construction of Direct-Translation Tasks: The two language conditions are distribution-matched, either rendering the same latent instance or sampling independently from identical generator parameters.The three item-paired types render the same latent instance.
  • B Construction of Direct-Translation Tasks: Template pairs were checked for semantic correspondence, placeholder coverage, answer-format consistency, and unchanged constraints and scoring logic.These checks supplement the procedural matching of the language conditions.
  • B Construction of Direct-Translation Tasks: TOST found no systematic accuracy difference exceeding the prespecified ±10 pp margin between the two rendered conditions.The result is treated as evidence against a substantial difficulty shift, not as a substitute for linguistic validation.
  • C Calibration Details: 51 directly calibrated tiers all fall inside their target bands, with adjacent tiers separated by at least 10 pp.These tiers comprise 24 English direct-translation, 12 script-adaptation, and 15 Korean-only tiers.
  • C Calibration Details: 24 mirrored Korean direct-translation tiers inherit the English-calibrated generator configuration without re-tuning.They are evaluated alongside the directly calibrated tiers produced by the calibration protocol.

D Held-Out Regeneration Check

A held-out regeneration check tested whether calibrated difficulty persisted on fresh seeds across six representative puzzle types. All adjacent tiers remained separated, and most evaluated cells stayed within target accuracy bands while preserving the intended ordering.

  • Regeneration protocol: 50 fresh instances per tier were generated from the same calibrated configurations and evaluated with Gemini 3 Flash at medium reasoning effort across six representative puzzle types.The sample covered Cipher and Cryptarithmetic in English and Korean, Array Formula and Causal DAG in English, and Jamo Composition and an additional Korean task.
  • Difficulty separation: 16 of 16 adjacent-tier comparisons met the separation criterion, with Easy–Medium and Medium–Hard gaps of at least 10 percentage points.This confirms separation across the evaluated regenerated cells.
  • Target-band persistence: 20 of 24 evaluated cells (83%) remained within their target bands; the four exceptions missed the nearest boundary by 1–7 pp.The exceptions did not change the Easy > Medium > Hard ordering and may reflect sampling variation at n = 50 per cell.

E Equivalence Testing Details · F Model Configurations

The equivalence analysis finds matched English–Korean performance equivalent for all 12 non-floor models under both independent-sample and paired TOST procedures. Model configurations are summarized separately, with Gemini 3 Flash serving only as the calibration reference.

  • E Equivalence Testing Details: ±10 pp was the equivalence margin, matching the tolerance used for difficulty bands.The analysis used two one-sided tests for independent samples.
  • E Equivalence Testing Details: 12 non-floor models achieved equivalence, with a maximum Holm-corrected p-value of 3.1×10−4.This result comes from the independent-sample TOST analysis.
  • E Equivalence Testing Details: 12 models also reproduced equivalence in paired TOST on 900 paired items per model from three identically seeded tasks.The tasks were Causal DAG, Inequality, and Minesweeper; the maximum Holm-corrected p-value was 2.5 × 10−3.
  • E Equivalence Testing Details: Held-out regeneration accuracy was evaluated on 50 fresh instances per tier from unseen seeds using calibrated configurations and the Table 5 reference model.A dash indicates that a language condition was not included in the subsample; an asterisk marks results outside the target band by at most 7 pp.
  • E Equivalence Testing Details: ≤4.8 pp effects were statistically distinguishable in some cases but remained practically equivalent within the equivalence margin.Per-model gaps and test statistics are reported in Table 7.
  • F Model Configurations: Table 8 summarizes inference configurations by model category, while Gemini 3 Flash is listed separately as the calibration reference only.The configuration summary belongs to the Model Configurations section.

G Full Per-Task Results

The section presents accuracy for all 25 tasks across three difficulty tiers and 15 models, organized by bilingual direct translations, script adaptations, and Korean-only tasks. It also records presentation-language equivalence and non-monotone per-task behavior, including a Kinship tier inversion.

  • Per-task results: Table 9 reports accuracy (%) for every task, language, difficulty tier, and evaluated model across 25 tasks, three tiers, and 15 models.Bilingual tasks place English and Korean on adjacent rows, while horizontal rules separate direct translations, script adaptations, and Korean-only tasks.
  • Presentation-language comparisons: ±10 pp equivalence holds for every non-floor model under both two-sample and paired tests, with maximum corrected p-values of 3.1×10−4 and 2.5×10−3.The tests cover all 8 Direct tasks and the three seed-matched tasks, respectively.
  • Aggregation: Macro avg is defined as the per-task macro average over all 25 task–languages.This aggregation is reported alongside the full per-task results.
  • Difficulty behavior: 59/54/77 on Kinship is a tier inversion for Opus 4.8, among 45 non-monotone model–task pairs reported in Figure 3.The values correspond to Easy, Medium, and Hard tiers.
Loading 2608.04397v1…