CulturalRiddles: A Multicultural Benchmark of Riddles

1 Introduction

• Whether models adequately represent cultures worldwide remains open, while existing cultural benchmarks mostly assess cultural knowledge rather than its application to contextual inference.Prior cultural-riddle benchmarks also have limited linguistic and geographic coverage, data validation, and scalable human-informed evaluation.Source: §1 · pp. 2–3
• Cultural riddles test whether models combine cultural clues and knowledge through compositional and abductive reasoning, beyond factual recall.Solvers weigh alternative associations, combine clues coherently, and infer the answer that best explains them.Source: §1 · p. 3
• CULTURALRIDDLES contains 6,191 original riddles from 62 communities in 51 languages and 45 countries, with native-speaker verification and evaluation of model answers.Native speakers labeled 68,067 unique answers across 57 communities and 48 languages, covering 59.10% of model responses.Source: §1 · p. 4
• The best model reaches 79.4% overall accuracy, but no model exceeds 22% in Lumasaba and Mongondow communities; translation closes only about one-fifth of the hardest communities’ gap.Exact-match scoring also misses valid answers and changes model rankings.Source: §1 · p. 4

Cultural Benchmarks

• Most cultural benchmarks test knowledge, norms, values, or bias, often through closed-form recognition or recall that says little about applying cultural knowledge in reasoning.Some combine cultural knowledge with multi-step reasoning, but human evaluation in cultural QA has generally covered only 120–700 answers across fewer than 10 languages.Source: §2 · p. 5

Riddle Benchmarks

• General riddle benchmarks predominantly use English puzzles and rarely require cultural knowledge, while cultural-riddle benchmarks span only one to three languages per benchmark.Existing cultural-riddle approaches rely on multiple choice, unvalidated LLM judges, or human scoring alone, and cover few topics and model families.Source: §2 · p. 5
• CULTURALRIDDLES scales Boczka (2026)’s one-language pilot to 62 communities, with human and validated LLM-judge evaluation.Existing benchmarks also vary widely in coverage and in how much content is written and evaluated by humans.Source: §2 · p. 5

3 CulturalRiddles Benchmark

• The benchmark contains 6,191 original riddles written by 167 native speakers for 62 national-linguistic communities, targeting 100 riddles per community across shared and community-chosen topics.Eight riddles were removed for harmful or discriminatory content, and one was excluded from human evaluation for using the wrong language.Source: §3 · p. 5, §3.1 · p. 5

3.1 Participatory Benchmark Construction

• The working unit is a national-linguistic community, an imperfect but scalable basis for data collection and coverage reporting; recruitment relied on volunteers identifying with a community.Over six months, 167 writers contributed, typically one or two per community.Source: §3.1 · p. 5
• Each riddle pairs a topic and indirect riddle with a short local-language reference answer; optional hints and comments may accompany it, and English translations are reading aids only.Translations may lose information needed for solving, and some concepts have no English equivalent.Source: §3.1 · p. 6
• Writers created riddles from scratch within shared topics, using indirect clues and community-recognizable language to make solving require more than a lookup.In a survey, 97.4% reported drawing on personal knowledge and experience; 46.1% tested riddles on AI tools without revealing answers.Source: §3.1 · p. 6
• A native speaker other than the author verifies every riddle for solvability, unambiguous clues, and a correct reference answer.Verification does not require every community member to solve a riddle or agree on cultural “truth”; some riddles rely on generational, regional, or interest-specific knowledge.Source: §3.1 · p. 6

3.2 Contents and Coverage

• Coverage includes languages absent from prior cultural benchmarks and underrepresented locales, with multiple languages in some countries and multiple countries represented for some languages.Seven languages appear in several countries, seven countries contribute several languages, and Arabic appears in five varieties.Source: §3.2 · pp. 6–7
• Each community could choose two wildcard topics alongside eight shared topics; Food was the most common community-chosen theme at 27.4%.Politics was rated the hardest shared topic to write at 3.41/5, while Everyday Life was the easiest at 2.37/5.Source: §3.2 · p. 7
• Riddle forms were not fixed, and communities often favored distinct forms, including first-person questions in Luganda and short “What is it?” questions in Japanese.Some riddles draw on local traditions or collective memories, so solving can require familiarity with how an event is remembered, not only the event itself.Source: Figure 3 · p. 7, §3.2 · p. 8

4.1 Models

• Each configuration answers 6,090 riddles across 61 communities once, using a shared English prompt with country and topic context.The configurations range from 0.8B to 1.6T disclosed parameters and yield 170,520 responses.Source: §4.1 · p. 8

4.2 Evaluation

• Accuracy is the share of scored responses receiving score 3, while the four-point scale ranges from incorrect (0) to correct (3).The evaluation also defines 13 response types for finer distinctions within score categories.Source: §4.2 · p. 8
• Responses are scored by rules, native speakers, or a human-validated LLM jury, allowing open-ended answers beyond exact reference matches to be assessed.Exact matches receive score 3; empty outputs and placeholder copies receive 0, and explicit “I don’t know” responses receive 1.Source: §4.2 · p. 8
• Error-type counts cover responses scored by rules or human annotators, excluding responses scored only by the jury or still unscored.The dominant type within each score is highlighted in blue.Source: Table 3 · p. 9
• Community annotators label scores and error types, recognizing valid local answers; their labels cover 100,769 responses, or 59.10% of model outputs.They provide 68,067 unique labels, while a second speaker independently relabels a subset in 19 communities.Source: §4.2 · p. 10
• Two LLM judges score remaining answers with model names hidden, and only responses with matching 0–3 scores count as scored in this evaluation.The jury is assessed on doubly annotated pairs, human-labeled pairs, and pairs without valid human labels; unresolved responses are excluded from accuracy computation.Source: §4.2 · p. 10

5.1 LLM judge agreement approaches human agreement

• Human annotators agree strongly on accuracy and 0–3 scores, but less on the 13 error types, while LLM judges approach human agreement on accuracy and scores.Human–human κ is 0.80 ± 0.04 for accuracy and quadratic-weighted κ is 0.80 ± 0.03 for scores; judge agreement is 0.79–0.81 and 0.79–0.83, respectively.Source: §5.1 · pp. 11–12, Figure 6 · p. 12

5.2 Frontier models perform well, but most models lag behind

• Gemini 3.1 Pro Preview leads overall at 79.4%, followed by GPT-5.5 at 76.3% and Claude Opus 4.7 Thinking at 66.0%.The best open model, DeepSeek V4 Pro with high reasoning, ranks fifth at 63.8%; every model other than configurations of the leaders and Gemini 2.5 Flash scores below 40%.Source: §5.2 · p. 12

5.3 Exact match misses half of correct answers

• Exact match misses about 47% of responses ultimately scored correct, and human and jury review raises accuracy by 64–166% relative to exact match for nearly every model.Exact match can exaggerate performance gaps and alter rankings; Claude Opus 4.7 Thinking moves from fifth to third after review.Source: §5.3 · p. 13

5.4 Models hallucinate rather than abstain

• Hallucinated answers are the most frequent error at 24.19% of scored responses, while only 0.80% admit not knowing.Abstention remains rare for every model, at no more than 3.81%.Source: §5.4 · p. 14

5.5 High averages hide cultural blind spots

• No model exceeds 22% accuracy on Lumasaba or Mongondow, and even Gemini 3.1 Pro and GPT-5.5 average only 43% and 39% across the eight hardest communities.Those eight communities have mean accuracy of 4–14%, while the eight easiest have means of 47–66%.Source: Figure 7 · p. 13, §5.5 · p. 14
• Accuracy falls from 37.0% for high-resource languages to 12.7% for extremely low-resource languages, yet community accuracies still vary by 8–13 percentage points within each resource level.Six of the eight hardest communities use low- or extremely low-resource languages; Swahili and Sudanese Arabic are exceptions within the hardest group.Source: §5.5 · p. 14

6.1 What makes a riddle difficult?

• About 60% of benchmark-associated variance is riddle-level, versus 24% language, 13% community within language, and 2% topic.Model identity accounts for 38% of total variance, while interactions between models and benchmark factors account for less than 2%.Source: §6.1 · p. 14, Figure 9 · p. 15
• Rare answer terms are associated with lower accuracy: riddles with very rare English reference-answer words have adjusted odds of a correct answer of 0.54 (95% CI 0.46–0.65) versus common-answer riddles.Riddle features explain only 3.1% of riddle-level variance; a Konkani answer of bebinca was confused with kuswar, the larger dish family.Source: §6.1 · pp. 14–15
• Low- and extremely low-resource languages have adjusted odds of a correct answer of 0.10 (95% CI 0.02–0.48) and 0.04 (0.01–0.15) times those of high-resource languages.The medium-resource estimate is 0.49 (95% CI 0.20–1.21) and is less certain; language resources do not explain all community differences.Source: §6.1 · p. 15, Figure 8 · p. 15
• Communities sharing a language and script still differ by 5–26 pp; Simplified Chinese averages 28.5% versus 46.3% for Chinese in Singapore, a 17.8 pp gap.Language-level averages can conceal differences between communities.Source: §6.1 · p. 16
• Translation raises accuracy from 22.1% to 32.4% (+10.3 pp, 95% CI 7.6–13.0) on five hardest communities, closing only about one-fifth of their gap to high-resource communities.The comparison used 2,049 responses scored in both conditions; Sudanese Arabic showed no detectable gain, and English responses lacked native-speaker review.Source: §6.1 · p. 16

6.2 Beyond open-ended answers: Crosswords

• Crossword structure changes reference agreement by only +1.6 points (95% CI −0.3 to 3.7) but raises consistent crossings from 53.0% to 60.1% (+7.2, 95% CI 4.8–9.6).The study used 53 text grids and compared crossword input with a condition that provided the same answer lengths.Source: §6.2 · p. 16

6.3 Effort of benchmarking new models

• Rules, prior human labels, and an LLM jury cover about 80% of leading models’ responses, but only 51% for the median model and 2–13% for the smallest models.Roughly 100–115 answers per leading model still need native review; Chinese varieties, Thai, Kannada, and Buginese were excluded because their grids could not be built reliably.Source: §6.3 · p. 16

7 Discussion & Conclusion

• The benchmark contains 6,191 original riddles from 62 communities, 51 languages, and 45 countries, with native speakers writing, verifying, and scoring answers.Human scoring recognizes valid local answers beyond the reference answer, which exact-match scoring can miss.Source: §7 · p. 17
• No model exceeds 22% accuracy in Lumasaba or Mongondow, despite the best model reaching 79.4% overall.The top three proprietary configurations each score below 50% in 6–9 of 57 human-evaluated communities; the best open configuration does so in 11.Source: §7 · p. 17
• Translation closes only about one-fifth of the hardest communities’ gap to high-resource communities, while same-language, same-script communities differ by up to 26 pp.The authors conclude that stronger multilingual ability alone is unlikely to close these gaps and that much of what models lack appears to be content from communities themselves.Source: §7 · p. 17
• The largest difficulty lies in community-shared associations, traditions, perceptions, and memories; final-answer scores cannot show whether models lack facts or misread clues.The authors estimate that native-speaker review of about 100 answers per leading model supports extending the benchmark to new models and communities.Source: §7 · p. 17

8 Ethics Statement

• Community members wrote and annotated riddles as unpaid volunteers; released data credits writers by community, omits individual attribution and contact details, and reports surveys in aggregate.After content review, 8 riddles were removed and 56 received content notes; some retained riddles refer to painful histories.Source: §8 · p. 17

A.1 Benchmark Coverage

• Table 5 identifies each benchmark community by a language and a country.Source: §A.1 · p. 27

A.2 Benchmarking Prompt and Models

• The evaluation includes open-weight and open-source models from 0.8B to 1.6T parameters, proprietary models, reasoning variants, and one search-enabled configuration.Source: §A.2 · p. 27
• The runs used about 217M tokens in total, and the authors report tokens rather than cost because hosted APIs and locally served models were mixed.Source: §A.2 · p. 27
• Each configuration answered each riddle once with one prompt, so rankings could change under prompt variations and scores might not reproduce under stochastic decoding.Source: §A.2 · p. 27

A.3 Human Evaluation Protocol

• Human scoring assigns a consistent score and error type to each unscored answer, with validation rejecting inconsistent rows and routing them to the jury.Annotators can consult community members or the author about unclear references.Source: §A.3 · p. 27
• The second native-speaker annotation covered 4,018 pairs from 1,028 test riddles, but volunteer availability left many communities, including most of the hardest ones, without double annotation.The second annotator independently relabeled 4,246 pairs from 1,074 riddles in 19 communities.Source: §A.3 · pp. 27, 30
• The guidelines distinguish hallucinated entities, answers from closely related cultures or regions, and explicit admissions of not knowing.A score of 0 requires a 0.x type, while a score of 2 requires a 2.x type.Source: §A.3 · p. 30, §A.4 · p. 31
• The 13 error types extend Boczka (2026), but annotators and judges disagree most at boundaries such as hallucinated versus too broad or wrong entity.The authors note that clearer definitions could improve reproducibility.Source: §A.3 · p. 30
• The analysis scores final answers rather than reasoning traces, so accuracy cannot identify which reasoning step failed.Correct answers can result from a misreading or lucky guess, while wrong answers can follow a sound interpretation that misses a local fact.Source: §A.3 · p. 30

A.4 LLM Judge Protocol

• The judges receive the riddle, community, topic, bilingual reference answer, candidate answer, and guidelines, and return a validated JSON verdict with score and error category.Setting B samples at least 200 human-labeled test pairs per community, or all pairs when fewer are available.Source: §A.4 · p. 30, Figure 11 · p. 31
• Hybrid agreement reaches κ = 0.92 for high-resource and 0.88 for extremely low-resource languages, while the hybrid retains human labels when judges disagree.The hybrid is not a human-free scoring pipeline.Source: §A.4 · p. 32

• Hallucinations dominate the reported errors, while abstention remains rare.Shares use rule- and human-scored responses and therefore differ from Figure 6’s accuracy population.Source: Figure 12 · p. 33
• The error-share panels use different scales, so their heights should not be compared as if they shared one scale.Rows are ordered by exact-match share.Source: Figure 12 · p. 33
• The plotted error shares exclude jury and unresolved responses, unlike the accuracy population in Figure 6.Source: Figure 12 · p. 33

B.2 Riddle and Answer Features

• Riddle identity accounts for 61% of latent variance, language 24%, community within language 13%, and topic 2% across 5,688 riddles from 57 human-evaluated communities.Source: §B.2 · p. 33
• Longer riddles have higher adjusted odds of correctness (OR 1.66 per doubling of words, 95% CI 1.47–1.88), while low-resource levels have lower odds than high-resource languages.Adjusted resource-level odds ratios are 0.49 for medium (0.20–1.21), 0.10 for low (0.02–0.48), and 0.04 for extremely low (0.01–0.15). First-person statements have a modest positive association (OR 1.29, 95% CI 1.02–1.63); intervals for question form and clue structure include 1.Source: §B.2 · pp. 33–34
• Answer rarity is based on English-reference words and serves as a lexical proxy, not a direct measure of cultural knowledge.About a third of riddles in the feature model contain a reference term with Zipf frequency below 1.5.Source: §B.2 · p. 33

B.3 Crossword Comparison Details

• Reference agreement measures the share of a puzzle’s clues matching the reference after normalizing case, spacing, and written units; missing or incomplete answers count as wrong.Puzzles are equally weighted, and intervals resample 42 languages while keeping country variants together.Source: §B.3 · p. 34

B.4 English Translation Condition

• Buginese, Lumasaba, and Yoruba gain 14–18 pp with English translation, Mongondow gains about 6 pp, and Sudanese Arabic does not gain.The high-resource reference accuracy is about 69% on different riddles; this is not a matched control, and translation may change clues.Source: §B.4 · p. 35, Figure 14 · p. 35
• The paired translation comparison pools five models’ responses scored in both conditions: 2,049 of 2,290 responses across 458 retained riddles, with confidence intervals estimated by resampling riddles.Source: §B.4 · p. 35

B.5 Workload for a New Model

• For a new model, workload depends on repeated previously labeled answers; leading models are projected to leave about 1,200–1,400 responses for the jury and 100–115 unresolved for native review.These estimates exclude additional review recommended in communities with low judge agreement.Source: §B.5 · p. 35

B.6 Agreement by Community and Answering Model

• Figure 15 shows setting-B false-acceptance rates among human-rejected responses and accuracy shifts under the agreement-or-human rule; Figure 16 displays agreement by community and resource level.The hybrid rule retains human labels on disagreements; the four communities without valid human labels are omitted from these figures and main accuracy comparisons.Source: §B.6 · p. 35, Figure 15 · p. 36, Figure 16 · p. 37