Source-linked AI summary
EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges
Clinton J. Wang, Dean Lee, Cristina Menghini, Johannes Mols, Jack Doughty, Adam Khoja, Jayson Lynch, Sean Hendryx, Summer Yue, Dan Hendrycks
TL;DR
Existing puzzle benchmarks are narrow in scope, while established reasoning benchmarks use well-defined problem spaces that frontier models are rapidly saturating. ENIGMAEVAL builds a multimodal puzzle-hunt benchmark with original puzzles and human transcriptions to evaluate reasoning separately from parsing. Frontier models achieve very low accuracy, including 7.0% on normal puzzles and 0% on hard puzzles.
Problem
Existing puzzle benchmarks are often narrow or text-only, and established reasoning benchmarks use well-defined problem spaces that frontier models are rapidly saturating.
Source: §1 · p. 1Method
ENIGMAEVAL draws difficult text-and-image puzzles from puzzle hunts and provides original puzzles plus human transcriptions to evaluate end-to-end performance and reasoning abilities.
Source: §1 · pp. 1–2Results
Frontier models achieve only 7.0% accuracy on normal ENIGMAEVAL puzzles and 0% on hard puzzles.
Source: §1 · p. 2, §3.2 · p. 5Takeaways & Limitations
ENIGMAEVAL complements Humanity’s Last Exam by testing reasoning integration in creative problem-solving rather than structured academic knowledge.
Source: §1 · p. 2The benchmark is kept private and will be continuously updated because a structured release could enable solution memorization and some authors prefer limited distribution.
Source: §1 · p. 2
Abstract
from arXiv · showhide
As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce EnigmaEval, a dataset of problems and solutions derived from puzzle competitions and events that probes models' ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity -- each typically requiring teams of skilled solvers hours to days to complete -- with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity's Last Exam, unveiling models' shortcomings when challenged with problems requiring unstructured and lateral reasoning.
1. Introduction
ENIGMAEVAL addresses the need for challenging evaluation of creative reasoning on unstructured multimodal problems. The paper introduces a puzzle-hunt benchmark and reports very low frontier-model accuracy, while keeping the benchmark private to limit possible data leakage.
1. Introduction
- Existing puzzle benchmarks are often narrow in puzzle type or text-only, while established reasoning benchmarks use well-defined problem spaces that frontier models are rapidly saturating.The resulting evaluation gap concerns creative reasoning on unstructured multimodal challenges.Source: §1 · p. 1
- ENIGMAEVAL draws highly difficult, unstructured text-and-image problems from puzzle hunts and releases both original puzzles and human transcriptions.The two formats support evaluation of end-to-end capabilities and reasoning abilities.Source: §1 · p. 1
- The two-format design helps distinguish limitations in reasoning from limitations in parsing complex documents or processing different modalities.The paper notes that raw puzzle formats can substantially reduce some models’ performance, while transcribed puzzles remain challenging.Source: §1 · p. 2
- 7.0% accuracy on normal ENIGMAEVAL puzzles and 0% on hard puzzles leave frontier models far below experienced human puzzle hunters.The paper also reports that all models struggle on ENIGMAEVAL while many existing reasoning and multimodal benchmarks are largely solved.Source: §1 · p. 2, Figure 1 · p. 2
- The benchmark is kept private and will be continuously updated to reduce possible data leakage and respect some puzzle authors’ distribution preferences.The authors note that a structured dataset release could enable solution memorization.Source: §1 · p. 2
2. ENIGMAEVAL
ENIGMAEVAL contains 1,184 puzzles in original and standardized formats, drawn from diverse puzzle events. Human curation preserves puzzle structure and answers, while the collection excludes some modalities and puzzle types to support its evaluation scope.
2. ENIGMAEVAL
- ENIGMAEVAL represents 1,184 puzzles as original source PDFs or webpage screenshots and as structured text-image versions for targeted reasoning evaluation.Solutions are typically words or short phrases.Source: §2 · p. 2
2.1 Data Collection
- The dataset draws from eight online puzzle events spanning beginner-friendly puzzles through advanced competitions, with sources including PDF and HTML archives.Figure 2 samples puzzles and solutions from PuzzledPint, Labor Day Extravaganza, MIT Mystery Hunt, and CRUMS.Source: §2.1 · pp. 2, 4, Figure 2 · p. 3
- Hard puzzles typically require at least five non-trivial steps with little verification, and the sources combine text with visual elements such as grids, pictures, and diagrams.Source: §2.1 · p. 4
- The collection includes 77 meta-puzzles in the normal split but excludes puzzles relying on audio, video, or web-app interaction.Puzzles were also restricted to those with author consent or Creative Commons licenses.Source: §2.1 · p. 4
- The original and standardized formats provide separate evaluation paths to distinguish document-parsing ability from reasoning performance.The standardized format preserves semantic relationships and visual elements while reducing preprocessing load and distractors.Source: §2 · p. 2, §2.1 · p. 4
- Human annotators standardized puzzles while preserving complex layouts, removing identifying headers and footers, and reviewing puzzle and solution transcriptions for accuracy and completeness.Automating transcription with LLMs proved impractical because puzzle formats were diverse and complex.Source: §2.1 · pp. 4–5
3. Experiments
The experiments evaluate leading multimodal LLMs against verified puzzle answers, including in raw and transcribed formats. Models perform poorly overall, and results also reveal preprocessing and contamination-audit findings.
3.1 Evaluation Setup
- Evaluation compares model answers with ground-truth solutions by string matching, using format-specific prompts that request step-by-step reasoning and a standardized final answer.Meta-puzzle accuracy is measured with correct component solutions supplied to isolate meta-reasoning.Source: §3.1 · p. 5
3.2 Results
- Raw images preserve performance for some models but sharply reduce it for others, suggesting weaker OCR abilities in the latter group.Human-transcribed text-image versions are used to help isolate reasoning from visual parsing.Source: §3.2 · p. 5
- Despite publicly accessible solutions and possible pretraining exposure, low pass rates across the evaluated benchmarks suggest data contamination is not prominent.The authors conclude that performance generally reflects problem-solving capabilities despite possible training-data presence.Source: §3.2 · p. 5
- O1 reaches 7.0% accuracy on the normal split and 0% on the hard split, where all tested models score 0%.The authors describe the gap from expert-level puzzle-solving as substantial.Source: §3.2 · p. 5, Table 2 · p. 6
- Manual inspection found every case flagged by O1’s answer audit to be a false positive, suggesting the model independently reached those correct answers.The audit compared generated reasoning with official puzzle walkthroughs.Source: §3.2 · pp. 5–6
4. Related Work
Related work spans reasoning benchmarks, puzzle-solving benchmarks, and puzzle-hunt collections. ENIGMAEVAL is positioned as testing flexible reasoning across varied puzzle formats and modalities while drawing from collections with clearer licensing and organization.
4. Related Work
- Reasoning benchmarks cover mathematics, academic expertise, vision-language reasoning, and abstract pattern recognition; ENIGMAEVAL adds synthesis of disparate clues in unstructured multimodal information.The passage cites MATH, GPQA [9], FrontierMath, OlympiadBench, Humanity’s Last Exam, MMLU, MMMU, MathVista [15], VISTA, and ARC-AGI.Source: §4 · p. 6
- Existing puzzle benchmarks target varied but comparatively consistent formats, whereas ENIGMAEVAL uses puzzles that vary dramatically in format and modality.Examples include PUZZLES, PuzzlePlex, PuzzleBench [20], GridPuzzle [21], and RiddleSense.Source: §4 · p. 6
- Puzzle-hunt collections offer challenging source material, but decentralized authorship and varied licensing complicate dataset creation; ENIGMAEVAL selects collections with clear Creative Commons licenses and centralized organization.The passage names university, corporate, and community-organized hunts as source categories.Source: §4 · p. 6
A.1 Number of Images per Puzzle
The appendix notes that puzzles vary in visual content and describes how annotators converted source puzzles into solver-ready transcriptions and collected answers.
A.1 Number of Images per Puzzle
- Text-only puzzles are a small minority; most puzzles have one key visual component, though many include multiple graphics.Figure A.1 summarizes image counts in transcriptions and page counts in source PDFs.Source: §A.1 · p. 10, Figure A.1 · p. 10
A.2 Annotator Instructions
- Annotators reformatted each puzzle as text and images, preserving meaningful layout and formatting, capturing visual elements, and copying the final solution for grading.Instructions also excluded boilerplate and solution-check elements and marked puzzles requiring physical presence or audio/video as not answerable by an AI model.Source: §A.2 · pp. 10–11
A.3 Example Puzzle Transcription
The appendix presents a PuzzledPint puzzle transcription as an example. Its prompt asks solvers to pair television shows and identify what they share.
A.3 Example Puzzle Transcription
- The example asks solvers to match shows across two columns, find their shared similarity, and name the thing the narrator likes nearly as much as television.The figure caption identifies it as a PuzzledPint puzzle transcription and references the original PDF and solution.Source: §A.3 · p. 12, Figure A.2 · p. 12
B.1 Model Details
The appendix lists the evaluated model checkpoints, describes puzzle-solving prompt templates and their variants, and specifies a prompt for auditing possible plagiarism.
B.1 Model Details
- The evaluated checkpoints span OpenAI, Google, Anthropic, Mistral, and Meta models.The listed checkpoints include both general-purpose and multimodal models.Source: §B.1 · p. 12
B.2 Prompt Templates
- The default puzzle prompt asks for step-by-step solutions and a standardized final answer, with tips for exploring text, numbers, and images.Tips include trying codes and letter manipulations, rereading titles and flavor text, and inspecting images in different ways.Source: §B.2 · pp. 13–14
- Prompt variants adapt instructions for raw PDFs, text-only puzzles, meta-puzzles, MIT Mystery Hunt puzzles, and puzzles requiring multiple answers.For multi-answer puzzles, the requested answer is a comma-separated list.Source: §B.2 · p. 14
B.3 Cheating Audit Prompt
- The cheating audit prompt flags copied wording, incomplete or incorrect reasoning with a correct answer, and premature answers, then rates evidence from 0 to 3.A suspected plagiarism judgment must point to specific evidence.Source: §B.3 · p. 14
B.4 Answer Extraction
Answer extraction uses manually annotated formats to standardize comparisons between model outputs and ground-truth solutions.
B.4 Answer Extraction
- Manual annotations assign puzzles to single-answer, answer-pair, or comma-separated-list formats.Each format specifies how the final answer should be written.Source: §B.4 · p. 15
- Standardized answer formats support consistent extraction and reduce ambiguity when comparing model outputs with ground-truth solutions.Source: §B.4 · p. 15
C. Additional Results
Table C.1 reports solve rates for 77 meta-puzzles when models are given the correct answers to preceding puzzles.
C. Additional Results
- Table C.1 reports solve rates on 77 meta-puzzles with correct answers to preceding puzzles supplied to the models.Source: Table C.1 · p. 15