Source-linked AI summary
EnigmaForge: The Question Is Hidden in the Story
Daniel Eisner
TL;DR
Static benchmarks face saturation and leakage, and prior work had not supplied verified unique ground truth for discovering an unstated task. EnigmaForge generates and verifies hidden-task instances and evaluates matched conditions; intuition rankings spread much more widely than fact-recovery scores. The paper also reports condition-correlated refusals as a leaderboard concern.
Problem
Static benchmarks saturate and leak, while the paper reports no prior benchmark it found offering verified unique ground truth for discovering an unstated task.
Source: §2 · p. 2Method
Results
Intuition spans 22× while fact F1 spans 1.6×; Claude Opus 5.5, second on facts, ranks fourteenth on intuition.
Source: §6 · p. 6Takeaways & Limitations
The paper concludes that scoring refusals as failures can introduce a hidden filter-survival axis because refusals are condition-correlated.
Source: §6 · p. 7The benchmark has no human baseline, uses five genre packs, and evaluates each model in only one configuration.
Source: §6.1 · p. 7
Abstract
from arXiv · showhide
Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and logbook margins is a small logic puzzle whose solution is unique - proved by a SAT solver at generation time, with an ablation certificate showing every clue is load-bearing. Because instances are generated rather than collected, the corpus renews forever. The headline measure is intuition: task success when handed only the story, with world reconstruction as the secondary axis. Twenty-five frontier models ran over 600 instances (17,400 scored records) under three matched conditions. Intuition reshuffles the leaderboard: a 22x spread where fact recovery spans 1.6x, the second-best fact-recoverer ranks fourteenth, one model is indifferent to being told the question, and another is significantly better without it. Several models were blocked by their own content filters before reaching the puzzle - any benchmark scoring refusals as failure is quietly measuring filter behavior.
1 Introduction
EnigmaForge presents a generated benchmark in which models receive a story of documents without a stated question and must discover and solve the embedded task. Its design verifies unique solutions and load-bearing clues, matches conditions across world families, and evaluates 25 frontier models.
1 Introduction
- The benchmark hides a uniquely solvable logic puzzle in documents and marginal notes, requiring models to infer the question, recover the world, and choose the rule-governed action.Some marginal notes provide knowledge bridges needed to solve the puzzle, although that knowledge is not stated in the record.Source: §1 · pp. 1–2
- EnigmaForge builds a benchmark for models encountering old documents with no question stated.The example record contains ten numbered fragments, including an invoice, receipt, photograph, and witness remark, alongside marginal notes.Source: §1 · pp. 1–2
- A DPLL engine proves each intended solution unique, clue-removal certificates show every clue is load-bearing, and brute-force enumeration cross-checks the engine.Matched conditions present each world family with the question stated, the formal world stated, or neither.Source: §1 · p. 2
- The evaluation covers 600 instances, 120 families, 6 difficulty levels, and 17,400 scored records, with family-clustered bootstrap confidence intervals.Source: §1 · p. 2
2 Related work
Related work addresses benchmark saturation and leakage through renewal or verification, while EnigmaForge uses generation. The paper identifies verified ground truth for discovering an unstated task as an unfilled benchmark-design space.
2 Related work
- Renewal and verification are existing responses to static-benchmark saturation and leakage: LiveBench refreshes questions monthly, while P4Bench uses zero-knowledge proofs.EnigmaForge instead generates instances from seeds and proves uniqueness at generation.Source: §2 · p. 2
- Prior narrative deduction benchmarks use tabletop-puzzle formats, while EnigmaForge makes the task hidden and procedural.The paper reports that no prior benchmark it found offered verified unique ground truth for discovering an unstated task.Source: §2 · p. 2
3 The generator
The generator creates hidden formal worlds, proves their intended solutions unique, and renders them as varied stories. It scales difficulty through larger worlds and seeded design axes.
3.1 Hidden formal worlds
- Each hidden formal world has finite-domain variables, constraints, evidence across nine document channels, staged objectives, and a public register-or-hold policy.The true goal follows an apparent intermediate goal, and the policy also includes a superseded provisional rule the model must not follow.Source: §3.1 · pp. 2–3
3.2 Ground truth first, then squeeze
- The pipeline derives constraints from sampled worlds, strengthens and minimizes evidence, then proves uniqueness by requiring the negated intended solution to be UNSAT.A brute-force oracle validates the DPLL engine on every instance; disagreement rejects the instance.Source: §3.2 · p. 3
3.3 Certificates
- For every clue, an ablation certificate shows that removing it admits a second model, verifying both load-bearing clues and genuinely misleading distractors.Source: §3.3 · p. 3
3.4 Surfaces
- A compiler renders worlds as prose across five genre packs and at least two surface realizations, while story mode embeds the verified instance without an exhibit list or stated task.Clue clauses must survive verbatim, and span-search plus extraction-round-trip checks gate each scene.Source: §3.4 · p. 3
3.5 Difficulty
- Worlds scale from 8 variables and 10 evidence units at L00 to 42 variables and forward-walking chains at L05; design axes are seeded for reproducibility.The corpus does not repeat because condition, realization, genre, and burial are seeded axes.Source: §3.5 · p. 3
4 Evaluation design
The evaluation combines exact structured grading with derived measures that separate unguided task success from world reconstruction. It uses family-level paired comparisons and includes controls for grading validity.
4 Evaluation design
- The corpus contains 600 instances across 120 families, five realizations, six levels, three conditions, and five genres.Table 1 covers all 600 instances and sorts its leaderboard by all-item fact F1.Source: §4 · p. 3, Table 1 · p. 4
- Grading uses structured assignments, exact-match fact precision, recall, and F1, exact-world recovery, and policy-validated decisions, without lexical overlap or an LLM judge.Partial credit applies to facts but not actions.Source: §4 · p. 3
- 0.737 F1 is reached by copying out directly stated facts, showing that a substantial portion of fact score requires no inference.The paper describes this as a floor motivating the earned-decisions measure.Source: §4 · p. 4
- Intuition measures task success on the implicit condition, while earned decisions count correct actions only when accompanied by a fully correct world.Discovery retention is 100 minus the explicit–implicit F1 gap; reasoning discipline tracks whether the answer arrived before the thinking budget ran out.Source: §4 · p. 4
- Condition comparisons use paired family-level means with 2,000 bootstrap resamples, and the reported intervals are descriptive without multiplicity correction.Families, rather than instances, are treated as exchangeable units.Source: §4 · p. 4
5 Results
Across 25 frontier models, unguided task success varies much more than fact recovery, and several models struggle to turn recovered worlds into correct actions. Results also show question effects and content-filter interference, while controls perform as expected.
5.1 Nobody is close to the ceiling
- Fact F1 reaches 0.966 for the best model, while task success tops out at 80, far from the ceiling.The frontier spans 0.57 to 0.97 in fact F1.Source: §5.1 · p. 4
5.2 Intuition: the leaderboard the question cannot see
- Intuition spans 3.6 to 80.1, a 22× spread versus 1.6× for fact F1; GPT-6 Astra leads, while Claude Opus 5.5 ranks fourteenth.Across difficulty levels, GPT-6 Sol’s intuition stays between 0.95 and 0.98, while GPT-6 Astra’s falls from 0.98 to 0.57 at L5.Source: §5.2 · p. 5
5.3 Finding the problem is not the hard part anymore
- Discovery retention runs 58 to 102, while earned decisions run 28 to 88; only one model exceeds 80 earned decisions.For nearly every model, recovering the world does not consistently translate into the required action.Source: §5.3 · p. 5
5.4 What it costs not to be told the question
- GPT-6 Sol’s explicit−implicit task-success difference is −0.142 (CI [−0.204, −0.075]), significantly favoring performance without the question.Grok-4.6’s difference is 0.000 (CI [−0.062, +0.058]), indistinguishable across conditions.Source: §5.4 · p. 5
5.5 When the benchmark never reaches the model
- Content filters block some models before the puzzle is reached; Claude Fable 5.1 was blocked on 585 of 600 attempts, while Claude Opus 5 refused all 120 formal-condition presentations.These rows are lower bounds, not zeros; scoring refusals as failures would mix filter survival into benchmark rankings.Source: §5.5 · pp. 5–6, Table 2 · p. 6
5.6 Controls hold
- The perfect-information solver scores 1.0 and the story-copier 0.0, supporting the grading pipeline’s arithmetic validity.Source: §5.6 · p. 6
6 Discussion
Intuition diverges sharply from world reconstruction: fact-recovery leaders can rank much lower when the question is hidden, while some models retain or even improve unguided performance. The discussion also raises an all-or-nothing decision hypothesis, filter-related measurement concerns, and limits on generalization.
6 Discussion
- A 22× intuition spread contrasts with a 1.6× fact-F1 spread; Claude Opus 5.5 ranks second on facts but fourteenth on intuition, while GPT-6 Astra leads at 80/100 unguided tasks.GPT-6 Sol is second on intuition at 75, has fact F1 of 0.966, and is significantly better unguided than guided.Source: §6 · p. 6
- Unguided success falls faster with difficulty for most models, but GPT-6 Sol stays at 0.95–0.98 across levels and Grok-4.6's 21/100 intuition differs from its zero discovery tax.GPT-6 Astra falls from 0.98 to 0.57 at L5 as transport failures spike; DeepSeek 4 Pro retains 82/100 at L5.Source: §6 · p. 6
- The authors hypothesize that all-or-nothing decision scoring may explain why strong world recovery does not ensure successful action, making decision completion a distinct training target.The proposed target is decision completion under all-or-nothing validation, rather than more comprehension.Source: §6 · p. 7
6.1 Limitations
- Condition-correlated refusals can bias leaderboard rankings, while the study has no human baseline and its English, genre, configuration, and filtered-provider coverage constrain interpretation.The constraint grammar is learnable, so a lab trained on the generator could overfit its surface; tail rankings for filtered providers rely on partial coverage.Source: §6 · p. 7, §6.1 · p. 7
7 Conclusion
The conclusion argues that problem discovery can be benchmarked with verified generation and matched conditions. EnigmaForge makes intuition measurable, with most frontier models struggling when the question is hidden and one model bridging the gap.
7 Conclusion
- Problem discovery can be benchmarked using generation, proofs, certificates, and matched conditions.The conclusion says this construction makes scores mean what they claim and keeps the corpus renewable.Source: §7 · p. 7
- Most frontier models solve worlds they are told about but stumble when the question is withheld.The conclusion identifies one model that bridges the gap entirely.Source: §7 · p. 7
- The second-best fact-recoverer ranks fourteenth when the question is taken away.The conclusion points to harder worlds and an interactive variant as next steps.Source: §7 · p. 7
Reproducibility
The reproducibility materials include a command for regenerating an instance and public benchmark artifacts.
Reproducibility
- A small instance can be reproduced with the pipeline command using seed 2026.The command specifies size small and writes output to runs/demo.Source: Reproducibility · p. 7
- The command is presented as reproducing an instance from its seed.The example invocation uses the small setting and seed 2026.Source: Reproducibility · p. 7
- The public materials include the generator, seeds and settings, provenance hashes, scored results, and an interactive leaderboard.The passage directs readers to the title footnote for the URLs.Source: Reproducibility · p. 7
A A worked example
The worked example follows a maritime instance from story fragments and knowledge bridges to a hidden formal world and verification checks. It illustrates distinct surface realizations and certificates for uniqueness, clue importance, and distractor safety.
Realization 1 (what the model receives)
- The worked example is a real seed 2026, level L00 maritime instance reproduced byte for byte from pipeline output; the solver receives a complete record and is asked what it requires them to figure out.Its marginal notes mention Shakespeare and penicillin; fragments (9) and (10) are distractors, both supporting the true but non-load-bearing hypothesis that the harbor fee was never paid.Source: A A worked example · p. 7, Realization 1 (what the model receives) · pp. 7–8
Realization 2 (same world, different surface)
- The two realizations transmit the same constraint while changing its surface phrasing, including turning an assertion into a documented absence.The extraction round-trip must recover the formal model from either realization alone.Source: Realization 2 (same world, different… · p. 8
The hidden formal world
- The hidden world has eight variables, ten constraints, a verified solution, and staged objectives that distinguish an apparent goal from the true final action.The true objective is to determine the correct final action: act on the corrected record.Source: The hidden formal world · pp. 8–9
The certificates
- The verification artifact reports agreement between the SAT engine and brute-force oracle and a uniqueness pass for the solution.The ban clause formed from the negated solution is UNSAT.Source: The certificates · p. 9
- Ablation certificates show that removing any one of eight clue constraints admits a second model, while two harbor-fee distractors do not break uniqueness.The distractors nevertheless admit false hypotheses.Source: The certificates · p. 9