PaperlayerView exact revision on arXiv

Source-linked AI summary

RiddleBench: A New Generative Reasoning Benchmark for LLMs

Deepon Halder, Alan Saji, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan, Raj Dabre

arXiv:2510.24932v1cs.CL

TL;DR

Existing benchmarks leave integrated reasoning across deduction, spatial awareness, and constraint satisfaction underassessed. RiddleBench introduces 1,737 English puzzles to probe these capabilities; its evaluation finds fragile reasoning and substantial failures in cross-model correction.

  • Problem

    Many benchmarks emphasize structured skills, leaving flexible reasoning that integrates logical deduction, spatial awareness, and constraint satisfaction underassessed.

    Source: §1 · p. 1

  • Method

    RiddleBench is a benchmark of 1,737 challenging puzzles designed to assess complex, multi-step LLM reasoning.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    RiddleBench reveals fragile reasoning under reordered constraints or irrelevant information, and evaluators incorrectly validated flawed reasoning in 45.2% of cases.

    Source: §5.2 · p. 4, §6 · p. 5

  • Takeaways & Limitations

    RiddleBench provides a diagnostic tool for reasoning failures and a way to measure progress toward more dependable AI.

    Source: §6 · p. 5

    The benchmark is curated from Indian exam materials, which may introduce cultural bias, and contamination cannot be definitively ruled out for proprietary models.

    Source: §7 · p. 5

Abstract

from arXiv · show

Large Language Models have demonstrated strong performance on many established reasoning benchmarks. However, these benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a benchmark of 1,737 challenging puzzles in English designed to probe these core reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including hallucination cascades (accepting flawed reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable language models.

1 Introduction

Existing benchmarks often emphasize pattern recognition or memorization, leaving inferential capabilities difficult to assess. RiddleBench introduces 1,737 English puzzles and evaluates leading models to examine their reasoning strengths and failures.

1 Introduction

  • Existing benchmarks often measure pattern recognition or memorization rather than deep reasoning, limiting assessment of inferential capabilities.The paper argues that true intelligence requires reasoning, deduction, and inference.Source: §1 · p. 1
  • RiddleBench comprises 1,737 English puzzles designed to evaluate LLM reasoning.The puzzles require multi-step deduction, spatial reasoning, and constraint satisfaction.Source: §1 · pp. 1–2
  • The authors evaluate leading LLMs and analyze hallucination cascades and poor self-correction.They report that even the most powerful models struggle with a significant portion of the benchmark.Source: §1 · p. 2

2 Related Work

Prior benchmarks target structured, knowledge-retrieval, or rule-based reasoning, while RiddleBench focuses on integrated reasoning across logical, spatial, and constraint-based demands.

2 Related Work

  • Mathematical and algorithmic benchmarks such as GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) emphasize structured problems; CommonsenseQA (Talmor et al., 2019) and WinoGrande (Sakaguchi et al., 2020) emphasize implicit knowledge retrieval.LogiQA (Liu et al., 2020) and RuleTaker (Clark et al., 2020) measure rule-based deduction.Source: §2 · p. 2
  • Existing approaches mostly test explicit, single-path solutions, whereas RiddleBench requires satisfying textual constraints while constructing spatial or relational layouts and inferring entity relationships.The authors describe this synthesis as reflecting cognitive integration in natural, constraint-rich problems.Source: §2 · p. 2
  • RiddleBench offers a focused diagnostic at the intersection of logical, spatial, and constraint-based reasoning, examining reliability beyond accuracy.The authors relate its analysis of hallucination cascades and confirmation bias to approaches such as (Zheng et al., 2023).Source: §2 · p. 2

3 RiddleBench

RiddleBench is a 1,737-puzzle benchmark for complex reasoning beyond accuracy, organized into four categories. Its puzzles come from archived Indian government-service mock exams and undergo extraction, structuring, and human verification.

3 RiddleBench

  • RiddleBench contains 1,737 challenging puzzles targeting multi-step deduction, spatial reasoning, and constraint satisfaction across four categories.The categories are Sequential Reasoning, Seating Arrangements, Blood Relations, and Coding-Decoding.Source: §3 · p. 2

3.1 Data Collection and Curation

  • Puzzles were sourced from publicly archived Indian government-service mock examination papers, and all are in English.The collection process used OCR to digitize questions, answers, and official reasoning traces, then structured the raw output into problem components.Source: §3.1 · p. 2
  • Authors manually evaluated each data point for transcription accuracy and correctness; the dataset is released under CC0.The paper describes this process as ensuring high dataset fidelity.Source: §3.1 · pp. 2–3

4 Experimental Design

The experiments use zero-shot prompting to evaluate model accuracy and reasoning reliability. They examine cross-model correction, self-correction, and robustness to information order and irrelevant content.

4 Experimental Design

  • Evaluations use zero-shot prompting with temperature 0.7 and a thinking budget of 8192 tokens.The exact prompt format is given in Appendix C.Source: §4 · p. 3
  • The analysis examines whether LLMs detect and correct other models’ reasoning errors or instead enter a hallucination cascade.The paper presents puzzles’ logical and spatial constraints as producing clear, diagnosable reasoning chains.Source: §4 · p. 3
  • The research questions also test self-correction and robustness to reordered information or irrelevant information.These are posed as separate questions about model behavior.Source: §4 · p. 3
  • Cross-model correction and self-correction experiments primarily use Qwen QwQ 32B as evaluator and DeepSeek-R1 as generator.The authors selected them for strong baseline performance and API accessibility.Source: §4 · p. 3

5 Results and Analyses

RiddleBench results show that even the strongest evaluated model fails on nearly a third of puzzles, while validation and prompt perturbation tests expose unreliable, fragile reasoning. The analyses identify failures in cross-model correction, self-correction, and robustness.

5.1 Overall Performance

  • GPT-oss-120B leads with 69.26% accuracy, yet even state-of-the-art models fail on nearly a third of the benchmark; models universally struggle with Seating Arrangement puzzles.The benchmark table reports correct-answer percentages across four categories: Sequential Reasoning, Seating Arrangement, Blood Relations, and Coding-Decoding.Source: §5.1 · p. 3, Table 2 · p. 4

5.2 The ‘Hallucination Cascade’: Failures in Cross-Model Correction

  • Qwen QwQ 32B timed out in 55.0% of forced-choice checks; given flawed traces, it identified errors in 44.1% and incorrectly endorsed flawed reasoning in 45.2% of cases.The evaluator sometimes accepted a flawed premise instead of re-solving the puzzle.Source: §5.2 · p. 4
  • After incorrectly validating flawed outputs, Qwen QwQ 32B reversed its judgment in only 4.4% of re-evaluations.The authors report that iterative refinement was almost entirely ineffective at stopping a hallucination cascade.Source: §5.2 · p. 4

5.3 The Illusion of Self-Correction

  • Models failed to identify their own errors in 67.7% of trials, flagging their flawed logic in just 17.3%, compared with 44.1% when evaluating a peer’s reasoning, suggesting self-confirmation bias.Source: §5.3 · p. 4

5.4 Fragile Reasoning: The Lack of Robustness

  • Shuffling constraint sentences reduced Qwen QwQ 32B accuracy by 6.70 p.p. on Blood Relations and 3.69 p.p. on Seating Arrangements.The authors interpret this sensitivity as evidence of reliance on brittle sequential heuristics rather than robust comprehension.Source: §5.4 · p. 5
  • Adding an irrelevant sentence lowered accuracy on most categories, but Blood Relations accuracy increased by 2.74 p.p.The authors suggest the red herring may have disrupted a brittle heuristic and prompted a more robust reasoning path.Source: §5.4 · p. 5

6 Conclusion

RiddleBench tests complex, multi-step reasoning and reveals weaknesses in the specific models evaluated, including error propagation, poor self-correction, and sensitivity to prompt changes.

6 Conclusion

  • The tested models show hallucination cascades, error fixation, and poor self-correction associated with strong self-confirmation bias.The authors describe cascades as models uncritically adopting flawed reasoning from other evaluated models.Source: §6 · p. 5
  • LLM reasoning is fragile, with performance disrupted by reordered constraints or irrelevant information.The conclusion summarizes sensitivity to these prompt changes as a finding of the evaluation.Source: §6 · p. 5
  • RiddleBench is presented as a tool for diagnosing these issues and measuring progress toward more dependable AI.The authors also plan dataset growth and cross-lingual evaluations.Source: §6 · p. 5

7 Limitations

The authors identify limits related to evaluation versions, source-material bias, experiment coverage, and possible data contamination.

7 Limitations

  • Proprietary-model results use August 2025 API versions and may change with updates; Indian exam sourcing may introduce cultural bias.High API costs and computation also restricted cross-model and self-correction experiments to a subset of models.Source: §7 · p. 5
  • Contamination cannot be definitively ruled out for proprietary models whose training sets are undisclosed, despite sourcing puzzles from mock exam PDFs.The authors say the puzzles’ multi-step deductive nature resists memorization.Source: §7 · p. 5

8 Future Work

The authors plan to expand RiddleBench’s problem coverage and extend evaluation beyond English through multilingual versions.

8 Future Work

  • Future work may add problems within existing categories and introduce new logical-puzzle types to improve statistical robustness and broaden the reasoning skills assessed.The authors describe these as planned directions for benchmark expansion.Source: §8 · p. 5
  • The authors plan translations, initially into major Indian languages and eventually into other world languages, for cross-lingual assessment of LLM reasoning.They state that multilingual evaluation is important for assessing language-agnostic reasoning capabilities.Source: §8 · p. 5

9 Ethics Statement

The authors describe the benchmark’s public release and AI-use policy, and identify the proprietary and open-weight models evaluated using versions available in August 2025.

9 Ethics Statement

  • RiddleBench is released under CC0 for broad, unrestricted research access; generative AI assisted only with language refinement and boilerplate code.The authors say the original content was theirs.Source: §9 · p. 6

A Models Evaluated

Proprietary Models

Open-Weight Models

  • The cited model list names six entries: DeepSeek-R1, GPT-oss-120B, Mistral Small 24B it, Llama 3.3 70B, Gemma 3 27B it, and Qwen QwQ 32B.Source: Open-Weight Models · p. 7

B Dataset Examples

The examples span coding, numerical sequences, seating, and blood relations; the accompanying prompts specify evaluation and verification procedures, while analyses describe model profiles and a Gemini-specific visual strategy.

B.1 Coding-Decoding / B.2 Sequential Reasoning

  • The examples include a code-mapping question and a missing-term number sequence, illustrating coding-decoding and sequential reasoning tasks.Source: §B.1 · p. 7, §B.2 · p. 7

B.3 Seating Arrangement / B.4 Blood Relations

  • Other examples ask for the leftmost person in a constrained seating arrangement and how Bala is related to Roop in a family-relations puzzle.Source: §B.3 · p. 7, §B.4 · pp. 7–8

C.1 Zero-Shot Evaluation Prompt

  • The zero-shot evaluation prompt presents a question and optional answer choices, then requests a final answer in boxed format.Source: §C.1 · p. 8

C.2 Prompt for Forced-Choice Answer Verification

  • The forced-choice verification prompt asks a model to choose between a correct answer and a prediction for the same question.The prompt labels them Answer 1 and Answer 2 and requests “1” or “2” in boxed format.Source: §C.2 · p. 8

C.3 Prompt for Reasoning Validation / C.4 Sample Problems for Robustness Experiments

  • The reasoning-validation prompt checks both reasoning and answer correctness, and the robustness examples illustrate prompt perturbations.Source: §C.3 · p. 8, §C.4 · p. 8

D Model Reasoning Profiles

  • Figure 3 maps accuracy across Sequential Reasoning, Seating Arrangement, Blood Relations, and Coding-Decoding; GPT-oss-120B has a large, balanced profile, unlike more skewed models.The axes correspond to categories, and distance from the center reflects accuracy.Source: D Model Reasoning Profiles · p. 9

E Visual Aids in Solving Problems

  • Gemini models uniquely generated ASCII family trees for Blood Relations, translating textual constraints into spatial layouts without guaranteeing correct answers.The authors describe this as mimicking human diagram-drawing and as a potentially more interpretable reasoning strategy.Source: E Visual Aids in Solving Problems · p. 9, Figure 5 · p. 9

F Patterns in Thinking in Sequence Tasks

Most models use an arithmetic-first, geometric-second heuristic on numerical sequences; the authors caution that this procedural pattern may be vulnerable on unconventional problems. The costing section reports API access arrangements and total project spend.

F Patterns in Thinking in Sequence Tasks

G Model API Costing

  • The project incurred $314 in API costs; evaluated models were accessed through proprietary APIs or DeepInfra.Source: G Model API Costing · p. 10
Loading 2510.24932v1…