Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?

• Unlike general overthinking on well-defined questions, MiP-Overthinking arises when a question is unsolvable because a necessary premise or condition is missing.The authors present it as evidence of reasoning models’ weak critical thinking and a potential drawback of current training recipes.Source: §1 · p. 2
• The authors curate four MiP datasets spanning difficulty levels and three generation strategies, then evaluate reasoning and non-reasoning models across well-defined and MiP questions.The strategies are Rule-Based Generation, Body-Question Swapping, and Essential-Premise Removal; evaluation considers response length, accuracy on well-defined questions, and abstention rate on MiP questions.Source: §1 · p. 3
• Missing premises make reasoning models generate 2× to 4× more tokens than general overthinking on well-defined questions, without improving MiP identification and contrary to test-time scaling.The paper describes MiP-Overthinking as additional tokens generated because a question lacks information needed to solve it.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Non-reasoning models consistently produce shorter responses to MiP questions and quickly identify the missing information, showing greater robustness than reasoning models.A suitable critical response would request clarification or state that the question cannot be answered as posed.Source: §1 · pp. 2–3
• On MiP questions, reasoning models often enter self-doubt loops, repeatedly revisiting the question and guessing user intent, which drives token growth.They may notice or identify the missing premise early but hesitate to commit and continue ineffective reasoning.Source: §1 · p. 3

2.1 Definition of Missing Premise

• A MiP removes a premise essential to uniquely determining a logically valid answer; an ideal solver should stop once it recognizes this insufficiency.The paper notes that state-of-the-art reasoning models instead produce extensive, redundant chains without identifying the missing premise.Source: §2.1 · p. 3

2.2 Overview of Data Construction

• The controlled suite combines three math datasets and synthetic formulas, using rule-based generation, body-question swapping, or essential-premise removal across three difficulty levels.The three strategies create missing-premise questions through undefined variables, incompatible bodies and queries, or removal of premises needed for resolution.Source: §2.2 · p. 4
• MiP-Formula contains 50 recursively generated unsolvable formulas, with combinations of variables and operators limited to a maximum depth of three.Their undefined variables make the formulas’ unsolvability apparent despite their surface complexity.Source: §2.2 · p. 4
• For SVAMP (Patel et al., 2021), the authors randomly permute problem bodies and questions, then manually inspect the resulting clear inconsistencies.These mismatches are easy for humans to identify.Source: §2.2 · p. 4
• For GSM8K (Cobbe et al., 2021), they remove one of two or three numerical conditions and manually filter out questions that remain solvable.These cases require basic logical analysis to recognize as unsolvable.Source: §2.2 · pp. 4–5
• For MATH 500 (Hendrycks et al., 2021), they manually select 58 feasible questions and remove one necessary premise, requiring substantial mathematical reasoning to detect insufficiency.The authors note that rule-based filtering is difficult for this competition-level dataset.Source: §2.2 · p. 5

3.1 Evaluation Metrics

• Evaluation measures response length, MiP abstain rate, and accuracy on well-defined questions; datasets without reference answers are evaluated only for abstention.GPT-4o performs the automated response evaluation.Source: §3.1 · p. 5

3.2 Main Results

• Reasoning models produce 2 – 4× more tokens on MiP questions, often with low abstain rates, while non-reasoning models keep similar lengths and more often acknowledge missing premises.On the relatively harder MiP-MATH dataset, all models produce longer responses and have lower abstain rates.Source: Table 2 · p. 6, §3.2 · pp. 6–7

3.3 Thinking Patterns through Tokens

• On MiP-GSM8K, reasoning models show explosive growth in reasoning-related tokens and steps, while non-reasoning models typically use fewer steps, consistent with redundant, self-doubting thought.Reasoning models use tokens such as “alternatively,” “wait,” and “check” more often than non-reasoning models.Source: §3.3 · p. 7, Table 3 · p. 7

3.4 Step-level Similarities

• MiP responses have higher step-level similarity and lower variance than well-defined responses, indicating repetition and re-checking that inflate token use without improving abstention.Average similarity is 0.50 for MiP responses versus 0.45 for well-defined responses; variance is 8.2e-4 versus 7.9e-3, respectively.Source: §3.4 · p. 8, Figure 3 · p. 8

3.5 Thinking Patterns through Example

• A reasoning-model example shows five thinking patterns being overused to produce a long, redundant response that does not help the model abstain from a MiP question.The paper highlights the patterns in the example and directs readers to Appendix D for additional examples.Source: §3.5 · p. 8

4.1 Do Models know premises are missing?

• Reasoning models often suspect MiP questions are unsolvable early, but only a small proportion ultimately abstain.The analysis measures suspicion during reasoning and the step when it first appears; the reported results indicate that recognizing potential unsolvability does not reliably lead models to stop.Source: §4.1 · pp. 8–9, Table 4 · p. 10, Figure 5 · p. 10
• The authors conclude that models can identify unsolvable MiP questions but lack the critical thinking to act on that suspicion by abstaining.They characterize continued reasoning despite suspected unsolvability as MiP-Overthinking.Source: §4.1 · pp. 8, 10

4.2 What Caused MiP-Overthinking?

• The authors hypothesize that inadequate length constraints in rule-based reinforcement learning contribute to MiP-Overthinking and that distillation propagates it to supervised models.They also describe format and accuracy rewards, sometimes supplemented by step or length rewards, as potentially encouraging excessive reasoning patterns.Source: §4.2 · p. 10
• Fine-tuning Qwen-2.5-7B-Instruct on 50 DeepSeek-R1 MiP responses increased response lengths and reduced abstention on GSM8K, while creating a length disparity absent before tuning.The authors report these changes after tuning on MiP-Formula responses; the figure compares response length, MiP abstention, and accuracy on well-defined questions before and after tuning.Source: §4.2 · p. 10, Figure 6 · p. 10

5.1 Reasoning Large Language Model

• Reasoning research uses reinforcement learning and curated-data instruction tuning, while studies report redundant solutions, overthinking, and slowdown attacks exploiting verbose reasoning.Examples include reinforcement-learning approaches (Shao et al., 2024; Xiong et al., 2025), instruction tuning (Ye et al., 2025; Muennighoff et al., 2025), redundancy (Xia et al., 2025), overthinking (Qu et al., 2025; Liu et al., 2025), and slowdown attacks.Source: §5.1 · p. 11

5.2 Test-time Scaling

• Test-time scaling literature favors more inference tokens through parallel sampling, sequential refinement, or tree-based methods, but extended responses can waste computation and degrade performance.The cited approaches include parallel sampling (Brown et al., 2024; Levi, 2024), sequential refinement (Snell et al., 2024; Lee et al., 2025), and tree-based methods (Hou et al., 2025).Source: §5.2 · p. 11

5.3 Models’ Behavior Study in Ambiguous Condition

• Prior ambiguity benchmarks assess recognition of knowledge limits, whereas this study probes missing-premise questions and finds specialized reasoning models reason longer and perform worse.Prior work includes Cole et al. (2023), Amayuelas et al. (2024), and Zhou et al. (2023).Source: §5.3 · p. 11

6 Conclusion

• Reasoning models widely generate dramatically long responses with low abstain rates on ill-defined, unsolvable questions with missing premises.The paper names this phenomenon MiP-Overthinking.Source: §6 · p. 11
• Models may suspect early that a MiP question is unsolvable, yet typically fail to act on those suspicions.They instead produce repetitive, redundant traces, and their final answers do not address the missing premises.Source: §6 · p. 11
• The observed behavior indicates a lack of critical thinking capability in current reasoning models.The conclusion describes the phenomenon as widespread but still under-explored.Source: §6 · p. 11
• Current reasoning-model training recipes emphasize thorough chains of thought but do not sufficiently reward critical thinking or early exit from unsolvable tasks.Source: §6 · p. 11

• The setup covers models, evaluation metrics, and generation settings.Source: A Detailed Experimental Setup · p. 16

A.1 Models

• The non-reasoning model set includes Qwen2.5-32B-Instruct, Gemma-2-27B-it, Phi-3-medium-128k, GPT-4o, and Gemini1.5.The passage cites Team (2024), Team et al. (2024b), Abdin et al. (2024), OpenAI et al. (2024), and Team et al. (2024a), respectively.Source: §A.1 · p. 17
• The reasoning model set includes QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, S1.1, DeepSeek-R1, GPT-o1, GPT-o1mini, and GPT-o3mini.The passage cites Team (2025), DeepSeek-AI et al. (2025), Muennighoff et al. (2025), DeepSeek-AI et al. (2025), OpenAI (2024b), OpenAI (2024c), and OpenAI (2025), respectively.Source: §A.1 · p. 17

A.2 Evaluation Metrics

• Response length includes reasoning and answer tokens, while GPT-4o judges abstention from the final two paragraphs and correctness on GSM8K and MATH.Open-source token counts use model-specific tokenizers; proprietary-model generation lengths come from their APIs.Source: §A.2 · p. 17

A.3 Generation Setting

• Open-source models use greedy decoding, model-specific default chat templates, and no system prompt; proprietary models use API defaults, with GPT-o1mini and GPT-o3mini set to medium reasoning effort.Source: §A.3 · p. 17

B Data Construction Details

• The MiP suite draws on SVAMP, GSM8K, and MATH 500, plus 50 synthetic unsolvable formulas, each containing an undefined, non-constant variable.The formulas are generated recursively from variables and operators, with maximum recursion depth three.Source: B Data Construction Details · p. 17
• From SVAMP (Patel et al., 2021), the authors manually selected 300 randomly permuted body-question pairs after human review to remove accidentally solvable questions.The resulting pairs have logical inconsistencies between problem body and question.Source: B Data Construction Details · p. 17
• From GSM8K (Cobbe et al., 2021), removing one of two or three numerical conditions and human verification yielded 582 MiP questions requiring basic logical analysis to identify unsolvability.Source: B Data Construction Details · pp. 17–18
• From MATH (Hendrycks et al., 2021), the authors manually selected 58 MATH500 questions and removed a necessary premise, testing recognition of insufficiency in complex problems.Source: B Data Construction Details · p. 18

C Prompt Template for Evaluation

• Answer-accuracy and abstention evaluation use separate paired and non-paired GPT-4o prompt templates, passing the model’s final two paragraphs to avoid an empty input.The templates cover Formula and SVAMP, and GSM8K and MATH, respectively.Source: C Prompt Template for Evaluation · p. 18, Figure 7 · p. 18, Figure 8 · p. 18
• To locate the first paragraph suggesting a missing premise, GPT-4o judges paragraphs sequentially; because the process is unstable, the authors repeat it three times and use the median.Figure 9 provides the paragraph-level judgment prompt.Source: C Prompt Template for Evaluation · p. 18, Figure 9 · p. 19

D Examples of Model Response

• Non-reasoning models identify missing premises and abstain or request details, whereas reasoning models give verbose answers that may be absurd or hallucinated.Examples include Gemini 1.5 abstaining, GPT-4o requesting more information, and a DeepSeek-R1 answer based on an assumed discount rate.Source: D Examples of Model Response · p. 19, Figure 10 · p. 19, Figure 11 · p. 20
• s1.1 spends substantial effort on inefficient, redundant reasoning before producing a meaningless result.This example is from the MiP-Formula data.Source: Figure 12 · p. 21
• DeepSeek-R1 thinks for a long time, then hallucinates an answer by assuming a discount rate.The example is from the MiP-GSM8k dataset.Source: D Examples of Model Response · p. 19, Figure 13 · p. 22