Source-linked AI summary
Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?
Chenrui Fan, Ming Li, Lichao Sun, Tianyi Zhou
TL;DR
The paper examines why reasoning LLMs overthink ill-posed questions with missing premises and studies this behavior across curated MiP datasets and diverse models. It finds that reasoning models generate substantially longer, redundant responses and often fail to abstain, whereas non-reasoning models respond more briefly and identify the missing information more reliably. The results expose a limitation in current reasoning-model behavior and motivate more efficient critical thinking.
Problem
Existing reasoning models can overthink simple well-defined queries, but their behavior on unsolvable questions lacking necessary premises is insufficiently understood.
Method
The study curates MiP datasets using synthetic generation and modifications of established datasets, then evaluates diverse reasoning and non-reasoning LLMs using response length, accuracy, and abstain rate.
Results
Reasoning models produce 2× to 4× more tokens on MiP questions than during general overthinking, while non-reasoning models generate shorter responses and identify MiP more quickly.
Takeaways & Limitations
The findings indicate that current reasoning models often recognize missing premises but fail to act on that judgment, revealing limited critical thinking and inefficient reasoning.
Takeaways & Limitations
The paper’s empirical analysis is based on its curated MiP datasets and the evaluation metrics of response length, well-defined-question accuracy, and MiP abstain rate.
Abstract
from arXiv · showhide
We find that the response length of reasoning LLMs, whether trained by reinforcement learning or supervised learning, drastically increases for ill-posed questions with missing premises (MiP), ending up with redundant and ineffective thinking. This newly introduced scenario exacerbates the general overthinking issue to a large extent, which we name as the MiP-Overthinking. Such failures are against the ``test-time scaling law'' but have been widely observed on multiple datasets we curated with MiP, indicating the harm of cheap overthinking and a lack of critical thinking. Surprisingly, LLMs not specifically trained for reasoning exhibit much better performance on the MiP scenario, producing much shorter responses that quickly identify ill-posed queries. This implies a critical flaw of the current training recipe for reasoning LLMs, which does not encourage efficient thinking adequately, leading to the abuse of thinking patterns. To further investigate the reasons behind such failures, we conduct fine-grained analyses of the reasoning length, overthinking patterns, and location of critical thinking on different types of LLMs. Moreover, our extended ablation study reveals that the overthinking is contagious through the distillation of reasoning models' responses. These results improve the understanding of overthinking and shed novel insights into mitigating the problem.
1 Introduction
The paper introduces MiP-Overthinking, in which reasoning models respond excessively to ill-posed questions lacking necessary premises. Across curated datasets and model types, these models produce longer, less effective reasoning than non-reasoning models, revealing weaknesses in critical thinking and efficient inference.
- Problem: MiP-Overthinking occurs when ill-posed questions lack necessary premises or conditions, making them unsolvable.The paper distinguishes this setting from ordinary overthinking on well-defined questions.
- Method: The study constructs MiP datasets from synthetic and modified SVAMP, GSM8K, and MATH500 questions and evaluates reasoning and non-reasoning models.Evaluation uses response length, well-defined-question accuracy, and abstain rate on MiP questions.
- Contribution: The paper presents MiP-Overthinking as a widespread but under-explored limitation of current reasoning models with low abstain rates on unsolvable questions.Its contribution is a systematic comparison of model behavior on well-defined versus missing-premise questions.
- Key findings: 2× to 4× more tokens are generated for missing-premise questions than for general overthinking on well-defined questions, without improving MiP identification.This contradicts the expected benefit of additional test-time reasoning.
- Key findings: Non-reasoning models generate shorter responses and quickly identify missing-premise questions, showing greater robustness to absent critical information.The comparison suggests that extensive reasoning training does not automatically produce efficient critical thinking in this scenario.
- Key findings: Reasoning models often enter self-doubt loops, repeatedly revisiting the question and guessing user intentions instead of resolving the missing premise.These patterns produce redundant and ineffective reasoning traces.
- Key findings: Models may detect a missing premise early but hesitate to commit, continuing ineffective thinking rather than abstaining.The failure concerns acting on an appropriate judgment, not only recognizing the problem.
2 Missing Premise Definition and Construction
The paper defines a Missing Premise problem as a question whose available premises cannot uniquely determine a logically valid answer, then constructs datasets by creating such insufficiencies through several controlled transformations.
- Definition: A Missing Premise problem removes an essential premise so the remaining premises cannot uniquely determine a valid answer to the question.The definition uses a premise set P, a question Q, and a proper subset P′ formed by removing one premise.
- Definition: The study evaluates whether models recognize insufficient information and terminate inference rather than continuing with redundant reasoning.The authors state that ideal systems should identify the absent premise and stop when a unique solution cannot be derived.
- Dataset Construction: The curated benchmark combines three mathematical datasets with synthetic formulas containing unassigned variables and spans multiple difficulty levels.The sources include SVAMP, GSM8K, and MATH500, alongside a synthetic formula dataset.
- Dataset Construction: The construction methods are rule-based generation, body-question swapping, and essential-premise removal.These transformations create undefined variables, incompatible premises and queries, or otherwise unsolvable versions of well-formed questions.
- Dataset Construction: MiP-SVAMP permutes problem bodies and questions, while MiP-GSM8K removes one numerical condition and filters cases that remain solvable.MiP-MATH manually removes one necessary premise from selected competition-level problems.
- Evaluation: The evaluation framework compares response length, well-defined-question accuracy, and MiP abstain rate across reasoning and non-reasoning models.The figure description contrasts these metrics for well-defined and missing-premise questions.
3 Overthinking under Missing Premise
Across datasets, reasoning models respond to missing-premise questions with substantially longer and more repetitive reasoning, but this extra computation generally does not improve abstention; non-reasoning models often stop sooner.
- Main Results: Reasoning models often produce 2–4× more tokens on MiP questions than their already long responses to well-defined questions.QwQ-32B and DeepSeek-R1 increase from roughly 1,000 tokens on simple GSM8K questions to more than 3,000 tokens on MiP questions.
- Main Results: Reasoning models generate longer responses than non-reasoning models even on well-defined questions, with limited corresponding accuracy gains.Non-reasoning models use approximately 200 tokens, compared with about 1,000 for DeepSeek-R1 and 1,800 for QwQ-32B on the same questions.
- Main Results: Across MiP datasets, most reasoning models combine longer responses with low abstain rates, and the pattern is stronger on the harder MiP-MATH dataset.The reported preference is shorter responses with higher abstention.
- Thinking Patterns through Tokens: Reasoning models show explosive increases in tokens such as “alternatively,” “wait,” and “check,” as well as in step counts, whereas non-reasoning models typically use fewer steps for MiP questions.The authors interpret this contrast as redundant, self-doubting thinking rather than efficient identification of unanswerability.
- Step-level Similarities: MiP responses have higher step-level similarity and lower variance than well-defined responses, indicating repeated reasoning and re-checking.The analysis uses pairwise cosine similarity between response steps and finds repetitive content that inflates token usage without improving abstention.
- Thinking Patterns through Tokens: Example analyses identify multiple thinking patterns that reasoning models overuse, producing long responses that remain redundant and do not help them abstain.The reported patterns are presented as evidence of ineffective reasoning under missing-premise conditions.
4 Further Discussion
Reasoning models often suspect missing premises early but continue exploring unsolvable questions instead of abstaining. The analysis also examines recurring reasoning patterns and finds that this behavior can spread through distillation.
- 4.1 Do Models know premises are missing?: Most reasoning models detect potential unsolvability early, yet continue revisiting questions and definitions rather than abstaining.In-Process Suspicion Rate measures expressed doubt, while First Suspicion Index measures when doubt first appears.
- 4.2 What happens during reasoning?: Five recurring patterns—revisiting questions, consulting knowledge, proposing assumptions, self-doubt, and pausing—can produce redundant, unhelpful reasoning.
- 4.1 Do Models know premises are missing?: Models suspect many MiP questions but abstain from only a small proportion of them.
- 4.1 Do Models know premises are missing?: The study concludes that reasoning models can identify unsolvable MiP questions but do not confidently act on that recognition.
- 4.3 Why does MiP-Overthinking happen?: MiP-Overthinking appears across RL-based and SFT-based reasoning models, with distillation proposed as a route for behavioral transmission.
- 4.3 Why does MiP-Overthinking happen?: Fine-tuning on 50 DeepSeek-R1 MiP responses makes Qwen-2.5-7B-Instruct exhibit longer responses and a new length disparity between MiP and well-defined questions.
5 Related Work
Related work covers advances in LLM reasoning, analyses of reasoning quality and overthinking, test-time scaling strategies, and abstention under uncertainty. This paper distinguishes its MiP setting by examining prolonged reasoning and inferior performance on missing-premise questions.
- Reasoning in LLMs: Recent work improves LLM reasoning through post-training approaches including reinforcement learning and other methods.
- Reasoning Quality and Overthinking: Studies of reasoning quality report redundancy and identify overthinking as excessively long reasoning for queries requiring little reasoning.
- Test-Time Scaling: Test-time scaling research optimizes inference-time token generation through strategies such as parallel sampling.
- Abstention and Reliability: Prior abstention research evaluates whether LLMs recognize knowledge limits, whereas this study examines reasoning models under missing-premise conditions.
6 Conclusion
The paper identifies MiP-Overthinking as a widespread failure in which reasoning models generate long, repetitive traces for unsolvable questions and rarely abstain. It links the failure to insufficient support for critical thinking and early exit in current training recipes.
- 6 Conclusion: MiP-Overthinking occurs when reasoning models face ill-defined questions with missing premises, producing long responses and low abstain rates.
- 6 Conclusion: Models may suspect unsolvability early but usually continue repetitive reasoning and produce answers that do not address the missing premises.
- 6 Conclusion: The findings indicate that current reasoning-model training recipes do not sufficiently reward critical thinking or early exit from unsolvable tasks.
A Detailed Experimental Setup
The detailed experimental setup lists appendices covering models, evaluation metrics, and generation settings.
- A.1 Models: Appendix A.1 covers the models used in the experiments.
- A.2 Evaluation Metrics: Appendix A.2 covers the evaluation metrics.
- A.3 Generation Setting: Appendix A.3 covers the generation settings.
A Detailed Experimental Setup
The study evaluates both non-reasoning and reasoning models from open-source and proprietary sources, covering multiple training recipes.
- The evaluation includes non-reasoning models such as Qwen2.5-32B-Instruct, Gemma-2-27B-it, Phi-3-medium-128k, GPT-4o, and Gemini1.5.
- The reasoning-model group includes QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and S1.1.
A.2 Evaluation Metrics
The evaluation combines response-length measurement, abstention and accuracy judgments, missing-premise detection, and controllable MiP datasets spanning several mathematical benchmarks.
- Response length combines reasoning and answer tokens, using model-specific tokenizers for open-source models and API generation lengths for proprietary models.
- Abstain rates are evaluated from the model’s final two paragraphs against reference answers when available, using GPT-4o as judge.
- The study constructs MiP questions from SVAMP, GSM8K, and MATH 500, plus a synthetic rule-based Formula dataset.
- MiP-SVAMP: MiP-SVAMP contains 300 manually selected questions created by permuting problem bodies and associated questions from SVAMP.
- MiP-GSM8K: MiP-GSM8K removes one necessary numerical condition from questions with two or three conditions, followed by human verification.
- MiP-MATH: MiP-MATH manually selects 58 MATH500 questions and removes one necessary premise from each for MiP construction.
D Examples of Model Response
The examples contrast non-reasoning models that quickly recognize missing premises with reasoning models that produce lengthy, ineffective reasoning or hallucinated answers.
- Gemini 1.5 quickly identifies the missing premise in MiP-Formula and abstains from answering.
- GPT-4o quickly identifies the missing premise in MiP-GSM8K and asks the user for more information.
- The s1.1 example spends substantial effort on inefficient, redundant reasoning before producing a meaningless result.
- DeepSeek-R1 thinks for a long time and then hallucinates an answer based on an assumed discount rate.