PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking Inertia: LLMs Keep Thinking When Told Not To

Dianqiao Lei, Kevin Qinghong Lin, Pan Lu, Philip Torr, James Zou

arXiv:2610.11765v1cs.CL

TL;DR

Surface controls do not reliably reveal whether LLM responses contain visible inference, leaving no-thinking difficult to evaluate. The paper measures response-level compliance, relevance, and explicit inference across models and tasks, finding persistent inference under no-think controls and answer-space-dependent trade-offs.

  • Problem

    No-thinking is often inferred from settings, missing reasoning traces, or short outputs, although these proxies can fail to distinguish answer-only responses from visible inference.

    Source: §1 · p. 2

  • Method

    The paper evaluates pre-answer text and final answers separately, measuring answer-only compliance, question relevance, explicit inference, and task accuracy across models and answer spaces.

    Source: §2.2 · p. 5, §2.3 · p. 5

  • Results

    Explicit no-think controls do not reliably remove inference, which becomes more prevalent as answer spaces open; stronger suppression trades open-ended accuracy for answer-only compliance.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    No-thinking is a measurable response-level behavior, and stopping visible inference cannot be assumed from model settings or instructions alone.

    Source: §5 · p. 15

    The six-model, six-benchmark evaluation leaves generalization to multilingual, code, tool-use, multi-turn, stochastic, and evolving proprietary-endpoint settings open.

    Source: §5.1 · p. 15

Abstract

from arXiv · show

Large Language Models (LLMs) increasingly ship with explicit "thinking modes", yet their counterpart, "no-thinking", has received far less attention. We study LLMs' no-thinking behavior along two axes. a. How to measure no-thinking? Prior work typically defines no-thinking through proxies such as a disabled thinking mode or the absence of long traces. These proxies are unreliable: disabled thinking modes may still emit reasoning, while long traces may contain filler rather than genuine inference. We instead normalize each response into a pre-answer trace and final answer, and evaluate it at three levels: (i) Empty-Thinking Rate for strict answer-only compliance; (ii) instruction-aware Question-Pre-answer Relevance for similarity between the question and pre-answer trace; and (iii) LLM-as-judge Explicit Inference Rate for visible explicit inference. Together, these metrics distinguish answer-only output, relevant but non-inferential text, and explicit inference. b. How does no-thinking vary across tasks and models? We evaluate six prompting interventions on six LLMs across Boolean, multiple-choice, and open-ended questions. We find that explicit no-think controls cannot reliably eliminate visible inference. Models instead exhibit "Thinking Inertia": explicit inference persists even under strict controls and becomes more prevalent as the answer space opens. Accuracy remains stable on Boolean and multiple-choice tasks, whereas open-ended tasks reveal a trade-off between answer-only compliance and task accuracy. Rewriting the same questions across answer spaces shows that supplying candidate answers makes answer-only responses easier to produce. These findings establish no-thinking as a non-trivial capability: stopping explicit reasoning cannot be assumed from model settings or instructions alone and deserves systematic evaluation alongside reasoning ability.

1 Introduction

No-thinking is difficult to define because surface controls do not reliably indicate whether a response contains visible inference. The paper evaluates response-level behavior and finds that inference persists under no-think controls, varying with answer space and suppression strength.

1 Introduction

  • Surface proxies—including disabled thinking modes, absent reasoning tokens, and short outputs—can miss visible, task-specific inference.A generic lead-in can be mistaken for reasoning, while a compact equation can expose an inferential step.Source: §1 · p. 2, Figure 2 · p. 2
  • The framework evaluates strict answer-only compliance, pre-answer relevance, and explicit inference separately, while assessing final-answer accuracy independently.It uses Empty-Thinking Rate, Question–Pre-answer Relevance, and Explicit Inference Rate for the three response measures.Source: §1 · pp. 2–3
  • Human validation supports the three measures, and human annotations reproduce the main answer-space and intervention trends.Annotators’ explicit-inference labels achieve 98.8% mean binary agreement with the primary judge; an independent judge from another model family produces robust EIR results.Source: §1 · p. 3
  • Explicit no-think controls do not reliably eliminate visible inference, which becomes more prevalent as answer spaces open.On open-ended tasks, stronger answer-only instructions improve compliance but can reduce accuracy; supplying candidate answers makes answer-only responses easier for the same questions.Source: §1 · p. 3
  • The paper defines no-thinking as measurable response-level behavior and identifies thinking inertia, a compliance–accuracy trade-off, and answer-space dependence.These findings support the broader goal of LLMs adapting when to reason and when to answer directly.Source: §1 · p. 3

2 Measuring No-Thinking

The paper defines no-thinking as an observable response behavior rather than a prompting, format, or length proxy. It evaluates pre-answer text and final answers separately, then organizes interventions by how they change the thinking configuration, output constraints, or answer interface.

2.1 Observations

  • Surface proxies can miss visible reasoning: no-think instructions, removing reasoning tags, and shortening outputs do not establish that a response contains no inference.These controls describe prompting, format, or length rather than the content of the response.Source: §2.1 · pp. 3–4

2.2 Problem Formulations

  • The framework treats the visible response as pre-answer text T plus final answer A, and distinguishes instruction or output changes from changes to the question or answer space.The decomposition makes no claim about unobserved model computation.Source: §2.2 · pp. 4–5

2.3 Three-Level Response Evaluation

  • Three levels separate strict answer-only compliance (ETR), instruction-aware relevance of T to Q (QRel.), and judge-labeled explicit inference (EIR), while accuracy evaluates A independently.QRel. measures relevance rather than inference; a relevant assertion or paraphrase alone does not count as explicit reasoning.Source: §2.3 · pp. 5–6

2.4 Thinking Behavior Control

  • The six-mode intervention spectrum includes two thinking-side references and four no-thinking modes, from the native baseline to stronger suppression constraints.THINK-ON and COT elicit thinking; THINK-OFF, SOFT, STRICT, and PREFIX form the no-think intervention space.Source: Table 2 · p. 7, §2.4 · p. 7

3 Experiments

The experiments test native controls and prompt interventions for suppressing visible pre-answer content across task domains and answer spaces, with human annotations supporting the three-level evaluation. Across these analyses, answer-only compliance falls as answer spaces open, and visible inference remains common on open-ended tasks even under the strictest instruction.

3.1 Settings

  • The evaluation spans six LLMs, Boolean, multiple-choice, and open-ended benchmarks, six intervention modes, and accuracy plus three no-thinking metrics.Metrics are aggregated over all outputs; the human study samples 600 outputs and manually audits the empty-pre-answer boundary.Source: §3.1 · p. 8

3.2 Native Think-on vs. Think-off

  • Under native THINK-OFF, ETR is roughly 80–100% on Bool but collapses on Open-ended tasks, where QRel. stays near THINK-ON and EIR is highest.The six-model mean EIR rises from Bool through MCQ to Open-ended tasks. STEP-BY-STEP prompting often restores QRel. and EIR toward THINK-ON levels while preserving that ordering; GPT-5.4 is the main exception on Bool and MCQ.Source: §3.2 · pp. 8–9, Figure 4 · p. 9

3.3 Performance across No-Think Variants

  • On Open-ended tasks, M2-to-M5 raises mean ETR from near zero to roughly 40% while lowering mean accuracy by about 15 percentage points and EIR from 99.69% to 62.00% overall.Bool and MCQ accuracy stays largely flat; M2-to-M5 EIR falls from 5.15% to 0.12% on Bool and from 45.57% to 25.39% on MCQ. M6 does not reliably strengthen answer-only compliance: its mean ETR falls versus M5 on MCQ and Open-ended tasks, while mean QRel. and EIR rebound in every answer-space family.Source: Figure 5 · p. 10, §3.3 · p. 11

3.4 Performance across disciplines

  • Under the fixed four-choice MMLU interface, STEM has the highest accuracy but also the lowest ETR (0.3–36.4%) and highest QRel. (0.509–0.784) and EIR (59.7–85.5%) across all four models.This shows domain variation in no-thinking behavior even when answer format is held constant.Source: §3.4 · p. 11, Table 4 · p. 12

3.5 Performance under answer-space rewrites

  • On matched questions, the Boolean rewrite changes accuracy from 77.8% to 78.9% and raises ETR from 10.5% to 62.5%, while the open-ended rewrite lowers accuracy to 52.6% and ETR to 0%.For the Boolean rewrite, QRel. falls from 0.709 to 0.314 and EIR from 86.95% to 36.90%; for the open-ended rewrite, QRel. rises to 0.810 and EIR to 90.14%. Candidate-visible formats make answer-only responses easier than open-ended generation.Source: §3.5 · pp. 11–12, Figure 6 · p. 12

3.6 Human Validation

  • Human validation finds 94.2% agreement on strict-compliance decisions, QRel. correlation of ρ = 0.829, and explicit-inference agreement of 98.8% with GPT-5.5 and 96.7% with Claude Opus 4.8.Human rates preserve the experimental trends: explicit inference rises from Bool to MCQ to Open-ended under M2 and M5, while M5 lowers it from 100.0% to 57.1% on Open-ended tasks.Source: §3.6 · pp. 12, 14, Table 5 · p. 14
  • Human annotations reproduce the answer-space staircase and persistent visible inference on open-ended tasks under M5.Direct annotations also show that explicit inference becomes harder to suppress as answer spaces open and remains common under the strictest natural-language instruction.Source: §3.6 · p. 14
  • Human judgments corroborate ETR, show relevance judgments correlate with QRel., and agree strongly with primary and independent LLM judges on explicit-inference labels.The boundary audit supports ETR, while human labels validate the explicit-inference judgments.Source: §3.6 · p. 14

4 Related Work

Related work connects no-thinking to selective reasoning and instruction following, while this paper focuses on whether requested no-thinking is realized in observable responses rather than inferred from output proxies.

Efficient and Adaptive Reasoning

  • Selective-reasoning work allocates or compresses visible computation, while studies also find that useful reasoning length depends on task difficulty and model capability [1] [11] [18] [23] [24] [43] [45] [49].Other methods train models to decide whether extended reasoning is needed [9] [16] [34] [47].Source: §4 · p. 14
  • The paper distinguishes observable response behavior from hidden computation and measures whether no-thinking requests remove visible question-conditioned text, not whether latent reasoning disappears [20].Related work includes reconstructing traces from black-box inputs and answers [48] and decoding encrypted reasoning blocks [27].Source: §4 · pp. 14–15
  • Unlike prompt repetition work that evaluates accuracy, length, and latency [22], this study separates pre-answer text from final answers and assesses relevance, explicit inference, and answer-only compliance.It also uses same-question answer-space controls.Source: §4 · p. 15

Instruction Following

  • The paper treats producing no visible reasoning as a verifiable output constraint, connecting its interventions to instruction-following and compliance benchmarks [51].The focus is whether surface compliance also removes visible question-conditioned content.Source: §4 · p. 15

5 Discussion and Conclusion

The discussion frames no-thinking as a distinct response-level capability: visible inference persists under explicit controls and varies with answer-space structure. The authors delimit claims to observable outputs, note evaluation boundaries, and propose task-aware stopping and broader audits.

5 Discussion and Conclusion

  • Across Qwen3 models, scaled variants, and frontier APIs, explicit no-think controls do not reliably remove inference; thinking inertia varies with answer space, and open-ended tasks expose a compliance–accuracy trade-off.The discussion also says compressibility depends on question structure, not question difficulty or model capability alone.Source: §5 · p. 15

5.1 Scope and Limitations

  • The analysis compares observable responses across local models and proprietary APIs, so its claims concern visible output rather than whether task-relevant computation persists in hidden states.Provider signals are included when available but are not treated as equivalent descriptions of internal reasoning.Source: §5.1 · p. 15
  • Generality beyond the six-model, six-benchmark suite—including multilingual, code, tool-use, multi-turn, and stochastic settings—remains open; EIR also depends on its visible-text rubric and judge calibration.The paper reports independent-judge and copying, paraphrase, and response-order audits.Source: §5.1 · p. 15

5.2 Implications and Future Work

  • The authors suggest task-aware routing: answer directly when the interface provides sufficient support, while preserving explicit computation when suppression would sacrifice accuracy.Visible compliance alone does not establish lower inference cost when hidden reasoning may remain active.Source: §5.2 · p. 16
  • Future work could train calibrated stopping across accuracy, answer-only compliance, visible inference, latency, and actual reasoning-token cost, while examining internal computation and extending to interactive tasks.The paper argues against universally suppressing intermediate computation.Source: §5.2 · p. 16

A Appendix Overview

  • The appendix reports protocols, robustness checks, aggregate results, question-structure controls, and representative outputs underlying the main-text findings.Source: A Appendix Overview · p. 21

B.1 Dataset Sizes

The appendix describes dataset sizes and model-generation settings for its analyses, including API sampling and equal response budgets across modes.

B.1 Dataset Sizes

  • The appendix records dataset splits and sample counts, with dataset sizes summarized in Supplementary Table B.1.Source: §B.1 · p. 21, Table B.1 · p. 21
  • Each API model contributes 5,687 outputs per mode and 34,122 across six modes; local Qwen3-4B/32B runs use full benchmark splits.API sampling uses 1,000 items from five benchmarks and all 687 StrategyQA items per mode.Source: §B.1 · p. 21

B.2 Models and Inference Settings

  • The evaluated models expose public thinking-control interfaces; the appendix documents generation settings without inferring hidden computation policies.Interface and decoding settings are summarized in Supplementary Table B.2.Source: §B.2 · p. 21, Table B.2 · p. 21
  • Where endpoints expose temperature, decoding is deterministic, and all modes share an 8,192-token API rerun budget; at most 0.35% of equal-budget outputs reach the cap.Provider reasoning signals are used only for the strict |T| = 0 decision.Source: §B.2 · p. 21

C Evaluation Protocol

The appendix applies the response-level evaluation protocol to pre-answer text and final answers, then specifies how metrics and analyses are aggregated and scoped.

C Evaluation Protocol

  • Outputs are split into pre-answer text T and final answer A, with Acc., ETR, QRel., and EIR reported as in the main text.Exposed provider reasoning signals check the answer-only boundary when available.Source: C Evaluation Protocol · p. 22
  • Grouped Boolean, multiple-choice, and open-ended aggregates are item-weighted averages over two datasets per answer family; frontier aggregates use the 8,192-token rerun with seed 20260502.Supplementary Table C.1 summarizes evaluation scope for appendix result blocks.Source: C Evaluation Protocol · p. 22, Table C.1 · p. 22
  • QRel. uses the complete visible pre-answer text, while GPT-5.5 EIR covers global, frontier, scaling, and domain analyses; question-structure controls examine relevance and consistency.Source: C Evaluation Protocol · p. 22

D.1 Level 2: Encoder Robustness

Alternative retrieval encoders preserve the main answer-space and intervention relevance patterns, supporting Qwen3-Embedding-4B as the primary encoder while providing sensitivity checks.

D.1 Level 2: Encoder Robustness

  • Qwen3-Embedding-4B is the primary Level 2 encoder because its instruction-aware retrieval interface lets the query specify the intended relation between Q and T.The full-corpus comparison with all-MiniLM-L6-v2 focuses on M2/M5.Source: §D.1 · p. 22, Table D.1 · p. 23
  • BGE-base-en-v1.5 and E5-base-v2 are sensitivity checks on a balanced 3,600-output sample, not replacements for the primary metric.The sample contains 50 outputs from each of 72 model, answer-space, and intervention cells; the alternatives lack the primary encoder’s reasoning-focused instruction interface.Source: §D.1 · p. 23, Table D.2 · p. 23
  • Every encoder preserves Bool < MCQ < Open under M2 and M5 < M2 in every answer space.The alternative encoders preserve the same directions despite different absolute scales.Source: §D.1 · p. 23, Table D.1 · p. 23, Table D.2 · p. 23

D.2 Level 3: Cross-Family Judge Robustness

An independent judge reproduces the main answer-space and intervention ordering, while human annotations validate the relevance and explicit-inference rankings across sampled settings.

D.2 Level 3: Cross-Family Judge Robustness

  • Only 355 of 572,688 outputs (0.06%) are labeled paraphrase-only, and these do not contribute to EIR.The five-way rubric separates semantic paraphrases and relevant non-inferential text from explicit inference.Source: §D.2 · p. 23, Table D.4 · p. 24
  • Both judges preserve Bool < MCQ < Open under M2 and M5, and M5 < M2 in every answer space.Claude Opus 4.8 identifies explicit inference in 61.02% of Open M5 outputs, close to GPT-5.5’s 62.00%.Source: Table D.3 · p. 23, §D.2 · p. 24

D.3 Human Validation Protocol

  • The human study samples 600 outputs across benchmarks and hides dataset, model, mode, final answer, correctness, QRel., and judge labels from annotators.Five annotators review the same sample; empty T receives relevance score 1 and no visible inference, while the 402 non-empty texts are reviewed by each annotator.Source: §D.3 · pp. 24–25
  • Human relevance scores distinguish irrelevant or weakly related text from on-topic non-evidential content, question-specific support, and multi-step support.The separate five-way reasoning annotation counts only inferential acts such as deduction, computation, comparison, or evidence-to-conclusion as explicit reasoning.Source: §D.3 · p. 25
  • Rates use all sampled outputs per M2/M5 cell, and agreement is reported as exact five-way labels or binary explicit-reasoning classification.Krippendorff’s α is computed across the five annotators on the 402 independently reviewed non-empty texts.Source: §D.3 · p. 25
  • Across six sampled cells, mean human rates rank exactly with QRel. and GPT-5.5 EIR (ρ = 1.000), and nearly with Opus 4.8 EIR (ρ = 0.986).The rankings compare mean human rates with the respective aggregate metric rankings.Source: §D.3 · p. 26

E Measurement Robustness Audits

Audits find that copying, paraphrase-only labels, and answer-first ordering are rare, while an independent prompt check supports a literal answer-only reading.

E Measurement Robustness Audits

  • The audits cover 572,688 successful records and test copying, paraphrase mistaken for inference, and explanations appearing after an answer.The answer-first check uses outputs with a parseable boxed or fboxed span.Source: E Measurement Robustness Audits · p. 26
  • Only 301 of 572,688 responses (0.053%) are flagged for near-verbatim copying, limiting the worst-case global mean QRel. change to 0.00053.The bound assumes every flagged record received QRel. = 1 solely because of copying and should instead receive zero.Source: E Measurement Robustness Audits · p. 26, Table E.1 · p. 27
  • Only 355 of 572,688 outputs (0.06%) receive the paraphrase-only label, which is excluded from the EIR numerator.The separate label applies whether or not paraphrase wording overlaps lexically with the question.Source: E Measurement Robustness Audits · p. 26
  • Answer-first responses occur in 149 of 535,208 boxed outputs (0.028%), so T → A is dominant but not universal.The audit requires the first boxed answer to follow at most three lexical tokens and be followed by at least three.Source: E Measurement Robustness Audits · p. 26, Table E.1 · p. 27
  • An independent GPT interpretation reads the answer-only instruction as returning only the result without explanation, making prompt ambiguity an unlikely explanation for Figure 1.The interpretation matches M5 and the high ETR observed where models comply.Source: E Measurement Robustness Audits · p. 27, Figure E.1 · p. 28

F.1 Observed counterparts of the five regimes in Figure 3

Real outputs illustrate five distinct combinations of pre-answer length and question relevance, with four cases sharing an MMLU item and one using StrategyQA.

F.1 Observed counterparts of the five regimes in Figure 3

  • The five regimes are descriptive examples of length and relevance varying independently, not additional threshold-based labels.Cases 1–4 use the same sampled MMLU item; the fifth uses StrategyQA because no evaluated model–mode pair produced a genuinely long, low-QRel. response on the shared MMLU item.Source: §F.1 · p. 28, Table F.1 · p. 29
  • The examples include answer-only output, short high-QRel. text, and short low-QRel. text, all on the same MMLU question.The answer-only case has empty T; the short high-relevance case includes a question-specific fact, while the short low-relevance case says only that the answer is correct.Source: §F.1 · pp. 28–29
  • The long high-QRel. example gives a worked comparison on the shared MMLU question, while the long low-QRel. example uses a StrategyQA item.All five example final answers are correct.Source: §F.1 · pp. 29–30, Table F.1 · p. 29

F.2 Cases behind the aggregate findings

  • Four representative outputs illustrate aggregate patterns of residual visible information, re-elicitation, compression cost, and candidate visibility, but are not additional aggregate evidence.The cases are presented as examples behind the aggregate findings.Source: §F.2 · p. 30, Table F.2 · p. 30

G Additional Global Results

The local Qwen results reproduce the main-text pattern: stronger answer-only compliance can coexist with question-relevant, often inferential text, and interventions do not act monotonically. Dataset-level figures also summarize the M2-to-M5 accuracy and compliance changes.

Global six-mode profiles

Dataset-level trade-off

H Detailed Experiment Tables

The detailed results extend the answer-space pattern across providers, model sizes, datasets, and MMLU domains. Scaling improves accuracy but does not remove the answer-space structure, and domain compressibility varies.

Frontier native no-think behavior

Qwen scaling across all modes

MMLU domain view

I Question-Structure Details

These analyses vary question structure while holding model family and mode fixed, examining surface form, candidate visibility, and answer generation. The results distinguish those factors from simple label or option-order effects.

Question-structure controls

  • The controls vary task instances while holding model family and mode fixed, separating answer-support structure from surface order, candidate visibility, and numeric-source generation.Source: I Question-Structure Details · p. 34

MMLU subject view

Surface perturbations

Candidate visibility

Removing options from numeric-source items

  • When options disappear from numeric-source items, strict answer-only pressure either brings back question-relevant pre-answer content or costs accuracy; Parse and Exact separate generation from parsing effects.Source: I Question-Structure Details · p. 37, Table I.5 · p. 37

Related papers

Loading 2610.11765v1…