Are Large Reasoning Models Good Everyday Reasoners? A Comparison with Non-Reasoning Models on Logical Reasoning Tasks

• Whether reasoning capabilities transfer beyond formal domains remains unclear, despite models’ strong performance on mathematical and scientific benchmarks and growing real-world use.The introduction notes their deployment in conversational systems and digital agents.Source: §1 · p. 1
• The study introduces a pattern-controlled logical reasoning task and dataset to compare reasoning and non-reasoning models on everyday reasoning.The evaluation focuses on the gap between everyday reasoning and formal reasoning.Source: §1 · p. 1
• Reasoning models perform near-perfectly when everyday patterns align with formal logic, but their everyday alignment degrades when the two diverge.The authors characterize this as a systematic tendency to prioritize formal rules, even when they yield intuitively unacceptable conclusions.Source: §1 · pp. 1–2
• Some chain-of-thought-prompted non-reasoning models align better with everyday reasoning than reasoning models, while additional instructions provide limited, asymmetric control.Formal alignment can improve substantially, but everyday alignment remains constrained and meta-level prompting does not induce adaptive reasoning.Source: §1 · p. 2

2 Evaluating Language Models on Everyday Reasoning

• The section introduces a logical reasoning task and dataset to evaluate reasoning models in everyday contexts and compare them with non-reasoning models.Source: §2 · p. 2

2.1 Everyday Acceptability and Formal Validity

• Everyday acceptability means general acceptance under everyday reasoning or natural interpretation, whereas formal validity means validity under standard logical systems.The two judgments can diverge, creating the gap cases targeted by the evaluation.Source: §2.1 · p. 3

2.2 Evaluation Task

• The Pattern-Controlled Logical Reasoning Task tests whether models follow everyday patterns using inference problems drawn from all four combinations of everyday acceptability and formal validity.Instances present premises and a conclusion; answers are yes, no, or undetermined, with the last option covering judgments dependent on context or underdetermined by the available information.Source: §2.2 · p. 3

2.3 Control Reasoning Patterns

• The pattern labels yield four configurations; the two gap cases distinguish formal over-generation from under-generation relative to everyday reasoning.Formally valid but intuitively unacceptable cases contrast with intuitively acceptable cases not licensed by standard formal systems; the framework tests for formalist bias.Source: §2.3 · pp. 3–4

2.4 Dataset Construction

• The English dataset contains 200 instances—20 for each of 10 inference patterns—and uses a semi-synthetic generation and expert-selection procedure.Gemini 3 Flash generates additional instances from seed examples, and logic experts select representative examples for each pattern.Source: §2.4 · p. 4

3.2 Evaluation Metrics

• Alignment rate measures agreement with everyday labels across the dataset, while Formalist Bias measures the direction of preference on items where formal and everyday labels diverge.The metrics are complementary: high overall alignment can coexist with a systematic preference for formal reasoning on gap items.Source: §3.2 · p. 5

3.1 Models and Experimental Setup

• Reasoning models and non-reasoning models with CoT are compared using zero-shot prompting for the former and both zero-shot and CoT prompting for the latter.Two model pairs share base models, enabling controlled comparisons between reasoning and non-reasoning variants.Source: §3.1 · p. 5, §3.3 · p. 6

3.3 Experiment 1: Basic Model Evaluation and Comparison

• When formal validity and everyday acceptability diverge, reasoning-model alignment drops substantially, and same-base non-reasoning models with CoT often align better.Alignment is generally lower for valid-but-unacceptable than invalid-but-acceptable patterns.Source: Figure 2 · p. 6, §3.3 · p. 7
• For Ross, VT, and eSI, alignment is below 50% across models; reasoning-model alignment on VT is close to 0%.dFC and eFC have relatively higher alignment, while non-reasoning models vary more across patterns.Source: Figure 3 · p. 6, §3.3 · p. 7

3.4 Experiment 2: Evaluation with Context Instructions

• Everyday Context shifts all models somewhat toward everyday reasoning, but alignment remains incomplete and the effect varies across models.For olmo-3.1-32b-think, the prompt improves alignment by approximately 0.2 relative to olmo-3.1-32b-instruct with CoT; the trend does not hold for qwen3-30b-a3b-thinking-2507.Source: §3.4 · pp. 7–8
• Formal Context yields strong formal alignment for most models, whereas Meta Context produces limited, inconsistent shifts and does not reliably induce adaptive reasoning.Under Formal Context, most models except llama-3.3-70b-instruct have FB scores close to 1; under Meta Context, some shift toward everyday reasoning while others change little or shift slightly toward formal reasoning.Source: Figure 4 · p. 7, §3.4 · p. 8
• Reasoning-model traces have more explicit, structurally identifiable signals, while non-reasoning-model traces are more diffuse and ambiguous.Disagreement for reasoning models mainly concerns aggregating agreed-upon signals into one category; for non-reasoning models it persists after decomposition.Source: §3.4 · p. 8

Reasoning and Non-Reasoning Models

• Post-training and Chain-of-Thought prompting have advanced LLM reasoning, but whether benchmark gains extend to everyday logical reasoning remains unclear.Examples of reasoning models include DeepSeek-R1 and OpenAI’s o1 and o3.Source: §4 · pp. 8–9

Biases and Limitations in LLM Reasoning

• Prior work studies content bias, while this work asks which notion of validity models prioritize when formal validity and everyday acceptability diverge.The distinction is whether models ignore logical structure or favor formal validity over everyday acceptability.Source: §4 · p. 9

Everyday vs. Formal Reasoning in LLMs

• Holliday et al. (2024) examine diverse conditional and modal inferences, whereas this work treats their formal-versus-everyday divergence as a unified evaluation perspective.The cited survey analyzes discrepancies across reasoning patterns without treating the gap as one unified phenomenon.Source: §4 · p. 9
• Here, everyday reasoning means everyday logical entailment under natural interpretations, excluding external world knowledge and non-deductive inference.The stated aim is to isolate how models resolve conflicts between formal logical structure and natural interpretation.Source: §4 · p. 9

5 Discussion

• Reasoning models are not necessarily well aligned as everyday reasoners and exhibit a systematic formalist bias.The discussion frames this bias as a gap between everyday acceptability and formal validity.Source: §5 · p. 9
• Current reasoning models appear specialized toward formal reasoning, while non-reasoning models can produce more natural everyday judgments and arguments.This challenges the assumption that reasoning models are a strict improvement in reasoning ability apart from generation speed.Source: §5 · p. 9
• More general reasoning models require training and evaluation beyond formal, logic-based mathematical and scientific tasks.The discussion also identifies flexible, context-sensitive reasoning as an ongoing challenge because appropriate reasoning may vary by context.Source: §5 · p. 9
• The dataset uses canonical patterns where formal validity and intuitive judgment diverge, but whether individual instances match human judgments remains open.The authors call for empirical grounding through human judgments at scale.Source: §5 · pp. 9–10

A Details of Reasoning Patterns

• The dataset covers all four combinations of formal validity and everyday acceptability, with each pattern characterized by both properties.The patterns are summarized in Table 1.Source: A Details of Reasoning Patterns · p. 12
• Modus Ponens and Modus Tollens are both formally valid and intuitively acceptable inference patterns.Modus Ponens is described as canonical deductive reasoning; Modus Tollens is another standard valid inference.Source: A Details of Reasoning Patterns · p. 12
• Vacuous Truth and Ross’s Paradox are formally valid but intuitively unacceptable patterns.Vacuous Truth makes a conditional true when its antecedent is false; Ross’s Paradox derives an obligation to tell the truth or lie from an obligation to tell the truth.Source: A Details of Reasoning Patterns · p. 12
• Free Choice Permission treats permission of A or B as supporting permission of A, an intuitively acceptable inference that is not formally valid in SDL.The example is “You may have cake or fruit,” from which one naturally infers “You may have cake”; accounts differ without a unified treatment.Source: A Details of Reasoning Patterns · p. 12
• Epistemic Free Choice similarly treats “It might be A or B” as supporting “It might be A,” though the inference is not formally valid.The source notes that various accounts exist but no unified treatment is agreed upon.Source: A Details of Reasoning Patterns · pp. 12–13
• Deontic and epistemic scalar implicatures infer permission or possibility of the alternative from permission or possibility of A.Both are described as pragmatic inferences based on reasoning over alternatives.Source: A Details of Reasoning Patterns · p. 13
• Epistemic Disjunction Elimination is formally invalid and intuitively rejected: necessity of A or B does not establish necessity of A.The example contrasts “It must be that it is raining or snowing” with the unsupported conclusion that it must be raining.Source: A Details of Reasoning Patterns · p. 13

B Evaluation Metrics

• Everyday alignment rate measures how well model predictions match everyday labels across the full dataset.The metric is defined over all N items.Source: B Evaluation Metrics · p. 13
• Gap items are cases where formal validity and everyday judgments disagree, and their signed scores distinguish formal alignment, everyday alignment, and abstention.The score is 1 for following the formal label, −1 for following the everyday label, and 0 for abstaining.Source: B Evaluation Metrics · p. 13
• Formal Bias summarizes preference direction: positive values indicate formal reasoning, while negative values indicate everyday reasoning.The section also defines pattern-wise Formalist Bias to compare that preference across inference patterns.Source: B Evaluation Metrics · p. 14
• FA and EA measure agreement with formal and everyday labels, respectively, while RR measures commitment to either label rather than abstention.The section characterizes FB as preference direction and RR as degree of commitment.Source: B Evaluation Metrics · p. 14

C Prompts

• Tables 3 and 4 provide prompt examples for Experiments 1 and 2.Table 3 is for Experiment 1; Table 4 provides additional examples for Experiment 2.Source: C Prompts · p. 14, Table 3 · p. 14, Table 4 · p. 15

D.1 Experiment 1

• Figure 5 presents results across reasoning patterns and models, including non-reasoning models with and without chain-of-thought prompting.The figure distinguishes everyday-aligned, undetermined, and formal-aligned predictions.Source: §D.1 · p. 16
• Tables 5–8 report model alignment rates by reasoning pattern and by everyday-acceptability and formal-validity labels.Source: §D.1 · p. 16
• Table 9 reports average everyday-reasoning alignment rates by reasoning pattern.Source: §D.1 · p. 16

D.2 Experiment 2

• Non-reasoning models show negative Formalist Bias under the base condition, while reasoning models show positive bias, indicating everyday versus formal preferences, respectively.Table 10 reports Formalist Bias scores across models and prompting conditions.Source: §D.2 · p. 16, Table 10 · p. 18
• Table 11 breaks down gap-item behavior using formal alignment, everyday alignment, Formalist Bias, and resolution rate.Formalist Bias is the difference between formal and everyday alignment; resolution rate is the proportion of cases where a model commits to either label.Source: §D.2 · p. 16, Table 11 · p. 19
• The section also provides average everyday-alignment rates by reasoning pattern and a distribution of prediction alignments across patterns.Table 9 reports averages; Figure 5 distinguishes everyday-aligned, undetermined, and formal-aligned predictions.Source: Figure 5 · p. 17, Table 9 · p. 18

E Analysis of Reasoning Traces

• The trace analysis aimed to characterize reasoning style rather than evaluate correctness, using 34 responses each from two same-architecture Qwen models.Sampling sought coverage of correct and incorrect cases across reasoning patterns; labels were Formal, Everyday, Undetermined, or Mixed, with auxiliary flags for explicit cues.Source: E Analysis of Reasoning Traces · p. 20
• Primary-label agreement was low (κ = 0.29 for reasoning models; κ = 0.15 for non-reasoning models), with disagreement concentrated between Formal and Mixed.Two annotators independently labeled each trace.Source: E Analysis of Reasoning Traces · p. 20
• Relabeling from explicit reasoning signals raised agreement for reasoning models from κ = 0.29 to κ = 0.48, but non-reasoning-model agreement remained low, moving from κ = 0.15 to κ = 0.19.The authors interpret the improvement for reasoning models as evidence that their traces contain explicit, consistently identifiable signals.Source: E Analysis of Reasoning Traces · p. 20
• Examples show that traces can mix or diverge in style across cases where formal validity and everyday acceptability conflict.The examples include everyday-aligned predictions labeled both Everyday and Mixed, and formal-aligned predictions labeled Formal or Mixed.Source: Figure 6 · p. 21, Figure 7 · p. 22, Figure 8 · p. 23