PaperlayerView exact revision on arXiv

Source-linked AI summary

Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worlds

Anish R Joishy, Ishwar B Balappanawar, Vamshi Krishna Bonagiri, Manas Gaur, Krishnaprasad Thirunarayan, Ponnurangam Kumaraguru

arXiv:2505.22318v2cs.CLcs.LG

TL;DR

LLMs can struggle to reason when premises conflict with parametric knowledge, and existing benchmarks did not systematically test multi-step logic under such conflicts. The paper introduces CounterLogic and proposes Flag & Reason (FaR), which reduces the measured performance gap in evaluations across six reasoning datasets.

  • Problem

    Existing benchmarks did not systematically disentangle logical validity from knowledge alignment in multi-step reasoning, despite LLMs’ performance degradation when information conflicts with parametric knowledge.

    Source: §I · p. 1, §III · p. 3

  • Method

    The paper introduces CounterLogic to evaluate multi-step reasoning under knowledge conflicts and proposes FaR, which prompts models to flag factual believability before reasoning.

    Source: §I · p. 2

  • Results

    14%: Across six reasoning datasets, accuracy fell on average in counterfactual scenarios; FaR narrowed the gap from 14% to 7% and increased overall accuracy by 4%.

    Source: C. Results · p. 7

  • Takeaways & Limitations

    FaR helps models compartmentalize knowledge conflicts and anchor deduction in the provided logical context.

    Source: §VI · p. 7

    The study focuses on foundational logical structures, tested models, and English common-knowledge concepts; results may differ across other languages, cultures, or specialized domains.

    Source: §VII · p. 7

Abstract

from arXiv · show

A fundamental challenge in reasoning is navigating hypothetical, counterfactual worlds where logic may conflict with ingrained knowledge. We investigate this frontier for Large Language Models (LLMs) by asking: Can LLMs reason logically when the context contradicts their parametric knowledge? To facilitate a systematic analysis, we first introduce CounterLogic, a benchmark specifically designed to disentangle logical validity from knowledge alignment. Evaluation of 11 LLMs across six diverse reasoning datasets reveals a consistent failure: model accuracy plummets by an average of 14% in counterfactual scenarios compared to knowledge-aligned ones. We hypothesize that this gap stems not from a flaw in logical processing, but from an inability to manage the cognitive conflict between context and knowledge. Inspired by human metacognition, we propose a simple yet powerful intervention: Flag & Reason (FaR), where models are first prompted to flag potential knowledge conflicts before they reason. This metacognitive step is highly effective, narrowing the performance gap to just 7% and increasing overall accuracy by 4%. Our findings diagnose and study a critical limitation in modern LLMs' reasoning and demonstrate how metacognitive awareness can make them more robust and reliable thinkers.

I. INTRODUCTION

LLMs can struggle to follow logically valid premises that conflict with their parametric knowledge, and existing benchmarks had not systematically tested multi-step reasoning under counterfactual assumptions. The paper introduces CounterLogic to study this gap and proposes Flag & Reason (FaR), which reduces the measured performance gap.

I. INTRODUCTION

  • 14%: Across 11 LLMs and six reasoning datasets, accuracy dropped on average in counterfactual scenarios compared with knowledge-aligned ones.Source: §I · p. 2
  • The gap exceeded 30% on tasks requiring negation and was minimal on simpler logical forms.Source: §I · p. 2
  • CounterLogic was introduced to disentangle logical validity from knowledge alignment and probe multi-step reasoning under counterfactual assumptions.Source: §I · pp. 1–2
  • FaR prompts models to flag whether a conclusion is factually believable before reasoning, anchoring deductions in the acknowledged conflict.Source: §I · p. 2
  • FaR narrowed the performance gap to 7% and increased overall accuracy by 4% on average.Source: §I · p. 2

A. Logical Reasoning in LLMs

Prior work describes persistent reasoning errors in LLMs, especially when premises conflict with stored knowledge, while metacognitive approaches offer ways to separate belief evaluation from logical inference. The paper introduces CounterLogic to evaluate this issue in controlled multi-step tasks.

A. Logical Reasoning in LLMs

B. Knowledge Conflicts and Counterfactual Reasoning

C. Metacognition and Human Reasoning

III. THE COUNTERLOGIC DATASET

  • CounterLogic fills a benchmark gap with 1,800 examples across nine logical schemas, scaling from one to nine reasoning steps while balancing knowledge-conflicting scenarios.Source: Table I · p. 2, §III · p. 3

A. Dataset Construction

CounterLogic is generated from hierarchical entity relationships and formal logical schemas, with controls for reasoning depth, validity, and knowledge alignment. The evaluation applies self-consistency checks to 11 LLMs across CounterLogic and five other datasets, using standard chain-of-thought prompting as a baseline.

A. Dataset Construction

  • Hierarchical entity triples generate paired sentence templates, with half using correct entity hierarchies and half using inverted ones.Source: A. Dataset Construction · p. 3
  • The generated propositions instantiate nine logical schemas and balance examples by logical validity and whether conclusions align with parametric knowledge.Source: A. Dataset Construction · p. 3
  • Reasoning chains scale from one to nine steps, and GPT-4o paraphrases final logical queries into natural-language questions.Source: A. Dataset Construction · p. 3

A. Experimental Setup

  • The evaluation tests 11 LLMs on CounterLogic and five other reasoning datasets, using self-consistency checks and standard CoT prompting without FaR as the baseline.Source: A. Experimental Setup · p. 3

B. Consistent Performance Gap

LLMs show a substantial accuracy gap between knowledge-aligned and counterfactual reasoning, and the gap persists across reasoning depths and logical schemas. It is especially pronounced for negation and structurally complex forms.

C. Deconstructing the Failure

  • The performance gap is largely independent of reasoning depth, persisting as additional reasoning steps are added.The analysis therefore attributes the gap to unresolved conflict with parametric knowledge rather than losing track of longer deductions.Source: C. Deconstructing the Failure · p. 4
  • The gap occurs across all nine logical schemas but exceeds 30 percentage points for Disjunctive Syllogism and Destructive Dilemma.The latter forms require reasoning over negated propositions.Source: C. Deconstructing the Failure · p. 4

B. Consistent Performance Gap

  • 14%: accuracy falls on average in counterfactual scenarios compared with knowledge-aligned scenarios.For valid problems, average accuracy is 72% on knowledge-aligned examples and 58% on counterfactual examples.Source: B. Consistent Performance Gap · p. 3, Figure 3 · p. 5

C. Deconstructing the Failure

  • 73%: baseline accuracy on schemas involving negation, versus 84% on schemas without negation.The paper characterizes the overall weakness as interference from parametric knowledge exacerbated by structural complexity and negation.Source: C. Deconstructing the Failure · p. 5

V. FLAG & REASON (FAR): A METACOGNITIVE INTERVENTION

The paper hypothesizes that unresolved conflict between premises and ingrained knowledge disrupts deduction, and proposes Flag & Reason (FaR), a separate metacognitive flagging step before reasoning. Across models and tasks, FaR narrows the average gap from 14% to 7% and raises overall accuracy by 4%.

A. The Belief Inhibition Hypothesis

  • The authors hypothesize that belief inhibition fails when models cannot suppress ingrained parametric knowledge that contradicts task premises.They relate this conflict to human belief bias and metacognition: recognizing a belief-premise conflict and setting belief aside to follow logic.Source: A. The Belief Inhibition Hypothesis · p. 5
  • Adding reasoning steps does not cause a significant performance drop, while the aligned-conflicting gap persists across addition counts.At zero additions, FaR improves performance from 80.22% to 83.4%, and the scenario gap is about 6%.Source: Figure 4 · p. 6

B. The Flag & Reason (FaR) Intervention

  • FaR first flags whether a premise or conclusion aligns with parametric knowledge, then reasons; combining both steps in one prompt removes the performance benefit.The authors interpret the separate flag as enabling epistemic compartmentalization before the reasoning step.Source: B. The Flag & Reason (FaR) Intervention · p. 6

C. Results

  • 7%: FaR narrows the average knowledge-aligned versus counterfactual performance gap from 14% to 7%, while increasing overall accuracy by 4%.Figure 6 compares baseline and FaR accuracy across models; the reported averages cover all models and tasks.Source: Figure 6 · p. 7, C. Results · p. 7
  • 5%: FaR increases CounterLogic overall valid accuracy; negation-task average accuracy rises from 73% to 79%, while FOLIO shows less improvement.The improvement is reported across logical schemas, including more complex schemas with lower baseline performance.Source: C. Results · p. 7

D. Thought Anchor Analysis

The analysis links Flag & Reason (FaR) to a shift in models’ reasoning anchors, while the conclusion reports a smaller counterfactual accuracy gap and higher overall accuracy. The paper also identifies limits in dataset coverage, tested models, language, domain, and the interpretation of human-cognition parallels.

D. Thought Anchor Analysis

  • With FaR, reasoning anchors shift from fact retrieval and uncertainty management toward plan generation and active computation.The analysis says this shift suggests a more stable path for deduction, with less interference from ingrained knowledge.Source: D. Thought Anchor Analysis · p. 7

VI. CONCLUSION AND FUTURE WORK

  • FaR halves the counterfactual accuracy gap from 14% to 7% and boosts overall accuracy by 4%.The conclusion attributes the mechanism to compartmentalizing knowledge conflict so deduction stays anchored in the provided logical context.Source: §VI · p. 7

VII. LIMITATIONS

  • The study is bounded by foundational CounterLogic structures, tested models, English and common-knowledge concepts, conceptual cognition parallels, and possible synthetic-query biases.Future work could examine internal representations, more complex real-world domains, and methods to resolve tensions between parametric memory and in-context logic.Source: §VII · p. 7, §VI · p. 7

APPENDIX

The appendix describes how CounterLogic examples are constructed and how prompting adapts logical tasks for reflection and reasoning. It also explains the benchmark’s balance criteria and how reasoning depth is varied.

A. Hierarchical Entity Triples

  • CounterLogic represents entities as triples of strict subset relationships, such as a ⊂ b ⊂ c.The benchmark builds examples from these hierarchical entity relations.Source: A. Hierarchical Entity Triples · p. 10

B. Sentence Templates

  • Sentence pairs instantiate four complementary or contradictory quantifier templates, including “All A are B” and “Some A are not B.”The listed templates pair universal and existential statements with their negations or complements.Source: B. Sentence Templates · p. 10

C. Dataset Statistics

  • The 1,800-example dataset covers nine logical schemas, balancing knowledge alignment, validity, entity relationships, and sentence templates.Each schema has 200 instances; half are knowledge-aligned and half knowledge-conflicting, and half are logically valid and half invalid.Source: C. Dataset Statistics · p. 10

D. Varying Reasoning Depth

  • Reasoning depth increases by replacing implications with longer chains of intermediate propositions while preserving knowledge alignment.The method excludes Disjunctive Syllogism and Commutation because their forms do not involve implications.Source: D. Varying Reasoning Depth · p. 10

E. Prompt for Natural Language Reformulation of Logical Structures

  • The prompting procedure reformulates formal rules into natural-language contexts and asks models to judge logical validity regardless of premise factuality.The standard condition proceeds directly to reasoning; the metacognitive condition first asks whether a statement is factually correct.Source: E. Prompt for Natural Language… · p. 11
  • FaR first assesses isolated plausibility, then supplies the full reasoning context, with task-specific inputs for syllogisms, KNOT, FOLIO, LogicBench, and CounterLogic.For datasets without alignment labels, the study uses LLM factuality judgments as proxy labels; KNOT’s aligned-to-conflicting ratio is approximately 1:15.Source: E. Prompt for Natural Language… · pp. 11–12

F. Model Access

The section describes the access route and standardized inference settings used across model providers.

F. Model Access

  • Models were accessed through the OpenRouter API to standardize evaluation across providers.The setup covered different model providers.Source: F. Model Access · p. 12
  • The access setup standardized inference parameters across model providers.The reported shared settings were temperature 0.7, top-p 0.95, and a 4096-token maximum.Source: F. Model Access · p. 12
  • Inference used temperature 0.7, top-p 0.95, and a maximum length of 4096 tokens.These settings were used to standardize inference parameters across providers.Source: F. Model Access · p. 12

G. Model Selection Criteria

Across the evaluated models, FaR improved average reasoning accuracy; the study selected a diverse set of systems and reports their access and evaluation details. Model descriptions span OpenAI, Google, Meta, Alibaba, and DeepSeek.

G. Model Selection Criteria

  • Model selection targeted state-of-the-art performance, architectural and parameter-scale diversity, and varied training paradigms.The selected set spans models from 7B to 72B+ parameters.Source: G. Model Selection Criteria · p. 12

1) OpenAI Models:

  • The OpenAI group includes GPT-4o, GPT-4o-mini, and o3-mini, with o3-mini described as a reasoning model using specialized tokens for internal CoT.GPT-4o is described as the standard non-reasoning model and GPT-4o-mini as a lighter, faster version.Source: 1) OpenAI Models · p. 12

2) Google Models:

  • Gemini-Flash-1.5 is described as a lightweight Gemini 1.5 model optimized for quick responses while maintaining strong reasoning capabilities.Source: 2) Google Models · p. 12

3) Meta AI Models:

  • The Meta group includes Llama 3.3-70B and Llama 3.1 variants at 70B and 8B, spanning larger and smaller parameter scales.The passage describes the 3.1 series as an upgrade with enhanced instruction-following and reasoning capabilities.Source: 3) Meta AI Models · p. 12

4) Alibaba Models:

  • The Alibaba group includes Qwen-2.5-72B and Qwen-2.5-7B, described as strong on benchmarks, particularly multilingual reasoning.Source: 4) Alibaba Models · p. 12

5) DeepSeek Models:

  • The DeepSeek group includes DeepSeek-V3, a 671B dialogue model, and DeepSeek-R1-Distill-Llama, fine-tuned from DeepSeek-R1 using Llama-3.3-70B-Instruct outputs.Source: 5) DeepSeek Models · p. 12

I. Model Inference Parameters

  • Evaluations used OpenRouter for non-OpenAI models and the OpenAI API for three OpenAI models, with system-prompt-capable models receiving a standard instruction-following prompt.Source: §I · p. 13

J. Cost of the Evaluations

  • FaR improved average reasoning accuracy across all 11 evaluated models, with GPT-4o gaining approximately 7%.Figure 6 shows FaR outperforming the baseline for every model; the ablation caption says CoT always beats zero-shot, while few-shot does not for some Syllogistic models.Source: J. Cost of the Evaluations · p. 13, Figure 7 · p. 13

K. Key Performance Patterns

FaR improves accuracy across evaluated tasks, though its impact varies by dataset and model. Further analyses show that prompting and factual evidence continue to influence logical judgments.

K. Key Performance Patterns

L. Comparison of Prompting Strategies

  • Across prompting strategies, aligned datapoints outperform gibberish, which outperform conflicting datapoints; CoT improves accuracy, while few-shot effects vary by model.OpenAI models including GPT-4o-mini degrade in few-shot settings, while most others maintain or slightly improve performance.Source: L. Comparison of Prompting Strategies · p. 14

M. Perturbing with Model-Generated Evidence

  • For logically valid datapoints, supporting generated evidence raises accuracy and contradictory evidence lowers it, despite instructions to judge premise-conclusion logic alone.Source: M. Perturbing with Model-Generated… · p. 15

N. Quantifying Counterfactual Bias of Models

  • SPD values are broadly positive across datasets, indicating a bias toward factually aligned answers even when they do not follow from the context.SPD compares positive-answer likelihood in believable versus unbelievable scenarios, independently of logical correctness.Source: N. Quantifying Counterfactual Bias of… · p. 16

O. Single Prompt FaR

The appendix reports that combining FaR and evaluation prompts in one prompt does not improve performance over baseline. It also outlines hypotheses and open questions about knowledge interference and its possible resolution.

O. Single Prompt FaR

  • Combining the FaR and evaluation prompts in one prompt shows no performance gains over baseline, unlike standard FaR with separate prompts.Source: O. Single Prompt FaR · p. 16

P. Competing Pathways in Knowledge Interference

  • The proposed dual-path hypothesis is that failures may reflect competition between retrieved parametric knowledge and controlled reasoning from current premises.Source: P. Competing Pathways in Knowledge… · p. 16
  • The hypothesized conflict’s exact neural implementation remains unknown; pathway analysis is proposed as a way to test whether distinct neural substrates mediate it.Source: P. Competing Pathways in Knowledge… · p. 16
  • Positive SPD values across datasets indicate bias toward factually correct answers even when they may not fit the given context.Source: Figure 11 · p. 17

Q. Metacognitive Dissonance in Artificial Systems

R. Epistemic Compartmentalization Mechanisms

  • Metacognitive prompts may transiently suppress parametric knowledge; the authors connect this possibility to knowledge editing and hippocampal indexing approaches.The figure frames believability assessments as guiding inference strategies for resolving conflict.Source: R. Epistemic Compartmentalization… · p. 17, Figure 12 · p. 17
  • A proposed research direction is to identify mechanisms for temporary belief suspension; layer-wise relevance propagation could test for distinct activation patterns in higher transformer layers.Source: R. Epistemic Compartmentalization… · p. 17

S. Parametric Knowledge as Malleable Belief Systems

The analyses describe sensitivity to entity order despite identical logical structure, and compare how reasoning steps and token use differ across examples and with FaR. They also raise questions about how contextual evidence may affect model beliefs.

S. Parametric Knowledge as Malleable Belief Systems

Loading 2505.22318v2…