Language models show human-like content effects on reasoning

Motivation

• Although abstract reasoning is ideally content-independent, human reasoning is entangled with prior knowledge and beliefs about the problem’s content.Humans reason more readily about realistic situations than abstract attributes, and this content sensitivity also appears in probabilistic reasoning.Source: §1 · pp. 1–2
• Language models’ training-shaped prior expectations often reflect human knowledge and beliefs, which may influence their answers to reasoning problems.The paper links this possibility to the “pre-train and adapt” or “foundation models” paradigm (Bommasani et al., 2021).Source: §1 · p. 2

Research question

• The paper asks whether language-model biases on abstract reasoning resemble human content effects, building on concerns about models’ inconsistent abstract reasoning.Prior critiques include Rae et al. (2021), Razeghi et al. (2022), Patel et al. (2021), and Valmeekam et al. (2022).Source: §1 · pp. 1–2

Key findings

• Across NLI, syllogism validity, and Wason tasks, language models show content effects resembling patterns observed in humans.Models are influenced by prior knowledge and abstract reasoning in NLI, by conclusion believability in syllogisms, and perform better with realistic Wason framing.Source: §1 · p. 2

Implications

• The findings highlight content as a factor in model reasoning and suggest that human-like biases may make some reasoning errors difficult for people to detect.The paper also argues that cognitive-science theories and experimental paradigms can help probe complex machine-learning behavior.Source: §1 · p. 3

2.1. Natural Language Inference

• NLI classifies whether a hypothesis entails, contradicts, or is neutral to a premise; the authors use logically generated comparisons because crowdsourced datasets may not follow strict logical definitions.Because this task is relatively simple, adult humans would likely perform well, and human content effects had not been examined on it.Source: §2.1 · p. 3

2.2. Syllogisms

• Human syllogism judgments are biased by whether the conclusion is believable, even when the argument’s logical validity is unchanged.Prior work reports that participants mistakenly judged invalid syllogisms valid 90% of the time when conclusions were believable.Source: §2.2 · p. 3

2.3. The Wason Selection Task

• In the Wason task, people often choose the converse rather than the logically implied contrapositive, while realistic social-rule content improves accuracy.The abstract task remains difficult even for mathematically educated participants, who achieved less than 50% accuracy in cited studies.Source: §2.3 · p. 4

Creating datasets

• The authors generate new task datasets to reduce the risk that familiar cognitive-science stimuli appeared in language-model training data.They follow approaches from prior work and refer to Appendix A.1 for full details.Source: §3 · p. 4
• Across the tasks, logical structure stays fixed while entities and their relationships vary across belief-consistent, belief-violating, and nonsense conditions.Figure 2 depicts the task datasets by rows and content versions by columns.Source: §3 · p. 4, Figure 2 · p. 5
• Wason conditions include realistic, arbitrary, and nonsense rules, plus shuffled-realistic and belief-violating realistic controls.The controls distinguish implausible relationships between realistic components from relationships that directly contradict expectations.Source: §3 · pp. 4–5

Models & evaluation

• The evaluation uses the 70-billion-parameter Chinchilla model across datasets and conditions, and also tests its 7-billion-parameter version on NLI.The smaller-model comparison is intended to assess how scale interacts with content on the easier NLI task.Source: §3 · p. 5
• Models score each possible answer by likelihood; DC-PMI corrects for baseline answer likelihood, but NLI uses maximum likelihood to avoid controlling away the bias under study.The authors report that syllogism and Wason results show the same qualitative effects under either scoring method.Source: §3 · p. 5

4. Results

• NLI accuracy favors belief-consistent content, syllogism judgments are strongly belief-biased, and Wason accuracy is higher for realistic rules.The supplied figures and results show content-sensitive performance across all three tasks.Source: Figure 3 · p. 6, Figure 4 · p. 6, §4 · p. 6

4.1. Zero shot performance

• Only the largest model exceeds chance on consistent NLI items; both models show stronger performance for consistent than violate or nonsense content.A mismatched-control analysis also found above-chance performance, while the consistent condition remained better; nonsense performance stayed poor.Source: §4.1 · pp. 6–7
• The zero-shot syllogism bias qualitatively resembles human belief bias; when comparing arguments to decide which is valid, performance achieves 70-80% accuracy (chance is 8%), with minimal accuracy differences by conclusion-reality consistency.Source: §4.1 · p. 7
• Shuffled realistic Wason rules remain above chance but below realistic rules, whereas violate-realistic rules are at chance.Performance is therefore sensitive both to realistic propositions and to whether their relationship fits expectations.Source: §4.1 · p. 8

4.2. Few-shot training on the logic task

• Nonsense-example prompting improves NLI performance across entity types, and five-shot performance becomes comparable across conditions, washing out the larger model’s content effect.The authors report no substantial differences when other entity types are used in the prompts.Source: Figure 6 · p. 8, §4.2 · p. 8
• Few-shot examples improve syllogism calibration, but substantial belief-bias effects remain.With examples, models more often answer “valid” for valid than invalid syllogisms in every condition.Source: Figure 7 · p. 8, §4.2 · p. 8
• For Wason, realistic prompts are most effective overall; nonsense prompts yield only weak gains, and arbitrary or nonsense probes remain difficult.Other prompt types help in some probe conditions, but each is worse than realistic examples in some conditions.Source: §4.2 · pp. 8–9, Figure 8 · p. 9

4.3. Details of Wason response patterns

• The model usually selects one antecedent and one consequent card, but its errors are more evenly split than humans’ and violate-realistic errors cluster on the consequent.The authors suggest the model may be ignoring the subtle rule change in the violate-realistic condition.Source: §4.3 · p. 9

5. Discussion

• Models mirror human content effects, reasoning less reliably when stimuli are abstract or conflict with prior understanding.The discussion emphasizes that performance depends on content and context.Source: §5 · pp. 9–10
• Wason errors on violate-realistic rules often treat the rule as realistic, while realistic examples bring the largest performance gains.The discussion also identifies error patterns on violate-realistic rules and relative ease of shuffled-realistic rules as effects not previously investigated in humans.Source: Figure 9 · p. 10, §5 · p. 10
• A single model can show biased or consistent reasoning depending on context; in NLI, examples and instructional prompts can reduce bias.The discussion suggests a context-dependent attention mechanism could arbitrate between conflicting responses.Source: §5 · p. 11
• Content biases can emerge from training a transformer to imitate human-produced language, without incorporating human-specific internal mechanisms.The authors frame abstract reasoning as a graded, content-sensitive capacity in both humans and models.Source: §5 · pp. 10–11

Why might model reasoning patterns differ

• Prompting uncertainty and imperfect human-model matches complicate interpretation, although tested task-framing variations generally made little difference.Model Wason errors differ somewhat from human errors, and the model lacks the significant believability-by-validity interaction observed in humans.Source: Why might model reasoning patterns… · p. 12
• Models lack humans’ direct, grounded experience and active social use of language, which may contribute to reasoning differences.The discussion notes physical simulation and gesture as possible supports for human reasoning.Source: Why might model reasoning patterns… · p. 12
• Formal education is associated with more consistent human logical reasoning, but causal evidence is scarce; targeted model training is a proposed direction.Prior work cited includes synthetic reasoning training, instruction tuning, self-correction, and iterative fine-tuning; the authors suggest reasoning may be supported by environmental and educational factors.Source: Why might model reasoning patterns… · pp. 12–13

A.1. Datasets

• The authors generated new datasets for each task to avoid training-data contamination.Source: §A.1 · p. 19

A.1.1. NLI task generation

• NLI comparisons were generated from real-world statements and manually filtered to remove false or subjective comparisons.The authors prompted Gopher with six true comparisons to generate 100 statements, then selected those matching the desired form.Source: §A.1.1 · p. 19
• NLI items pair premises with logically valid or invalid hypotheses, varying whether the conclusion agrees with real-world beliefs or uses nonsense entities.Models select between answer options by likelihood; a separate format asks whether the inference is valid or invalid.Source: §A.1.1 · pp. 19–20

A.1.2. Syllogisms data generation

• Syllogism stimuli form 12 four-argument clusters crossing validity with belief consistency while reusing the same three entities.The design approximately balances entities across conditions and includes nonsense versions made by replacing realistic entities.Source: §A.1.2 · p. 20, Figure 11 · p. 21
• Syllogisms are presented as forced choices between valid and invalid, with prompt and example variations aggregated because their effects were similar.Examples, when used, came from distinct argument clusters.Source: §A.1.2 · p. 21

A.1.3. Wason data generation

• The dataset includes 12 realistic and 12 arbitrary rules, each paired with four instances matching or violating the antecedent and consequent.Realistic rules include contextual framing; answers are forced choices over pairs of instances.Source: §A.1.3 · p. 21, Figure 12, Figure 13
• The authors report that basic inferences about the rule propositions have similar difficulty across rule types.This comparison is described as a control for proposition difficulty; see Figures 12 and 13 for the rules and instances.Source: §A.1.3 · p. 21
• Control rules shuffle realistic antecedents and consequents or reverse the consequent to create an especially implausible rule.The shuffled rules are intended to be more arbitrary or belief-neutral than the realistic rules.Source: §A.1.3 · p. 22

A.2. Evaluation

• DC-PMI selects the answer whose likelihood increases most relative to a baseline prompt.The baseline prompt is “Answer:”; p(x|y) denotes the model’s evaluated likelihood of continuation x after prompt y.Source: §A.2 · pp. 22–23

B.1. Directly estimating model beliefs via log likelihood, in the NLI task

• Model beliefs are estimated from average per-token log likelihoods of comparison statements presented without prior input.Generated comparisons and their inverted versions are pooled rather than assigned only to assumed consistent and violating categories.Source: §B.1 · p. 23
• Zero-shot NLI accuracy is highly positively related to the unconditioned likelihood of the conclusion, and the relationship diminishes for one- and five-shot problems.The analysis uses the likelihood difference between correct and incorrect conclusions as a logistic-regression input; Figure 14 plots the fits and answer distributions.Source: §B.1 · p. 23, Figure 14 · p. 24
• The analysis uses Chinchilla, with correct-versus-incorrect conclusion likelihood differences entered into a logistic regression predicting answer correctness.The regression also includes a bias term.Source: §B.1 · p. 23

B.2. Classifying natural language inferences as valid or invalid produces a belief-bias effect.

• The larger model’s NLI validity accuracy is highest for belief-consistent hypotheses and lowest for nonsense entities, while the smaller model is not above chance.The authors note that the consistent and violate conditions are slightly confounded by the requirement to include invalid inferences.Source: §B.2 · p. 24, Figure 16 · p. 25
• After DC-PMI correction, NLI performance with mismatched premises remains significantly above chance, indicating that the premise contributes to responses.The control retains hypotheses that are consistent with or violate prior expectations while mismatching them with the premise.Source: §B.2 · p. 25

B.3.1. NLI

• Explicit argument-evaluation instructions improve performance on violating NLI cases, but both models still struggle with nonsense and perform best on consistent cases.The “pretend” prompt gives similar findings, and belief-bias effects appear across prompts.Source: §B.3.1 · p. 25, Figure 17 · p. 26

B.3.2. Syllogism

• Syllogism instructions shift overall response tendencies, but the patterns of belief bias remain similar across prompt conditions.The authors therefore aggregate across prompts in other figures and include prompt as a random effect in regressions.Source: §B.3.2 · p. 26, Figure 18 · p. 26

B.4. The Wason rule propositions have similar difficulty across conditions

• Wason proposition difficulty shows no substantial differences across rule types, leaving component-proposition difficulty an unlikely explanation of the observed effects.Arbitrary rule premises were numerically slightly easier, but the differences were not significant.Source: §B.4 · p. 27, Figure 19 · p. 27

• The model usually selects one antecedent and one consequent card across Wason experimental conditions.Source: §B.5 · p. 27
• This matches the majority of participants in Wason (1968), who test both parts of the rule.Source: §B.5 · p. 27
• The model nevertheless does not always choose the correct card within each category.Source: §B.5 · p. 27

• Bias is relatively weak in this forced-choice design, echoing findings that humans show less bias when choosing between two arguments than judging one argument’s validity.Source: §B.6 · pp. 27–28
• In this case, for Syllogisms and Wason, Domain-Conditional PMI scoring produces much lower accuracy than raw likelihoods and minor bias differences; without correction, accuracy is around 35–40%, regardless of belief consistency.Source: §B.6 · p. 28
• The model performs far above chance when selecting a valid syllogism conclusion from all possible conclusions, especially with raw log-likelihoods.Source: §B.6 · p. 27, Figure 21 · p. 28

• In zero-shot Syllogisms and Wason, raw answer likelihoods produce much worse performance than DC-PMI; syllogism answers also show greater invalid-response bias.Source: §B.7 · p. 28, Figure 22 · p. 29
• With few-shot raw-likelihood scoring, syllogism invalid-response bias falls somewhat at five shots, but belief bias persists and grows as invalid-response bias eases.Source: §B.7 · pp. 28–29, Figure 23 · p. 30
• On Wason, raw scoring hurts realistic-rule performance but can improve arbitrary or nonsense rules; the realistic-rule advantage over those rules remains.At five shots, shuffled realistic rules can become competitive with realistic rules, though scoring each type by its most favorable metric still favors realistic rules.Source: §B.7 · p. 29, Figure 23 · p. 30

B.8. Varying few-shot examples

• For NLI and syllogisms, different prompt-example types yield similar probe performance in most cases; NLI reaches close to ceiling with five shots.Source: §B.8 · p. 30, Figure 25 · p. 31
• For Wason, realistic examples benefit realistic and shuffled-realistic probes, while arbitrary and nonsense rules tend to benefit more from matching examples.Nonsense examples provide limited benefit; the authors use matching examples in later Wason follow-up experiments.Source: §B.8 · p. 30, Figure 26 · p. 31

C. Statistical analyses

• The statistical analyses use hierarchical (multilevel) logistic regressions to account for dependencies in the data.The authors describe these methods as supporting generalizable inferences.Source: C. Statistical analyses · p. 32