PaperlayerView exact revision on arXiv

Source-linked AI summary

A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners

Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, Dan Roth

arXiv:2406.11050v2cs.CLcs.AI

TL;DR

The paper asks whether LLMs genuinely reason or whether their performance reflects sensitivity to semantic cues. It tests token bias with controlled synthetic tasks and statistical hypothesis tests, finding that LLMs often rely on token bias rather than consistent reasoning.

  • Problem

    Whether LLMs possess genuine reasoning capabilities remains debated, with evidence that reasoning-task performance correlates with semantic content that supports correct inference.

    Source: §1 · p. 1

  • Method

    The paper uses synthetic reasoning data, systematic token perturbations, and statistical hypothesis tests to assess whether LLM responses shift when the underlying logic stays constant.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Statistical evidence indicates that LLMs do not always reason consistently and primarily rely on token bias when generating responses.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Chain-of-thought prompting and in-context learning may elicit semantic shortcuts that imitate desired behavior without actual reasoning.

    Source: §6 · p. 9

    The framework applies to multiple-choice or yes/no questions, and the study focuses on conjunction and syllogistic fallacies and the twenty-five-horses problem and variants.

    Source: §7 · p. 10

Abstract

from arXiv · show

This study introduces a hypothesis-testing framework to assess whether large language models (LLMs) possess genuine reasoning abilities or primarily depend on token bias. We go beyond evaluating LLMs on accuracy; rather, we aim to investigate their token bias in solving logical reasoning tasks. Specifically, we develop carefully controlled synthetic datasets, featuring conjunction fallacy and syllogistic problems. Our framework outlines a list of hypotheses where token biases are readily identifiable, with all null hypotheses assuming genuine reasoning capabilities of LLMs. The findings in this study suggest, with statistical guarantee, that most LLMs still struggle with logical reasoning. While they may perform well on classic problems, their success largely depends on recognizing superficial patterns with strong token bias, thereby raising concerns about their actual reasoning and generalization abilities. Codes and data are open-sourced at https://github.com/bowen-upenn/llm_token_bias.

1 Introduction

The paper reframes LLM reasoning evaluation around whether responses withstand changes to irrelevant tokens, since performance may track semantic cues rather than valid inference. It proposes controlled synthetic perturbations and statistical tests to assess token bias.

1 Introduction

  • Token bias is defined as predictable output shifts after systematic token changes that leave the underlying logic intact.The paper argues that strong bias suggests reliance on superficial patterns and could lead to brittle generalization.Source: §1 · p. 1
  • The framework combines synthetic data generation, systematic token perturbation, and statistical hypothesis testing to evaluate reasoning behavior.The resulting contingency table records outcome counts for statistical tests.Source: §1 · p. 2, Figure 2 · p. 2
  • Illustrations show GPT-4 confused by changing Linda to Bob and GPT-4 and Claude performing worse after animal-name or number perturbations.The twenty-five-horses figure reports significantly more cases solved originally but not after perturbation than the reverse.Source: Figure 1 · p. 1, Figure 3 · p. 2
  • The study targets token bias rather than overall fallacy-identification accuracy, using conjunction, syllogistic, and twenty-five-horses problems as examples.These examples are selected because their problem statements exhibit identifiable token biases.Source: §1 · p. 2
  • Comprehensive experiments are reported to identify token bias, rather than genuine reasoning advances, as the largest contributor to any reasoning-task performance gains.The introduction qualifies the gains with “if any.”Source: §1 · pp. 2–3

2 The General Framework

The framework tests whether an agent reaches consistent conclusions when task framing changes but the underlying logic remains intact. It combines controlled synthetic tasks and token perturbations with matched-pair statistical tests.

Testing assumptions

  • The null hypothesis treats consistent reasoning as invariant to framing changes, with procedural reasoning mistakes assumed to occur independently and identically.For each matched pair, π_ab denotes the probability of original outcome a and perturbed outcome b; n_ab/n estimates that probability.Source: §2 · p. 3

Synthetic data generation

  • Synthetic task descriptions are generated from curated entities and templates, creating novel combinations while preserving the task’s underlying logic.The data can be generated dynamically and scaled to the sample size needed for statistical validity.Source: §2 · p. 3

Token perturbation

  • Systematic token changes create matched pairs so model outcomes on original and perturbed tasks can be compared.The perturbations are intended to reveal whether token bias predicts a consistent shift in performance.Source: §2 · p. 3

Statistical testing

  • Matched original and perturbed samples form a 2×2 contingency table, and group-level tests use Benjamini–Hochberg to control the false discovery rate.For small samples, an exact binomial test is available; when n* > 10, the reference distribution is treated as approximately normal and a standardized statistic is used.Source: Table 1 · p. 3, §2 · pp. 3–4

3 A Peek into Token Bias

The section treats reasoning as invariant to superficial changes that preserve logical structure, then proposes token perturbations to test that expectation. It applies this framework to fallacies, entity names, quantifiers, narratives, and explicit hints.

3 A Peek into Token Bias

  • The framework tests whether LLMs remain consistent under changes to irrelevant tokens while the underlying logical structure is preserved.Its null hypotheses assume genuine reasoning; token bias is assessed by systematically changing tokens predicted to affect answers.Source: §3 · p. 4

3.1 Preliminaries

  • Conjunction fallacies violate the rule that a conjunction cannot be more probable than either event alone, while invalid syllogisms draw conclusions not implied by their premises.The section uses these cognitive-science problems to construct synthetic tasks for perturbation.Source: §3.1 · p. 4

3.2 Lost in Irrelevant Context

  • For misleading conjunction options, replacing a context-relevant added event with an irrelevant one tests whether performance shifts as predicted by token bias.The null expects π12 = π21; the proposed alternative is π12 < π21, while π12 > π21 is treated as invalid.Source: §3.2 · pp. 4–5

3.3 Token Bias on Widely Cited Examples in Classic Literature

  • Changing the one-shot Linda exemplar’s name to Bob tests whether performance depends on a familiar name rather than the reasoning task.The null expects equal matched-pair probabilities, with π12 > π21 proposed as the alternative.Source: §3.3 · p. 5

3.4 Token Bias on Well-Known Entity Names

  • Replacing celebrity names with generic names, or altering classic entity names in mathematical problems, tests sensitivity to familiar entities.The examples include a celebrity-name change and a horses-to-bunnies change in the classic counting problem.Source: §3.4 · pp. 5–6

3.5 Token Bias in Reasoning about Sets

  • Syllogism tests rephrase quantifiers such as “all” and “some” or alter source narratives without changing the logic.The narrative variants include reputable institutions and satirical or less reputable sources.Source: §3.5 · p. 6

3.6 Leaking Hint Tokens

  • Adding a fallacy label or detailed reasoning guidance tests whether LLMs rely on explicit hints to reach correct inferences.The paper also describes manually crafted chain-of-thought instructions that teach reasoning steps and potential logical pitfalls.Source: §3.6 · pp. 6–7

4 Experiment

The experiments use controlled token changes in synthetic reasoning tasks to test whether LLM outputs remain consistent when the underlying logic is unchanged. Across conjunction fallacies and syllogisms, the reported results show sensitivity to misleading options, names, quantifier synonyms, reputable entities, and hints.

4 Experiment

  • Each model–prompting-method pair was tested over n trials, with McNemar tests, Benjamini–Hochberg correction, and α = 0.05 for rejecting null hypotheses.The number of trials depended on how many synthetic samples related to each hypothesis.Source: §4 · p. 7

4.1 Models

  • The study included commercial and open-source models from OpenAI, Meta, Anthropic, and Mistral.The listed models include GPT-3.5-Turbo, GPT-4-Turbo, GPT-4o, Llama-3, Llama-2, Claude-3, and Mistral Large.Source: §4.1 · p. 7

4.2 Synthetic Dataset Generation

  • Synthetic tasks were generated from curated entity lists drawing on occupational statistics, stories, news, disease-symptom pairs, names, objects, and media sources.The authors describe a controlled data-generation process and provide the samples for each hypothesis in Appendix C.Source: §4.2 · p. 7

4.3 Prompting Methods

  • The prompting conditions included direct-answer baselines, zero-shot chain-of-thought, one-shot and three-shot examples, and hint variants for Hypothesis 6.The zero-shot chain-of-thought prompt asks the model to think step by step.Source: §4.3 · p. 7

4.4 Hypothesis Testing Results

  • Across conjunction-fallacy tests, replacing misleading options with random alternatives reduced success, and replacing “Linda” with “Bob” in exemplars consistently reduced performance.The celebrity-name variants also frequently misled models; the authors rejected all null hypotheses for that test.Source: §4.4 · p. 7, Figure 4 · p. 8
  • On syllogisms, most models were not robust to quantifier synonyms, some were misled by reputable names, and all tested models relied heavily on hints.Llama-3-70b-instruct was the exception to the quantifier-synonym result; reputable names especially misled GPT-4 and LLaMA-3-70B.Source: §4.4 · pp. 7, 9, Figure 4 · p. 8

5 Related Work

The paper positions its work as a fine-grained, statistically grounded study of token bias, distinct from accuracy-focused reasoning benchmarks and coarser fallacy analyses. It examines whether superficial token changes can explain apparent reasoning performance.

The Reasoning Capabilities of LLMs

  • Unlike accuracy-based benchmarks, the study examines token biases that may make logical reasoning performance appear stronger while remaining vulnerable to irrelevant token perturbations.The authors frame this as a more nuanced analysis of reasoning generalization and provide statistical guarantees through hypothesis testing.Source: §5 · p. 9

Cognitive Biases and Logical Fallacies in LLMs

  • Prior work on cognitive biases and fallacies often emphasizes accuracy at a coarse level, whereas this study uses tuned hypotheses and quantitative analyses with statistical guarantees.It contrasts its general prompting strategies with Rationality of Thought’s six predefined steps and hand-crafted prompt engineering.Source: §5 · p. 9

6 Discussion

The authors interpret their statistical evidence as showing that LLMs do not reason consistently in every decision and instead primarily rely on token bias. They caution that further investigation is needed to understand the mechanisms and limits of LLM reasoning.

6 Discussion

  • The authors conclude that LLMs primarily rely on token bias, and that chain-of-thought or in-context learning may produce semantic shortcuts rather than actual reasoning.They call for further investigation into the underlying mechanisms and limitations of LLM reasoning.Source: §6 · p. 9

7 Limitations

The framework is limited to multiple-choice or yes/no responses and the study tests a selected set of fallacies and variants. The appended examples and prompts illustrate the Linda problem and the hint conditions used in the experiments.

7 Limitations

  • The framework targets multiple-choice or yes/no questions, not open-ended responses, and weaker instruction-following models may introduce confounders beyond token bias.The authors therefore mainly focus on state-of-the-art models; identifying token biases also requires manual effort.Source: §7 · p. 10
  • The study focuses on conjunction fallacies, syllogistic fallacies, the “twenty-five horses” graph problem, and variants, while the framework could cover broader tasks and modalities.The authors acknowledge that other hypotheses and assumptions may also be relevant to genuine reasoning.Source: §7 · p. 10

A The Original Linda Problem in Tversky and Kahneman (1983)

  • The original Linda problem asks whether Linda is more likely to be a bank teller or a bank teller active in the feminist movement.The options contrast a single event with a conjunction of events.Source: A The Original Linda Problem in… · p. 14

B.1 Weak Hint

  • The weak-hint prompts identify the task as a Linda problem for conjunction questions and as a syllogistic-fallacy problem for syllogism questions.The conjunction prompt requests an explicit option; the syllogism prompt requests “Yes” or “No.”Source: §B.1 · p. 14

B.2 Strong Hint

  • The strong-hint prompts explain the conjunction probability rule and direct attention to syllogistic quantifiers and logical structure.For conjunction problems, the prompt advises selecting the single-event option; for syllogisms, it advises checking whether the conclusion follows from the premises.Source: §B.2 · pp. 14–15

C Synthetic Data Generation

The section describes generating synthetic examples for conjunction-fallacy and syllogistic problems, with 100 samples per variant. The variants provide controlled ways to examine how changes in wording or framing affect model responses.

C Synthetic Data Generation

C.1 Conjunction Fallacy

  • Conjunction-fallacy variants draw on stories, biographies, disease symptoms, and celebrity scenarios to create subset-related options.Some variants alter the second option’s relevance or use different connecting words.Source: §C.1 · pp. 15–17

C.2 Syllogistic Fallacy

  • Syllogism problems use sampled objects and category traits, while news-source names provide a framing perturbation.The news-source perturbation dataset contains 200 samples.Source: §C.2 · pp. 17–18

D.1 Hypothesis 1

Hypothesis 1 tests whether changing contextually relevant options to irrelevant alternatives affects performance on conjunction-fallacy problems. The reported results conclude that LLMs fail to reason against misleading options.

  • LLMs fail to reason against contextually misleading options in conjunction-fallacy problems.Source: Figure 5 · p. 20
  • The test uses 400 problems and compares options that are contextually relevant with irrelevant alternatives across prompt methods.Source: Figure 5 · p. 20
  • The accuracy comparison between original and perturbed problems is shown in Figure 6.Source: Figure 6 · p. 21

D.2 Hypothesis 2

The experiments report token sensitivity across syllogism tasks: models show bias toward the name Linda, celebrity names, and the “All..., Some..., Some...” pattern. The reported framing effects also include reputable names and equivalent All/Some expressions.

D.2 Hypothesis 2

  • LLMs show a strong token bias toward “Linda,” a name frequent in classic literature.The in-context exemplars use the name change; n = 500.Source: Figure 7 · p. 22

D.3 Hypothesis 3

  • LLMs are frequently misled by celebrity names that are irrelevant to the logical essence of the problem.The celebrity name is replaced by a generic name; n = 100.Source: Figure 9 · p. 24

D.4 Hypothesis 4

  • Most LLMs rely on the “All..., Some..., Some...” pattern when reasoning about syllogisms.The test alternates “All” and “Some” with different but equivalent expressions; n = 200.Source: Figure 11 · p. 27

D.5 Hypothesis 5

The experiments report that LLMs may respond to reputation cues in syllogisms and continue to rely heavily on hint tokens in logical-fallacy problems. The two framing contrasts test names of differing reputability after removing classic syllogism patterns.

D.5 Hypothesis 5

  • LLMs might be misled by reputable news-agency and university names irrelevant to syllogism structure.Classic “All..., Some..., Some...” patterns are removed; n = 200.Source: Figure 13 · p. 29
  • LLMs might be less likely to be misled when reputable names are replaced by less reputable ones, although such shifts should not occur for genuine reasoners.The classic syllogism patterns are removed in this framing comparison.Source: Figure 15 · p. 32

D.6 Hypothesis 6

  • LLMs still heavily rely on hint tokens when solving logical-fallacy problems.The problems leak weak or strong hints; the experiment uses zero-shot and one-shot prompts with n = 800.Source: Figure 17 · p. 35
Loading 2406.11050v2…