Source-linked AI summary
Revealing Algorithmic Deductive Circuits for Logical Reasoning
Phuong Minh Nguyen, Tien Huu Dang, Naoya Inoue
TL;DR
The paper addresses how LLMs understand and execute individual reasoning steps and overall graph-traversal strategies from few-shot demonstrations. It aligns reasoning steps with token logits and applies causal mediation analysis to localize responsible attention heads and their information flow. The analysis finds low-confidence reasoning-critical tokens, specialized heads for factual and rule-based retrieval, and higher-layer integration of global reasoning strategies.
Problem
It remains unclear how LLMs internally understand individual reasoning steps and overall algorithms from limited few-shot demonstrations.
Method
The study aligns symbolic-aided CoT reasoning components with token logits and uses causal mediation, activation patching, and path patching to identify attention-head circuits.
Results
The analysis finds low-confidence reasoning-critical tokens, specialized attention heads for sub-reasoning information retrieval, and higher layers that integrate information and coordinate global strategies.
Takeaways & Limitations
The identified circuits are important and general across deductive-reasoning datasets, while general-knowledge performance decreases only slightly after ablation.
Takeaways & Limitations
The study primarily uses synthesized datasets, focuses on attention mechanisms, and evaluates circuits whose generalization across architectures, scales, and domains remains open.
Abstract
from arXiv · showhide
Recent studies have shown that Large Language Models (LLMs) can achieve strong reasoning performance by incorporating functional symbolic representations that abstractly describe graph traversal algorithms and step-by-step reasoning in few-shot learning settings. However, it remains unclear how LLMs genuinely understand the abstract meaning of each reasoning step and the overall algorithm from only a limited number of demonstrations. This work aims to localize the attention heads responsible for individual reasoning steps and characterize the types of information transferred among them. We first align constituent reasoning steps with their corresponding token logits under a symbolic-aided Chain-of-Thought (CoT) prompting framework. Our analysis shows that token positions that steer the reasoning process are associated with low confidence scores caused by constraints on satisfying reasoning behavior patterns in demonstrations. We then adopt causal mediation analysis techniques to identify the attention heads responsible for these patterns. In addition, our findings indicate that LLMs retrieve factual and rule-based information for individual sub-reasoning tasks through specialized attention heads (approximately 3% total heads), whereas higher layers predominantly facilitate information integration and the emergence of global reasoning strategies (e.g., graph traversal algorithms) that coordinate multiple intermediate reasoning steps to solve the overall task.
1 Introduction
The paper investigates how LLMs internally execute multi-step logical reasoning from few-shot demonstrations. It localizes attention-head circuits associated with reasoning components and examines their information flow.
- Research question: The study asks what mechanisms LLMs internally employ to solve logical reasoning problems.The question targets internal reasoning mechanisms rather than a new state-of-the-art reasoning framework.
- Research hypothesis: Few-shot demonstrations may enable LLMs to abstractly learn the meaning of inference steps and follow a graph-traversal algorithm.Here, “learn” refers to activating attention heads that mediate inference-step execution and algorithm adherence.
- Token-level analysis: Reasoning-critical tokens are low-confidence positions because decoding them must satisfy multiple implicit constraints.These constraints include factual validity, rule applicability, and consistency with the traversal pattern demonstrated in the prompt.
- Circuit discovery: The analysis identifies premise selection, premise-selection termination, and rule selection as major reasoning components within inference steps.These components are investigated as interacting circuits using causal mediation analysis.
- Circuit validation: Activation and path patching locate attention heads responsible for reasoning components and characterize information flow among them.Circuit ablation substantially reduces deductive-reasoning performance while causing only a slight decrease on general-knowledge tasks.
2 Preliminary Experiment
The preliminary experiment constructs controlled symbolic reasoning data, identifies uncertain tokens, and finds that uncertainty concentrates in three components guiding inference-step generation.
- Dataset construction: The synthesized dataset uses uppercase-letter premises, few-shot demonstrations, 8–18 randomly generated facts and rules, and BFS-generated reasoning chains.Ambiguous samples are filtered out before analysis.
- Experimental setup: Four LLMs are evaluated using cached token probabilities, with probabilities below 0.8 marking uncertain tokens.The models are Llama-3.1-8B-Instruct, Qwen3-8B, Phi-4, and Qwen3-4B.
- Results: Premise selection, premise-selection termination, and rule selection account for most uncertainty in the reasoning chains.Other reasoning-chain tokens are mostly assigned high probabilities.
- Scope: The synthesized dataset preserves multi-hop logical challenges but does not capture some real-world complexities, such as premise-complexity variation.
3 Methodology
The methodology uses causal mediation analysis with activation and path patching to identify attention heads responsible for reasoning components and quantify causal information flow between them.
- Circuit discovery: Causal mediation analysis discovers important attention heads and circuits for premise selection, termination, and rule selection.The target components are defined as r ∈ R.
- Background and notation: The model generates an output sequence yq from a k-shot prompt, with each reasoning token assigned a probability at its output position.For a token tr at position pos(tr), f(p)[tr] denotes its assigned probability.
- Background and notation: The residual stream combines the previous hidden state, projected MLP output, and projected attention-head outputs across layers.The paper focuses on attention components because attention primarily routes information and composes relations.
- Activation patching: Activation patching restores clean activations in corrupted prompts to measure each head’s contribution to recovering clean behavior at a target reasoning token.Clean and corrupted prompts share structure but differ in causal context.
- Activation patching: Top-k heads are selected by causal indirect effects, distinguishing information-reading heads from decision heads by their patching positions.Reading heads are patched at causal spans, whereas decision heads are patched before the focal reasoning component.
- Path patching: Path patching corrupts an emitting head and restores the resulting activation at a receiving head to quantify causal dependence between head pairs.The resulting score ranks average causal information flow among reasoning-responsible heads.
4 Experimental Results
The experiments localize specialized attention heads for deductive reasoning components and reveal a layered, interactive circuit structure. Ablating these heads substantially harms logical reasoning, with the strongest effects from jointly removing all three component groups.
- Attention-head localization: Reading heads concentrate in earlier layers than decision heads, consistently across evaluated LLMs.Information-reading heads extract and propagate relevant facts to higher layers for subsequent logical operations.
- Attention-head localization: Rule-condition matching heads cluster in middle layers, while rule-selection heads are sparse and high-impact.The highest AIE score exceeds 30% in Llama-3.1-8B-Instruct and 12% in Qwen models.
- Attention-head localization: The layer sequence follows a consistent temporal structure from matching rule conditions toward later reasoning decisions.The reported line graphs show this structure across all evaluated LLMs.
- Circuit networks: Top-5 head circuits transfer role-specific information from reading heads to deeper decision heads that integrate sub-reasoning information.Examples include rule-condition information flowing into deeper layers for rule-condition matching, rule selection, and premise selection.
- Circuit networks: Polysemantic attention heads read multiple information types, while polysemantic decision heads integrate multiple reasoning subtasks.The circuit network therefore includes both specialized role transfer and multi-task integration.
- Ablation results: Jointly ablating approximately 3% of heads across three reasoning roles causes reasoning ability to collapse to nearly zero on synthesized data.Individual role ablations use approximately 1% of heads each, and their effects exceed uniformly random ablation despite using fewer heads.
- Ablation results: The identified heads generalize across ProofWriter and ProntoQA, while individual component ablations on MMLU produce comparable performance drops.Collective LR-head ablation causes substantially larger degradation than random ablation on general knowledge tasks.
5 Related Work
Prior work has examined LLM reasoning limitations, improvement techniques, and specialized reasoning circuits. This study distinguishes itself by investigating circuits for deductive reasoning.
- Logical reasoning in LLMs: LLMs struggle with complex reasoning tasks, motivating prompt engineering, fine-tuning, and symbolic Chain-of-Thought approaches.These approaches target reasoning capability limitations through prompting, training, or external symbolic representations.
- Research focus: Unlike these prior approaches, this work investigates deductive reasoning circuits in LLMs.Its focus is the internal circuitry supporting deductive logical inference.
- Reasoning circuits in LLMs: Existing circuit studies cover factual recall, indirect object identification, temporal information, arithmetic, abstract, syllogistic, and propositional reasoning.The cited work spans multiple specialized reasoning and information-retrieval tasks.
6 Conclusion
The paper identifies sparse, modular attention-head circuits underlying multi-step deductive reasoning. Early-to-middle layers retrieve and process information, while higher layers integrate it and coordinate global reasoning strategies.
- Mechanistic findings: Causal mediation analysis localizes specialized attention heads associated with different components of multi-step logical inference.The identified components include premise selection, premise-selection termination, and rule selection.
- Mechanistic findings: Early-to-middle layers retrieve factual and rule-based information, whereas higher layers integrate information and coordinate global reasoning strategies.The conclusion describes this organization as hierarchical.
- Reasoning constraints: Reasoning-critical token positions consistently have low confidence because few-shot demonstrations impose strict reasoning constraints.These positions steer the reasoning process despite being less confidently decoded.
- Ablation evidence: Ablating identified reasoning heads causes substantially larger degradation than random ablation across synthesized, deductive benchmark, and general knowledge datasets.The result supports the necessity of these circuits for deductive reasoning and their broader recruitment in the evaluated tasks.
Limitations
The analysis is limited by its reliance on synthesized, explicitly structured logical datasets and attention mechanisms. The discovered circuits are also specific to the studied models and datasets, so broader generalization remains unresolved.
- The synthesized datasets may not fully generalize to free-form text reasoning or implicit chain-of-thought in naturalistic contexts.
- The analysis focuses on attention mechanisms, although MLP layers may also contribute to reasoning.
- The identified circuits are specific to the studied models and datasets, requiring investigation across architectures, scales, and reasoning domains.
B Preliminary experiments
The preliminary experiments examine uncertain reasoning components and their probability distributions across four language models. The reported trend is consistent across the evaluated models.
- Four models—Llama-3.1-8B-Instruct, Qwen3-8B, phi-4, and Qwen3-4B—are included in the uncertainty analysis.
- The experimental results show the same trend across LLMs.
- Figure 6 groups low-probability uncertain tokens by reasoning component for Llama-3.1-8B-Instruct and Qwen3-8B.
- Figure 7 presents the corresponding uncertain-token distribution for phi-4 and Qwen3-4B.
C Data Synthesizing Algorithm
The synthesized dataset uses structurally identical clean and corrupted prompt pairs to identify reasoning-circuit components. Additional circuit-discovery results are reported across multiple supplementary figures.
- The dataset constructs structurally identical prompt pairs consisting of clean prompts and corrupted prompts.
- Corruption systematically modifies causal elements, such as fact values and rule definitions, that influence token selection at reasoning positions.
- Additional circuit-discovery results across LLMs are provided in Figures 8–14 because they could not fit fully in the main paper.
E Ablating LR Heads Across Datasets
Additional experiments ablate logical-reasoning heads to validate their importance across both the synthesized dataset and ProofWriter.
- The experiments knock out logical-reasoning heads in language models.
- The validation covers the synthesized dataset and the ProofWriter logical-reasoning dataset.
- The experiments further validate the importance of deductive-reasoning heads across datasets.
F Synthesize Data Example
This section illustrates corruption-based prompt pairs for examining premise and rule selection, termination, and traversal-algorithm effects, alongside attention-head analyses across models.
- Synthesize Data Example: Clean-corrupt prompt pairs in Dpair(c) cover multiple corruption types used to examine the synthesis process.The examples are presented in Tables 3–6.
- Synthesize Data Example: Modifying facts affects premise selection, with causal-span changes evaluated through logit differences at the reasoning component.Table 3 distinguishes clean causal content, corrupted content, and ignored text.
- Synthesize Data Example: Changing rule content affects premise-selection termination and the rule-selection component.These corruption types are illustrated separately in Tables 4 and 5.
- Attention-Head Analysis: Attention-head analyses report AIE-score distributions across layers and heads for Llama-3.1-8B-Instruct, Qwen3-8B, Qwen3-4B, and phi-4 models.The Llama-3.1-8B-Instruct analysis highlights five highest-scoring heads and identifies the layer most responsible for rule selection.
- Attention-Head Analysis: Circuit networks visualize the top-five attention-head scores associated with reasoning components, while inference-step accuracy is evaluated after knocking out top-k logical-reasoning heads.The knockout evaluation uses synthesized data and the ProofWriter Logical Reasoning task, with k = 8 for phi-4 and k = 5 for other models.
- Synthesize Data Example: Changing traversal algorithms in demonstrations affects the premise-selection component.This corruption type is shown in Table 6.