Source-linked AI summary
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi, Sadegh Mohammadian, Fatemeh Askari, Mobin Bagherian, Amirmohammad Izadi, Mahdieh Soleymani Baghshah
TL;DR
LLMs’ implicit counting becomes less reliable for large counts, and the paper attributes this limitation to architectural constraints. It proposes a test-time strategy that partitions large counting tasks and analyzes how counts move through the model; experiments report effectiveness across models, alongside a mechanistic account of the process.
Problem
Large-count failures in LLMs are attributed to capacity-limited System-1-like processing and transformer architectural constraints.
Source: §1 · pp. 1–2Method
The test-time strategy decomposes large counting tasks into smaller sub-tasks for reliable counting and aggregates their results, while attention analysis and causal mediation examine the mechanism.
Source: §1 · p. 2Results
Takeaways & Limitations
The study presents structured test-time strategies as a way to extend existing models’ computational abilities and deepen interpretability, with possible application to other tasks where implicit representations saturate.
Source: §7 · p. 9The study uses text-based counting with limited object diversity and synthetic repeated-noun contexts, and the approach is not expected to generalize to tasks with strongly interdependent subproblems.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from the architectural limits of transformers, where counting is performed across layers, leading to degraded precision for larger counting problems due to depth constraints. To address this limitation, we propose a simple test-time strategy inspired by System-2 cognitive processes that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. We evaluate this approach using observational and causal mediation analyses to understand the underlying mechanism of this System-2-like strategy. Our mechanistic analysis identifies key components: latent counts are computed and stored in the final item representations of each part, transferred to intermediate steps via dedicated attention heads, and aggregated in the final stage to produce the total count. Experimental results demonstrate that this strategy enables LLMs to surpass architectural limitations and achieve higher accuracy on large-scale counting tasks. This work provides mechanistic insight into System-2 counting in LLMs and presents a generalizable approach for improving and understanding their reasoning behavior.
1 Introduction
LLMs’ layerwise counting loses precision at larger counts, a limitation attributed to transformer depth and capacity-limited implicit processing. The paper proposes decomposing large tasks at test time and examines the strategy’s behavior and internal mechanism.
1 Introduction
- System-2 counting maintains high accuracy across problem sizes while System-1 collapses beyond approximately 30 items; experiments report effectiveness across LLMs without model changes or fine-tuning.Figure 1 contrasts the two strategies as problem size increases.Source: Figure 1 · p. 1, §1 · p. 2
- Prior work describes layerwise counting as saturating with item count and numerical representations as losing resolution with magnitude, while large-count failures are attributed to architectural constraints.The introduction characterizes the underlying single-pass approach as fast, automatic, and capacity-limited.Source: §1 · pp. 1–2
- Partition-level counts are localized at final-item or separator tokens, transferred through residual streams and attention pathways to intermediate reasoning tokens, then aggregated into the final answer.The proposed mechanism is examined using attention analysis and causal mediation techniques.Source: Figure 2 · p. 2, §1 · p. 2
- The strategy divides large counting tasks into smaller, independent sub-tasks that the model can reliably count, then aggregates their results.System-2 denotes deliberate computation extending across token generation beyond fixed layerwise processing.Source: §1 · p. 2
2 Problem Setup and Methodology
The method evaluates counting repeated-item lists using structured or unstructured inputs and short-answer or chain-of-thought outputs. Its structured strategy partitions the list, counts each part, and aggregates the partial counts.
2 Problem Setup and Methodology
- The task presents repeated-item lists and asks the model to report the total, with prior work reporting high accuracy below 10 items and deterioration beyond that range.The counting framework follows prior research.Source: §2 · p. 3
- An external vertical-bar separator divides the input into partitions, which the model counts individually before aggregating their partial counts.Partition sizes are randomly selected from a range the model can count accurately; the strategy requires neither fine-tuning nor external tools.Source: §2 · p. 3
- The baselines cross structured versus unstructured input with short-answer versus chain-of-thought output formats.Short answers correspond to an immediate System-1-like process, while chain-of-thought includes intermediate reasoning steps.Source: §2 · p. 3
- Beyond 10 items, Qwen2.5-7B’s decoded-output heatmap shows fading diagonal entries, indicating reduced confidence as target counts increase.The probabilities are averaged over item types for target counts from 1 to 25.Source: Figure 3 · p. 3
3 Behavioral Results
Across the evaluated models, structured inputs paired with intermediate reasoning were the only configuration consistently successful on long contexts. The analysis also identifies intermediate partition counts—not final addition—as the main source of structured-CoT errors.
3 Behavioral Results
- For Qwen2.5-7B, prediction confidence decreased beyond 10 items, with systematic biases toward more frequent counts such as 15, 20, and 25.Counts of 16 and 24 were often predicted as 15 or 25, while 21 and 22 were typically predicted as 20.Source: §3 · p. 3
- Performance decreased as context size increased and was associated with model scale, especially depth; larger and deeper models showed higher accuracy.Closed-source models were evaluated on longer contexts because performance was near-perfect below 50 items.Source: §3 · pp. 3–4
- Structured inputs with intermediate reasoning were the only configuration that consistently succeeded on long contexts across the evaluated models.CoT without structured input showed no notable improvement, and structured input without intermediate steps was ineffective or sometimes harmful.Source: §3 · p. 4, Table 1 · p. 4, Table 2 · p. 4
- In structured short-answer errors, the predicted count often matched the largest partition: 13% for Qwen2.5-7B and 43.6% for Llama3-8B errors at context sizes 11–30.The authors suggest that the latent counter resets at each partition, making its maximum count correspond to a partition size rather than the total.Source: §3 · p. 4
- The main source of structured-CoT failure was incorrect intermediate partition counts; given correct intermediate counts, most models could reliably add them.Table 3 distinguishes total accuracy, final-step accuracy conditional on generated intermediate counts, and intermediate-step accuracy.Source: §3 · p. 4, Table 3 · p. 5
- The study reports that its overall trends hold across tokenizers and input formats, and that the observed trends generalize across model scale and domain specialization.The robustness and generalization checks are reported in the appendices.Source: §3 · pp. 4–5
4 Attention Analysis
Attention analyses suggest a staged process: partition-local counts draw on each partition’s final items, then the final answer attends to intermediate counts for aggregation. These pathways peak in middle-to-late layers, with influential heads concentrated in layers 21–23.
4 Attention Analysis
- Intermediate counting tokens attend most strongly to the final item and final comma of their corresponding partition, especially around layers 19–23.Earlier items in the same partition receive substantially lower attention; the analysis examines selected tokens and localizes relevant layers and heads.Source: §4 · p. 5
- Attention peaks in layers 19–23 for both intermediate counts attending to partition-final items and final answers attending to partition counts.The head-level analysis concentrates influential heads in layers 21–23, including head 13 in layer 22.Source: Figure 5 · p. 5, §4 · p. 6, Figure 6 · p. 6
- During final-answer generation, the colon and preceding whitespace attend strongly to intermediate reasoning numbers, while attention to original input items is weak.Attention is concentrated in the same layers identified for intermediate counting.Source: §4 · pp. 5–6
- The observations support a staged mechanism: partition-final tokens encode local counts, intermediate reasoning tokens receive them, and attention to those tokens supports final aggregation.The authors present this separation of counting, information transfer, and aggregation as a hypothetical mechanism consistent with their observations.Source: §4 · p. 6
5 Causal Mediation Analysis
Causal mediation experiments trace how the structured counting strategy stores partition counts, routes them through intermediate steps, and combines them into a final answer. Interventions identify the representations and attention pathways involved at each stage.
5 Causal Mediation Analysis
- The causal mediation experiments combine activation patching, masking ablation, and attention knockout to locate stored counts, their transfer, and components required for aggregation.CountScope is used because logit-lens and tuned-lens are not reliable for decoding numerical information in this analysis.Source: §5 · p. 6, §5.1 · p. 6
5.1 Token-Level Information Probing
- CountScope decoding finds partition counts chiefly at the final item and comma; counts reset at boundaries, keeping each implicit count within the model’s reliable range.For later partitions, confidence at the final item falls while confidence at the vertical bar rises, but the final comma reliably stores the partition count.Source: §5.1 · pp. 6–7, Figure 7 · p. 6
5.2 Token-Level Causal Interventions
- Zero-ablating a partition’s final item and comma sharply reduces the probability of generating its correct count in the corresponding reasoning step.For example, ablating the final “, apple” token of a four-item partition reduces the likelihood of generating “4”.Source: §5.2 · p. 7, Figure 8 · p. 7
5.3 Information Pathway Localization
- For Qwen2.5-7B, influential components concentrate in layer 22: head 13 matters most for intermediate steps, while head 1 matters most for final aggregation.The knockout pattern is sparser than the attention pattern, plausibly because parallel pathways can compensate; transfer pathways to intermediate steps and final answers need not be the same.Source: §5.3 · p. 7, Figure 10 · p. 8
5.4 Cross-Context Activation Patching
- Swapping intermediate-step embeddings between contexts causally changed the corresponding steps and final totals, which followed the transferred numbers.In the example, totals changed from 19 and 14 to 21 and 12; the affected embeddings included “:”, whitespace, and the partition number.Source: §5.4 · pp. 7–8
6 Related Work
Prior work documents brittleness in LLM counting, studies numerical representations and mechanistic signals, and explores decomposition for reasoning. This work positions its structured prompting as a way to investigate how count information forms, moves, and combines across stages.
Counting in LLMs
- Counting can vary with surface form, tokenization, and minor prompt or content changes, while exact transformer counting is feasible only in specific regimes.Prior work links feasibility to representation capacity and context-length scaling.Source: §6 · p. 8
Numerical representations and mechanistic views; Decomposition and reasoning traces
- Prior studies examine numerical representations and localize latent counter-like signals; decomposition has improved visual counting, but its underlying mechanism remains underexplored.Hasani et al. (2025b) use target-context probes and layerwise and token-level analyses with causal interventions.Source: §6 · pp. 8–9
Interpretability tools
- Interpretability research uses causal interventions and patch-based inspection, alongside readout probes such as the Logit Lens and Tuned Lens.These tools provide complementary approaches to inspecting computation and tracking when information becomes linearly decodable across layers.Source: §6 · p. 9
Positioning of This Work
- This work combines marked input partitions with segment-level subcounts and aggregation, creating explicit stage boundaries to study how structured decomposition reshapes computation on long contexts.It connects behavioral brittleness observations to a stage-wise account of count information being formed, routed, and combined.Source: §6 · p. 9
7 Conclusion
The paper attributes large-scale counting failures to capacity-limited implicit counting and presents a test-time decomposition strategy. Its conclusion describes a mechanism for partition counts to reach intermediate steps and be aggregated, with applicability bounded by the reliable range of each subproblem.
7 Conclusion
- System-2-style test-time decomposition enables accurate counting over long contexts without modifying model parameters or training procedures.The conclusion attributes large-scale counting failures to capacity-limited System-1-like implicit counting, not fundamental limits of numerical reasoning.Source: §7 · p. 9
- Partition-level counts are encoded at boundary tokens, transferred through attention pathways to intermediate reasoning steps, and aggregated in middle-to-late layers.Intermediate reasoning tokens mediate numerical information flow from input partitions to the final output.Source: §7 · p. 9
- The procedure removes a fixed upper bound on countable size in principle, provided each subproblem stays within the model’s reliable regime.The paper suggests the analysis framework may extend to other reasoning tasks where implicit representations saturate and explicit decomposition helps correct behavior.Source: §7 · p. 9
Limitations
The study’s scope and method rely on simplified text inputs and structured decomposition, with applicability bounded by task structure and knowledge of the model’s reliable counting range.
Limitations
- The experiments focus on text-based counting with limited object diversity and synthetic contexts using repeated nouns rather than natural free-form text.These choices support systematic mechanistic analysis but narrow the evaluated setting.Source: Limitations · p. 10
- The approach depends on structured prompts and prior knowledge of the model’s reliable counting range, which may limit applicability across domains..Source: Limitations · p. 10
- The strategy is most effective when tasks divide into near-independent subproblems and is not expected to generalize to strongly interdependent tasks.The paper gives multi-step arithmetic and sequential planning as examples of tasks that may be decomposed.Source: Limitations · p. 10
A.1 Details of the Experimental Setup
The evaluation compares flat lists with partitioned lists and tests whether intermediate reasoning helps models count repeated items. It reports exact-match accuracy and mean absolute error across context lengths.
Evaluation design
- The study varies input structure and whether prompts request intermediate reasoning, measuring exact-match accuracy and mean absolute error across context-length ranges.The four settings combine structured or unstructured inputs with or without intermediate steps.Source: §A.1 · p. 12
Items
- Inputs use repeated single-word nouns drawn from fruit and animal categories.The candidate set includes 16 nouns.Source: §A.1 · p. 12
Input formats
- Partitioned inputs divide the same item multiset into segments separated by “|”, enabling local counting followed by summation.The unstructured format is a flat, comma-separated list.Source: §A.1 · p. 12
Partitioning and prompts
- Partition sizes are 6–9 items for open-source models and 15–25 for closed-source models, summing to the target length.The structured prompts request either a final total or separate partition counts followed by their sum.Source: §A.1 · p. 12
A.2 Behavioral Results on Additional Models
Additional-model evaluations find that structured input with intermediate reasoning generally remains the strongest setting, with a math-specialized exception. Robustness tests and attention analyses examine formatting, tokenization, and partition-related attention; causal mediation experiments are described as a method for testing count-information pathways.
A.2 Behavioral Results on Additional Models
- Structured input with intermediate reasoning remains strongest overall in smaller models and DeepSeekMath-7B, while Qwen2.5-Math-7B performs strongly in both structured settings, often without reasoning steps.Smaller models show lower absolute accuracy, especially at longer contexts; these tests cover four models and context sizes from 11 to 50 items.Source: §A.2 · p. 13
A.3 Robustness to Tokenization and Input Structure
- Across tokenizer and vocabulary differences, behavioral and mechanistic patterns remain consistent; for Qwen2.5-7B, structured inputs with reasoning outperform unstructured or no-reasoning variants, and Structure A outperforms Structure B.The two alternative formats vary part labels, separators, and item spacing.Source: §A.3 · p. 14, Table 6 · p. 14, Table 7 · p. 15
A.4 Attention Analysis
- Across Qwen2.5-7B, Llama3.2-8B, and Gemma3-4B, the token before a generated partition count attends most strongly to that partition’s final item and trailing comma.The selected layer ranges are 21–23 for Gemma3-4B and 13–18 for Llama3.2-8B; the figures show layer-wise attention and full maps.Source: §A.4 · p. 14, Figure 12 · p. 15, Figure 13 · p. 16
A.5 Causal Mediation Analysis
- CountScope is used to probe token-level latent counts, while attention-knockout experiments test how blocking heads affects intermediate-count transfer and final summation accuracy.The knockout prompt explicitly separates partition counting from aggregation and requires intermediate counts.Source: §A.5 · pp. 14–15