Looking Inside LLMs: Small-World Connectivity as a Signature of Reasoning Performance

1 INTRODUCTION

• The paper asks whether measurable internal organization is associated with stronger LLM reasoning, beyond what behavioral benchmarks quantify.The motivation draws on neuroscience findings associating stronger small-world organization in functional brain networks with higher intelligence.Source: §1 · p. 1
• Across models and training checkpoints, higher small-world index (SWI) consistently correlates with better fluid reasoning performance.The paper constructs functional graphs with attention heads as nodes and activation similarities as edge weights.Source: §1 · p. 2
• Functional graphs of attention-head activity capture relationships expressed in trained model responses, unlike architecture-level graphs or weight spectra that characterize design or weight statistics.The paper presents attention-head activation similarities as a way to study functional relationships among component activities.Source: §1 · pp. 1–2
• Across six LLMs, SWA better preserves small-world organization and model performance than competing allocation strategies, reducing WikiText perplexity by up to 20%.SWA uses core and bridge scores to guide hierarchical sparsity allocation across layers and attention heads.Source: §1 · p. 2
• Important heads tend to have more connection weight within their communities and a more concentrated distribution of weight across communities, corresponding to high core and low bridge scores.These observations motivate using the scores as structural indicators of head importance for reasoning.Source: §1 · p. 2

2.1 SMALL-WORLD ORGANIZATION OF BRAIN NETWORKS

• Small-world organization combines high clustering with short paths, and in neuroscience is viewed as balancing segregated specialization with integrated communication.Functional brain networks link statistical dependencies among regional activities to network organization.Source: §2.1 · p. 2
• The small-world index compares a graph’s average clustering coefficient and global efficiency with those of degree-preserving randomized graphs.A larger SWI indicates stronger local clustering and more efficient global communication relative to degree-matched random graphs.Source: §2.1 · pp. 2–3

2.2 REPRESENTATION SIMILARITY

• SVCCA measures similarity between mean-centered representations by retaining high-variance singular directions before comparing their projections with canonical correlation analysis.The procedure can retain directions explaining a high proportion of variance, for example 99%.Source: §2.2 · p. 3
• The paper’s SVCCA variant sums Fisher z-transformed canonical correlations instead of using their mean.The canonical correlations are computed between the projected representations.Source: §2.2 · p. 3

3 KEY FINDINGS

• Small-world organization combines high local clustering with efficient short-path communication.The section presents the functional-graph construction and SWI analysis as a framework for examining reasoning performance.Source: §3 · p. 3, Figure 1 · p. 4

3.1 CONSTRUCTING FUNCTIONAL GRAPHS OF ATTENTION HEADS

• For each head, the analysis averages its token-level query activations over generated answer tokens for sampled questions.The sampled questions are excluded from downstream evaluations.Source: §3.1 · p. 4
• Head-level functional graphs use activation similarity, estimated with Fisher z-transformed and aggregated canonical correlations, as edge weights.The method adapts SVCCA because attention heads produce vector-valued activations rather than scalar activity time series.Source: §3.1 · p. 4
• The graph retains the strongest pairwise similarities as edges at a fixed target density.Attention heads are the graph nodes.Source: §3.1 · pp. 4–5

3.2 CONNECTION OF SMALL-WORLD INDEX TO FLUID INTELLIGENCE

• Higher SWI is associated with better fluid reasoning performance across Pythia-6.9B checkpoints and different LLMs.Performance is measured as mean answer-token NLL, with lower values indicating better performance.Source: Figure 2 · p. 5, Figure 3 · p. 5, §3.2 · p. 5

4.1 LOCAL CONNECTIVITY PATTERNS OF IMPORTANT HEADS

• Head importance is the absolute gradient of mean answer-token NLL with respect to a head-output multiplier, averaged across test examples.The multiplier is applied to each head’s output, with gradients evaluated when all multipliers equal one.Source: §4.1 · p. 5
• A high core score means a larger share of a head’s connection weight lies within its own community, while a low bridge score means weight is concentrated in fewer communities.Communities are obtained using Louvain community detection.Source: §4.1 · pp. 5–6
• Low bridge scores and high core scores are each significantly more prevalent among important heads than among the remaining heads.Table 1 compares the top 25% of gradient-ranked heads with the remaining 75% in Llama-3.2-3B on DRE-Bench.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 VALIDATING HYPOTHESIS VIA PRUNING

• SWA uses core and bridge scores to allocate sparsity hierarchically across transformer blocks and individual attention heads.Its layer-level scores set block budgets, while head-level scores distribute attention sparsity within each block.Source: §4.2 · p. 6
• Heads with low bridge or high core scores receive lower sparsity, and layers with more structurally important heads receive a lower sparsity budget.SWA normalizes and rescales the relative pruning ratios to match the target global sparsity.Source: §4.2 · p. 6
• SWA achieves the best WikiText perplexity across all evaluated models and sparsity levels among the compared pruning methods.The comparison covers Wanda and SparseGPT and variants augmented with FARMS, ATP, and SWA at sparsity 0.5 and 0.7.Source: Table 3 · p. 7

5.1 EXPERIMENTAL SETUP

• The setup compares SWI with answer NLL/token on six open-source LLMs, using DRE-Bench Level 1 and 2 questions as calibration inputs.Levels 3 and 4 are excluded because current open-source models often fail on them, making answers unreliable for activation-based analysis.Source: §5.1 · pp. 7–8

5.4 ABLATION OF HIERARCHICAL SPARSITY ALLOCATION

• Removing either allocation level raises WikiText perplexity in every tested setting; head-only allocation degrades more than layer-only, especially at 0.7 sparsity.The ablation tests Qwen3 8B with SparseGPT and Wanda at sparsity levels 0.5 and 0.7; the table reports absolute SWA perplexity and increases for ablated variants.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 SENSITIVITY TO THE GRAPH-CONSTRUCTION DATASET

• At 70% sparsity, SWA using GSM8K, ARC-C, or MMLU graphs yields lower perplexity than the base method across four reported models, though the best source varies.On Llama-3.2-3B, perplexity falls from 70.70 to 48.95–50.54 across the three graph sources; each graph uses 400 training examples or all available examples if fewer.Source: Table 5 · p. 8, §5.5 · p. 9

5.2 EFFECTIVENESS OF PRESERVING SWI

• At 0.5 sparsity, SWA retains higher relative SWI than competing allocation strategies across all six models, although pruning substantially lowers absolute SWI.The comparison applies SparseGPT and recomputes SWI after pruning.Source: §5.2 · p. 8

5.3 EFFECTIVENESS OF SWA

• SWA consistently outperforms pruning baselines on WikiText perplexity across models and sparsity levels, with a wider gap at higher sparsity.The evaluation uses the WikiText validation set.Source: §5.3 · p. 8

Structural Analysis of Neural Networks

• Architecture graphs, weight spectra, and representation-similarity methods capture distinct aspects of network structure, but do not explicitly model interactions among model components.Prior work also finds transformer attention heads heterogeneous, with some specialized behaviors and many redundant or weakly used heads.Source: §6 · p. 9
• This study builds activation-defined functional graphs over attention heads and uses SWI to test whether their organization is associated with reasoning capability.The approach focuses on functional connectivity rather than prescribed architecture or representation comparisons alone.Source: §6 · p. 9

Functional Connectivity and Small-world Organization

• Small-world organization is characterized by high clustering and short paths, or high local and global efficiency, and is framed as balancing specialization with integrated communication.Prior machine-learning work also uses activation-based graphs and topological summaries to study neural-network organization and generalization.Source: §6 · p. 9

7 CONCLUSION

• Across models and training checkpoints, higher SWI is consistently associated with better fluid reasoning performance, while important heads tend to have high core and low bridge scores.SWA better preserves small-world organization and model performance than competing allocation strategies.Source: §7 · p. 9
• The authors report using generative AI for writing, editing, and proofreading, with the authors reviewing the assisted text and taking responsibility for the paper.Source: AI USE STATEMENT · p. 10
• The study uses publicly available models and datasets, reports no additional ethics concerns specific to its experiments, and states that methodological and experimental details are provided for reproduction.The authors also state that they will release code.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 JUSTIFICATION OF THE HEAD SIMILARITY MEASURE

• The measure is a variant of SVCCA that replaces mean aggregation with a sum of Fisher z-transformed canonical correlations.It produces a scalar similarity score summarizing shared variation across questions between vector-valued attention-head representations.Source: §A.1 · p. 14
• Before CCA, the method retains leading singular directions explaining at least 99% of each representation's variance, focusing comparison on subspaces capturing most observed variation.The stated motivation is to remove low-variance directions that may reflect noise.Source: §A.1 · p. 14
• Fisher z aggregation gives greater emphasis to strongly correlated canonical directions, and the resulting sum depends on both correlation strengths and the number of retained directions.The authors use the transformation as an aggregation rule without assuming it stabilizes sampling variance.Source: §A.1 · p. 14
• Unlike CKA, the measure identifies correlated linear projections and aggregates their correlation strengths to represent functional connectivity between heads.The choice reflects the intended graph-edge interpretation, not a claim of general superiority over CKA.Source: §A.1 · p. 14

• The section covers functional-graph and SWI settings, pruning methods and allocation baselines, and evaluation protocols.Source: §A.2 · p. 14

A.2.1 GRAPH CONSTRUCTION AND SWI ANALYSIS

• Graph construction uses 400 DRE-Bench questions, with additional GSM8K, ARC-C, and MMLU samples for sensitivity analyses.Head activations are reduced by SVD and compared pairwise with CCA; graphs retain the top similarities at fixed edge density δ = 0.25, and SWI uses degree-preserving randomized references.Source: §A.2.1 · pp. 14–15

A.2.2 PRUNING EVALUATION

• SWA is compared with uniform, FARMS, and ATP sparsity allocations when integrated with SparseGPT or Wanda.SparseGPT uses second-order information to select masks and update remaining weights; Wanda scores weights using magnitude and input activations without updating survivors.Source: §A.2.2 · p. 15
• Evaluation includes WikiText perplexity and mean zero-shot accuracy at 50% and 70% sparsity, averaged over three seeds.ARAOC is an additional evaluation-only benchmark; Qwen3-8B with SparseGPT is tested at 50% sparsity using output-grid shape mismatch across six task types.Source: §A.2.2 · pp. 15–16

A.3 NUMERICAL RESULTS FOR THE SWI–PERFORMANCE ASSOCIATION

• Across Pythia-6.9B checkpoints and six Qwen and Llama models, lower DRE-Bench answer-token NLL is associated with higher SWI on both levels.On Level 2, Pythia’s NLL falls from 0.8255 to 0.3749 as SWI rises from 0.4587 to 1.1864 between steps 4,000 and 143,000.Source: Table 6 · p. 16, §A.3 · p. 16, Table 7 · p. 16

A.2.3 COMPUTING INFRASTRUCTURE

• Experiments use a cluster equipped with NVIDIA RTX PRO 6000 GPUs and AMD EPYC 9655 processors.Source: §A.2.3 · p. 16

A.4 ZERO-SHOT PERFORMANCE OF SWA

• At 70% sparsity, SWA raises Wanda-pruned Llama-3-8B average zero-shot accuracy from 27.57% to 37.89%, beating ATP by 1.99 percentage points.SWA also achieves the highest mean accuracy across six models at both sparsity levels with Wanda or SparseGPT.Source: Table 8 · p. 17, §A.4 · p. 17

A.5 PERFORMANCE ON ARAOC REASONING TASKS

• On ARAOC Fill Internal, SWA reduces output-grid shape mismatch from SparseGPT’s 34.25% to 28.69%, versus 32.66% with ATP.Across all six task types, SWA has lower mismatch rates than pruning baselines, though rates remain above the unpruned model.Source: Table 9 · p. 17, §A.5 · p. 17