PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds

Md. Faiyaz Abdullah Sayeedi

arXiv:2608.17950v1cs.CL

TL;DR

How LLMs connect physically distant and semantically disjoint concepts during long-context reasoning remains poorly understood. This paper analyzes hidden-state geometry as sparse proximity graphs and finds that deep layers form Small-World pathways of fewer than six semantic hops, while hallucinations show topological collapse and grounded outputs remain near three hops.

  • Problem

    How Transformers route and reason across physically distant, semantically disjoint concepts in expanding contexts remains poorly understood.

  • Method

    The paper sparsifies hidden-state similarity matrices into latent proximity graphs and measures connectivity and shortest paths between disjoint semantic anchors using breadth-first search.

  • Results

    Deep layers transition into connected Small-World manifolds averaging ≤5 semantic hops, while RAG hallucinations collapse topologically and grounded outputs remain at approximately 3 hops.

  • Takeaways & Limitations

    Hidden-state topology provides a geometric framework for interpreting long-context reasoning and a zero-shot mechanistic signature for hallucination detection.

  • Takeaways & Limitations

    Evaluations cover only 1.5B–8B parameter models because exact O(N^2) token-wise proximity graphs are resource-intensive for frontier models and unavailable for proprietary APIs.

Abstract

from arXiv · show

Large Language Models (LLMs) demonstrate remarkable multi-hop reasoning capabilities over long contexts, yet the internal mechanisms enabling these distant cognitive leaps remain poorly understood. Traditional attention-based interpretability often fails to capture true semantic proximity due to routing artifacts like attention sinks. In this paper, we bypass attention weights to directly analyze the dynamic geometry of the hidden state manifold, proving that deep LLM latent spaces natively organize into Small-World networks. By sparsifying the continuous similarity matrices of long-context representations into unweighted graphs, we trace the connectivity between highly disjoint semantic anchors across two distinct architectures. Our findings reveal a sharp topological phase transition: while early syntactic layers remain entirely fractured, deep reasoning layers abruptly compress massive conceptual distances into highly navigable pathways strictly bounded by the "Six Degrees of Separation" limit (=< 6 semantic hops). Furthermore, we demonstrate the practical efficacy of this framework by applying it to zero-shot hallucination detection within Retrieval-Augmented Generation (RAG) using the RAGognize dataset. We show that factually grounded generations maintain structural integrity with their source context (approximately 3 hops), whereas hallucinations induce severe topological collapse. Ultimately, this work mathematically formalizes how transformers execute abstract reasoning and provides a novel, strictly geometric signature for evaluating factual reliability.

1 Introduction

The paper asks whether LLMs develop Small-World latent-space topologies that enable routing across physically distant, semantically disjoint concepts. It replaces attention-based proxies with direct geometric analysis and evaluates the resulting topology for zero-shot hallucination detection in RAG.

  • Motivation: The study investigates how Transformer models route and reason across physically distant, semantically disjoint concepts in expanding long-context windows.It frames this as a question of whether LLMs possess their own Small-World topology.
  • Motivation: Attention weights are unreliable information-flow proxies because attention collapse can route deep reasoning heads toward structural sink tokens instead of semantic bridges.Examples of sink tokens include <bos> and punctuation.
  • Method: The proposed Semantic Anchor methodology analyzes the latent data manifold using external embeddings while controlling physical and conceptual distance biases.The framework is designed to avoid statistical bias from physical token proximity and enforce distance constraints.
  • Findings: At τ ≈0.81, deep reasoning layers become densely connected while early syntax layers remain entirely disconnected with 0% connectivity.This abrupt change is characterized as a topological phase transition at strict Cosine Similarity thresholds.
  • Findings: Across architectures, LLMs compress distant, opposed concepts into an average of ≤5 semantic hops, while RAG hallucinations show ∞hops versus ≈3 hops for grounded outputs.The RAGognize application treats these contrasting structures as a mechanistic signature for evaluating reliability.

2 Background

The section defines Small-World networks through local clustering and short path lengths, then motivates analyzing LLM hidden-state geometry rather than attention weights. It introduces dynamic graph sparsification of long-context similarity manifolds to quantify semantic compression.

  • Small-World Networks and Cognitive Topology: Small-World networks combine high local clustering with strictly bounded, short average path lengths that typically scale logarithmically with network size.The concept was mathematically formalized by Watts and Strogatz (1998) and popularized as “Six Degrees of Separation.”
  • Transformer Routing and the Attention Fallacy: Attention-based graphs can collapse semantic pathways because deep reasoning heads route excess attention to structurally necessary but semantically hollow sink tokens.Examples include the <bos> token, punctuation, and line breaks, motivating a shift away from attention routing as a semantic map.
  • Geometry of Latent Representation Spaces: Transformer hidden states become increasingly contextualized and anisotropic through deep layers, supporting geometric analysis of latent representations.Prior work also advances linear representation hypotheses for high-level concepts.
  • Geometry of Latent Representation Spaces: The methodology dynamically sparsifies a long-context window’s continuous similarity manifold into unweighted adjacency matrices and applies graph traversal to measure semantic compression.This extends cosine-similarity analysis from static word vectors or prompt embeddings into network theory.

3 Methodology

The methodology bypasses attention analysis by mapping hidden-state manifolds into dynamic latent proximity graphs anchored on maximally disjoint concepts. It evaluates Small-World connectivity through shortest paths, threshold sweeps, and comparisons across layers and structurally diverse LLMs.

  • Latent proximity graphs: Hidden states are converted into layer-wise unweighted graphs by thresholding pairwise token cosine similarities, producing G(l) = (V, E).This directly maps the dynamic latent manifold rather than relying on attention matrices, which can suffer sink-token collapse.
  • Semantic anchors: 150 ≤N ≤300-token sequences are analyzed with an independent, task-agnostic dense embedding model to reduce confounding from physical token proximity.The external evaluator projects semantically valid tokens into an external vector space before graph analysis.
  • Semantic anchors: δ = 20 selects source and target anchors as the most semantically disjoint concepts while reducing bias from textual proximity.Anchors are chosen by minimizing cosine similarity in the external embedding space.
  • Topological evaluation: BFS measures shortest anchor paths, defining disconnected cases as L = ∞ and Small-World success through high connectivity with L ≤6.Connectivity rate Crate is the percentage of sequences with finite anchor paths.
  • Phase-transition analysis: [0.75, 0.91] threshold sweeps locate phase-transition boundaries across shallow syntax and deep reasoning layers and are repeated across structurally diverse LLMs.The comparison includes l = 2 and l = L −4.

4 Experimental Setup

The experimental pipeline evaluates latent proximity graphs across distinct transformer architectures while controlling for vocabulary, architectural, and pretraining-distribution effects. It uses long contiguous text, independently selected distant semantic anchors, threshold sweeps, and BFS-based path analysis, alongside the RAGognize hallucination-detection dataset.

  • Models: The study compares Qwen2.5-1.5B at layers l = 2 and l = 24 with Phi-3-Mini-4k-Instruct at layers l = 2 and l = 30.The models differ substantially in architecture and training data, enabling a cross-paradigm evaluation of latent topology.
  • Evaluation Corpus: The evaluation corpus uses wikitext-2-raw-v1, filtered to contiguous paragraphs containing between 150 and 300 tokens to preserve long-range conceptual separation.The filtering is intended to prevent short-term syntactic dependencies from artificially inflating connectivity metrics.
  • Hallucination Detection: RAGognize supplies closed-domain, token-level hallucination annotations from Wikipedia-based generations isolated from models’ parametric knowledge.Its source articles contain references time-stamped strictly after May 23, 2024, and include responses from modern target models such as Llama-3.1.
  • Semantic Anchors: Semantic anchors are selected by all-MiniLM-L6-v2 as the alphabetically restricted token pair with minimum external cosine similarity, subject to δ ≥20 tokens of separation.The embedding model functions as an independent judge, while the distance constraint avoids localized phrase selection.
  • Graph Construction and Analysis: The pipeline sweeps τ ∈[0.75, 0.91] in steps of 0.02, sparsifies hidden-state similarity matrices into unweighted graphs, and uses BFS to measure shortest anchor-path length.Repeating the procedure across thresholds identifies where latent representations fracture into disjoint subgraphs or form a continuous Small-World network.

5 Experimental Results

Across two model architectures, deep latent spaces formed navigable Small-World networks through abrupt topological phase transitions, while early syntactic layers remained disconnected. The same geometric framework separated grounded from hallucinated RAG generations and achieved promising zero-shot detection performance.

  • Topological organization: Deep reasoning layers, rather than early syntax layers, constructed Small-World geometry across both evaluated architectures.Across τ ∈[0.75, 0.91], Layer 2 showed Crate ≈0.0%, despite processing the same context window.
  • Topological phase transition: Both models underwent an abrupt topological phase transition as sparsification thresholds changed.Qwen2.5-1.5B reached 7.5% connectivity at τ = 0.81 and 48.5% at τ = 0.75; Phi-3-Mini reached 12.3% at τ = 0.85 and 42.0% at τ = 0.81.
  • Topological compression: Deep-layer routing compressed physically and conceptually distant anchors into paths bounded by the Six Degrees limit.The anchors were separated by an average of 246 tokens, while Phi-3-Mini compressed 200+ tokens into ≈3 to 5 hops and separation ablations yielded Lavg ≈4.0 to 5.5.
  • Layer-wise evolution: Layer-wise analysis localized compression to the deepest abstract reasoning blocks rather than showing gradual emergence.At τ = 0.81, Layers 1 through 12 had near-zero connectivity, Layer 16 reached Crate ≈12%, and Layer 22 exceeded 40%.
  • RAG hallucination detection: In RAGognize, grounded generations retained strong context connectivity, whereas unanswerable distractors produced structural fracturing.At τ = 0.81, answerable generations reached 92.4% connectivity for Qwen and 96.1% for Phi-3, with ≈3-hop paths; the classifier achieved AUROC of 0.89 on Qwen2.5-1.5B.

6 Discussion

The mathematical proof of a Small-World phase transition in deep transformer latent spaces establishes a foundational geometric framework for understanding long-context multi-hop reasoning. It shows that models dynamically compress disjoint premises into adjacent latent nodes, enabling robust cognitive leaps within a bounded number of steps.

  • Geometric framework: Deep transformer latent spaces undergo a mathematically proven Small-World phase transition.This provides a foundational geometric framework for advancing NLP research.
  • Long-context reasoning: Long-context reasoning dynamically compresses disjoint premises into adjacent latent nodes rather than preserving a fragile linear logic chain.The framework explains how models reason across thousands of tokens.
  • Reasoning mechanism: Bounded latent-space steps enable robust cognitive leaps between otherwise disjoint premises.This geometric organization demystifies the mechanics of long-context multi-hop reasoning.

7 Conclusion

Deep LLM latent spaces undergo a phase transition into Small-World networks, compressing long contexts so highly disjoint concepts connect in fewer than six semantic hops. On RAGognize, hallucinations appear as measurable topological collapses, enabling zero-shot mechanistic hallucination detection.

  • Topological compression: Deep LLM latent spaces undergo a phase transition into Small-World networks that connect highly disjoint concepts in fewer than six semantic hops.This geometric organization compresses long contexts into navigable semantic pathways.
  • Hallucination detection: On the RAGognize dataset, hallucinations manifest as measurable topological collapses rather than mere semantic errors.The finding characterizes hallucination as a structural failure in the latent geometry.
  • Hallucination detection: The geometric framework provides a zero-shot, mechanistic signature for hallucination detection.The conclusion presents this signature as an application of the latent-space analysis.

Limitations

The study’s geometric framework is limited by model scale, tested context length, and the empirically calibrated phase-transition threshold, motivating further research.

  • Model scale and access: Evaluations cover only 1.5B to 8B parameter models, while exact O(N^2) token-wise proximity graphs are resource-intensive for larger models and impossible for proprietary APIs.GPT-4 is cited as an example of an API whose internal hidden states are obfuscated.
  • Context length: The Small-World hypothesis was validated only for context separations up to 250 tokens, leaving its ≤6-hop limit across millions of tokens unresolved.Such ultra-long contexts may eventually fracture into localized sub-networks.
  • Calibration: The phase-transition threshold τ ≈0.81 was established empirically and may shift with embedding dimensionality or normalization techniques.Applying the framework to entirely novel architectures therefore requires a brief calibration sweep.

A Appendix

The appendix formalizes latent topology as a structural hallucination verifier and compares it with lexical and statistical baselines. On RAGognize, Small-World Topology uses deep-layer hidden-state graphs and structural routing features to detect disconnected hallucinated claims.

  • Method: Small-World Topology extracts deepest-layer hidden states, computes token-wise cosine similarities, and sparsifies them into an unweighted latent graph.For Qwen2.5-1.5B, the example deepest reasoning layer is Layer 24; the graph uses a phase-transition threshold of τ = 0.81.
  • Method: A linear SVM classifies context-response pairs using Connectivity Rate (Crate) and Average Path Length (Lavg) between source and generated-claim anchors.Crate measures valid, unbroken context-to-claim paths, while Lavg counts shortest-path semantic hops and assigns unreachable pairs a large penalty M.
  • Comparative Analysis of Detection Mechanisms: AUROC 0.89: Small-World Topology outperforms ROUGE-L, which achieves AUROC 0.68 on the RAGognize test set.ROUGE-L can misclassify abstractive correct reasoning because of low string overlap and can be fooled by verbatim repetition of irrelevant distractor text.
  • Comparative Analysis of Detection Mechanisms: Hallucinated claims fracture the latent graph, producing Crate →0 and Lavg →∞ as generated entities detach from the RAG context.The resulting structural separation is linearly classified by the SVM, according to the appendix’s geometric interpretation.
  • Threshold Sensitivity Analysis: AUROC ≈0.55: Detection remains near random chance at low thresholds τ < 0.60, where overly dense graphs create spurious connections.The appendix states that detection capability peaks sharply at the phase-transition threshold, although the supplied passage truncates the peak’s exact value.
Loading 2608.17950v1…