Source-linked AI summary
The Geometry of Knowledge Accessibility in Large Language Models
Lihu Chen
TL;DR
LLMs store broad knowledge but cannot access it reliably, and answer correctness can conflate recall with reasoning. This paper models query representations around an accessibility center and finds that distance tracks accessibility and intervention benefits differ by region.
Problem
LLMs contain broad knowledge but cannot access all of it reliably, and the geometry organizing knowledge accessibility is an open question.
Source: §1 · p. 1Method
The paper models query representations with an accessibility center and measures knowledge distance as a pre-generation signal.
Source: §3.1 · p. 3, §7 · p. 9Results
Takeaways & Limitations
Pre-generation knowledge distance may help models choose whether to answer directly, reason more, or retrieve information.
Source: §7 · p. 9Evidence is correlational and focused on single-fact queries; correctness is a behavioral proxy, and stored-but-unreached knowledge is not distinguished from knowledge that was never stored.
Source: §2.1 · p. 2, §7 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) contain broad knowledge, but they cannot access all of it reliably. We study this problem through knowledge accessibility, which describes whether the knowledge needed for a query can be recalled from the model. We find that knowledge accessibility has a simple geometric structure in the model's representation of the query alone, before any generation. More accessible queries are closer to a center in the representation space, while less accessible queries are farther away. This geometry reveals a knowledge boundary that separates more accessible queries from less accessible ones. Accessibility consistently decreases with distance from the center, and this distance-based ordering transfers across datasets even when the centers differ. Controlled experiments further show that the centered geometry is more closely related to knowledge accessibility than to reasoning difficulty. The geometry also reveals when different interventions are useful. Query rewriting helps more for accessible queries, chain-of-thought reasoning helps more near the boundary, and retrieval gives larger gains beyond the boundary. These findings not only provide a new geometric view of how knowledge is organized in language models, but also suggest a useful pre-generation signal for adaptive inference.
1 INTRODUCTION
The paper asks how knowledge accessibility is organized in LLM representations, separating factual access from reasoning failures. It proposes a center-based geometry and reports that distance from the center tracks accessibility and helps distinguish which interventions benefit queries.
1 INTRODUCTION
- Prior work often defines the knowledge boundary through answer correctness, conflating knowledge access with reasoning, and does not explain how accessibility is organized in representation space.The paper distinguishes this question from whether knowledge is merely encoded in hidden states.Source: §1 · p. 1
- The authors study accessibility with single-fact queries, which require a factual association rather than multi-step reasoning.They define accessibility as whether the knowledge required by a query can be recalled from the model’s parameters.Source: §1 · p. 1
- The accessibility center hypothesis places reliably accessible queries in a localized region, with greater distance indicating lower accessibility and defining a boundary between regions.The center is used to measure query distance; the passage does not present it as a location where facts are stored.Source: §1 · pp. 1–2
- Across two LLMs and four QA datasets, accessibility decreases with distance from a learned center, while a flexible MLP gains an advantage on reasoning-heavy tasks.On knowledge-focused tasks, centered geometries predict accessibility better than competitive probing baselines; controlled experiments support the distinction between accessibility and reasoning.Source: §1 · p. 2
- Rewriting helps most for accessible queries, chain-of-thought reasoning near the boundary, and retrieval beyond it; distance-based ordering transfers across datasets despite differing centers.Queries also become less stable in correctness as they approach the boundary.Source: §1 · p. 2
2 KNOWLEDGE ACCESSIBILITY
The paper treats accessibility and reasoning as distinct contributors to correctness, using single-fact recall as a cleaner behavioral proxy for accessibility. It then asks whether accessibility has a center-based organization in query representations.
2 KNOWLEDGE ACCESSIBILITY
- Correctness is conceptually decomposed into knowledge accessibility, reasoning capability, and other task-specific capabilities, while the study focuses on factual recall using single-fact queries.These queries mainly depend on recalling a single target fact, helping isolate factual recall from multi-step reasoning.Source: §2 · p. 2
2.1 QUERY CORRECTNESS AND FACTUAL RECALL
- For single-fact queries, correctness under a decoding strategy serves as a behavioral observation of factual recall, but low accessibility may reflect either unstored knowledge or stored knowledge that is not reached.This proxy is less clean for mathematical or multi-step tasks, where reasoning and other capabilities also affect correctness.Source: §2.1 · p. 2
2.2 A GEOMETRIC HYPOTHESIS OF KNOWLEDGE ACCESSIBILITY
- Prior probing shows accessibility-related information in hidden states but not its organization; the hypothesis is that recall becomes less likely as query distance from an accessibility center increases.The center summarizes a region associated with reliable factual access, not a location where individual facts are stored.Source: §2.2 · p. 3
3 MODELING THE GEOMETRY OF KNOWLEDGE ACCESSIBILITY
The paper models accessibility using centered distances in query-representation space, comparing isotropic and anisotropic geometries with linear and nonlinear references. A learned radius partitions queries into interior, boundary, and exterior regimes.
3.1 CENTERED ACCESSIBILITY GEOMETRY
- The isotropic model measures Euclidean distance from a shared center, while an anisotropic extension weights dimensions and yields hyperellipsoidal level sets.The comparisons include a linear baseline and a more expressive MLP, contrasting centered geometry with linear separability and flexible nonlinear predictability.Source: §3.1 · p. 3
3.2 THE KNOWLEDGE BOUNDARY
- A learned boundary radius and transition width divide queries into interior, boundary, and exterior regimes, with accessibility expected to be higher inside and lower outside.The boundary marks an unstable transition; signed distance expresses the same regimes relative to the learned radius.Source: §3.2 · p. 4
4 EVALUATING THE ACCESSIBILITY GEOMETRY
Across benchmark comparisons and controlled tests, centered query geometry tracks knowledge accessibility more directly than reasoning-heavy correctness. Distance from the center also marks changing accessibility and stability, and preserves an accessibility ordering across datasets even when centers differ.
4.1 EXPERIMENTAL SETUP
- The evaluation uses two instruction-tuned models and four QA benchmarks, extracting each query’s final-token representation before generation and testing across five random seeds.TriviaQA and SciQ emphasize factual recall; MuSiQue and GSM8K require more reasoning.Source: §4.1 · p. 4
4.2 COMPARING KNOWLEDGE ACCESSIBILITY GEOMETRIES
- On TriviaQA and SciQ, A-Centered is strongest on three of four model–dataset pairs, while Linear performs substantially worse than centered models.Simple Centered is close to MLP on knowledge-focused tasks; on reasoning-heavy MuSiQue and GSM8K, MLP performs best, while A-Centered remains competitive with O(d) rather than O(dm) parameters.Source: §4.2 · pp. 4–5, Figure 1 · p. 5
4.3 SEPARATING KNOWLEDGE ACCESSIBILITY FROM REASONING
- On MQuAKE, A-Centered distance aligns more closely with single-hop knowledge accessibility than final multi-hop correctness: knowledge failures lie outside the boundary, while reasoning failures remain inside.The geometry learned on TriviaQA is applied to MQuAKE without relearning.Source: Table 3 · p. 5, §4.3 · p. 6
4.4 BEHAVIOR OF KNOWLEDGE ACCESSIBILITY
- Accessibility and answer consistency decline with distance, while correctness stability is lowest near the boundary and rises again beyond it, where repeated failures dominate.The stability pattern is independent evidence because repeated generations were not used to learn the boundary.Source: §4.4 · pp. 6–7
4.5 TRANSFERABILITY OF ACCESSIBILITY DISTANCE
- Distance ordering transfers across datasets without relearning, although the relation is weaker on reasoning-heavy tasks and the learned centers and dimension weights do not transfer.Different task types appear to have their own accessibility centers.Source: §4.5 · p. 7
5 KNOWLEDGE ACCESSIBILITY INTERVENTIONS
On TriviaQA, the most useful intervention depends on a query’s accessibility region: rewriting helps in the interior, extra computation near the boundary, and retrieval in the exterior. The paper uses query-only distance as a pre-generation signal for these differences.
5 KNOWLEDGE ACCESSIBILITY INTERVENTIONS
- Query-only distance is available before generation and may guide whether to answer directly, reason more, or retrieve information.The paper presents this as a potential signal for adapting inference and reducing cost.Source: §5 · pp. 7–9
- The interventions operate at inference without retraining: rewriting changes wording, CoT adds computation, and RAG supplies external information.The study examines them on TriviaQA as complementary ways to improve effective accessibility.Source: §5 · p. 7
- On TriviaQA, rewriting gains peak in the interior, chain-of-thought gains near the boundary, and retrieval gains in the exterior.Figure 5 compares the interventions across accessibility regions.Source: §5 · p. 7, Figure 5 · p. 8
- Query rewriting changes wording without adding information, and its interior gains can recover failures by offering another path to accessible knowledge.Rewriting becomes less effective farther into the exterior, where parametric access is increasingly unreliable.Source: §5 · pp. 7–8
- CoT yields its largest average accuracy gain near the boundary, where access is unstable but knowledge remains accessible.Its gain decreases in the exterior because CoT adds computation but no new information.Source: §5 · p. 8
- RAG’s average benefit increases with knowledge distance and is largest in the exterior, where parametric access is unreliable.Unlike the other interventions, retrieval supplies external information.Source: §5 · pp. 8–9
6 RELATED WORK
Prior work uses internal states to estimate knowledge boundaries, including from the query before generation. This paper distinguishes accessibility from reasoning and studies how accessibility is organized geometrically.
6 RELATED WORK
- Prior studies probe internal states for knowledge-boundary information, and some estimate the boundary from the query before generation.These signals can help inform how to answer a query.Source: §6 · p. 9
- Prior approaches predict observed correctness without separating limited knowledge recall from reasoning failures.The paper identifies this as a limitation of correctness-based boundary estimates.Source: §6 · p. 9
- This work studies the geometric organization of knowledge accessibility in query representations rather than only whether boundary information is encoded in hidden states.The distinction addresses a gap left by prior probing studies.Source: §6 · p. 9
7 CONCLUSION
The conclusion presents accessibility as geometrically organized around a center and links distance to intervention choice. It also identifies single-fact scope, correctness as a proxy, and correlational evidence as limitations.
7 CONCLUSION
- Accessible queries lie close to a common center, accessibility declines with distance, and distance-based ordering transfers across datasets despite differing centers.Rewriting helps in the interior, reasoning near the boundary, and retrieval beyond it; the pre-generation signal may guide intervention choice.Source: §7 · p. 9
- The study focuses on single-fact queries and uses correctness as a behavioral proxy for accessibility.It does not distinguish knowledge never stored from knowledge stored but not reached, and its evidence is correlational.Source: §7 · p. 9
A.1 REPRESENTATIONS AND CORRECTNESS LABELS
- For each query, the representation is the final non-padding query-token hidden state, extracted before any answer token is generated.Source: §A.1 · p. 13
- Each representation dimension is standardized using the training split’s mean and standard deviation before geometry fitting.The paper uses h(x) to denote the standardized representation.Source: §A.1 · p. 14
- A response is labeled correct when Mistral-Large judges it semantically equivalent to an answer in the reference set; greedy decoding is used unless stated otherwise.Source: §A.1 · p. 14
A.2 GEOMETRY MODELS AND TRAINING
The appendix details training and geometry analyses, including evidence that Linear+Norm tends to learn a centered structure even without an explicit center. The analyses also report accessibility and stability patterns for Qwen-14B.
A.2 GEOMETRY MODELS AND TRAINING
- The Linear baseline is logistic regression with C = 1 and L-BFGS; the MLP has one 128-unit ReLU hidden layer, dropout 0.1, and validation-AUROC early stopping.Source: §A.2 · p. 14
- Centered initializes its center from correctly answered training queries and uses a soft-margin objective; A-Centered initializes from Centered and trains with the inherited validation-selected margin.Source: §A.2 · p. 14
- The methods share correctness-stratified 40%/10%/50% train/validation/test splits across layers, with five seeds; Adam optimizes MLP, Centered, and A-Centered at 10^-2.Source: §A.2 · p. 14
B DETAILS OF THE GEOMETRY ANALYSES
- For Qwen-14B on TriviaQA, accessibility and consistency decrease with distance from the center, while correctness is least stable near the boundary.Source: Figure B.2 · p. 15
B.1 TESTING THE CENTERED INDUCTIVE BIAS
- Linear+Norm’s learned coefficient α averages negative across layers, and its score is equivalent to a rescaled Centered score with an implied center.Together with Centered’s stronger performance, this supports centered structure as an inductive bias rather than a model-capacity effect.Source: §B.1 · p. 15
B.2 CONTROLLED MQUAKE EVALUATION
The appendix evaluates cross-dataset distance–accuracy transfer and describes a controlled MQuAKE setup that distinguishes accessible knowledge from reasoning success. In the reported diagnostics, the distance-based ordering transfers even when dataset-specific geometric parameters differ.
B.2 CONTROLLED MQUAKE EVALUATION
- MQuAKE separates knowledge accessibility from reasoning using multi-hop queries and their required single-hop fact questions.The evaluation uses 1,000 examples with Llama-3.1-8B-Instruct.Source: §B.2 · p. 15
- Correctness on all required single-hop questions serves as a proxy for accessible knowledge, while multi-hop correctness measures reasoning success.Their combination yields Success, Reasoning Failure, Knowledge Failure, and Recovered Success groups.Source: §B.2 · p. 15
- MQuAKE queries are evaluated using a frozen A-Centered geometry learned on TriviaQA, with no geometry fitted on MQuAKE.Queries are represented at the selected TriviaQA layer and their distances are computed under that geometry.Source: §B.2 · p. 15
B.3 BEHAVIOR OF KNOWLEDGE ACCESSIBILITY
- The appendix estimates correctness probabilities and stability from repeated stochastic responses, and separately measures consistency of surface-form answers.High answer consistency is used to label confidently correct or wrong queries for visualization only.Source: §B.3 · pp. 15–16
- The distance-based ordering transfers across datasets, although learned centers and dimension weights differ by dataset.For Qwen-14B, accuracy generally decreases with distance across target datasets.Source: Figure B.4 · p. 16, Figure B.3 · p. 16
B.4 CROSS-DATASET TRANSFER AND DATASET-SPECIFIC GEOMETRY
The section tests whether distance–accuracy relations transfer across datasets and whether the learned geometric parameters are shared. The ordering transfers despite dataset-specific centers and dimension weights.
B.4 CROSS-DATASET TRANSFER AND DATASET-SPECIFIC GEOMETRY
- Each of TriviaQA, SciQ, MuSiQue, and GSM8K serves in turn as the source, whose A-Centered geometry is applied without relearning.The best-performing layer is selected for Llama-8B.Source: §B.4 · p. 16
- For each target dataset, queries are grouped into distance bins under the source geometry, and mean distance and accuracy are used to summarize their relation.The relation is summarized with Spearman correlation.Source: §B.4 · p. 16
- Center similarity is measured by mapping learned centers to the original representation space and normalizing their distance by representation scale.Smaller normalized values indicate more similar centers.Source: §B.4 · p. 17
- Dimension-weight similarity is compared using Spearman correlation after mapping the weights back to the original representation space.Higher correlation means the geometries emphasize similar dimensions.Source: §B.4 · p. 17
- Transferred geometries retain much of target-specific AUROC, and distance–accuracy correlations are negative across source–target pairs, while centers and dimension weights differ across datasets.The negative relation means target queries farther from the source center are answered less accurately; transfer requires preserving relative ordering, not sharing a center or shape.Source: §B.4 · pp. 16–17
C INTERVENTION EXPERIMENTS
The intervention experiments evaluate query rewriting, chain-of-thought, and retrieval on TriviaQA test queries using a frozen geometry and signed distance to the learned boundary. Each intervention is assessed against original greedy-answer correctness.
C.1 SHARED SETUP
- All interventions use TriviaQA test queries, the same frozen A-Centered geometry and selected layer, and correctness comparisons against the original greedy answer.CoT and retrieval answers are judged without showing the reasoning trace or retrieved context to the judge.Source: §C.1 · p. 17
- Queries are divided into 20 equal-frequency bins by signed boundary distance, with negative values inside and positive values outside.Reported curves use raw bin-level statistics without smoothing or interpolation.Source: §C.1 · p. 17
C.2 QUERY REWRITING
- Query rewriting uses five Mistral-Large paraphrases per query, and its gain is the change in correctness relative to the original query.The model answers each paraphrase independently using internal knowledge and greedy decoding; correctness fraction and paraphrase stability are also measured.Source: §C.2 · p. 17
C.3 CHAIN-OF-THOUGHT
- Chain-of-thought prompts the model to reason step by step using internal knowledge, adding computation but no external information.The experiment records answer correctness and generated-token count to compare benefit and cost across boundary regions.Source: §C.3 · p. 17
C.4 RETRIEVAL-AUGMENTED GENERATION
- Retrieval prepends TriviaQA evidence to the question and asks the model to answer using it, without requesting chain-of-thought.Queries lacking a matching example or usable retrieval context are excluded from the RAG analysis.Source: §C.4 · p. 18