PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Reasoning: Flowing Logics in Representation Space

Yufa Zhou, Yixiao Wang, Xunjian Yin, Shuyan Zhou, Anru R. Zhang

arXiv:2510.09782v2cs.AIcs.CLcs.LGcs.LO

TL;DR

The paper addresses whether LLM reasoning has geometric structure beyond semantic surface form. It models reasoning as representation-space flows, tests logical skeletons across varied carriers, and finds that velocity and curvature preserve logical invariants while positional similarity remains more semantic. The authors conclude that this framework offers tools for interpretability, while noting that generation accuracy and the origins of these patterns remain outside scope.

  • Problem

    The paper asks whether LLMs internalize logical structure independently of semantic carriers and whether reasoning can be understood geometrically.

  • Method

    The paper models reasoning as smooth representation-space flows and uses a controlled dataset with shared logical skeletons across topics and languages, analyzed through velocity and Menger curvature.

  • Results

    Velocity and curvature reveal logic as the principal organizing factor beyond surface form, while shuffled logical order collapses these higher-order similarities but preserves positional similarity.

  • Takeaways & Limitations

    The framework provides a conceptual foundation and practical tools for analyzing reasoning dynamics and opens new directions for interpretability.

  • Takeaways & Limitations

    The work studies understanding rather than generation, so its geometric measures cannot be meaningfully related to generation-specific properties such as task output accuracy.

Abstract

from arXiv · show

We study how large language models (LLMs) ``think'' through their representation space. We propose a novel geometric framework that models an LLM's reasoning as flows -- embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. Our findings indicate that training solely via next-token prediction can lead LLMs to internalize logical invariants as higher-order geometry in representation space, challenging the ``stochastic parrot'' argument. Experiments across Qwen and LLaMA model families further suggest the presence of a general, possibly universal, representational law underlying machine understanding and human linguistic regularities, largely independent of specific training recipes or model architectures. Our work serves as both a conceptual foundation and practical tools for studying reasoning phenomena, offering a new lens for interpretability and formal analysis of LLMs' behavior.

1 INTRODUCTION

The paper frames LLM reasoning as structured geometric flows and tests whether logical form persists across semantic carriers. Its experiments find that higher-order representation geometry tracks logic more strongly than surface meaning.

  • 1 INTRODUCTION: The paper models reasoning as a trajectory or flow on a low-dimensional concept manifold, with local velocities governed by logical operations.The framework records representations as reasoning prefixes are extended and analyzes their geometric dynamics.
  • 1 INTRODUCTION: Formal logic is treated as a carrier-invariant skeleton, motivating tests across topics and languages to separate logical structure from semantic surface.The dataset preserves abstract reasoning steps while varying words, contexts, and languages.
  • 1 INTRODUCTION: Velocity and Menger-curvature similarities remain consistent for flows sharing logical skeletons across unrelated topics and languages, while differing logic produces lower similarity.The comparison reverses when semantic carriers match but logical structures differ.
  • 1 INTRODUCTION: Shuffling logical statement order collapses velocity and curvature similarity while preserving positional similarity, indicating that logic is encoded in higher-order geometry.Raw representation position is therefore more semantically organized than first- and second-order differences.
  • 1 INTRODUCTION: The work contributes formal definitions, analytic tools, a logic dataset, and empirical validation for studying LLM reasoning dynamics.The authors present the framework as a foundation for interpretability and practical analysis.

2 RELATED WORK

Related work motivates geometric representations of concepts and complementary accounts of reasoning as computation, trajectories, or graph dynamics. This paper builds on those perspectives by emphasizing structured flows in representation space.

  • 2 RELATED WORK: Concept geometry ranges from linear directions to multidimensional or manifold-like structures, especially for complex features such as colors, dates, and antonyms.The literature supports geometric representations while limiting strict linearity as a universal assumption.
  • 2 RELATED WORK: Existing reasoning accounts address test-time scaling, chain-of-thought computation, superposed trajectories, hidden planning, inductive biases, and graph dynamics.These perspectives explain reasoning through computation, model bias, or structural path aggregation.

3 PRELIMINARIES

The preliminaries map discrete language sequences into continuous representations and define geometric tools for tracing reasoning trajectories. Menger curvature supplies a metric-sensitive measure of local trajectory structure.

  • 3 PRELIMINARIES: An LLM defines a conditional token distribution, and chain-of-thought reasoning generates a token sequence through recursive sampling from prior context.At each step, the next token is sampled conditioned on the prompt and preceding tokens.
  • 3 PRELIMINARIES: A representation operator maps token sequences to continuous vectors, making discrete language inputs analyzable in an embedding space.The operator may select a token position, pooling rule, or internal layer state.
  • 3 PRELIMINARIES: The representation space is the range of this operator, consisting of embeddings that serve as an empirical proxy for geometric reasoning analysis.In practice, representations can come from pretrained encoders or hidden states extracted from LLMs.
  • 3 PRELIMINARIES: Menger curvature measures reasoning-flow geometry from triples of embedding points as the reciprocal radius of their circumcircle.Unlike angle-only measures, it captures angular deviation together with distance variation.

4 REASONING AS GEOMETRIC FLOWS IN REPRESENTATION SPACE

The paper models reasoning as trajectories through representation and concept spaces, with semantic content evolving continuously and logical structure regulating local motion. It formalizes cumulative flows, their alignment across spaces, and the role of logic as a controller of semantic velocity.

  • 4 REASONING AS GEOMETRIC FLOWS IN REPRESENTATION SPACE: LLM reasoning is formalized as a trajectory through representation space, while logical structure acts as a local controller of the flow.The framework connects semantic trajectories, formal logical space, and representation-logic dynamics.
  • 4.1 CONCEPT SPACE AND SEMANTIC TRAJECTORIES: Concept space is modeled as a smooth geometric space in which coherent semantic content unfolds along continuous trajectories.Discrete prefixes are aligned with points on a curve representing the gradual evolution of conceptual content.
  • 4.1 CONCEPT SPACE AND SEMANTIC TRAJECTORIES: The formal logical space maps semantically different expressions with the same natural-deduction proposition to the same structural representation.This mapping separates logical form from the particular semantic carrier used to express it.
  • 4.2 REPRESENTATION SPACE: A context-cumulative flow records the embedding of each growing prompt-and-reasoning prefix, producing a sequence of representation states.The trajectory is constructed by appending each reasoning step to prior context and applying the representation map.
  • 4.2 REPRESENTATION SPACE: The alignment between linguistic inputs and representation trajectories is only guaranteed on restricted domains where the conceptual trajectory map is injective.Global injectivity over natural language is explicitly left as an open problem.
  • 4.3 LOGIC AS DIFFERENTIAL CONSTRAINTS ON FLOW: Logic is treated as a differential regulator: discrete logical steps integrate local semantic velocity and govern both its magnitude and direction.The framework links local representation increments to derivatives of continuous embedding trajectories.
  • 4.3 LOGIC AS DIFFERENTIAL CONSTRAINTS ON FLOW: Flows sharing a natural-deduction skeleton are predicted to retain correlated curvature across changes in topics or languages.Cross-carrier similarity in first-order differences and curvature is proposed as empirical evidence connecting formal and representational logic.

5 FORMAL LOGIC WITH SEMANTIC CARRIERS

The paper uses natural deduction to define logical scaffolds and constructs matched reasoning tasks whose topical and linguistic carriers vary while the underlying inference structure remains fixed. This design enables comparison of position, velocity, and curvature similarities to determine which geometric quantities reflect logic rather than surface semantics.

  • 5.1 LOGIC AND NATURAL DEDUCTION SYSTEM: A natural deduction system is defined as a formal language of formulas together with inference rules governing derivations.The rules include introduction and elimination rules for logical constants.
  • 5.1 LOGIC AND NATURAL DEDUCTION SYSTEM: Natural-deduction proofs are trees of derivable-formula judgements whose edges follow inference rules, with temporary assumptions discharged by specific rules.Paired introduction and elimination rules determine the proof-theoretic meaning of each connective.
  • 5.1 LOGIC AND NATURAL DEDUCTION SYSTEM: Table 1 compares position, velocity, and curvature similarities grouped by logic, topic, and language.Logic groups examples by deduction skeleton, whereas topic and language capture surface carriers.
  • 5.1 LOGIC AND NATURAL DEDUCTION SYSTEM: The experiments show that position similarity is dominated by surface carriers, whereas velocity and curvature identify logical structure as the primary invariant.This is the central comparison reported in Table 1.
  • 5.2 DATA DESIGN: The dataset preserves identical logical scaffolding while varying topical domains and linguistic realization to separate logical structure from semantic content.The same abstract reasoning steps are instantiated across multiple carriers, so persistent similarities can be attributed to logic.
  • 5.2 DATA DESIGN: The dataset uses a two-stage pipeline that generates abstract logical templates and then rewrites them for domains and languages.It contains 30 logical structures with 8–16 reasoning steps, instantiated across 20 topics and four languages.

6 PLAY WITH LLMS

Experiments compare reasoning-flow similarities across logic templates, topics, languages, model scales, and families. Position representations reflect surface semantics, while velocity and curvature reveal stable logical structure and sensitivity to reasoning order.

  • 6.2 RESULTS ANALYSIS: Logical similarity is low at position level but becomes dominant in velocity and curvature, while topic and language show low velocity similarity.This pattern indicates that logical structure transcends surface carriers and may occupy a distinct higher-order geometric organization.
  • 6.2 RESULTS ANALYSIS: Randomly permuting logical steps preserves high position similarity but degrades velocity and curvature similarity, indicating that reasoning-sequence order is crucial to flow structure.Surface semantic effects remain strong after shuffling, whereas higher-order flow measures depend on the original ordering of reasoning steps.
  • 6.2 RESULTS ANALYSIS: Similarity patterns remain stable as Qwen models scale from 0.6B to 4B and across Qwen and LLaMA families, suggesting a model- and training-recipe-independent property.The reported stability concerns the similarity measures across both model size and family axes.
  • 6.2 RESULTS ANALYSIS: At position level, embeddings cluster by topic and language; first-order flows align for shared logical skeletons, while second-order curvature amplifies their separation.The heatmap shows this pattern for Qwen3 0.6B across logic templates instantiated with different semantic carriers.
  • 6.2 RESULTS ANALYSIS: The results provide evidence that LLMs internalize latent logical structure beyond surface form, challenging a purely surface-based account of next-token-trained language models.The paper attributes this evidence to logical structure appearing in higher-order representation-space geometry.

7 DISCUSSION

The discussion frames the framework as a post-hoc account of understanding rather than generation, while identifying practical uses and unresolved representation-level limitations. It contrasts smooth reasoning flows with graph-based random-walk views and highlights directions for control, analysis, retrieval, and architecture.

  • 7 DISCUSSION: The framework studies natural language understanding only, so its geometric measures are not related to generation-specific properties such as task output accuracy.Explaining training origins, learnability, training dynamics, and generation remains outside the paper’s scope.
  • 7 DISCUSSION: Smooth, directed reasoning flows provide a different account from graph models that treat chain-of-thought as random walks between discrete nodes.The paper argues that graph views fit isolated-embedding noise but do not capture the observed cumulative-context dynamics.
  • 7 DISCUSSION: Learned representations encode factors beyond logic, including semantic objects, discourse tone, language identity, and higher-level cognitive signals, whose interactions remain difficult to disentangle.The paper identifies systematic isolation and characterization of these components as a major future challenge.
  • 7 DISCUSSION: Trajectory-level control could support steering, alignment, and safety, while geometric analysis could examine reasoning efficiency, stability, and failure modes.The discussion also proposes flow-aware retrieval and representation methods for RAG, reranking, and search.

8 CONCLUSION

The conclusion positions the work as a differential-geometric framework for dynamic reasoning, distinct from static concept geometry, formal-logic expressivity analyses, and training-dynamics studies. Its central contribution is modeling logic as a constraint on reasoning-flow motion.

  • 8 CONCLUSION: The paper models LLM reasoning as smooth flows in representation space, with logic acting as a controller of local velocities.A controlled dataset separates logical skeletons from semantic carriers so velocity and curvature can expose their organizing role.
  • 8 CONCLUSION: Formal logic serves here as a validation tool for representation-space geometry, not as the paper’s end task or primary object of study.This separates the contribution from prior work characterizing transformer expressivity or compiling formal logics into architectures.
  • 8 CONCLUSION: Unlike static concept-space studies, this work analyzes how representations evolve step by step during reasoning using velocity and curvature rather than static geometric relations.The comparison distinguishes the paper’s kinematic geometry from prior structural geometries based on static concepts and relations.
  • 8 CONCLUSION: The framework targets a post-hoc, model-agnostic law of reasoning, whereas related work studies generation correlations or how geometric patterns arise during training.The paper therefore treats the trained model as a fixed dynamical system rather than analyzing learning dynamics.

B ADDITIONAL EXPERIMENTS

Additional experiments test whether the geometric framework generalizes across model families and sizes while formalizing the spaces and maps used to represent reasoning flows.

  • More Similarity Heatmap: Findings generalize across Qwen3 1.7B, Qwen3 4B, and LLaMA3 8B under the same experimental settings.The corresponding results are shown in Figures 3, 4, and 5.
  • Spaces and Maps: The roadmap aligns input sequences, semantic curves, representation curves, formal logic, and representation-based logic through a commuting structure.Canonical alignment identifies semantic and representation trajectories one-to-one under the stated assumptions.
  • Spaces and Maps: Representation-based logical space encodes reasoning increments Δy_t and evaluates geometric descriptors such as Menger curvature.This non-symbolic space serves as the model’s internal analogue of logic.
  • Spaces and Maps: The framework distinguishes formal reasoning in concept space from local representation differences extracted from embedding trajectories.The semantic operator maps curves to formally valid natural-deduction steps, while the representation operator extracts local increments.
  • Geometric Foundations: Menger curvature couples angular change with distance variation to quantify the geometric intensity of reasoning transitions.The framework uses it as a geometry-aware proxy for reasoning-step strength.

D.1 CONTINUITY OF REPRESENTATION TRAJECTORIES

The appendix constructs smooth representation trajectories from relaxed prefix masks and explains how Menger curvature measures turns in discrete embedding paths.

  • Continuity of Representation Trajectories: The encoder representation combines token embeddings, positional information, contextual transformations, and mask-aware processing into sentence-level hidden states.The hidden state is the outcome of the full prefix encoding process rather than a simple sum of embeddings.
  • Continuity of Representation Trajectories: A smoothstep relaxed prefix mask and a C1 encoder yield a C1 embedding trajectory whose sentence embeddings are recovered at sentence boundaries.The construction uses a continuous progress parameter while matching the discrete prefix states at boundary points.
  • Continuity of Representation Trajectories: The construction justifies treating stepwise sentence representations as samples from a smooth differentiable curve and defining flow velocity.The authors note that alternative continuous realizations are also possible.
  • Menger Curvature: Menger curvature is the reciprocal of the circumcircle radius through three points, providing a computable curvature descriptor for representation trajectories.For three consecutive states, the measure is computed from their triangle geometry and is zero for collinear points.
  • Menger Curvature: Unlike cosine similarity, Menger curvature distinguishes triples with the same angle but different length scales because it incorporates both angular and distance information.This enables separation of different curvature regimes that cosine similarity treats identically.

E DATA GENERATION

The data-generation pipeline creates abstract logical scaffolds and rewrites them into domain- and language-specific reasoning sequences while preserving their structure.

  • Data Generation: The pipeline uses GPT-5 in two stages: abstract logical-template construction followed by domain- and language-specific rewriting.The prompts separately specify formal scaffold generation and natural-language instantiation.
  • Data Generation: The abstract generator produces exactly N internally coherent symbolic steps using propositional or first-order logic notation.Optional line-end justifications can reference earlier step indices.
  • Data Generation: Generated reasoning sequences must preserve the abstract scaffold’s step count, ordering, and logical dependencies.The rewriting instructions prohibit merging, splitting, adding, or removing steps.
  • Data Generation: Multilingual generation creates one section per requested language with aligned step meanings and identical indexed structure.Each language section uses the same bracketed indices from [1] through [N].

E.2 DATA EXAMPLES

The data examples instantiate a shared nine-step logical scaffold in weather and finance domains, with parallel English and German formulations.

  • Data Examples: Table 2 presents one nine-step logical scaffold instantiated across weather and finance topics in English and German.The examples demonstrate carrier variation while retaining the same abstract reasoning structure.
  • Weather Example: Later weather steps derive heavy rain, falling airport temperatures, and their conjunction from earlier indexed premises.The English and German formulations preserve the same dependencies and conclusions.
  • Data Examples: The paired examples show how the same logical dependencies can be expressed with different semantic carriers and languages.Weather and finance statements differ in content while preserving indexed inference relationships.
  • Finance Example: The finance instantiation derives approval of a new term loan, acceptability of Bond A as repo collateral, and their conjunction.These conclusions use the corresponding earlier scaffold steps.
Loading 2510.09782v2…