PaperlayerView exact revision on arXiv

Source-linked AI summary

REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model

Bo Li, Guanzhi Deng, Ronghao Chen, Junrong Yue, Shuo Zhang, Qinghua Zhao, Linqi Song, Lijie Wen

arXiv:2509.22518v1cs.AIcs.LG

TL;DR

REMA addresses the lack of a general geometric framework for locating diverse reasoning failures. It compares erroneous representations with a manifold approximated from correct reasoning states, and experiments report low-dimensional structure, significant deviations, and layer-specific failure origins.

  • Problem

    A general framework for localizing diverse, naturally occurring reasoning failures from internal representation geometry is lacking.

    Source: §1 · p. 2

  • Method

    REMA treats correct reasoning representations as an approximation of a Reasoning Manifold, measures error representations’ k-nearest-neighbor deviation, and tracks it across layers against correct-state fluctuations.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · pp. 4–5

  • Results

    Experiments find low-dimensional internal states for correct and incorrect reasoning, significant error deviations from the correct manifold, and model- and task-dependent failure-origin layers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments provide empirical evidence for reasoning manifolds and show that deviations can indicate errors and help localize potential failure points.

    Source: §5 · p. 9

    The true Reasoning Manifold is unknown, so REMA uses correct-reasoning representations as its empirical approximation.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Understanding how Large Language Models (LLMs) perform complex reasoning and their failure mechanisms is a challenge in interpretability research. To provide a measurable geometric analysis perspective, we define the concept of the Reasoning Manifold, a latent low-dimensional geometric structure formed by the internal representations corresponding to all correctly reasoned generations. This structure can be conceptualized as the embodiment of the effective thinking paths that the model has learned to successfully solve a given task. Based on this concept, we build REMA, a framework that explains the origins of failures by quantitatively comparing the spatial relationships of internal model representations corresponding to both erroneous and correct reasoning samples. Specifically, REMA first quantifies the geometric deviation of each erroneous representation by calculating its k-nearest neighbors distance to the approximated manifold formed by correct representations, thereby providing a unified failure signal. It then localizes the divergence points where these deviations first become significant by tracking this deviation metric across the model's layers and comparing it against a baseline of internal fluctuations from correct representations, thus identifying where the reasoning chain begins to go off-track. Our extensive experiments on diverse language and multimodal models and tasks demonstrate the low-dimensional nature of the reasoning manifold and the high separability between erroneous and correct reasoning representations. The results also validate the effectiveness of the REMA framework in analyzing the origins of reasoning failures. This research connects abstract reasoning failures to measurable geometric deviations in representations, providing new avenues for in-depth understanding and diagnosis of the internal computational processes of black-box models.

1 INTRODUCTION

REMA addresses the lack of a general geometric framework for locating diverse, naturally occurring reasoning failures. It models correct reasoning representations as a Reasoning Manifold and analyzes errors as deviations from it.

1 INTRODUCTION

  • REMA targets a gap left by analyses that rely on error-specific signals or controlled input contrasts: a general framework for localizing diverse, naturally occurring failures.The introduction situates this gap alongside work on activation structure, mechanistic interpretability, and failure analysis.Source: §1 · pp. 1–2
  • REMA defines the Reasoning Manifold as the structured, relatively low-dimensional space of correct reasoning paths and treats errors as deviations from it.It measures each error’s k-nearest-neighbor distance to the approximated manifold, then tracks that distance across layers against normal fluctuations among correct representations.Source: §1 · p. 2
  • The framework locates a failure’s divergence point by finding the layer where its deviation first exceeds the baseline of fluctuations among correct representations.This marks where the reasoning chain begins to go off-track.Source: §1 · p. 2
  • Experiments across language and multimodal models find low-dimensional correct and erroneous states, significant error deviations, and model- and task-dependent failure origins.These findings support REMA’s use for diagnosing reasoning failures and their originating stages.Source: §1 · p. 2

2 RELATED WORK

Related work studies activation geometry, mechanisms, and failure modes, but the paper distinguishes REMA by analyzing diverse reasoning errors through their representations’ distance from correct reasoning states.

2 RELATED WORK

  • The Manifold Hypothesis describes meaningful data variations as lying on low-dimensional structures within high-dimensional spaces; prior work applies geometric and information-theoretic analyses to LLM activations.Examples include studies of intrinsic dimension, representation entropy, and decomposition of hidden states (Bengio et al., 2013; Facco et al., 2017; Pope et al., 2021; Tan et al., 2024; Song & Zhong, 2023).Source: §2 · p. 2
  • Mechanistic interpretability uses attention analysis, probing, feature decomposition, sparse autoencoders, and causal methods to study representations and internal computation.The passage cites representative work across these approaches, including Vaswani et al. (2023), Tenney et al. (2019), Cunningham et al. (2023), and Meng et al. (2023).Source: §2 · p. 3
  • Failure studies examine hallucinations, hierarchical stages, and geometric trajectories, while REMA defines errors as deviation from correct representations rather than input shifts or trajectory curvature.The paper says existing methods often use probes for specific error types or controlled input pairs; REMA focuses on reasoning outcomes across logical, factual, or other errors (Ko et al., 2024; Manson, 2025).Source: §2 · p. 3

3 METHODOLOGY: THE REMA FRAMEWORK

REMA analyzes layer-wise representations to quantify how errors differ from correct reasoning states and to locate where those differences first become significant.

3.1 PROBLEM FORMULATION AND REPRESENTATION EXTRACTION

  • Samples are partitioned by exact-match comparison of model predictions with ground truth, and each layer’s last-token hidden states are mean-pooled across generation steps.The analysis focuses on hidden states in the model’s core language module.Source: §3.1 · pp. 3–4

3.2 REASONING MANIFOLD

  • The Reasoning Manifold is hypothesized to be a low-dimensional structure primarily constituted by correct-sample representations, which approximate the unknown true manifold.The framework also characterizes the approximation using intrinsic dimension and mutual information.Source: §3.2 · p. 4

3.3 DEVIATION AND SEPARABILITY ANALYSIS

  • REMA measures each error’s k′-nearest-neighbor distance to correct representations, compares error and correct internal distances, and tests their separability with a classifier.Deviation significance is assessed with a two-sample Welch t-test; separability uses an RBF-kernel support vector classifier and five-fold cross-validation.Source: §3.3 · p. 4

3.4 LOCALIZATION OF DIVERGENCE POINT

  • A divergence point is the earliest layer where an error’s deviation exceeds the correct-sample mean internal distance plus a multiple of its standard deviation.The threshold uses a preset factor α (2 in this study); the distribution of these layer indices indicates stages where failures tend to originate.Source: §3.4 · pp. 4–5

4 EXPERIMENTS

REMA is evaluated across text and multimodal reasoning benchmarks, with experiments characterizing representation geometry, measuring errors’ deviation from correct reasoning, and locating divergence across layers. The findings support low-dimensional reasoning representations, greater deviation for errors, and model- and task-dependent failure origins.

4.1 EXPERIMENTAL SETUP

  • Experiments span seven text and multimodal reasoning datasets and Qwen3, Qwen2.5-VL, Llama3.2, and LLaVA-OneVision models ranging from 3B to 90B parameters.The evaluation uses official validation or test splits.Source: §4.1 · pp. 5–6

4.2 CHARACTERIZATION ANALYSIS

  • Correct and erroneous reasoning representations have intrinsic dimensions far below the hidden-space dimensions, while early-layer mutual information with the correct answer is higher for correct samples.The relative magnitudes of intrinsic dimension and mutual information vary across model families and tasks.Source: §4.2 · pp. 6–7

4.3 DEVIATION ANALYSIS

  • In final-layer SNLI-VE UMAP visualizations, correct samples form concentrated clusters while errors appear at cluster peripheries or in separate clusters.Figure 2 visualizes the tendency for the two sample types to separate.Source: Figure 2 · p. 7, §4.3 · p. 7
  • Error-representation deviation distances exceed correct-sample internal distances across tasks, and relative deviation is negatively correlated with zero-shot accuracy (ρ = 0.598, p < 0.01).The authors interpret the correlation as indicating greater deviation on more difficult tasks.Source: §4.3 · p. 7
  • Layer-wise SVM separability is generally above chance and typically rises with depth, reaching a high, stable level in mid-to-late layers.Some model-task combinations approach perfect separation.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 LOCALIZING REASONING DIVERGENCE

  • Divergence points vary across models on the same task and across tasks for the same model, rather than concentrating at one fixed processing stage.Figure 4 bins the earliest significant deviations in intervals of eight decoder layers.Source: Figure 4 · p. 8, §4.4 · pp. 8–9

4.5 ABLATION STUDY

  • Mean-pooling and attention-weighted pooling achieve the highest t-statistics, while last-token pooling is slightly inferior and max-pooling is least stable.Across tested model, task, and pooling combinations, error deviation remains significantly greater than correct-representation internal distance; mean-pooling is the authors’ efficient choice.Source: §4.5 · p. 9, Table 2 · p. 9

5 CONCLUSION

The conclusion presents REMA as evidence that reasoning errors correspond to deviations from a low-dimensional Reasoning Manifold, with potential future interventions to improve models. The background describes intrinsic dimension and mutual information as tools for analyzing representation structure and task relevance.

5 CONCLUSION

  • REMA’s experiments support the existence of Reasoning Manifolds and show that deviations indicate reasoning errors and can localize potential failure points.The conclusion proposes future work on regularization or editing to encourage desirable manifold geometry or pull deviating representations back onto the manifold.Source: §5 · p. 9

A BACKGROUND

  • Intrinsic dimension estimates a dataset’s local effective degrees of freedom; values below ambient dimension provide geometric evidence for low-dimensional structure.The paper estimates intrinsic dimension from first- and second-nearest-neighbor distance ratios using a linear fit to the empirical cumulative distribution.Source: A BACKGROUND · p. 13
  • Mutual information measures representation–objective dependence, and the KSG estimator approximates it through k-nearest-neighbor counts in joint and marginal spaces.The estimator is used because the true underlying distributions are generally unknown and direct MI computation is intractable.Source: A BACKGROUND · pp. 13–14

B DATASETS

The evaluation spans seven text and multimodal reasoning datasets, selected to cover tasks ranging from general visual question answering to multi-step mathematical deduction.

Text reasoning

  • The text benchmarks are GSM8K (1,000 test problems), MATH (500 problems), and GPQA (448 expert-written questions across biology, physics, and chemistry).GSM8K contains grade-school math word problems, MATH challenging competition problems, and GPQA multiple-choice questions.Source: B DATASETS · p. 14

Multimodal reasoning

  • The multimodal benchmarks include VQAv2, using 2,000 image-question pairs, and SNLI-VE, using 2,000 entailment or contradiction instances.SNLI-VE neutral samples were excluded; VQAv2 samples came from the official validation split, with each question corresponding to a unique image.Source: B DATASETS · p. 14
  • MathVista tests visual mathematical reasoning across puzzle figures, functional plots, and academic-paper figures; the study uses its 1,000-pair test mini split.MathVista comprises IQTest, FunctionQA, and PaperQA.Source: B DATASETS · p. 14
  • AI2D evaluates diagram and chart understanding; the study uses approximately 3.09K question-diagram pairs from a dataset of about 5,000 diagrams.The diagrams cover grade-school science and include annotated segments and relationships.Source: B DATASETS · p. 14

C MODELS

The study evaluates open-source language and multimodal models from several families and parameter sizes, using zero-shot inference to establish benchmark accuracy baselines.

C MODELS

  • The selected open-source models span multiple families and sizes from 3B to 90B parameters, and all models were evaluated with zero-shot inference.Benchmark accuracy was used as a baseline and to partition samples for subsequent interpretability analyses.Source: C MODELS · p. 15
  • The evaluated families include Qwen3 (4B, 30B) and Qwen2.5-VL (3B, 72B), covering language and vision-language models.The passage describes Qwen3 as a language-model family and Qwen2.5-VL as a vision-language model.Source: C MODELS · p. 15
  • The model set also includes Llama3.2 (3B, 11B, 90B) and LLaVA-OneVision (7B).Llama3.2-Vision models are described as optimized for visual recognition, image reasoning, captioning, and image questions; LLaVA-OneVision-7B Chat was included.Source: C MODELS · p. 15

D IMPLEMENTATION DETAILS

Implementation uses specified compute and analysis settings for representation metrics, deviation, separability, and visualization. Table 4 reports deviation comparisons for larger and Mixture-of-Experts models.

D IMPLEMENTATION DETAILS

  • Experiments ran on NVIDIA A100 (40G) GPUs with CentOS Linux 7.5, Python 3.9.12, PyTorch 2.1.0, and Transformers 4.50.0.Source: D IMPLEMENTATION DETAILS · p. 15
  • The analysis uses mean-pooled decoder-layer states, TwoNN for intrinsic dimension, Sentence-BERT answer embeddings, KSG with k=5, and Euclidean deviation distances with k′=5.Separability uses an RBF-kernel SVC with five-fold cross-validation; divergence localization uses α=2.0 and k′=5.Source: D IMPLEMENTATION DETAILS · p. 15
  • UMAP visualizations use 5 neighbors, min_dist 0.1, spread 1.0, and cosine distance.Source: D IMPLEMENTATION DETAILS · p. 15
  • Table 4 covers average deviation distances on larger and Mixture-of-Experts models and reports robust geometric deviation of error states.Source: Table 4 · p. 16

E LIMITATIONS AND DISCUSSION

The study’s interpretation is bounded by how it approximates the reasoning manifold and labels reasoning samples. The authors identify sample coverage and exact-match partitioning as limitations, and suggest finer alternatives.

E LIMITATIONS AND DISCUSSION

  • The correct-sample point cloud only approximates the manifold, and its fidelity depends on sample diversity and density.The authors suggest more precise manifold-learning models as future work.Source: E LIMITATIONS AND DISCUSSION · p. 16
  • Exact-match answer labels create a binary correct/error split that can misclassify nearly correct reasoning in either direction.The authors propose exploring finer-grained partitioning.Source: E LIMITATIONS AND DISCUSSION · p. 16
  • Confidence-based or soft-scoring-based partitioning methods are suggested as alternatives to the current split.These approaches are proposed for future work.Source: E LIMITATIONS AND DISCUSSION · p. 16

F.1 SCALING ANALYSIS ON LARGER LLMS

Validation on larger and differently architected models supports the observed geometric deviation of erroneous reasoning from correct representations. t-SNE projections show this separation across models and reveal its increasing prominence across layers.

F.1 SCALING ANALYSIS ON LARGER LLMS

  • Across all three larger models, error-sample average deviation distances significantly exceed correct-sample internal distances, with t-statistics from 8.64 to 29.96.For Llama3.2, Derror = 2.93 vs. Dcorrect = 1.79; for Qwen2.5-VL, Derror = 25.36 vs. Dcorrect = 14.18.Source: §F.1 · p. 16
  • For Llama3.2, scaling from 11B to 90B raises Derror from 1.45 to 2.93 and Dcorrect from 0.91 to 1.79, while Derror/Dcorrect remains similar at approximately 1.59 and 1.64.For Qwen2.5-VL, scaling from 3B to 72B raises Derror from 5.50 to 25.36 and Dcorrect from 3.70 to 14.18; the ratios are approximately 1.49 and 1.79.Source: §F.1 · p. 16
  • On GPQA, Qwen3 (30B-A3B) has smaller absolute distances than Qwen3 (4B): Derror = 7.78 vs. 38.40 and Dcorrect = 6.46 vs. 30.22.The authors suggest the MoE model may learn a more geometrically compact representation space, while noting potentially better performance in Table 3.Source: §F.1 · p. 17
  • The deviation phenomenon generalizes beyond the main-text models to larger models and different architectural types.The authors describe this as strengthening REMA’s credibility and applicability.Source: §F.1 · p. 17

F.2 T-SNE VISUALIZATION OF DEVIATIONS

  • Final-layer t-SNE shows clear separation between correct and error representations across Llama3.2, LLaVA-OneVision, and Qwen2.5-VL.Correct samples cluster tightly; error samples are more diffuse and often peripheral or in less-structured clusters.Source: §F.2 · pp. 17–18, Figure 5 · p. 17
  • In Llama3.2’s early layers, correct and error representations are heavily intermingled, with clusters emerging and errors peeling away in mid-layers.At Layer 0, the model has not yet effectively differentiated successful and failed trajectories.Source: Figure 6 · p. 17, §F.2 · p. 18
  • By Layer 39, correct samples form tight clusters while error samples occupy their outskirts or separate regions, consistent with UMAP and increasing layer-wise separability.The paper describes this progression as divergence emerging and becoming amplified in deeper representations.Source: Figure 6 · p. 17, §F.2 · p. 18

F.3.1 SENSITIVITY ANALYSIS OF DEVIATION DISTANCE TO NUMBER OF NEIGHBORS

The sensitivity analyses find that the deviation gap persists across tested neighbor counts, while threshold choice affects divergence-point counts and can alter the layer interval identified as the peak.

F.3.1 SENSITIVITY ANALYSIS OF DEVIATION DISTANCE TO NUMBER OF NEIGHBORS

  • Across tested k′ values and layers, error deviation distances remain significantly greater than correct internal distances, although absolute distances rise with larger k′.The analysis reports that the gap remains substantial at Layer 20; k′ = 5 was chosen for the main experiments to balance local structure and stability against outliers.Source: §F.3.1 · p. 18, Table 5 · p. 19

F.3.2 SENSITIVITY OF DIVERGENCE POINT LOCALIZATION TO THRESHOLD FACTOR

  • Using α = 2.0 generally identifies fewer divergence points than α = 1.0 because it requires greater deviation from correct-sample internal distance.Counts are aggregated into 8-layer intervals for Qwen2.5-VL (3B) on AI2D and MathVista.Source: §F.3.2 · pp. 18–19, Table 6 · p. 20
  • For AI2D, Interval 0-7 remains the most frequent divergence-origin stage under both thresholds, though its count falls with the stricter threshold.The authors suggest many failures begin with relatively small early deviations, while some still originate from more pronounced early deviations.Source: §F.3.2 · p. 19
  • For MathVista, raising α from 1.0 to 2.0 shifts the primary peak from Interval 0-7 (100 to 35) to Interval 24-31 (51 to 60).The authors interpret the shift as distinguishing minor early deviations from more severe deviations identified later.Source: §F.3.2 · p. 20, Table 6 · p. 20
  • The broader identification of early and mid-to-late processing stages as susceptible to divergence remains robust despite threshold-dependent counts.The authors use α = 2.0 in the main experiments to focus on more pronounced deviations.Source: §F.3.2 · p. 20

F.3.3 ROBUSTNESS OF DEVIATION DISTANCE TO SAMPLE SIZE PERTURBATIONS

In the Qwen3 GPQA subsampling analysis, correct representations have greater internal dispersion than error representations across the tested sample sizes. The distinction remains stable, though its direction differs from the simple intuition that errors should be more dispersed.

F.3.3 ROBUSTNESS OF DEVIATION DISTANCE TO SAMPLE SIZE PERTURBATIONS

  • Across all layers and tested sample ratios, correct-sample internal distance exceeds error-sample internal distance for Qwen3 on GPQA.The experiment randomly subsampled 50%, 70%, and 100% of available correct and error samples at each layer.Source: §F.3.3 · p. 20, Table 7 · p. 21
  • At Layer 35 with 100% of the data, Dcorrect is approximately 176.86 while Derror is 139.63, and the relationship also holds with 50% of the data.The table reports layer-wise average distances for the three subsampling ratios.Source: §F.3.3 · p. 20, Table 7 · p. 21
  • The authors suggest correct GPQA reasoning may explore a more diverse representational space, while errors may reflect compact representations from premature, high-confidence convergence.They present this as a potential failure mode for this complex, knowledge-intensive model-task combination.Source: §F.3.3 · p. 20
  • The difference in geometric properties remains significant across sample sizes, although which set is more dispersed depends on the task and model.The authors state that this stability indicates the finding is not an artifact of using the full dataset for this task.Source: §F.3.3 · p. 20
Loading 2509.22518v1…