PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Reasoning Chains Reduce Intrinsic Dimensionality

Archiki Prasad, Mandar Joshi, Kenton Lee, Mohit Bansal, Peter Shaw

arXiv:2602.09276v2cs.CLcs.AIcs.LG

TL;DR

The paper addresses the limited quantitative understanding of why reasoning strategies generalize differently. It measures intrinsic dimensionality by varying low-dimensional LoRA adaptations while holding the model fixed and the reasoning strategy varied. Across GSM8K evaluations, lower intrinsic dimensionality is strongly associated with better generalization, although the measure may reflect artifacts of particular models or datasets.

  • Problem

    Existing explanations of reasoning effectiveness remain qualitative, while a consistent quantitative link between reasoning strategy and generalization is not established.

  • Method

    The study fixes the model, varies reasoning-strategy supervision, and estimates intrinsic dimensionality as the minimum LoRA-trainable parameter count reaching a common accuracy threshold.

  • Results

    Intrinsic dimensionality shows a strong inverse correlation with generalization across CoT variants on GSM8K for Gemma-3 1B and 4B, in-distribution and out-of-distribution.

  • Takeaways & Limitations

    Effective reasoning chains are associated with lower-dimensional, more compressible task learning, and intrinsic dimensionality predicts generalization better than perplexity and length.

  • Takeaways & Limitations

    Intrinsic dimensionality is a statistical measure that may reflect artifacts of specific model families or datasets and should be interpreted with care.

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning and its variants have substantially improved the performance of language models on complex reasoning tasks, yet the precise mechanisms by which different strategies facilitate generalization remain poorly understood. While current explanations often point to increased test-time computation or structural guidance, establishing a consistent, quantifiable link between these factors and generalization remains challenging. In this work, we identify intrinsic dimensionality as a quantitative measure for characterizing the effectiveness of reasoning chains. Intrinsic dimensionality quantifies the minimum number of model dimensions needed to reach a given accuracy threshold on a given task. By keeping the model architecture fixed and varying the task formulation through different reasoning strategies, we demonstrate that effective reasoning strategies consistently reduce the intrinsic dimensionality of the task. Validating this on GSM8K with Gemma-3 1B and 4B, we observe a strong inverse correlation between the intrinsic dimensionality of a reasoning strategy and its generalization performance on both in-distribution and out-of-distribution data. Our findings suggest that effective reasoning chains facilitate learning by better compressing the task using fewer parameters, offering a new quantitative metric for analyzing reasoning processes.

1. Introduction

The paper asks why reasoning strategies differ in effectiveness and proposes intrinsic dimensionality as a quantitative lens for their generalization. Across GSM8K experiments, lower intrinsic dimensionality is associated with better generalization across model sizes and distribution shifts.

  • Motivation: CoT variants improve language-model performance by generating textual rationales before final answers.The strategies differ in length, structure, and generation method, including code-based solutions, decomposition, and verification loops.
  • Research gap: Existing explanations emphasizing reasoning structure or relevance remain qualitative and difficult to use predictively.The paper argues that a quantifiable measure could support reasoning-data annotation, model alignment, and regularizer design.
  • Proposed perspective: Intrinsic dimensionality is treated as a task-and-model measure of the effective parameter capacity needed to fit an objective.The study fixes the model while varying supervision through different reasoning strategies, keeping the underlying capability constant.
  • Main result: A strong inverse correlation links intrinsic dimensionality with generalization across CoT variants on GSM8K.The relationship holds for Gemma-3 1B and 4B models on both in-distribution and out-of-distribution evaluations.
  • Implications: Intrinsic dimensionality predicts generalization more strongly than trajectory length and likelihood-based alternatives.The measure is presented as a quantitative predictor with potential guidance for data annotation, model alignment, and training optimization.

2. Intrinsic Dimensionality of Reasoning

The method defines intrinsic dimensionality through low-dimensional LoRA adaptation and estimates it by sweeping trainable parameter counts. A common performance threshold enables fair comparison of reasoning strategies while preserving their relative ordering.

  • 2.1. Background on Intrinsic Dimension: Intrinsic dimensionality identifies the minimum dimensions needed for a task to reach a target performance under a fixed model.The parameterization separates pretrained parameters from a lower-dimensional trainable subspace.
  • 2.2. Lower-Dimension Projection for LLMs: LoRA constrains updates to low-rank subspaces while keeping pretrained weights frozen and optimizing learned low-rank matrices.It can target attention modules, MLP layers, or all transformer layers.
  • 2.2. Lower-Dimension Projection for LLMs: LoRA provides a structured, architecturally informed projection whose trainable capacity is controlled by rank and the number of adapted weight matrices.This connects LoRA parameterization to intrinsic-dimensionality measurement for LLM fine-tuning.
  • 2.3. Measuring Intrinsic Dimension of Reasoning: The intrinsic dimension d_int is the minimum trainable parameter count d for which training accuracy A(d) reaches threshold τ.The estimate comes from sweeping LoRA ranks and target matrices, with parameter counts distributed uniformly on a log scale.
  • 2.3. Measuring Intrinsic Dimension of Reasoning: A common threshold τ is used across reasoning strategies so their relative intrinsic-dimensionality ordering can be compared fairly.The threshold may be based on a fixed validation accuracy or full-capacity training accuracy after one epoch.

3. Experimental Setup

The experiments compare diverse reasoning strategies using Gemma-3 models trained on GSM8K-derived rationale datasets. Generalization is evaluated in-domain and on stress tests, while intrinsic dimensionality is compared with length and likelihood-based metrics.

  • Datasets: The evaluation uses GSM8K training data and measures performance on GSM8K Test plus GSM-Symbolic, GSM-IC, and GSM-Hard stress sets.The stress tests probe robustness to symbolic perturbations, irrelevant information, and harder problem variants.
  • Reasoning Strategies: The study varies reasoning length, structure, and generation method across No CoT, filler-token baselines, natural-language CoT variants, and other strategies.The strategy set is designed to test whether intrinsic dimensionality explains effectiveness beyond trajectory length.
  • Reasoning Strategies: Each strategy receives its own GSM8K-based training dataset, and a full-capacity model is fine-tuned to generate the strategy’s rationale followed by the final answer.This creates matched training conditions while changing the reasoning supervision.
  • Baseline Metrics: The baselines include trajectory length, token perplexity, and sequence KL divergence as predictors that avoid test-time evaluation.Intrinsic dimensionality is generally computed using 90% of maximum first-epoch training accuracy as the threshold.
  • Setup: Gemma-3 1B and 4B models are fine-tuned with consistent reasoning-strategy comparisons but different step counts and learning rates.The 1B model uses 8,000 steps at 10^-3, while the 4B model uses 6,000 steps at 10^-4.
  • Setup: The main table reports ID accuracy, OOD geometric-mean accuracy, overall geometric-mean accuracy, and Spearman correlations between metrics and overall accuracy.For downward metrics, positive correlations indicate that lower values predict higher accuracy; † marks p < 0.01.

4. Main Results and Analysis

Across Gemma-3 models and reasoning strategies, intrinsic dimensionality most strongly predicts generalization, with effective chains requiring fewer parameters and maintaining this relationship across thresholds, domains, and data quality conditions.

  • Intrinsic Dimension with Gemma-3 4B: Intrinsic dimensionality correlates 0.93 with average accuracy for Gemma-3 4B, outperforming length and KL divergence as predictors of reasoning effectiveness.Length correlates 0.31 and KL divergence −0.17, while token perplexity reaches 0.82.
  • Intrinsic Dimension with Gemma-3 1B: Intrinsic dimensionality correlates 0.75 with generalization for Gemma-3 1B, exceeding token perplexity at 0.63 and remaining stronger than length or KL divergence.The relationship persists despite the 1B model’s substantially lower absolute performance.
  • Robustness to Threshold Selection: Intrinsic dimensionality remains strongly correlated with performance, ranging from 0.72 to 0.94 across thresholds and models.The analysis uses epoch-1 training thresholds of 70%, 80%, and 90%, or 90% validation accuracy.
  • Additional Results and Analysis: For Gemma-3 4B, Executed PoT crosses the 63.0% threshold at 1.49M parameters and achieves 43.40% OOD accuracy, outperforming the compared strategies.Its threshold crossing occurs earlier than Short CoT, Gemma 27B CoT, Very Short CoT, and No CoT.
  • Additional Results and Analysis: Effective strategies require lower intrinsic dimensionality in the 4B model, whereas ineffective or noisy strategies can require much more capacity than in the 1B model.Very Short CoT requires over 500M parameters on 4B versus 31M on 1B; the paper associates this reversal with less compressible patterns.
  • Additional Results and Analysis: Mixing 50% incorrect reasoning chains increases intrinsic dimensionality and reduces accuracy, with Short CoT rising over 500× and losing 12.0 points.Gemma 27B CoT rises approximately 125× and loses 12.8 points under the same mixed-data condition.
  • Additional Results and Analysis: Across Reasoning Gym categories, intrinsic dimensionality remains the most reliable predictor, while KL divergence and length show weaker and less consistent correlations.Executed PoT is strongest on algorithmic tasks, whereas No CoT is strongest on cognitive tasks.

5. Related Work

Prior work studies how reasoning structure affects language-model performance, but existing explanations are often difficult to quantify. Intrinsic dimensionality instead measures the task-relevant degrees of freedom needed for learning.

  • CoT effectiveness is task-dependent and is driven more by structural coherence than local numerical precision.
  • Intrinsic dimensionality measures the minimum degrees of freedom required to optimize a network for a specific objective.
  • The paper estimates learning-objective complexity involving both the model and reasoning task, rather than training-data complexity alone.
  • Effective reasoning strategies are hypothesized to simplify the rule connecting inputs to answers, allowing task fitting in a lower-dimensional subspace despite longer outputs.

6. Discussion and Conclusion

The discussion links reasoning-chain effectiveness to reduced intrinsic dimensionality and better generalization, while identifying computational cost and broader-setting validation as open constraints.

  • Generalization correlates with how much reasoning chains reduce task intrinsic dimensionality.
  • Intrinsic dimensionality correlates better with generalization than perplexity and trajectory length, potentially because it captures coherence across complete trajectory sets.
  • The study focuses on fine-tuning from reasoning trajectories, leaving other post-training settings for future work.
  • The estimate requires fine-tuning adapters of various sizes, making direct optimization computationally expensive.

Impact Statement

The paper aims to improve understanding of LLM reasoning through intrinsic dimensionality while providing analytical tools for characterizing and evaluating reasoning reliability.

  • The work is primarily analytical and interpretive and is intended to advance understanding of LLM reasoning through intrinsic dimensionality.
  • Intrinsic dimensionality may reflect artifacts of particular model families or datasets and should therefore be interpreted carefully.
  • The authors present the measure as a complementary tool for characterizing, evaluating, and improving reasoning reliability.

A.1. LoRA Sweeps for Computing Intrinsic Dimensions

The LoRA sweep estimates intrinsic dimensionality by sampling parameter counts across adaptation configurations and selecting configurations that closely match each target count.

  • Intrinsic dimensionality is computed with LoRA configurations whose parameter counts are uniformly distributed on a logarithmic scale.
  • The sweep compares four adaptation groups: attention q v, attention all, mlp all, and all layers.
  • For each target count, the procedure evaluates module groups, estimates and clips rank, computes actual parameters, and minimizes absolute count error.
  • Configurations with actual parameter counts are stored, retaining one configuration per unique count to handle collisions.
  • Training runs plateaued before the selected limits of 8,000 steps for 1B and 6,000 steps for 4B models.

A.2. Intrinsic Dimension Computation for Gemma-3 1B

For Gemma-3 1B, intrinsic dimensionality varies substantially across reasoning strategies and preserves their relative effectiveness ordering across model scales.

  • 1.03M parameters: Executed PoT crosses the 24.3% threshold, while Very Short CoT requires 31.45M parameters.The threshold is lower for the 1B model than for the 4B model because of its reduced reasoning capability.
  • The relative ordering of strategies by intrinsic dimensionality remains consistent between Gemma-3 1B and 4B.This consistency indicates that intrinsic dimensionality captures strategy effectiveness independently of model scale.

B.1. GSM8K

The GSM8K evaluation combines in-distribution and out-of-distribution splits with multiple reasoning-strategy datasets and comparison metrics. The section also illustrates how the evaluated strategies differ in output structure, from direct answers to decomposition, verification, and execution-based reasoning.

  • Evaluation Splits: Six test splits span in-distribution GSM8K and out-of-distribution GSM-based variants, including Symbolic, P1, and P2.The Symbolic, P1, and P2 splits contain 5K, 5K, and 2.5K instances, respectively.
  • Training Splits: Training data are generated by prompting teacher models with different reasoning strategies and retaining correct outputs, yielding 6.2K–7.5K valid instances per strategy.No CoT and No CoT with extra tokens retain the full training set.
  • Reasoning Gym: Reasoning Gym evaluation uses easy ID, hard OOD, and held-out OOD configurations, with held-out tasks unseen during training.The held-out OOD set includes needle haystack, modulo grid, letter counting, and count primes; OOD and Overall scores are geometric means.
  • Comparison Metrics: LongPPL evaluates two configurations, K=64 and K=256, as alternatives to length-normalized token perplexity.LongPPL down-weights short-context prefix tokens and assigns higher weight beyond position K.
  • Reasoning Strategies: The evaluated strategies range from direct answers and filler text to concise equations, brief explanations, distractors, unconstrained reasoning, decomposition, and program execution.All examples use the same base question to illustrate differences in generation structure.

F. Detailed Results across all Test Splits

The detailed-results tables report performance across six test splits for Gemma-3 4B and Gemma-3 1B. They distinguish hard, OOD, and Overall aggregates using geometric means.

  • Table 8 reports detailed Gemma-3 4B performance across test splits, including GSM-Symbolic, P1, P2, and IC.The table caption identifies Symb as GSM-Symbolic and P1/P2 as Symbolic Perturbations.
  • Hard is GSM-Hard, OOD is the geometric mean of five stress tests, and Overall is the geometric mean across all six splits.These aggregate labels apply to the reported detailed performance tables.
  • Table 9 reports detailed Gemma-3 1B performance across the corresponding test splits.Its caption uses the same Symb, P1/P2, and IC abbreviations.
Loading 2602.09276v2…