Source-linked AI summary
Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
Chengshuai Zhao, Zhen Tan, Pingchuan Ma, Dawei Li, Bohan Jiang, Yancheng Wang, Yingzhen Yang, Huan Liu
TL;DR
The paper asks when CoT reasoning genuinely generalizes beyond the conditions in which it was learned. It introduces a data distribution lens and the controllable DataAlchemy environment, finding that CoT is effective near the training distribution but fragile under moderate shifts, with apparent reasoning sometimes behaving like a mirage.
Problem
Existing evaluations do not adequately explain when or why CoT succeeds because they use narrow settings, entangled data, and potentially leaked pretrained models.
Method
The paper models CoT through distribution shifts in task, length, and format, and tests the hypothesis with DataAlchemy using controlled experiments and models trained from scratch.
Results
CoT works effectively on near in-distribution data but becomes fragile under moderate distribution shifts, sometimes producing fluent yet logically inconsistent reasoning.
Takeaways & Limitations
The findings suggest that apparent structured reasoning can arise from inductive biases shaped by in-distribution training data rather than logical inference.
Takeaways & Limitations
The paper acknowledges limitations in its controlled investigation and identifies them as avenues for future research.
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) prompting has been shown to be effective in eliciting structured reasoning (i.e., CoT reasoning) from large language models (LLMs). Regardless of its popularity, recent studies expose its failures in some reasoning tasks, raising fundamental questions about the nature of CoT reasoning. In this work, we propose a data distribution lens to understand when and why CoT reasoning succeeds or fails. We hypothesize that CoT reasoning reflects a structured inductive bias learned from in-distribution data, enabling models to conditionally generate reasoning trajectories that approximate those observed during training. As such, the effectiveness of CoT reasoning is fundamentally governed by the nature and degree of distribution discrepancy between training data and test queries. Guided by this lens, we dissect CoT reasoning via three dimensions: task, length, and format. To test the hypothesis, we introduce DataAlchemy, an abstract and fully controllable environment that trains LLMs from scratch and systematically probes them under various distribution conditions. Through rigorous controlled experiments, we reveal that CoT reasoning is a brittle mirage when it is pushed beyond training distributions, emphasizing the ongoing challenge of achieving genuine and generalizable reasoning.
1. Introduction
The paper frames CoT reasoning as effective near the training distribution but vulnerable to distribution shifts across task, length, and format. It introduces a data distribution lens and DataAlchemy to test this account under controlled conditions.
- CoT prompting elicits structured intermediate reasoning and has shown effectiveness in logical, mathematical, and commonsense tasks.
- Existing studies report failures in planning, puzzle-solving, and complex mathematical reasoning, challenging optimistic views of CoT.
- Current evaluations are limited by narrow settings, entangled real-world data, and leakage from pretrained models.
- The proposed lens attributes CoT effectiveness to structured inductive biases learned from in-distribution data and identifies task, length, and format as shift axes.
- DataAlchemy abstracts NLP tasks into controllable components and supports fine-grained experiments across distribution discrepancies.
- CoT works near the training distribution but becomes fragile under moderate shifts, sometimes producing fluent yet logically inconsistent reasoning.
- Across models varying in size, architecture, and temperature, CoT effectiveness consistently varies with distribution discrepancy.
2. Related Work
Related work establishes both the usefulness and fragility of CoT, while OOD studies suggest that generalization depends on shared structures and training-induced biases. This motivates analyzing CoT through task, length, and format distribution shifts.
- CoT methods include zero-shot prompting, self-consistency, Auto-CoT, Tree-of-Thought, and symbolic extensions for eliciting or organizing reasoning.
- Studies show that irrelevant perturbations, surface patterns, and increasing task complexity can substantially undermine CoT performance.
- Prior OOD research finds that pretrained models struggle to adapt when prompted to learn novel functions, although CoT can partially improve OOD generalization for long-reasoning tasks.
- Other findings attribute generalization to positional algorithmic biases, finer-grained CoT supervision, or shared latent structures across distributions.
3. The Proposed Data Distribution Lens
The data distribution lens models CoT as an inductive bias learned from training data whose effectiveness depends on train–test discrepancy. A theoretical bound motivates examining task, length, and format shifts.
- The lens hypothesizes that models generate reasoning trajectories approximating those observed in in-distribution training data.
- CoT effectiveness is governed by the nature and degree of discrepancy between training and test distributions rather than model architecture or scale.
- The formalization defines training and test distributions over problems and solution sequences, including intermediate reasoning traces.
- Distribution discrepancy is introduced through unseen tasks, varying text or reasoning lengths, and prompt-format variants.
- Theorem 3.1 provides a generalization bound linking expected test risk to empirical training risk and distribution discrepancy under bounded loss.
- The framework identifies task, length, and format as the three critical dimensions of distribution shift.
4. DataAlchemy: A Controllable Environment
DataAlchemy creates a clean, controllable environment by representing NLP tasks with atoms, elements, and transformations. Its compositional transformations support systematic variation of operations and explicit CoT reasoning steps.
- DataAlchemy distills real-world NLP tasks into atoms, elements, and transformations for controlled data curation.
- Atoms are alphabet symbols, while elements are ordered atom sequences whose length can be manipulated to vary the text space.
- Transformations abstract text operations and include ROT and Cyclic Position Shift functions.
- Compositional transformations apply a sequence of operations with configurable type, parameters, order, and length.
- Decomposing intermediate transformation processes provides explicit CoT reasoning steps for probing multi-step reasoning.
- The framework supports training and probing models across task, length, and format distributions, including models trained from scratch and real-world LLMs.
5. Task Generalization
Task generalization tests show that CoT effectiveness declines as transformations become less familiar, with reasoning traces often replicating training patterns rather than reliably solving novel compositions.
- Setup: Task generalization evaluates transformation and element combinations under progressively larger distribution discrepancies.The transformation scenarios are in-distribution, compositional, partially out-of-distribution, and fully out-of-distribution.
- Transformation generalization: CoT exact-match accuracy falls from 100% in ID to 0.01% in CMP and 0% in both POOD and OOD.This pattern accompanies increasing distribution discrepancy across the four transformation scenarios.
- Fine-grained analysis: Models often reproduce training patterns, producing correct reasoning with incorrect answers or incorrect paths with correct answers.Commutative transformations can yield accidental answer correctness despite unfaithful reasoning traces.
- Overall finding: Transformation generalization therefore deteriorates as task distribution discrepancy increases.The figure summarizes the same trend across task-distribution conditions.
- Supervised fine-tuning: A small unseen-data fraction, λ = 1.5e−4, can quickly improve generalization to unseen transformations.Generalization is easier when training and testing distributions are more similar.
6. Length Generalization
Length generalization is strong near the training distribution but fragile under shifts in text length or reasoning-step composition, while limited unseen data can improve adaptation.
- Supervised fine-tuning: A small portion of unseen data helps CoT reasoning quickly generalize.The result is summarized by the supervised-fine-tuning experiment.
- Text length generalization: Text-length experiments train on l = 4 and evaluate across lengths from l = 2 to l = 6.This isolates length distribution discrepancy while fixing other factors.
- Text length generalization: CoT performs excellently at l = 4 but degrades as text-length discrepancy increases, even failing under mild shifts such as l = 3 or l = 5.Models attempt to preserve the training-length reasoning pattern by adding or removing tokens.
- Reasoning step generalization: Increasing the proportion of k = 1 training examples improves one-step reasoning while compromising two-step reasoning.Performance changes with the composition of training data across reasoning-step lengths.
- Reasoning step generalization: When training covers only one reasoning-step length, CoT performs well on that length but fails to generalize to unseen cases.This demonstrates fragility under reasoning-step distribution shifts.
7. Format Generalization
Format perturbations reduce CoT effectiveness, with degradation depending on noise level and perturbation type; insertion is especially disruptive.
- Setup: Format generalization introduces insertion, deletion, modification, and hybrid perturbations at varying noise levels.These perturbations model surface-level distribution discrepancies in test queries.
- Findings: CoT effectiveness decreases when perturbations are introduced, and degradation grows with noise level.The result follows the proposed data distribution lens.
- Findings: Insertion causes the greatest performance difference among the perturbation methods.Across applied areas, elements and transformations matter more than changes to other tokens.
8. Generality of Data Distribution Lens
The data distribution lens remains consistent across model architectures, sizes, temperatures, and pretrained or state-of-the-art models, supporting its broader validity.
- Experimental scope: Experiments vary LLM architecture, size, and temperature to test the generality of the data distribution lens.The controlled evaluations cover task, length, and format generalization.
- Internal validity: Models with different sizes and architectures behave similarly under distribution shifts across task, length, and format generalization.This supports internal validity across GPT and LLaMA settings.
- External validity: DataAlchemy enables controlled evaluation by generating data unseen during training and fine-tuning models on it.This addresses the difficulty of identifying training-test discrepancies in opaque state-of-the-art models.
- External validity: LLaMA3-8B and Qwen3-14B-Instruct show trends similar to the DataAlchemy pretrained-model results across all three generalization dimensions.These findings indicate external validity of the distribution lens.
9. Discussion and Implication
The discussion argues that CoT effectiveness is distribution-dependent and can become a brittle mirage outside training-like conditions. The data-distribution lens frames this fragility as a consequence of learned patterns rather than genuine logical inference.
- CoT reasoning is effective near the training distribution but becomes fragile under moderate distribution shifts.
- The apparent reasoning capability can arise from memorized or interpolated training patterns rather than logical inference.
- The data-distribution lens is presented as valid for analyzing task, length, and format generalization.
10. Conclusion
The conclusion attributes perceived CoT reasoning largely to inductive biases shaped by in-distribution training data. DataAlchemy provides a controllable platform for testing this claim across multiple distribution dimensions and model configurations.
- Structured reasoning capability largely arises from inductive biases shaped by in-distribution training data and is bounded by distribution discrepancies.
- DataAlchemy systematically probes task, length, and format distribution discrepancies in a fully controllable framework.
- Experiments confirm that the data-distribution pattern is invariant across LLMs with different architectures and sizes.
Limitations
The paper identifies three principal limitations: DataAlchemy abstracts away natural-language complexity, proprietary training distributions remain opaque, and the analysis covers only three generalization dimensions.
- DataAlchemy’s symbolic abstraction may not fully capture natural-language semantic richness, ambiguity, and compositional diversity.
- The training-data distributions of commercial and proprietary models remain inaccessible, limiting precise quantitative validation of the lens.
- The analysis focuses on task, length, and format shifts and does not explicitly model cross-lingual, multimodal, or cultural-context shifts.
Ethical Considerations
The paper situates CoT reasoning within a broader debate about robustness, faithfulness, and evaluation. It describes DataAlchemy’s abstract components and controlled transformation-based settings for studying generalization.
- Discussion of CoT reasoning: Prior work reports that irrelevant perturbations can substantially degrade CoT performance and that models may favor surface patterns over logical validity.
- Discussion of CoT reasoning: Intervention studies indicate that final answers can remain unchanged when intermediate reasoning steps are falsified or omitted.
- DataAlchemy: DataAlchemy represents tasks using atoms, elements, and transformations, enabling controlled variation of task and distribution factors.
- Evaluation framework: The proposed evaluation framework targets narrow settings, data entanglement, and data leakage through abstraction and training from scratch.
- DataAlchemy components: An element is an ordered sequence of atoms, while a transformation is an operator acting on an element.
- Generalization settings: Compositional transformations apply multiple transformations sequentially to an element.
- Generalization settings: Task generalization decomposes into element generalization and transformation generalization.
- Generalization settings: Format generalization changes test inputs through addition, deletion, or modification mechanisms.
C.1. Task Discrepancy Measure
Task discrepancy is mapped monotonically from task novelty, and increasing distribution discrepancy consistently degrades CoT performance. Across task, element, format, length, and model-scale analyses, models preserve interpolation near training data but fail to generalize reliably beyond it.
- Task discrepancy measure: Task-axis discrepancy is obtained by applying a monotone mapping to task novelty, with zero novelty producing zero discrepancy.The mapping feeds task discrepancy into the distribution-discrepancy term of Theorem 3.1.
- Transformation generalization: Non-commutative transfer produces 0 exact-match accuracy across reasoning, answers, and full chains, with edit distance increasing and BLEU approaching zero.Removing superficial commutative equivalences causes reasoning traces and answers to fail together.
- Element generalization: Exact-match accuracy drops from 1 to 0 across ID, CMP, and OOD under element-distribution shift, while BLEU also collapses to 0.Models fail to produce sensible outputs when test elements contain novel atoms.
- Format generalization: Exact-match accuracy for the four format perturbations declines from 100% at p=0% to near-zero at p=30%, with edit distance and BLEU degrading in lockstep.Insert perturbations typically cause the largest drop, while Hybrid follows the envelope of the individual modes.
- Model scale and distribution shift: Larger models improve near-ID performance but show the same OOD collapse once the available training or fine-tuning support is exhausted.Scaling accelerates interpolation within the expanded training distribution rather than producing qualitatively different robustness.