PaperlayerView exact revision on arXiv

Source-linked AI summary

What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective

Ming Li, Yanhong Li, Tianyi Zhou

arXiv:2410.23743v2cs.CLcs.AIcs.LG

TL;DR

Layer-wise gradients in dynamic LLM training remain underexplored, particularly their scaling and distribution across layers. The study uses spectral analysis of layer-wise gradients to compare slow- versus fast-thinking rationales and correct versus irrelevant responses across tasks. Slow CoT yields similar gradient norms across layers and distinguishes correct from irrelevant reasoning; fast thinking produces larger gradients and layer differences, while longer knowledge responses do not show the same patterns.

  • Problem

    Layer-wise gradients in dynamic LLM training remain underexplored, particularly their scaling and distribution across layers.

    Source: §1 · p. 1

  • Method

    The study uses spectral analysis of layer-wise gradients to compare slow- versus fast-thinking rationales and correct versus irrelevant responses across tasks.

    Source: §1 · p. 2

  • Results

    Slow CoT yields similar gradient norms across layers and distinguishes correct from irrelevant reasoning; fast thinking produces larger gradients and layer differences, while longer knowledge responses do not show the same patterns.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The observed gradient patterns offer insights into LLM training stability and efficiency within the studied settings.

    Source: §1 · p. 2, §6 · p. 8

    The analysis focuses mainly on layer-wise gradient strength, and the paper notes that other metrics could also be used.

    Source: Limitations · p. 9

Abstract

from arXiv · show

What makes a difference in the post-training of LLMs? We investigate the training patterns of different layers in large language models (LLMs) through the lens of the gradient. We are specifically interested in how fast vs. slow thinking affects the layer-wise gradients, given the recent popularity of training LLMs on reasoning paths such as chain-of-thoughts (CoT) and process rewards. In our study, fast thinking without CoT leads to larger gradients and larger differences of gradients across layers than slow thinking (Detailed CoT), indicating the learning stability brought by the latter. Additionally, we study whether the gradient patterns can reflect the correctness of responses when training different LLMs using slow vs. fast thinking paths. The results show that the gradients of slow thinking can distinguish correct and irrelevant reasoning paths. As a comparison, we conduct similar gradient analyses on non-reasoning knowledge learning tasks, on which, however, trivially increasing the response length does not lead to similar behaviors of slow thinking. Our study strengthens fundamental understandings of LLM training and sheds novel insights on its efficiency and stability, which pave the way towards building a generalizable System-2 agent. Our code, data, and gradient statistics can be found in: https://github.com/MingLiiii/Layer_Gradient.

1 Introduction

Layer-wise gradients during LLM training remain underexplored, while common layer-probing methods do not directly reveal the gradients driving learning. This study compares gradients across thinking styles, response correctness, model types, and task types, finding distinct patterns for slow versus fast reasoning and a boundary between reasoning and knowledge-learning tasks.

1 Introduction

  • The study directly analyzes layer-wise gradients because prior layer-wise work mainly probes behavior through layer modifications or removal, rather than gradients driving learning.Source: §1 · p. 1
  • The analysis compares 5 base and 5 instruction-finetuned LLMs across math, commonsense reasoning, and Wiki knowledge, using gradient norms and their differences to assess training patterns and sensitivity.Source: §1 · p. 2, Table 1 · p. 2
  • Slow thinking with Detailed CoT yields similar gradient norms across layers, whereas fast thinking produces larger gradients and sharper layer-to-layer differences.Source: §1 · p. 2
  • Slow-thinking gradients distinguish correct from irrelevant responses, but instruction-finetuned models show no superior ability to identify incorrect reasoning paths.Source: §1 · p. 2
  • The reasoning-task gradient patterns do not extend to knowledge learning, where simply lengthening responses does not reproduce slow-thinking patterns.Source: §1 · p. 2

2 Methodology

The methodology measures gradients from instruction-tuning examples and summarizes their layer-wise behavior through spectral properties and curve-comparison metrics. It uses these measures to characterize gradient strength, variation across layers, and differences between gradient curves.

2.1 Preliminaries

  • Instruction tuning minimizes a per-sample loss over response tokens for each instruction-response pair.Source: §2.1 · p. 3

2.2 Gradient Representation

  • The study analyzes weight-shaped Q, K, V, and O projection gradients at each attention layer, using nuclear norms to capture gradient magnitude and spectral concentration.Source: §2.2 · p. 3

2.3 Metrics of Gradient Difference

  • MAD summarizes average absolute changes between consecutive layer values, while relative difference compares two curves at the same projection and layer.Source: §2.3 · pp. 3–4

3 Experimental Setup

The experiments examine ten models across math, commonsense reasoning, and wiki-knowledge tasks, sampling 500 instances per task. Table 2 reports consistently lower gradient MAD for more detailed reasoning paths.

3.1 Models

  • The study covers five base models and their instruction-tuned counterparts; main illustrations use Qwen2-1.5B, with other models reported in the appendix.Source: §3.1 · p. 4

3.2 Datasets

  • Datasets cover math, commonsense reasoning, and wiki knowledge; experiments randomly sample 500 instances per task, and the wiki data separates popular from unpopular pages.Source: §3.2 · p. 4
  • Gradient MAD decreases consistently across layer regions when models are trained to produce more detailed reasoning paths.Source: Table 2 · p. 4

4 Empirical Analysis

Across math and commonsense reasoning tasks, the analysis compares layer-wise gradients under different reasoning paths, response correctness, and model initializations. It also tests whether response length and popularity affect gradients on knowledge-intensive tasks.

4.1.1 Slow vs. Fast Thinking

  • Fast-thinking training has the largest layerwise MADs, while CoT lowers them most under Detailed CoT, whose gradient norms tend to be similar across layers.The authors suggest that large MADs might cause training instability.Source: §4.1.1 · p. 5, Figure 1 · p. 5

4.1.2 Effect of Response Correctness

4.1.3 Effect of Initial Models

  • Detailed CoT lowers and smooths gradients in both base and instruction-tuned LLMs, but Simplified-CoT patterns diverge and instruction-tuned models are not much better at detecting irrelevant responses.The authors suggest that Simplified-CoT patterns may reflect discrepancies with instruction data, and that gradient curves may measure sample alignment.Source: Figure 3 · p. 6, §4.1.3 · p. 7

4.2.1 Response Length & Popularity

  • On knowledge-intensive tasks, response length does not alter gradient scale or fluctuation, whereas unpopular knowledge raises both; the authors distinguish this from detailed-CoT reasoning effects.Source: §4.2.1 · p. 8

4.2.2 Response Correctness & Initial Models

  • On knowledge learning, neither base nor instruction-tuned gradient curves distinguish correct from nonsense responses, while instruction tuning raises gradient scale and fluctuation, especially for unpopular knowledge.Source: Figure 4 · p. 7, §4.2.2 · p. 8

5 Further Discussion

The discussion proposes a possible explanation for why detailed CoT stabilizes gradients, notes variation across model families, and outlines potential applications of gradient patterns.

5.1 Why is Slow Thinking more Stable?

  • Prystawski et al. (2023) propose detailed CoT creates overlapping concept neighborhoods, helping prior context support token inference and more stable gradients; this is a partial explanation.Source: §5.1 · p. 8

5.2 Behaviors of Different LLMs

  • Gradient-curve shapes vary across Gemma2, Qwen2, and Llama3 models, and the authors say generalization awaits further examination across LLMs.Source: §5.2 · p. 8

5.3 Potential Applications

  • The authors suggest gradient patterns could help assess data quality and model-data compatibility, identify critical layers for efficient training, and inform understanding of model differences.Source: §5.3 · p. 8

6 Conclusion

Slow-thinking training is associated with more similar gradient norms across layers, while fast thinking produces larger gradients and greater cross-layer fluctuation. Slow-thinking gradients distinguish correct from irrelevant responses, but longer knowledge-learning responses do not reproduce these reasoning-task patterns.

6 Conclusion

  • Slow thinking yields stable gradient norms across layers, whereas fast thinking produces larger gradients and greater cross-layer fluctuation.These knowledge-learning tasks do not show similar patterns from simply increasing response length.Source: §6 · pp. 8–9
  • Slow-thinking gradients distinguish correct from irrelevant responses, while no-CoT gradient patterns for the two response types are similar.Source: §6 · p. 9
  • In knowledge-learning tasks, simply increasing response length does not reproduce the slow-thinking gradient patterns found in reasoning tasks.Source: §6 · p. 9

Limitations

Page limits restrict how many results the paper presents, which the authors say might weaken its findings. They include further visualizations and statistics in the appendix and note that other metrics could extend the analysis.

Limitations

  • Page limits allow only a small proportion of results to be presented, which the authors say might weaken the findings.Source: Limitations · p. 9
  • The appendix contains additional visualizations and statistical results beyond those presented in the paper.Source: Limitations · p. 9
  • The analysis focuses mainly on layer-wise gradient strength, while other metrics could also be used.Source: Limitations · p. 9

A Dataset Description

The dataset description covers math and commonsense reasoning with answer-only and CoT training configurations, including GPT4o-generated detailed paths. Wiki knowledge experiments vary article popularity and response length to examine gradient patterns.

A.1 Math Reasoning

  • Math experiments use AQuA, GSM8K, and selected MATH question types, comparing original ground truths with CoT-plus-answer training and GPT4o-generated detailed paths.AQuA ground truths are answer options, GSM8K ground truths are resulting digits, and MATH ground truths are complete solutions.Source: §A.1 · p. 14

A.2 Commonsense Reasoning

  • Commonsense experiments use StrategyQA, ECQA, CREAK, and Sensemaking, testing original ground truths, human-annotated CoTs, and GPT4o-generated CoTs.Source: §A.2 · p. 14

A.3 Wiki Knowledge Learning

  • Wiki experiments vary article popularity and response length, using popular-page extracts of 100, 500, or 1,000 tokens and full unpopular articles unless they exceed 1,024 tokens.The study assumes modern LLMs learned the articles; under that assumption, unpopular knowledge triggers larger gradients, and gradient norm cannot identify irrelevant responses in these knowledge-intensive tasks. Table 4 presents average response lengths by dataset configuration.Source: §A.3 · p. 14, Table 4 · p. 15

B Data Examples

This appendix presents dataset examples and catalogs visualizations and statistical results for reasoning and Wiki tasks, spanning multiple models, datasets, and response categories, including irrelevant Wiki responses.

B Data Examples

  • The appendix catalogs examples across the listed datasets and includes simplified and detailed chain-of-thought samples, including arithmetic and commonsense questions.Examples include an AQuA train-and-bridge problem and an ECQA question about remembering.Source: B Data Examples · pp. 16, 17, 19

C.1.1 Reasoning Tasks

  • The subsection catalogs visualizations and statistical results for MATH-Algebra, MATH-Counting, MATH-Geometry, and six other reasoning datasets.The listed datasets are AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking; tables are identified for the three MATH tasks and AQuA.Source: §C.1.1 · p. 28, Table 5 · p. 28, Table 6 · p. 28

C.1.2 Wiki Tasks

  • The subsection identifies visualizations and statistical results for Wiki tasks.The cited materials are Figure 144 and Table 14.Source: §C.1.2 · p. 29, Table 14 · p. 34

C.2.1 Reasoning Tasks

  • The subsection catalogs visualizations and statistical results for MATH-Algebra, MATH-Counting, MATH-Geometry, and the listed additional reasoning tasks.The additional tasks are AQuA, StrategyQA, ECQA, CREAK, and Sensemaking; tables are identified for GSM8K and StrategyQA.Source: §C.2.1 · p. 29, Table 9 · p. 29, Table 10 · p. 29

C.2.2 Wiki Tasks

  • The subsection includes correct-response visualizations and statistical results, plus irrelevant-response statistics for the three MATH tasks, AQuA, and StrategyQA.The visualizations include Wiki tasks; the irrelevant-response tables are identified for MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, and StrategyQA.Source: Figure 25 · p. 33, Table 15 · p. 35, Table 16 · p. 35
  • Wiki-task visualizations and statistical results are presented for gemma-2-2b on irrelevant responses.The examples include AQuA, StrategyQA, ECQA, CREAK, Sensemaking, and Wiki.Source: Table 20 · p. 36, Figure 29 · p. 37, Figure 34 · p. 39

C.3.1 Reasoning Tasks

  • MATH and other reasoning-task examples include visualizations and statistical results for gemma-2-2b-it on correct responses.The listed datasets include MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §C.3.1 · p. 41, Table 24 · p. 41, Table 27 · p. 42

C.3.2 Wiki Tasks; D.1.2 Wiki Tasks

  • For Wiki tasks, the visualizations and statistical results are shown in Figure 144 and Table 53, with no model or response condition specified in the cited passage.Source: §D.1.2 · p. 56

C.4.1 Reasoning Tasks

  • MATH and other reasoning-task visualizations and statistics are listed for C.4.1, including AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.The MATH examples cover Algebra, Counting, and Geometry; the supplied captions also identify correct-response tables for GSM8K and StrategyQA using gemma-2-2b-it.Source: §C.4.1 · p. 42, Table 28 · p. 42, Table 29 · p. 42

C.4.2 Wiki Tasks

  • The C.4.2 appendix includes gemma-2-2b-it visualizations and statistical results for Wiki and reasoning tasks, covering correct and irrelevant responses.The passages identify Wiki and datasets including MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: Figure 38 · p. 43, Figure 44 · p. 46, Table 33 · p. 47

D.1.1 Reasoning Tasks

  • Llama-3.1-8B examples in D.1.1 provide visualizations and statistical results for MATH tasks and other reasoning datasets on correct responses.The listed tasks include MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §D.1.1 · p. 55, Table 44 · p. 55, Table 47 · p. 56

D.2.1 Reasoning Tasks

  • D.2.1 catalogs visualizations and statistical results for Llama3.1-8B reasoning tasks on correct responses.The listed tasks are MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §D.2.1 · p. 56, Table 48 · p. 56, Table 49 · p. 56

D.2.2 Wiki Tasks

  • Figure 64 visualizes Wiki tasks using Llama-3.1-8B on correct responses, and Table 53 reports Wiki statistical results; Figures 65 and 70 visualize MATH-Algebra and StrategyQA irrelevant responses.Source: Figure 64 · p. 60, Table 53 · p. 61, Figure 65 · p. 62
  • A visualization shows Wiki-task results for irrelevant responses using Llama-3.1-8B.Source: Figure 74 · p. 67
  • Statistical results for irrelevant responses are tabulated for ECQA, CREAK, Sensemaking, and Wiki using Llama-3.1-8B.The supplied captions do not report the tables’ values or conclusions.Source: Table 60 · p. 68, Table 61 · p. 68, Table 62 · p. 68

D.3.1 Reasoning Tasks

The appendix lists visualizations and statistical results for math, other reasoning, and Wiki tasks across four subsections. Its entries include correct- and irrelevant-response cases.

D.3.1 Reasoning Tasks

  • D.3.1 reports visualizations and statistical results for MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.The listed correct-response examples use Llama-3.1-8B-Instruct.Source: §D.3.1 · p. 69, Table 64 · p. 69, Table 67 · p. 70

D.3.2 Wiki Tasks

  • D.3.2 presents visualizations and statistical results for Wiki tasks, including a correct-response table for Llama-3.1-8B-Instruct.Source: §D.3.2 · p. 70, Table 73 · p. 75

D.4.1 Reasoning Tasks

  • D.4.1 lists visualizations and statistical results for MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §D.4.1 · p. 70

D.4.2 Wiki Tasks

  • D.4.2 presents visualizations and statistical results for Wiki tasks.Source: §D.4.2 · p. 70

E.1.1 Reasoning Tasks

These sections catalog visualizations and statistical results across reasoning and Wiki tasks, using Qwen2-1.5B and Qwen2-1.5B-Instruct responses labeled correct or irrelevant.

E.1.1 Reasoning Tasks

  • E.1.1 covers visualizations and statistical results for MATH and other reasoning tasks, including correct Qwen2-1.5B responses.The listed tasks include MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §E.1.1 · p. 83, Figure 95 · p. 83, Figure 96 · p. 83

E.1.2 Wiki Tasks

  • For Wiki tasks, visualizations and statistical results are shown in Figure 144 and Table 93; Figure 104 visualizes correct responses from Qwen2-1.5B.Source: §E.1.2 · p. 84, Figure 104 · p. 88

E.2.1 Reasoning Tasks

  • E.2.1 covers visualization and statistical results for reasoning tasks using irrelevant Qwen2-1.5B responses.The listed reasoning tasks include MATH datasets, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §E.2.1 · p. 84, Figure 105 · p. 90, Figure 106 · p. 90

E.2.2 Wiki Tasks

  • The visualizations and statistical results for Wiki tasks are shown in Figure 154 and Table 103; the cited passage does not identify a subsection for those results.Figure 114 is a visualization for Wiki tasks using Qwen2-1.5B on irrelevant responses.Source: §E.2.2 · p. 84, Figure 114 · p. 95

E.3.1 Reasoning Tasks

  • E.3.1 covers visualization and statistical results for reasoning tasks using correct Qwen2-1.5B-Instruct responses.The listed tasks include MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §E.3.1 · p. 97, Figure 115 · p. 97, Figure 116 · p. 97

E.3.2 Wiki Tasks

  • E.3.2 presents visualization and statistical results for Wiki tasks using correct Qwen2-1.5B-Instruct responses.The paragraph identifies Figure 164 for Wiki-task visualizations; the table number is unresolved in the supplied text.Source: §E.3.2 · p. 98, Figure 124 · p. 102

E.4.1 Reasoning Tasks

  • E.4.1 covers visualization and statistical results for reasoning tasks using irrelevant Qwen2-1.5B-Instruct responses.The listed reasoning tasks include MATH datasets, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §E.4.1 · p. 98, Figure 125 · p. 104, Figure 126 · p. 104

E.4.2 Wiki Tasks

  • The visualizations and statistical results on Wiki tasks are shown in Figure 174 and Table 123; Figure 134 visualizes Wiki tasks using Qwen2-1.5B-Instruct on irrelevant responses.Source: §E.4.2 · p. 98, Figure 134 · p. 109

F.1.1 Reasoning Tasks

These sections catalog visualizations and statistical results across reasoning and Wiki tasks, using Llama-2-7b-hf and Llama-2-7b-chat-hf responses labeled correct or irrelevant.

F.1.1 Reasoning Tasks

  • F.1.1 covers visualizations and statistical results for MATH and other reasoning tasks, including correct Llama-2-7b-hf responses.The listed tasks include MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §F.1.1 · p. 111, Figure 135 · p. 111, Figure 136 · p. 111

F.1.2 Wiki Tasks

  • The visualizations and statistical results for Wiki tasks are presented in Figure 144 and Table 133.Source: §F.1.2 · p. 112

F.2.1 Reasoning Tasks

  • F.2.1 covers visualization and statistical results for reasoning tasks using irrelevant Llama-2-7b-hf responses.The listed reasoning tasks include MATH datasets, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §F.2.1 · p. 112, Figure 145 · p. 118, Figure 146 · p. 118

F.2.2 Wiki Tasks

  • The visualizations and statistical results for Wiki tasks are shown in Figure 154 and Table 143; Figure 154 visualizes Wiki-task responses from Llama-2-7b-hf on irrelevant responses.Source: §F.2.2 · p. 112, Figure 154 · p. 123

F.3.1 Reasoning Tasks

  • F.3.1 covers visualization and statistical results for reasoning tasks using correct Llama-2-7b-chat-hf responses.The listed tasks include MATH-Algebra, MATH-Counting, MATH-Geometry, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §F.3.1 · p. 125, Figure 155 · p. 125, Figure 156 · p. 125

F.3.2 Wiki Tasks

  • F.3.2 presents visualization and statistical results for Wiki tasks using correct Llama-2-7b-chat-hf responses.The Wiki-task visualization and statistics are identified as Figure 164 and Table 153.Source: §F.3.2 · p. 126, Figure 164 · p. 130

F.4.1 Reasoning Tasks

  • F.4.1 covers visualization and statistical results for reasoning tasks using irrelevant Llama-2-7b-chat-hf responses.The listed reasoning tasks include MATH datasets, AQuA, GSM8K, StrategyQA, ECQA, CREAK, and Sensemaking.Source: §F.4.1 · p. 126, Figure 165 · p. 132, Figure 166 · p. 132

F.4.2 Wiki Tasks

  • Visualizations and statistical results for Wiki tasks are shown in Figure 174 and Table 163, and Figure 174 specifically visualizes Wiki tasks using Llama-2-7b-chat-hf on irrelevant responses.Source: §F.4.2 · p. 126, Figure 174 · p. 137

Related papers

Loading 2410.23743v2…