Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

1 Introduction

• The study trains a GPT2-like model from scratch on synthetic problems to control contamination and increase solution diversity while focusing on logical reasoning.Existing pretraining data may contain benchmark or near-identical problems, while GSM8K's 7.5K examples offer too few templates for training from scratch.Source: §1 · p. 2
• The synthetic framework yields more than 90 trillion solution templates, making template memorization insufficient to solve its problems.The authors say this count holds even after ignoring arithmetic, English, variable names, and unused parameters.Source: §1 · p. 3
• GPT2 achieves 99% accuracy on same-distribution problems and generalizes to longer reasoning lengths unseen during training, supporting learned reasoning rather than template memorization.The authors compare this pattern to length generalization in arithmetic [6] [13].Source: §1 · p. 3
• The model learns to generate shortest solutions while avoiding unnecessary computations, suggesting it plans before generating.The proposed plan avoids computing quantities that are not needed to solve the problem.Source: §1 · p. 3
• Probing suggests the model mentally preprocesses the necessary parameters before generating a solution, as humans do with scratch-pad work.The authors introduce six probing tasks to examine the hidden reasoning process.Source: §1 · p. 3
• The model learns all-pair dependency, a skill unnecessary for solving the training problems, whose solutions compute only necessary variables.The authors describe this as evidence of useful skills beyond those needed to fit the pretraining data.Source: §1 · p. 3
• Probing explains systematic errors, and some can be predicted before answer generation, independently of random generation.The authors report similar errors from GPT-4/4o, though they cannot probe those models' internal states.Source: §1 · p. 3
• A 16-layer, 576-dim transformer solves harder reasoning-length problems than a twice-as-large 4-layer, 1920-dim model, even with CoT.The authors do not claim direct applicability to foundation models such as GPT-4 or to more challenging mathematical reasoning tasks.Source: §1 · pp. 3–4

2 Result 1: Data Generation

• The generator represents direct, instance, and implicit dependencies, including abstract quantities inferred from English concepts such as which items count as fruit.Problems describe dependency graphs in shuffled English sentences, with a selected parameter posed as the question.Source: §2 · p. 4, §2.1 · p. 5

2.1 Step 1: Graph Construction and Problem Generation

• Structure graphs encode which items can occur under other items, while dependency graphs specify how instance parameters depend on up to four parameters and possibly a random-number generator.The layered categories require hierarchical access to items when calculating abstract parameters.Source: §2.1 · pp. 4–5

2.2 Step 2: Solution Construction (CoT)

• Solutions list only parameters needed for the query in topological order, using a sentence per computation and breaking calculations into binary operations.The model assigns random letters to necessary parameters; the chosen format reduces arithmetic complexity, which is not the paper’s focus.Source: §2.2 · p. 6

2.3 Difficulty Control

• Difficulty increases with the number of instance parameters (ip) and solution operations (op), the two controls used to define the iGSM dataset.The dataset is named iGSM to reflect that the synthetic dataset can be of infinite size.Source: §2.3 · p. 6

2.4 Train and Test Datasets

• Training uses iGSM-med with op≤15 and ip≤20 or iGSM-hard with op≤21 and ip≤28; evaluation includes larger-op OOD cases and template-disjoint test data.The med family tests op 20–23 and the hard family op 28–32; test templates are separated from training by a hash split.Source: §2.4 · pp. 6–7
• At least 7 billion templates occur in iGSM-medop=15 and at least 90 trillion in iGSM-hardop=21.The template estimates ignore unused parameters, numerics, sentence order, English words, and letter choices.Source: §2.4 · p. 7

Result 2: accuracy

• GPT2 performs well on harder-op out-of-distribution problems, including reasoning lengths absent from training, supporting the authors’ conclusion that it learns reasoning rather than merely memorizing templates.The authors also describe iGSM as suitable for pretraining their investigation of how language models solve grade-school math problems.Source: §3 · p. 8

Result 3: solution redundancy

• GPT2 predominantly uses level-1 reasoning, computing only necessary parameters even out of distribution; the authors contrast this with humans’ usual backward reasoning and scratch-pad use.Level-1 means generating shortest solutions by avoiding unnecessary computations.Source: §3 · p. 8

Model’s behavior process

• The study does not repeat all experiments across architectures: same-size Llama performance is described as similar, while resources constrained broader comparisons.The authors frame their focus as general behavior of autoregressive language models rather than small model differences.Source: §3 · pp. 7–8
• The probing tasks represent parameter necessity, dependency, known status, value, and whether a parameter can be computed next; probes are placed at problem, question, and solution-step boundaries.The probing setup is used to investigate the model’s mental reasoning process.Source: Figure 5 · p. 9

4 Result 4-5: Discover Model’s Mental Process

• The six probes test necessity, dependency, known status, value, next-step computability, and whether a computable parameter is necessary.The probes are designed to examine representations linked to human problem-solving strategies.Source: §4 · p. 9

4.2 Probing Results and Findings

• V-probing accuracies are high across tasks relative to majority-guess and random-model baselines, except for very hard out-of-distribution cases.Figure 7 reports beam=1 results; beam=4 results are described as almost identical.Source: Figure 7 · p. 10, §4.2 · pp. 10–11

4.1 V-Probing: A Nearly-Linear Probing Method

• V-probing freezes the pretrained model while training a classifier and a rank-8 input-embedding update to account for conditional parameter descriptions.The authors compare its accuracies with those from a randomly initialized transformer to assess whether probe signals come from pretrained weights.Source: §4.1 · p. 10

4.2 Probing Results and Findings

• During solution generation, the model tracks computed parameters and values and identifies which necessary parameters can be computed next.By the end of the problem description, it has identified the necessary parameters, indicating planning before solution generation.Source: §4.2 · p. 11
• Models learn all-pair parameter dependencies, including unnecessary ones, although solving the problems requires only necessary parameters.The authors describe this as a level-2 reasoning skill distinct from human problem-solving and suggest it may help organize relationships for future tasks.Source: §4.2 · p. 11
• The findings suggest that abundant data can teach backward planning without requiring it to be directly included in training examples.The paper distinguishes this planning of computation order from chain-of-thought decomposition into simpler steps.Source: §4.2 · p. 11

5 Result 6: Explain Model’s Mistakes

• Many reasoning mistakes are systematic errors in internal states, and some can be detected before the first solution step.Figure 8 examines probing accuracies for unnecessary parameters in correct solutions and the first wrong parameter in incorrect solutions.Source: Figure 8 · p. 12, §5 · p. 12
• Unnecessary steps in correct solutions often correspond to parameters incorrectly judged necessary, while first wrong parameters are often misjudged as ready to compute.The latter errors involve incorrect predictions of nece next(A) or can next(A) as true when the parameter is not ready.Source: §5 · p. 12
• GPT-4/4o also make similar errors by outputting unnecessary parameters, although their internal states could not be probed.Source: §5 · p. 12

Result 7

• A 4-layer transformer with 1920 hidden dimensions underperforms, while a deeper 20-layer, 576-dim model performs very well on the math datasets.The authors infer that depth is likely essential for reasoning tasks; this contrasts with suggestions that depth and width may have minimal universal impact [14] [5].Source: §6 · p. 13

Result 8

• Deeper layers more accurately predict whether parameters farther from the query are necessary, consistent with layer-by-layer dependency planning.The probing task examines necessary parameters at distances 1 through 8; Figure 10 plots probing accuracy by distance and layer for a 20-layer GPT2 model.Source: §6 · p. 13, Figure 10 · p. 14
• The authors propose that t-step mental reasoning may require deeper models as t grows, assuming other hyperparameters remain constant.They connect depth to the complexity of hidden reasoning processes, rather than asserting a fixed depth requirement for every task.Source: §6 · p. 13

Disclaimers

• Adding backward-thinking processes as chain-of-thought data can reduce the depth requirement, but the authors do not claim t reasoning steps require a depth-t transformer.A single layer might implement multiple mental steps, possibly with reduced accuracy as the number of steps increases; the relationship depends heavily on the data distribution.Source: §6 · p. 14

7 Conclusion

• The synthetic setting demonstrates grade-school math problem-solving through generalization rather than data contamination or template memorization.The paper also reports probing hidden reasoning, human-aligned and novel thinking processes, and a method to predict errors before solving.Source: §7 · p. 14
• Models trained purely on iGSM make mistakes similar to GPT-4/4o, which the authors cite as evidence that their hidden-reasoning findings connect to practice.They note that contemporary GPT-4/4o cannot few-shot solve iGSM-medop=11, while suggesting future LLMs may use synthetic math pretraining.Source: §7 · p. 14
• The paper situates its math-reasoning study alongside work on language structures and world knowledge in Parts 1 and 3.Part 1 examines mental dynamic programming; Part 3 addresses learning world knowledge.Source: §7 · p. 14

A Result 1 — An Example in iGSM-hard with op = 21

• Figure 11 gives an iGSM-hardpq training example with op=21; testing evaluates models at the harder op=28.Source: Figure 11 · p. 15

Appendix figure

• Figure 12 restricts probing accuracies to positive and negative labels, complementing Figure 7’s all-label results.Source: Figure 12 · p. 16

B Results 4-5 — Details on V-probing

• V-probing tests the model at selected positions: after the problem description for nece and dep, and after each solution sentence for other tasks.For dep, probing can occur before the solution or even before the question; the model is frozen while a small number of parameters are trained for each task.Source: B Results 4-5 — Details on V-probing · p. 16
• The setup freezes pretrained parameters except for a rank-8 embedding update and a trainable linear classification head over the last-layer hidden state.Inputs are truncated at the probe position and prefixed with [START] and the relevant parameter name or names.Source: B Results 4-5 — Details on V-probing · p. 16
• V-probing supports arbitrary variable inputs, such as parameter names, to test functions inside the transformer.The figure caption describes it as a nearly-linear method for testing whether a model knows an arbitrary function of a parameter at a specific input position.Source: B Results 4-5 — Details on V-probing · p. 16, Figure 13 · p. 17
• Unlike fixed-task linear probing, V-probing appends variable names to inputs and uses a low-rank embedding update to accommodate the probing task.The paper distinguishes its variable inputs from prior nearly-linear methods that do not support them; for qp data, input order is reversed, except for dep(A, B).Source: B Results 4-5 — Details on V-probing · pp. 16–17
• Because dep and nece next labels are imbalanced, Figure 12 reports probing accuracy on positive and negative labels separately.Guessing all false would yield 83% accuracy for dep(A, B) and 92% for nece next(A).Source: B Results 4-5 — Details on V-probing · p. 16

• Probing accuracy for nece(A) increases with layer depth.Source: Figure 14 · p. 18
• The x-axis gives parameter A’s distance from the query parameter.Source: Figure 14 · p. 18
• Colors transition from light to dark to represent layers 1 through the maximum layer.Source: Figure 14 · p. 18

D.1 Generate Structure Graph

• The structure graph has d layers, e edges, and between w0 and w1 items per layer; its layer sizes and edges are randomly generated subject to the edge count and connectivity constraints.The method first chooses layer sizes that permit the requested number of edges, then creates connecting edges and adds edges between adjacent layers until the count reaches e.Source: §D.1 · p. 19

D.1.1 Attach English

• English labels are assigned by randomly choosing a category hierarchy, 2–4 consecutive layers, a subcategory, and item names for each layer.Source: §D.1.1 · p. 21

D.2 Generate Dependency Graph

• Dependency edges encode which parameters depend on others; the necessary graph supports the solution description, while the full graph supplies the problem description and includes additional parameters and edges.Abstract-parameter dependencies are inherited from the structure graph, while dependencies for instance parameters are added randomly.Source: §D.2 · p. 21

D.2.1 Construction of Gnece1

• The necessary graph is built incrementally by adding abstract parameters while keeping its operation count at most n, prioritizing parameters spanning more layers, then adding instance parameters.A single abstract parameter can require recursively adding its dependencies, so the operation count is not necessarily exactly n.Source: §D.2.1 · p. 22

D.2.2 Construction of Gnece3

• A random query parameter and topological ordering guide incremental edge creation so every parameter is necessary for the query; if construction fails, the structure graph is regenerated.Source: §D.2.2 · pp. 22–23

D.2.3 Construction of Gnece

• The final necessary graph adds edges and an RNG vertex so parameters have randomly chosen in-degrees from 1 to 4.Before this step, the constructed graph has in-degree at most 1 for each instance parameter.Source: §D.2.3 · p. 24
• Generated dependency graphs target an exact solution-operation count, while each instance parameter is capped at three operations by limiting its dependencies.Abstract parameters are placed late in the topological order to avoid generation failures, while adding some edges to existing parameters can increase solution depth.Source: §D.2.3 · p. 24

D.2.4 Construction of Gd

• The complete graph adds unnecessary dependency edges and parameters to the necessary graph, including every instance parameter while allowing some abstract parameters to remain absent.New dependencies are randomly selected from previously known parameters, and new unnecessary abstract parameters may also be introduced.Source: §D.2.4 · p. 25

D.3 Generate English: Problem, Question and Solution

• Problem descriptions list and shuffle instance-parameter sentences, omit abstract-parameter descriptions, and ask for a query value; solutions follow topological order and break computations into binary steps.The omission of abstract descriptions is intended to make models learn category relationships across multiple examples; the query question can appear at the beginning or end of the problem.Source: §D.3 · p. 26

D.4 Putting Altogether

• Generation parameters control maximum instance count and operation count; a force switch sets the latter exactly, while randomized operation counts favor easier pretraining problems and template hashes separate training from evaluation.The hash split assigns templates with hash < 17 (mod 23) to training and the rest to evaluation.Source: §D.4 · p. 27

E Data Details: Probing Data Preparation

• Probing examples randomly choose parameters and eligible positions, with at most 10 samples per problem and task; solution-template hashes separate probing training from testing.Dependency and necessity probes occur at the end of the problem and question, while value-related probes may occur after solution sentences.Source: E Data Details: Probing Data Preparation · p. 28

F Experiment Details

• Experiments primarily use GPT2-12-12 with rotary positional embeddings and generate training data on the fly without limiting its amount.The setup also defines roughly equal-size model variants with different depths and widths, and twice-size variants; sample complexity is not explored.Source: F Experiment Details · pp. 28–29

F.1 Pretrain Experiment Details

• Pretraining used AdamW with mixed-precision fp16, cosine learning-rate decay, and 1000 linear ramp-up steps.For iGSM-med, training ran 100,000 steps with peak learning rate 0.002, weight decay 0.05, batch size 512, and context length 768; iGSM-hard used 200,000 steps, weight decay 0.03, batch size 256, and context length 1024.Source: §F.1 · p. 29
• Pretraining problems and solutions were randomly generated, concatenated, and truncated to fit the context window; longer problems were rarely discarded.Source: §F.1 · p. 29
• Evaluation used context length 2048 and either greedy beam-1 or sampled beam-4 decoding; larger beams brought no further improvement.The preferred settings were dosample=False for beam 1 and dosample=True for beam 4.Source: §F.1 · p. 29
• Main accuracy tests averaged 4096 problems per type and checked fully correct intermediate solution steps, while redundancy tests used 4096 problems per case among fully correct solutions.Source: §F.1 · p. 29

F.2 V-probing

• V-probing freezes the pretrained model and trains an output linear head plus a small rank-r input-layer update.The paper uses r = 8; the authors state this helps ensure high probing accuracy mostly comes from the pretrained model.Source: §F.2 · pp. 29–30
• V-probing tests used at least 4096 random problem-parameter pairs per cell, and layer-wise nece(A) tests used at least 73728 pairs per case.Layer-wise results were binned by parameter A’s distance from the query.Source: §F.2 · p. 30

• The few-shot evaluation used GPT-4 Turbo and GPT-4o on iGSM-medpq across operation counts 2–20.Source: G Failure Examples on GPT-4 / GPT-4o · p. 30
• To reduce arithmetic and language-comprehension confounds, the authors replaced mod23 with mod5 and simplified the English categories and descriptions.They also provided arithmetic examples, a background paragraph describing the structure graph, and five-shot problem-solution examples.Source: G Failure Examples on GPT-4 / GPT-4o · p. 30
• GPT-4o was almost randomly guessing for op ≥11, and GPT-4 Turbo for op ≥9.The evaluation checked whether the final output number, between 0 and 4, matched the correct answer.Source: G Failure Examples on GPT-4 / GPT-4o · p. 30
• GPT-4 and GPT-4o failures mostly involved computing unnecessary parameters or parameters not yet ready to compute, rather than format errors or misunderstanding.The paper relates these errors to its findings about models’ hidden reasoning process.Source: G Failure Examples on GPT-4 / GPT-4o · p. 30, Figure 15 · p. 31