PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning to Learn from Latent Thoughts

Yangjun Ruan, Neil Band, Chris J. Maddison, Tatsunori Hashimoto

arXiv:2503.18866v2cs.LGcs.AIcs.CL

TL;DR

As compute scaling outpaces the growth of human-written text, the paper asks how to improve LM pretraining efficiency when data is constrained. It infers latent thoughts behind compressed text and augments pretraining with them, using EM-based BoLT to bootstrap the generator. In math-focused continued pretraining, latent thoughts improve performance over raw-data and synthetic-paraphrase baselines, while bootstrapping improves across iterations and benefits from additional inference compute.

  • Problem

    Compute scaling is outpacing human-written text, creating a data-constrained LM pretraining regime in which compressed web text may limit data efficiency.

    Source: §1 · p. 1

  • Method

    The approach infers latent thoughts underlying observed text, augments the data with them, and uses the EM-based BoLT algorithm to iteratively improve latent synthesis.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    25.4% on MATH with GPT-4o-mini latent thoughts outperformed 5.74% from raw data and 19.4% from synthetic Chain-of-Thought paraphrases, while BoLT improved across bootstrap iterations.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Latent-thought inference and iterative bootstrapping can improve pretraining data efficiency and downstream performance in data-constrained math setups.

    Source: §1 · p. 2, §8 · p. 18

    The experiments are a proof of concept limited to a 1B-parameter LM and a few billion mostly mathematical tokens, with larger-scale and general-domain evaluation left for future work.

    Source: §7.2 · p. 17

Abstract

from arXiv · show

Compute scaling for language model (LM) pretraining has outpaced the growth of human-written texts, leading to concerns that data will become the bottleneck to LM scaling. To continue scaling pretraining in this data-constrained regime, we propose that explicitly modeling and inferring the \emph{latent thoughts} that underlie the text generation process can significantly improve pretraining data efficiency. Intuitively, our approach views web text as the compressed final outcome of a verbose human thought process and that the latent thoughts contain important contextual knowledge and reasoning steps that are critical to data-efficient learning. We empirically demonstrate the effectiveness of our approach through data-constrained continued pretraining for math. We first show that synthetic data approaches to inferring latent thoughts significantly improve data efficiency over training on the same amount of raw data. Furthermore, we demonstrate latent thought inference without a strong teacher, where an LM \emph{bootstraps its own performance} by using an EM algorithm to iteratively improve the capability of the trained LM and the quality of thought-augmented pretraining data. We show that a 1B LM can bootstrap its performance across at least three iterations and significantly outperform baselines trained on raw data, with increasing gains from additional inference compute when performing the E-step. The gains from inference scaling and EM iterations suggest new opportunities for scaling data-constrained pretraining.

1 Introduction

The paper proposes reasoning to learn: augmenting compressed web text with inferred latent thoughts to improve data-efficient pretraining. Its BoLT algorithm uses EM to bootstrap latent-thought synthesis, with experiments showing stronger math performance and iterative self-improvement on limited data.

Motivation

  • The method targets a data-constrained regime motivated by compute growing faster than human-written web text, which may make data a bottleneck for LM scaling.The paper frames human-written text as a compressed outcome of an underlying thought process that LMs usually do not model explicitly.Source: §1 · p. 1

Method

  • The approach models observed text X and underlying thoughts Z jointly, using a latent generator q(Z | X) to synthesize thoughts that augment pretraining data.The latent thought generator can share weights with the LM, reducing changes to the standard pretraining pipeline.Source: §1 · p. 2
  • BoLT uses EM to iteratively improve the latent-thought generator, and its Monte Carlo E-step benefits from additional inference samples.The approximate posterior approaches the true posterior as sample count increases; experiments report benefits from at least four samples and at least three bootstrap iterations.Source: §1 · p. 2

Empirical results

  • 25.4% on MATH with GPT-4o-mini latent thoughts exceeded 5.74% from raw data and 19.4% from synthetic Chain-of-Thought paraphrases.These results evaluate continued pretraining of a 1.1B TinyLlama model on limited FineMath data.Source: §1 · p. 2

Bootstrapping

  • BoLT bootstraps performance across iterations on likelihood metrics and MATH without task-specific data, as latent-thought quality improves.The gains are attributed to improved latent-thought quality across bootstrap iterations.Source: §1 · p. 3

Implication

  • The authors suggest that LM reasoning primitives can extract more capability from limited, task-agnostic pretraining data.This conclusion follows the reported gains from synthetic latent thoughts and self-generated bootstrapping.Source: §1 · p. 3

2 Related Work

The related work spans synthetic-data training, externally supervised reasoning, and reasoning learned from pretraining data. The paper distinguishes its approach by augmenting each document with latent thoughts and bootstrapping the generator for data-efficient pretraining.

Synthetic data for training LMs

  • The paper augments each document chunk X with latent thoughts Z rather than replacing real data, and reports this distinction as crucial to downstream performance.It also studies bootstrapping the data generator rather than the teacher-student distillation settings common in synthetic-data work.Source: §2 · p. 3
  • Prior synthetic-data methods curate prompts or seed corpora for diversity, while insufficient curation can cause mode collapse.Paraphrasing and relationship-description approaches retain real data as a seed corpus to mitigate this risk.Source: §2 · p. 3

Learning to reason using external supervision

  • Externally supervised reasoning methods commonly use verifiable rewards, reinforcement learning, self-play, or supervised data to internalize chains of thought.Related latent-variable and EM approaches also rely on domain-specific tasks or external supervision, unlike this work.Source: §2 · p. 3

Learning to reason using pretraining data

  • Pretraining-data approaches improve reasoning without explicit external rewards, including iterative hidden-state processing and thought-token reinforcement learning.The paper instead targets data-efficient pretraining with scalable synthetic latent generation and a simple training method.Source: §2 · pp. 3–4

Latent thought models

  • The latent thought model uses q(Z | X) to decompress thoughts and trains with paired Z, X sequences using standard next-token prediction.Randomly placing Z before or after X allows one LM to model both p(Z, X) and q(Z | X) with minimal pipeline changes.Source: Figure 2 · p. 4, §2 · p. 4

3 Reasoning to Learn with Latent Thought Models

The paper treats web text as compressed human thought and trains language models to infer and learn from latent thoughts alongside observed data. Synthetic latent thoughts support latent-space reasoning and downstream transfer using standard language-model training with minor formatting changes.

3.1 Latent Thought Models

  • Human-written web text compresses underlying knowledge and reasoning, motivating latent-thought augmentation to improve learning efficiency.The paper hypothesizes that models can learn to infer and improve these latent thoughts after training on them.Source: §3.1 · p. 4
  • Modeling latent thoughts Z alongside observed text X trains the joint p(Z, X) and posterior q(Z | X), approximating reasoning-to-learn from compressed data.The latent thoughts use the same discrete text space as observed data and are incorporated through next-token prediction.Source: §3.1 · p. 4

3.2 Training with Synthetic Latent Thoughts

  • GPT-4o-mini can synthesize latent thoughts that supply missing reasoning steps and background knowledge for training.The synthetic thoughts are presented as consistent, clean natural-language text used to augment raw data.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
  • The same autoregressive model trains q(Z | X) and p(Z, X) by placing latent text after or before observed chunks and distinguishing modes with special tokens.This formatting can be fed directly into the standard language-model pretraining pipeline with next-token predictions.Source: §3.2 · p. 6

3.3 Prediction with Latent Thoughts

  • Latent thought models can place few-shot chain-of-thought prompts in latent space Z, where they achieve better downstream performance than reasoning in observed text space X.The approach supports downstream tasks such as MATH through few-shot latent-space prompting.Source: §3.3 · p. 6, Figure 4 · p. 6

4 Bootstrapping Latent Thought Models

BoLT removes dependence on a fixed strong teacher by iteratively improving latent thoughts with Monte Carlo-enhanced EM. More inference samples tighten the training bound and support increasingly capable models, while the algorithm begins with a small synthetic-latent warmstart.

4 Bootstrapping Latent Thought Models

  • The method’s final-model capability is bounded by the surrogate posterior used to synthesize latent thoughts, motivating self-improvement through BoLT.Without bootstrapping, the quality of the surrogate q~(Z | X) limits the achievable performance ceiling.Source: §4 · pp. 6–7
  • BoLT uses an EM loop in which improved latent thoughts at the E-step increase learning efficiency at the M-step, enabling successive model improvement.The procedure bootstraps latent-thought generation rather than relying only on an external surrogate posterior.Source: Figure 5 · p. 7, §4 · p. 7

4.1 Expectation-Maximization with Monte Carlo Sampling

  • Monte Carlo sampling selects latents with importance weights, producing a surrogate posterior that is provably better than the current posterior.The weights favor latents that are simple and predictive while downweighting overly obvious explanations.Source: §4.1 · p. 7
  • Increasing the number of Monte Carlo samples K tightens the evidence lower bound and makes the surrogate posterior approach the true posterior as K →∞.This provides a mechanism for scaling inference compute to select better latents.Source: §4.1 · p. 7

4.2 Full algorithm

  • BoLT starts from warmstart data with synthetic thoughts, then alternates E-step latent synthesis and M-step training across new or fixed corpora.The algorithm can initialize each iteration from the previous model or retrain from scratch on improved latents.Source: §4.1 · pp. 7–8, end · p. 8, §4.2 · p. 8

5 Synthetic Latent Thoughts Improve LM Data Efficiency

Synthetic latent thoughts improve language-model data efficiency in data-constrained continued pretraining, outperforming raw-data and paraphrase baselines. The experiments also show that separating thoughts into a latent space and using them during evaluation is important to the method’s gains.

5.1 Experimental Setup

  • The study fixes 480M unique raw FineMath tokens and an 8B-token compute budget while comparing repetition, fresh raw data, paraphrases, and reasoning-trace paraphrases.Synthetic latent thoughts add approximately 1.1B tokens, with a 2.3:1 synthetic-to-raw token ratio and roughly five epochs of thought-augmented training.Source: §5.1 · pp. 9–10

5.2 Results

  • In a data-constrained setup, synthetic latent thoughts substantially improve TinyLlama-1.1B’s downstream performance during continued pretraining on 480M FineMath tokens with an 8B training-token budget.The method outperforms both 8B unique raw tokens and paraphrased data with synthetic reasoning traces; synthetic methods use approximately 1.1B tokens.Source: Figure 6 · p. 9, §5.2 · p. 10
  • Training with synthetic latent thoughts substantially outperforms raw-data and rephrasing baselines, including Raw-Fresh, while remaining robust to prompt variations.The method achieves this while seeing raw data 3.3x fewer times than baselines; gains are larger when normalized by effective raw tokens.Source: Figure 6 · p. 9, §5.2 · pp. 10–11
  • Separating synthetic thoughts into latent space Z and jointly modeling paired (Z, X) data is critical, because mixing unpaired thoughts with text performs substantially worse and suppressing latent thoughts at evaluation degrades performance.The ablation indicates that gains depend on the latent-space design and on using latent thoughts for downstream reasoning, not merely on synthetic thought quality.Source: Table 1 · p. 10, §5.2 · p. 11

6 LMs Can Self-Improve by Bootstrapping Their Latent Thoughts

BoLT studies whether language models can self-improve by generating increasingly better latent thoughts during fixed-data and continual-training bootstrapping. Across both settings, iterative latent improvement strengthens likelihood and downstream performance, while additional inference compute improves latent quality and a GSM8K side effect limits the scope of gains.

6.1.1 Experimental Setup

  • The fixed-data setup isolates latent quality by retraining on one corpus, whereas continual bootstrapping uses each model to generate latents for the next corpus.Continual training initializes each model from its predecessor, while the fixed-data experiment retrains from scratch on the same corpus.Source: §6 · pp. 11–12
  • The experiments use a 1.1B TinyLlama model with FineMath data, evaluating ELBO or NLL alongside few-shot and finetuned MATH and GSM8K performance.Fixed-data bootstrapping uses 1.92B raw tokens plus a 240M-token warmstart, while continual training uses increasing corpus subsets and a warmup-stable-decay schedule.Source: §6.1.1 · pp. 12–13, §6.2.1 · pp. 15–16

6.1.2 Results

  • Across fixed-data bootstrapping, ELBO and downstream MATH improve through the third iteration, while likelihood gains remain observable through the fourth.Few-shot MATH shows diminishing returns by the fourth iteration, but task-specific finetuning preserves gains through that iteration.Source: Figure 8 · p. 12, Figure 9 · p. 13, §6.1.2 · p. 14
  • BoLT-trained models significantly outperform both FLOP-matched and raw-token-matched baselines on downstream MATH, with gains persisting in MATH and GSM8K finetuning.The FLOP-matched baseline uses approximately 3.4× more corpus passes, while the raw-token-matched baseline uses two epochs over the corpus.Source: §6.1.2 · p. 14
  • Increasing Monte Carlo samples from K = 1, 2, 4, 8 monotonically improves ELBO and downstream MATH accuracy, indicating inference compute can improve pretraining efficiency.The samples are reweighted to select a latent, functioning as a policy improvement operator; Figure 10 reports robustness across GSM8K and prompt variations.Source: §6.1.2 · pp. 14–15, Figure 10 · p. 14
  • Few-shot GSM8K performance deteriorates across later BoLT iterations, although it remains above raw-data baselines and the degradation disappears after task-specific finetuning.The authors suggest that continued training on math-heavy FineMath may worsen general language understanding and word-problem interpretation.Source: §6.1.2 · p. 15

6.2.2 Results

  • In continual bootstrapping, updated latent generators produce steeper likelihood scaling and increasingly outperform fixed-generator baselines on downstream MATH.The comparison spans four iterations and uses three training runs; the performance gap becomes more pronounced at later iterations.Source: Figure 11 · p. 15, §6.2.2 · p. 16

6.2 Continual Bootstrapping

  • BoLT’s continual-learning results support a self-improvement loop in which more capable models generate higher-quality latents that improve subsequent learning efficiency.The continual setup addresses the wastefulness of retraining the latent model from scratch at every iteration.Source: §6.2 · p. 15, §6.2.2 · p. 16

7 Discussion

The discussion positions reasoning to learn as a potentially scalable alternative to task-specific reasoning training, while identifying experimental limits and directions for broader application.

7.1 Broader Implications

  • Reasoning to learn may complement task-specific RL by producing base models whose thoughts transfer to downstream task-specific reinforcement learning.The paper reports promising results in one reasoning-heavy domain and expects likelihood-based supervision to support broader applicability.Source: §7.1 · p. 16
  • Synthetic data generation can shift part of pretraining compute to asynchronous resources and motivate infrastructure for distributed, low-bandwidth workloads.More scalable BoLT sampling, including additional Monte Carlo samples, could provide another axis for asynchronous scaling.Source: §7.1 · pp. 16–17

7.2 Limitations

  • The experiments are constrained to a 1B-parameter model and a few billion mostly mathematical tokens, limiting evidence about larger-scale and general-domain pretraining.These choices are presented as a proof of concept enabled by faster latent-token synthesis and measurable downstream differences.Source: §7.2 · p. 17
  • The study leaves major design choices underexplored, including latent structure, warm-start models or prompts, Monte Carlo sampling, and chunk-size hyperparameters.The current latent generation is autoregressive for each text chunk.Source: §7.2 · p. 17
  • Bootstrapping synthetic data at scale may amplify biases and unintended behavior; GSM8K few-shot CoT performance degraded across BoLT iterations, with an alternative explanation involving mathematical-token optimization.The authors call for broader investigation of bootstrapping side effects.Source: §7.2 · p. 17

7.3 Future Directions

  • Future work may bootstrap general-domain reasoning without task-specific supervision and extend latent inference beyond text to modalities such as video.For complex long-form content, hierarchical latent structures could capture planning beyond the current myopic chunk-level autoregressive design.Source: §7.3 · p. 17

8 Conclusion

The conclusion presents reasoning to learn as a data-efficient pretraining approach that explicitly infers latent thoughts from observed data. Synthetic latents improve learning efficiency, while BoLT enables iterative self-improvement and steeper scaling across math-data bootstrap iterations.

8 Conclusion

  • Reasoning to learn improves data-constrained LM pretraining by deliberately thinking through observed data with explicit latent thoughts.The approach is validated through continued pretraining on mathematical data.Source: §8 · p. 18
  • BoLT uses EM-based self-improvement: stronger models synthesize higher-quality latents, which support greater learning efficiency across bootstrap iterations.The conclusion reports consistent downstream gains and steeper scaling over multiple iterations.Source: §8 · p. 18

A.1 Experimental Details

The WRAP baseline experiments select paraphrase-only training because omitting raw-data mixing gives the best observed performance.

A.1 Experimental Details

  • Ratio=0.0, using only paraphrased data, gives the best performance for both WRAP-Orig and WRAP-CoT.The mixing ratio was selected from {0.0, 0.25, 0.5}.Source: §A.1 · p. 25, Figure A.1 · p. 25

A.2 Additional Results

Additional evaluations show that the method’s gains remain robust across few-shot CoT prompt variants and become larger when performance is normalized by effective raw tokens.

A.2 Additional Results

  • The method maintains consistent gains over baselines across synthetic and standard few-shot CoT prompts, while several baselines degrade under standard prompts.The resulting performance gap is wider with standard prompts, especially for WRAP-CoT.Source: §A.2 · p. 25, Figure A.2 · p. 26
  • Normalizing by effective raw tokens reveals more significant gains over baselines, highlighting the method’s data efficiency.The comparison accounts for different numbers of raw documents processed because methods generate different training-token amounts.Source: §A.2 · p. 27, Figure A.3 · p. 27

B.1 Bootstrapping on Fixed Data

Across fixed-data and continual bootstrapping, latent-thought models improve likelihood-based and downstream performance over iterations, with gains robust across prompts and evaluation settings.

B.1 Bootstrapping on Fixed Data

  • NLL decreases smoothly through the fourth iteration, although raw-data baselines achieve substantially better NLL because they directly optimize that metric.Source: §B.1 · p. 28, Figure B.1 · p. 28
  • MATH gains remain robust across prompt variations, become more significant against raw-data baselines under Minerva CoT, and may plateau after the third iteration.Source: §B.1 · p. 28, Figure B.2 · p. 29
  • Increasing MC samples preserves performance gains across prompt variations and on GSM8K.Source: §B.1 · p. 30, Figure B.4 · p. 30
  • GSM8K few-shot performance can deteriorate across iterations, potentially because increased FineMath optimization reduces natural-language understanding, despite remaining above raw-data baselines.Source: §B.1 · p. 30, Figure B.5 · p. 30
  • Bootstrapping improves ELBO from about -1590 to -1510 nats/sample by substantially improving simplicity while slightly reducing utility and diversity.Simplicity changes from about -3090 to -2920 nats/sample, while utility changes from about -1290 to -1320 and diversity from about 2790 to 2730.Source: §B.1 · p. 31

B.2 Continual Bootstrapping

  • Continual bootstrapping produces steeper likelihood-based scaling and consistent MATH gains than fixed-latent-generator baselines, including on Minerva CoT.Source: §B.2 · p. 32, Figure B.7 · p. 32

C Prompts

This appendix presents prompts for generating latent thoughts, rephrasing documents, and evaluating few-shot chain-of-thought behavior across a range of reasoning tasks. It includes synthetic and standard few-shot examples with explicit intermediate steps and final answers, covering arithmetic, algebra, STEM, statistics, and multiple-choice problems.

C.1 Prompts for Synthetic Data Generation

  • C.1 Prompts for Synthetic Data Generation: Prompt C.1.1 asks GPT-4o-mini to insert latent thoughts between a document prefix and suffix, including missing background knowledge and step-by-step reasoning.The output should use concise declarative language, avoid repeating the prefix, and emphasize implicit reasoning and concrete derivations.Source: §C.1 · p. 34
  • C.1 Prompts for Synthetic Data Generation: Additional prompts rephrase websites using toddler-level, scholarly, Wikipedia-like, or conversational styles, following WRAP prompts.A separate prompt requests high-quality paraphrasing that preserves the original information while inserting detailed reasoning and mathematical or logical derivations.Source: §C.1 · pp. 34–35

C.2 Prompts for Few-Show CoT Evaluation

  • C.2 Prompts for Few-Show CoT Evaluation: Evaluation uses synthetic CoT prompts generated by GPT-4o-mini and standard prompts from prior work, with latent tokens removed for baselines.The standard prompts are Minerva CoT (Lewkowycz et al., 2022) for MATH, default CoT (Wei et al., 2022b) for GSM8K, and FLAN CoT (Wei et al., 2021) for MMLU-STEM.Source: §C.2 · p. 35
  • C.2 Prompts for Few-Show CoT Evaluation: Few-shot examples use Minerva’s questions for MATH, eight randomly selected training problems for GSM8K, and four randomly selected subject problems for MMLU-STEM.The GSM8K selection is intended to better match the test distribution, whereas FLAN uses subject-specific examples for MMLU-STEM.Source: §C.2 · p. 35

C.2.1 Synthetic Few-Shot CoT Prompts

  • C.2.1 Synthetic Few-Shot CoT Prompts: Mathematical examples place a <StartOfLatent><Prior> reasoning trace before the answer, deriving results such as det(AB) = 24 and a/b = −2/3.The traces explicitly state applicable properties, substitute values, factor expressions, and simplify intermediate steps.Source: §C.2.1 · pp. 36–37
  • C.2.1 Synthetic Few-Shot CoT Prompts: Arithmetic and word-problem examples expose calculations before final answers, including 16 lifts for Terrell and 16 chairs for Ben.The examples compute total quantities, formulate equations or divisions, and then mark the final answer after the latent reasoning.Source: §C.2.1 · pp. 36–37
  • Further GSM8K examples cover egg allocation and ship consumption, yielding 10 boys from 36 total eggs with 20 available for boys, and 121 people when 7x = 847.Source: §C.2.1 · p. 38
  • Across the prompt examples, latent reasoning is delimited by <StartOfLatent> and <EndOfLatent>, followed by an explicit final answer.This structure appears in both synthetic and standard few-shot CoT examples.Source: §C.2.1 · pp. 39–41, §C.2.2 · p. 43
  • A step-by-step arithmetic example translates relationships in a word problem into equations, sets equivalent expressions equal, and rearranges them to derive David’s age as 7.Source: §C.2.1 · pp. 39–40
  • Synthetic few-shot CoT prompts also cover STEM multiple-choice problems by applying formulas and deriving answers such as 240 W and -1/256.The examples explicitly use P = V × I and derive x = k × z^6 before selecting the corresponding choices.Source: §C.2.1 · p. 41
  • Additional synthetic prompts illustrate evaluating algebraic expressions and combining independent random variables, yielding option (B) and mean 518 grams with standard deviation 6.1 grams.The statistics example adds means and combines standard deviations using the square-root-of-sums-of-squares formula.Source: §C.2.1 · p. 42

C.2.2 Standard Few-Shot CoT Prompts

  • Standard few-shot CoT prompts are identified as Minerva Few-Shot CoT for MATH (Lewkowycz et al., 2022) and include domain and determinant problems.The examples conclude with the domain [2, 5) and det(AB) = 24.Source: §C.2.2 · p. 43
  • Standard few-shot CoT examples also include proportional reasoning, such as matching total lifted weight with lighter weights to obtain 16 repetitions.The reasoning equates 2 · 12 · 20 with 2 · 15 · n.Source: §C.2.2 · p. 43
  • Further standard prompts demonstrate concise arithmetic chains for counting computers, subtracting golf balls, combining chocolates, and adding toys.Their stated answers are 29, 33, 39, and 9, respectively.Source: §C.2.2 · p. 45
  • A watch-and-packaging problem demonstrates adding independent means and variances to obtain a 518-gram mean and 6.1-gram standard deviation, selecting (C).The latent reasoning computes 62 + 456 = 518 and 1.0^2 + 6.0^2 = 37.Source: §C.2.2 · p. 46
  • An orange-grove problem asks for the response variable when estimating the proportion of trees suffering more than 50 percent frost damage.The prompt samples 100 trees after frost and protective water spraying.Source: §C.2.2 · p. 46
  • The corresponding reasoning identifies each sampled tree’s frost-damage status as the response variable and selects (D).The distinction is between the measured tree-level outcome and aggregate counts or proportions.Source: §C.2.2 · p. 46
  • A random-variable problem asks for E(X + Y) and var(X + Y) given separate means and variances.The prompt supplies E(X) = 37, var(X) = 5, E(Y) = 62, and var(Y) = 12.Source: §C.2.2 · p. 46
  • The latent reasoning notes that covariance is required for the variance of a sum, so the problem has insufficient information and the answer is (D).It also states that means add regardless of independence.Source: §C.2.2 · p. 46
  • A comparison of three sets asks which has the smallest and largest standard deviation: {1,2,3}, {-10,10}, or {100}.The answer choices pair the set with the smallest deviation first and the largest second.Source: §C.2.2 · p. 46
  • The reasoning calculates variances of 2/3, 10^2, and 0, concluding that {100} has the smallest and {-10,10} the largest standard deviation, so the answer is (D).The calculation uses means 2 and 0 for the first two sets, while the singleton set has zero variance.Source: §C.2.2 · p. 46
  • A final statistics question contrasts classes with standard deviations 11.2 and 5.6 and asks for the most reasonable performance conclusion.The visible prompt provides answer choices concerning heterogeneity, homogeneity, relative performance, and test results.Source: §C.2.2 · p. 47

D.1 Examples of Downstream Transfer of Latent Thoughts

The examples show latent thoughts transferred across downstream mathematical tasks, including combinatorics, number theory, algebra, and word problems. These thoughts provide intermediate reasoning that leads to final answers such as 28, 10, 160, and sqrt(10).

Solving algebra problems on MATH

  • sqrt(10) is the distance between the intersections of y = x^2 and x + y = 1 after solving for the two points and applying the distance formula.Both coordinate differences equal sqrt(5), producing sqrt((sqrt(5))^2 + (sqrt(5))^2) = sqrt(10).Source: §D.1 · p. 47

Solving combinatorial problems on MATH

  • 28 assortments are obtained by applying stars and bars to six cookies across three distinguishable types.The derivation models cookie counts as x + y + z = 6 and evaluates 8 choose 2 = 28.Source: §D.1 · pp. 47–48

Solving number theory problems on MATH

  • 10 positive factors are found for 48 by factoring it as 2^4 * 3^1 and applying (e1 + 1)(e2 + 1).The calculation gives (4 + 1)(1 + 1) = 10.Source: §D.1 · p. 48

Downstream transfer examples

  • The examples also include a livestock word problem whose latent reasoning concludes that the total number of legs is 160.The calculation uses two legs per chicken and four legs per cow.Source: §D.1 · p. 49

D.2 Examples of GPT-4o-mini Synthesized Latent Thoughts

This section presents examples of GPT-4o-mini synthesized latent thoughts across PCA, physical explanations, mathematical derivations, and symbolic integration. The examples illustrate how these thoughts reconstruct background knowledge, connect intuition with formal reasoning, and provide explicit, verifiable solution steps.

Latent thoughts recover missing background knowledge and reasoning

  • The synthesized explanation recovers PCA’s full reasoning chain: centering data, forming covariance or correlation matrices, extracting eigenpairs, selecting leading components, and projecting data.It explains that eigenvectors define component directions, eigenvalues quantify captured variance, and projection yields a lower-dimensional representation while retaining essential structure.Source: §D.2 · p. 49
  • The latent reasoning derives PCA’s maximum-variance objective by linking squared projected distances to the covariance matrix’s Rayleigh quotient and its eigenvalues and eigenvectors.It explains that squaring prevents positive and negative deviations from canceling and that the largest eigenvalue identifies the maximum-variance direction.Source: §D.2 · pp. 50–51
  • PC1 captures 83% of the variance while PC2 captures 17%, so PCA prioritizes PC1 for compact representations and can use both components for three-dimensional data.The components are orthogonal, and using PC1 and PC2 is described as preserving underlying structure while reducing noise.Source: §D.2 · p. 51
  • The examples identify PCA’s limitations: principal components can be ambiguous, low-variance components may contain meaningful differences, and non-normal data can reduce effectiveness.These caveats qualify PCA’s dimensionality reduction and information-retention benefits.Source: §D.2 · pp. 51–52
  • When attributes use different units, the synthesized explanation recommends correlation coefficients because standardization makes relationships comparable; with identical units, covariance can be used directly.Correlation divides covariance by the product of standard deviations, removing scale effects from comparisons.Source: §D.2 · pp. 52–53

Example D.2.2: Latent thoughts elaborate physical knowledge

  • The latent thoughts explain frame dragging by connecting spacetime curvature, rotating masses, geodesics, and measurements of gyroscope precession.They emphasize that spacetime does not flow like water and that the analogy represents geometric rather than fluid-dynamic behavior.Source: §D.2 · pp. 53–54
  • The gyroscope explanation attributes periodic maxima in frame-dragging measurements roughly every 1000 days to the closest point of an elliptical orbit, where gravitational influence and precession are greatest.The latent thoughts relate this behavior to orbital characteristics and Kepler's laws.Source: §D.2 · p. 55
  • A second physical explanation argues that water-drain rotation can reflect the Coriolis effect, sink geometry, initial conditions, and angular-momentum conservation rather than purely random currents.It gives the Coriolis force as Fc = 2m(v)(Ω) sin(ϕ) and states that these factors can produce a consistent rotational direction.Source: §D.2 · p. 55

Example D.2.3: Latent thoughts elaborate mathematical derivations

  • The mathematical example derives the integral of tan(5x)tan(3x)tan(2x) by applying tangent identities, substitution, termwise integration, and logarithmic antiderivatives.The synthesized result is 5 log | cos(5x)| + 1 3 log | cos(3x)| −1 2 log | cos(2x)| + C.Source: §D.2 · pp. 55–56

Computation statistics

  • 118 antiderivative size, normalized size 1, five steps, and five rules characterize the compact integration result.The reported Rubi trace also gives time 0.214601 and integrand size 24.Source: §D.2 · pp. 57–58

Symbolic integration

  • The synthesized solution derives a three-term antiderivative combining square-root/polynomial terms with an ArcTanh term after transforming ArcCoth.The transformation uses a hyperbolic-function identity, while variable dependencies in c and a are simplified to isolate x for integration.Source: §D.2 · pp. 57–58

Verification

  • The antiderivative was successfully verified, confirming that the computed expression matches the expected integral behavior.The example notes that differentiation can be used to check correctness.Source: §D.2 · pp. 57–58

D.3 Examples of TinyLlama-1B Self-Generated Latent Thoughts

This section presents examples of TinyLlama-1B’s self-generated latent thoughts for explaining mathematical concepts and solving applied problems. The examples span definitions, recurrences, percentage calculations, and algebraic reasoning, including variations in numerical conclusions across iterations.

Example D.3.1: Models think through the data for learning

  • The model’s latent thoughts explain factorials through product notation, base cases, recurrence relations, applications, and worked values such as 5! = 120.The examples also list factorials from 1! = 1 through 10! = 3628800.Source: §D.3 · pp. 58–59

Example D.3.2: Models learn math problem solving with latent thoughts

  • The model derives percent increase and decrease by subtracting values, dividing by the original value, and multiplying by 100.It applies this procedure to pay, prices, staffing, item costs, and board lengths.Source: §D.3 · p. 60
  • 20%: Ann’s pay rises from $10.00 to $12.00, and the latent reasoning matches the written answer.The calculation uses a $2.00 increase divided by $10.00, then multiplied by 100.Source: §D.3 · p. 60
  • 27.5%: reducing company staff from 40 to 29 produces a percent decrease of 27.5% in the latent reasoning and written answer.The computation takes the absolute difference of 11, divides by 40, and converts 0.275 to a percentage.Source: §D.3 · pp. 60–61
  • The examples report an 8% increase calculated from 6 cents over 75 cents and a 33.3% decrease in board length.The latent reasoning gives approximately 33.33%, while the written answer rounds it to 33.3%.Source: §D.3 · pp. 61–62

Example latent thought

  • Using the equation 2x + 91 = 659 and the substitution y = x + 91, the example concludes that the dryer costs 380 dollars.It states y = 289 + 91 = 380.Source: §D.3 · p. 63

2nd iteration

  • The 2nd-iteration latent thought sets up the same linear relationship but incorrectly concludes that the dryer costs 284 dollars.It computes x = 568 / 2 = 284 and then states that the dryer costs 284 dollars.Source: §D.3 · p. 63

4th iteration

  • The 4th-iteration latent thought correctly models the costs as x + y = 659 and y = x + 91, deriving washer cost 284 and dryer cost 375.The derivation substitutes the relationship into the total-cost equation and then computes y = 284 + 91 = 375.Source: §D.3 · pp. 63–64

Reference answer

  • The source answer independently confirms the correct solution as dryer cost 375 and washer cost 284.The supplied answer explicitly gives x = 284 and y = 284 + 91 = 375.Source: §D.3 · p. 64

D.4 Failure Examples of Bootstrapped Models on GSM8K

The bootstrapped models exhibit failures in interpreting math word problems, carrying out intermediate calculations, and converting reasoning into final answers. Examples include incorrect group-count reasoning, an interest calculation yielding $6, and an average-roll-ups calculation yielding 1.

Failure Examples of Bootstrapped Models on GSM8K

  • These examples are preliminary qualitative observations of side-effects in fourth-iteration bootstrapped models rather than a rigorous analysis.The reported failures include cases that were correctly solved at earlier iterations.Source: §D.4 · p. 65

Failure of bootstrapped models in interpreting the math word problem

  • The models misinterpret a grouping problem, treating 54 students divided by 12 as 4.5 groups and concluding that 4 groups have been formed.They then report that 2 more groups are needed to reach 6 groups.Source: §D.4 · p. 65
  • In an interest problem, the model calculates $2 monthly interest for three months and incorrectly concludes that the total amount owed is $6.The intermediate interest values and final answer are shown explicitly in the generated reasoning.Source: §D.4 · p. 65

Failure of bootstrapped models in converting to the final answer

  • For a fruit-roll-up problem, the model computes totals of 48 and 42, sums them to 90, and then incorrectly divides 90 by 90 to obtain an average of 1 roll-up.The final response repeats the incorrect result as 1.Source: §D.4 · p. 66
Loading 2503.18866v2…