PaperlayerView exact revision on arXiv

Source-linked AI summary

Lasting Effects of Abstract Pretraining Beyond Perplexity

Zachary Shinnick, Hemanth Saratchandran, Damien Teney, Anton van den Hengel

arXiv:2609.38764v1cs.LG

TL;DR

Prior work leaves open which properties of abstract pretraining improve specific capabilities and how training timing affects transfer. The paper tests initial STACK training in small language models and finds capability-specific gains that depend on data structure and timing, despite perplexity not consistently reflecting them.

  • Problem

    Prior studies report benefits from brief abstract-data training, but which data properties, capabilities, and mechanisms drive those benefits remains unclear.

    Source: §1 · p. 1

  • Method

    The paper uses an initial STACK phase before language pretraining and task fine-tuning, then tests capability, data-structure, and timing variations.

    Source: §3 · p. 3, §3.2 · p. 4

  • Results

    STACK improves multi-hop and long-chain reasoning, while a matched queue fails to reproduce the gains and mixing or post-training exposure is less effective than using abstract data first.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Abstract pretraining can shape later capability acquisition, and capability-specific evaluations can reveal effects that perplexity does not consistently capture.

    Source: §1 · p. 1, §7 · p. 9

    The study tests only one type of abstract data and relatively small models, so other data types may have different effects.

    Source: §7 · p. 9

Abstract

from arXiv · show

Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated data can provide a better starting point for subsequent learning of natural language. In this paper, we show that in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity. Our warm-up uses an abstract stack-manipulation task that requires compositional and state-tracking capabilities. Allocating as little as 1% of pretraining tokens to this data improves multi-hop question answering by up to 3.9 F1 points on MUSIQUE, with additional gains on HOTPOTQA and 2WIKIMULTIHOPQA despite comparable language-modeling perplexity. Controlled experiments show that the warm-up substantially accelerates the acquisition of deeper reasoning chains. We also explore what drives this transfer. First, the structure of the data matters: replacing the stack task with a queue fails to produce the same gains. Second, the gains are specific: performance improves on sequential reasoning chains, with no consistent benefit on tasks that combine or compare independent facts. Third, timing matters: mixing abstract data with natural language is far less effective than an initial dedicated phase, and exposure after pretraining completely removes the benefits. The early advantage persists through billions of subsequent language tokens. These results show that early abstract training can reliably shape the capabilities language models later acquire.

1 INTRODUCTION

The paper examines which abstract-data properties and training conditions shape later capability acquisition, focusing on multi-hop reasoning in small language models. It finds that early STACK training improves dependent reasoning and long-chain learning, while effects vary with task structure, evaluation capability, and timing.

1 INTRODUCTION

  • The study uses controlled experiments to test which capabilities benefit, which abstract-data properties drive transfer, and how exposure timing affects it.Evaluations include MUSIQUE, HOTPOTQA, 2WIKIMULTIHOPQA, and the synthetic DEPO task.Source: §1 · pp. 1–2
  • STACK trains next-token prediction over push/pop sequences whose nested operations require tracking a changing memory state.The task’s simple structure lets the authors manipulate the abstract computation to test what transfers.Source: §1 · p. 2
  • STACK pretraining improves multi-hop reasoning on three Q&A benchmarks, especially dependent chains, and DEPO shows faster acquisition of longer chains.Benefits are not consistently coupled with language-modeling perplexity improvements.Source: §1 · p. 2
  • A matched queue task has similar perplexity but does not reproduce STACK’s downstream gains, indicating that transfer depends on the abstract data’s structure.The paper also finds that abstract training throughout or after language pretraining does not yield the same benefits as early exposure.Source: §1 · p. 2
  • The experiments use small models, SmolLM-135M and 360M, and identify capability, abstract-data structure, and pipeline position as factors affecting measurable benefits.The paper notes prior studies of larger models and longer training as complementary evidence.Source: §1 · p. 2

2 RELATED WORK

Prior work reports benefits from brief abstract-data pretraining, but leaves open which data properties and training schedules matter. This paper examines that gap for capability-specific transfer, including multi-hop reasoning.

Pretraining on abstract data

  • Prior work reports faster subsequent training and downstream-capability gains from abstract data, while this paper investigates which data properties and capabilities account for transfer.Reported benefits are not necessarily tied to better pretraining perplexity.Source: §2 · p. 3

Training order in language-model pretraining

  • Earlier studies of abstract data did not establish whether a dedicated initial phase is necessary; this paper compares exposure before, during, and after language pretraining.The comparison also tests whether effects persist through later training.Source: §2 · p. 3

Multi-hop reasoning

  • Multi-hop reasoning requires combining evidence through dependent steps, and the paper tests whether initial abstract training improves it while keeping architecture and fine-tuning protocol fixed.Prior approaches include context repetition, interleaved retrieval and reasoning, and RL fine-tuning.Source: §2 · p. 3

3 METHODS

The experiments compare an initial abstract-training phase with standard language pretraining and target-task fine-tuning, then vary the models, abstract tasks, and evaluations. STACK uses push/pop sequences to train memory-state tracking, while the target evaluations measure multi-hop reasoning at different levels of control.

3 METHODS

  • Abstract training precedes language pretraining and target-task fine-tuning, with the baseline omitting the abstract phase and token budgets matched across experiments.The abstract phase adds about 1% of the pretraining-token budget; perplexity is measured after language pretraining and downstream performance after fine-tuning.Source: §3 · p. 3

3.1 MODELS AND LANGUAGE PRETRAINING

  • The main experiments use SmolLM-135/360M with FINEWEB-EDU and 20 language tokens per parameter; some experiments use TINYSTORIES, and hyperparameters are tuned on the baseline.Hyperparameters are not retuned for abstract-data runs.Source: §3.1 · p. 3

3.2 ABSTRACT DATA

  • STACK predicts final memory contents after random push/pop sequences, while QUEUE and STACKRAND alter the removal order or final output order; other abstract tasks include SORTING, SET, NCA, and formal languages.STACK’s required state tracking motivates its role as the main abstract task.Source: §3.2 · p. 4

3.3 TARGET DATA AND EVALUATION

  • MUSIQUE, HOTPOTQA, and 2WIKIMULTIHOPQA evaluate multi-hop question answering, while DEPO tests controlled chains of one to four hops.The Q&A benchmarks report mean token-level F1; DEPO requires dereferencing ordered node pairs to answer each query.Source: §3.3 · p. 4

4 SPECIFIC IMPROVEMENTS BEYOND LANGUAGE MODELING

Initial STACK training improves multi-hop QA and accelerates learning on long reasoning chains, with gains concentrated in sequential composition rather than general language-modeling performance.

4.1 IMPROVEMENTS IN MULTI-HOP QUESTION ANSWERING

  • 3.9 F1 points is the largest gain, on MUSIQUE with SmolLM-135M; QA gains accompany comparable language-modeling performance at 135M and slightly worse performance at 360M.Mean F1 also improves on HOTPOTQA at both model sizes and on 2WIKIMULTIHOPQA at 135M, while the 360M result on that benchmark is largely unchanged.Source: Figure 2 · p. 5, §4.1 · p. 5

4.2 THE GAINS CONCENTRATE ON SEQUENTIAL COMPOSITIONS

  • STACK-trained models perform consistently better on chain-dominated question types across all three benchmarks, but show no clear effect on merge- and comparison-based types.One MUSIQUE question type shows slight degradation, close to inter-seed variability.Source: Figure 3 · p. 5, §4.2 · p. 5

4.3 FASTER LEARNING ON A CONTROLLED MULTI-HOP TASK

  • On DEPO, STACK-exposed models learn faster at every tested complexity and reach near-perfect accuracy, while language-only models still fail on four-hop queries.The improvement occurs despite similar or slightly worse language-modeling validation loss on both language corpora.Source: §4.3 · p. 6, Figure 4 · p. 6

5 TRANSFER DEPENDS ON TASK STRUCTURE AND PRECISE WEIGHTS

Transfer depends on the abstract task’s structure: STACK outperforms tested alternatives on DEPO, and its benefits cannot be reproduced by matching weight magnitudes alone.

5.1 TRANSFER VARIES ACROSS ABSTRACT TASKS

  • STACK enables the fastest learning to near-perfect DEPO accuracy among the tested abstract-data options, while alternatives learn more slowly or fail on 3- and 4-hop cases.The authors caution that this does not make STACK universally better, since abstract data can induce different capabilities.Source: Figure 5 · p. 6, §5.1 · p. 6
  • Replacing STACK with a queue performs worse than random initialization on QA and DEPO, while STACKRAND performs better than the queue but worse than STACK.STACKRAND randomly permutes the final stack contents while preserving similar surface statistics.Source: Figure 7 · p. 7

5.2 ADJUSTING WEIGHT MAGNITUDES DOES NOT REPRODUCE THE FULL BENEFITS

  • Matching per-tensor weight means and variances or shuffling values retains some DEPO benefit over random initialization, but neither matches intact STACK-trained weights, especially on longer chains.The authors conclude that learned information is more complex than a simple adjustment of weight magnitudes.Source: Figure 8 · p. 7, §5.2 · p. 7

6 EARLY EXPOSURE TO ABSTRACT DATA PRODUCES LASTING GAINS

Initial STACK exposure produces the strongest tested timing benefits: later insertion does not reproduce them, while QA gains persist through billions of language-pretraining tokens.

6.1 INITIAL EXPOSURE PRODUCES THE LARGEST GAINS

  • Initial STACK training produces the largest mean QA gains; insertion during language pretraining yields smaller gains, and post-training exposure provides no mean QA improvement.The timing experiment inserts STACK at 0%, 25%, 50%, 75%, or 100% of a fixed FINEWEB-EDU token budget.Source: §6.1 · p. 8, Figure 9 · p. 8
  • On both TINYSTORIES and FINEWEB-EDU, STACK before language pretraining accelerates four-hop DEPO learning, while STACK afterward lowers final accuracy below the language-only baseline.The comparison matches abstract- and language-training budgets and parameter-transfer protocols.Source: §6.1 · p. 8, Figure 10 · p. 8
  • Adding three further STACK phases improves mean F1 on HOTPOTQA and 2WIKIMULTIHOPQA but reduces it on MUSIQUE, so more abstract training is not necessarily better.Source: §6.1 · p. 8, Figure 11 · p. 8

6.2 BENEFITS PERSIST THROUGHOUT LANGUAGE PRETRAINING

  • STACK-initialized SmolLM-135M checkpoints achieve higher fine-tuned F1 at every evaluated checkpoint, with the advantage continuing through billions of language-pretraining tokens.Checkpoints are compared at matched language-token counts using the same downstream fine-tuning protocol.Source: §6.2 · pp. 8–9, Figure 12 · p. 9

7 DISCUSSION

The discussion finds that STACK’s benefits are strongest when abstract training comes first and are concentrated on sequential reasoning. It also identifies limits in the tested data and model scope, while leaving the mechanism and more efficient alternatives open.

7 DISCUSSION

  • Benefits are largest when abstract data comes first, smaller when mixed with language, and disappear when introduced after language pretraining.The early benefits persist through billions of language tokens.Source: §7 · p. 9
  • STACK’s improvements concentrate on multi-hop reasoning and long sequential chains, while a matched queue task does not produce the same benefits.Future work should identify and align structures shared by abstract and target data.Source: §7 · p. 9
  • The study tests only one type of abstract data and relatively small models, so other abstract tasks may have different effects.The authors note that results complement prior work on larger models and longer training.Source: §7 · p. 9
  • The paper does not explain how abstract training changes model parameters, identifying learned circuits and their effects on later language training as a target for analysis.The authors point to mechanistic analysis of circuits learned during the initial phase.Source: §7 · p. 9
  • Characterizing inductive biases could potentially replace explicit abstract training with new architectures or initializations.The paper frames this as a possible efficiency route, not a demonstrated alternative.Source: §7 · p. 9

A.1 PROCEDURAL TASKS

  • STACK uses push and pop sequences and predicts the final stack from top to bottom, requiring state tracking under last-in-first-out access.Its symbols are arbitrary and the target depends entirely on the operation sequence.Source: §A.1 · p. 13
  • The FIFO control changes pop to remove the oldest item, while the shuffled-output control preserves STACK updates but shuffles the final contents.Both controls use the same input sequences as STACK.Source: §A.1 · p. 13
  • The controlled task suite also includes sequence transformations, formal languages, and neural cellular automata.Examples include REVERSE, DYCK, and cellular automata of increasing complexity.Source: §A.1 · p. 13

A.2 NATURAL-LANGUAGE PROCEDURAL TRAINING

The natural-language procedural phase trains on STACK outputs, then transfers the full checkpoint directly into standard pretraining. Its setup uses a masked output-token loss and a specified 5,000-step optimization schedule.

A.2 NATURAL-LANGUAGE PROCEDURAL TRAINING

  • Each STACK example has 128 input tokens followed by a separator and final stack contents, with language-modeling loss applied only to output tokens.The input and separator are masked from the loss.Source: §A.2 · p. 13
  • The procedural checkpoint transfers directly into standard pretraining with its token embeddings and output head, without resizing or reinitialization.The task uses each architecture’s native vocabulary.Source: §A.2 · p. 13
  • STACK training runs for 5,000 steps with effective batch size 64, using Muon for hidden attention and MLP parameters and auxiliary AdamW for other parameters.Muon uses learning rate 4 × 10−3; auxiliary AdamW uses 10−4, with 250 warm-up steps and cosine decay.Source: §A.2 · p. 13

A.3 CONTROLLED PROCEDURAL TRAINING

The controlled procedural experiments use a small GPT-2 model and transfer its transformer blocks into later training. Standard language-pretraining settings include matched initialization controls and distinct recipes for the SmolLM and controlled experiments.

A.3 CONTROLLED PROCEDURAL TRAINING

  • Controlled procedural experiments use a 4-layer, 4-head, 512-dimensional GPT-2 model with about 12.7M parameters and AdamW.The procedural-task sweep configurations are summarized in Table 2.Source: §A.3 · p. 13, Table 2 · p. 14
  • Procedural input-output tasks train only on output tokens, and transfer reuses transformer blocks while reinitializing vocabulary-specific parameters when required.For direct transfer to DEPO, token embeddings, positional embeddings, and the language-modeling head are freshly initialized.Source: §A.3 · p. 14

B.1 NATURAL-LANGUAGE STANDARD PRETRAINING

  • SmolLM-135M and SmolLM-360M are trained on FINEWEB-EDU with approximately Chinchilla-optimal budgets of 20 tokens per parameter, using the tuned configuration selected on randomly initialized SmolLM-135M for all SmolLM language-pretraining runs.Source: §B.1 · p. 15
  • Random and procedural conditions use the same stage-(2) data, token budget, and optimization settings, while the baseline starts from random initialization.The procedural condition retains the full stage-(1) model, including embeddings and the language-modeling head.Source: §B.1 · p. 15

B.2 CONTROLLED STANDARD PRETRAINING

  • The controlled standard-pretraining setup uses a four-layer GPT-2 model and identical recipes for random and procedural initializations within each corpus.Training targets approximately 20 language tokens per parameter, with batch size 32 sequences and length 2,048.Source: §B.2 · p. 16

B.3 NATURAL-LANGUAGE MULTI-HOP EVALUATION

The evaluation combines natural-language multi-hop question answering with a controlled task that measures reasoning across increasingly long chains. The protocols define benchmark contexts, answer scoring, and the controlled task’s cycle-query structure.

B.3 NATURAL-LANGUAGE MULTI-HOP EVALUATION

  • Natural-language evaluation covers MUSIQUE with gold supporting paragraphs and HOTPOTQA and 2WIKIMULTIHOPQA with provided paragraphs and distractors.Models produce short answers without reasoning traces; final checkpoints are evaluated by greedy decoding using F1 and exact match.Source: §B.3 · p. 16
  • Question types are classified by their longest-path structure: chain-dominated types pass intermediate answers between dependent hops, while merge-dominated types join independent lookups.Examples exceeding 2,048 tokens are excluded, and each checkpoint is fine-tuned with three seeds.Source: §B.3 · p. 16

B.4 CONTROLLED MULTI-HOP EVALUATION

  • DEPO presents a randomly ordered directed cycle over 10 entities and asks queries for entities reached after one to four successive transitions.Evaluation uses 500 examples per hop at each checkpoint evaluation.Source: §B.4 · pp. 16–17

C ADDITIONAL RESULTS

Additional results compare language-modeling loss, abstract-data learning, and downstream F1 under post-language STACK training.

C ADDITIONAL RESULTS

  • STACK-initialized models have higher language-modeling validation loss than randomly initialized models throughout language pretraining.The comparison covers TINYSTORIES and FINEWEB-EDU.Source: Figure 13 · p. 18
  • STACK enables the fastest learning to near-perfect accuracy across all four hop counts on DEPO without intervening language pretraining.Other abstract data types yield slower or less complete learning, particularly on longer chains.Source: Figure 14 · p. 18
  • Table 6 reports a learning-rate sweep for post-language STACK training using SmolLM-135M and three pretraining seeds per configuration.Language-only and initial STACK exposure are included as references.Source: Table 6 · p. 18
Loading 2609.38764v1…