PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

Joseph Lee, Yidi Huang, Dokyoon Kim, Shu Yang, Li Shen

arXiv:2609.04180v1cs.CLcs.AI

TL;DR

The paper addresses how LLMs acquire knowledge during pre-training and whether knowledge representation matters beyond corpus-level properties. It uses controlled continued-pre-training experiments with repeated documents, paraphrases, and auxiliary views, finding that auxiliary views improve understanding and factual recall while changing how knowledge is encoded. The findings support treating diversity as complementary representations around individual knowledge, with scope limited by the study’s domains and training setting.

  • Problem

    Existing work emphasizes corpus characteristics, but the paper identifies a gap in understanding how knowledge should be represented during pre-training.

  • Method

    The paper conducts controlled continued-pre-training experiments on domain documents using repetition, paraphrases, auxiliary views, and contextual or prerequisite knowledge.

  • Results

    Auxiliary views improve understanding and factual recall despite replacing some document repetition, while inducing distinct layer-wise biases and compression in knowledge encoding.

  • Takeaways & Limitations

    Data diversity can be operationalized as complementary views around individual knowledge, offering principles for synthetic pre-training and a possible explanation for why diverse data helps.

  • Takeaways & Limitations

    The findings are based on three domains and a 100-step later-checkpoint experiment, limiting generalizability to pre-training from scratch.

Abstract

from arXiv · show

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and clarify that paraphrasing helps only at smaller batch sizes. Second, holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning, counterintuitively, even for factual recall. Third, the effectiveness of auxiliary views is not contingent on the strength of the teacher model that generates them. Fourth, we identify forms of knowledge, contextual and foundational, that aid learning in the presence of prior knowledge gaps. Finally, we examine how these effects manifest mechanistically via layer-wise biases and compression. Together, our findings suggest that auxiliary representations of knowledge, which arise naturally in large pre-training corpora, are a key factor in the success of pre-training and offer a plausible explanation for why data diversity matters.

1 Introduction

The paper asks how knowledge should be represented during pre-training, arguing that auxiliary views—conceptually diverse reformulations—improve acquisition beyond document repetition. Controlled experiments find benefits for understanding and factual recall, while clarifying when paraphrasing helps.

  • Pre-training research has emphasized corpus properties such as quality and diversity, but the paper asks how knowledge itself should be represented.
  • The experiments study natural-language representation and the surrounding contextual or prerequisite knowledge needed to acquire complex knowledge.
  • Repetition benefits new-knowledge acquisition, while paraphrasing adds benefit mainly at smaller batch sizes.
  • With a fixed token budget, reallocating tokens from document repetition to auxiliary views improves understanding and factual recall while changing layer-wise encoding and compression.
  • Auxiliary views remain effective regardless of the generating teacher model’s strength, and contextual or prerequisite knowledge helps when prior knowledge is missing.
  • The central conjecture is that conceptually diverse explanations, analogies, and reformulations produce more generalizable knowledge encodings and help memorize specific facts.
  • This provides an operational account of data diversity as complementary views around individual knowledge, with implications for synthetic pre-training in low-data domains.

2 Related Works

Prior work shows that knowledge acquisition depends on how information is formulated and repeated, but its evidence is narrow and sometimes conflicting. This paper extends the study to complex knowledge and investigates why reported paraphrasing effects differ.

  • Pre-training studies report conflicting findings about repetition because they vary in corpora, models, and experimental settings.
  • Prior controlled work finds that paraphrased augmentation can raise biographical-fact memorization from 9.7% to 96.6%.
  • Other work finds incremental acquisition and decay when fictional facts are intermittently injected during pre-training.
  • Earlier studies focus only on biographical facts and disagree about paraphrasing, whereas this paper examines complex knowledge and explains the differing results.
  • Continued pre-training remains difficult for domain adaptation; one cited example reports 62.7% factual recall for a 70B model trained on Wiki-style documents.

3 Experimental Setup

The study constructs controlled continued-pre-training experiments using documents from three domains and synthetic reformulations, then measures factual recall and inference with language-model probes. Training mixes isolate repetition, paraphrasing, and auxiliary-view effects under matched token budgets.

  • Objective: The autoregressive objective trains the model by next-token prediction over documents in the corpus.
  • Views: Auxiliary views are defined as diverse manifestations of the same knowledge that add contexts and forms beyond linguistic paraphrase.
  • Views: The experiments distinguish contextual knowledge, which the target knowledge references, from prerequisite knowledge, which it presupposes.
  • Documents: The study represents knowledge with 36 documents: 12 computer-science papers, 12 legal opinions, and 12 medical case reports.
  • Views: Auxiliary views are generated as textbooks, Stack Exchange–style Q&A, and blogs, while prerequisite views are textbooks and contextual views come from cited sources where available.
  • Probes: Factual probes test explicitly stated information, whereas inference probes combine facts to infer information not explicitly stated.
  • Probes: The probe pipeline yields 6,435 factual and 430 inference probes, plus 4,515 factual and 322 inference multiple-choice variants.
  • Training mixes: Source, paraphrase, and auxiliary-view conditions use single-batch injections; token matching reallocates tokens so auxiliary views replace some direct document repetitions.

4 Large Language Models Learn Better with Auxiliary Views

Under a matched token budget, auxiliary views improve inference and factual recall beyond source repetition or paraphrasing, with larger benefits at scale. The gains persist across reduced upsampling, pre-training-faithful settings, human-written views, and another model family, while mechanistic analyses indicate more selective parameter and layer changes.

  • The Benefit of Auxiliary Views: Auxiliary views improve both inference and factual recall when they replace some direct repetitions under a matched token budget.The ordering Para. 9 + Aux. > Para. 9 > Source holds across metrics, although Source learns faster on factual probes initially.
  • The Benefit of Auxiliary Views: Auxiliary views help factual recall even when factual targets are verbatim phrases from the source.This result is attributed to more generalized encoding rather than increased direct memorization.
  • The Benefit of Auxiliary Views: The auxiliary-view advantage widens from 1B through 32B models and is largely consistent across domains.Figure 2 reports gains over both Source and Para. 9 as model size increases.
  • The Benefit of Auxiliary Views: Textbooks, blogs, and Stack Exchange Q&A perform similarly, with a slight factual-recall benefit from mixing view types.The authors note that the distinct contributions of each view type require further study.
  • The Benefit of Auxiliary Views: Auxiliary views outperform despite lower probe-target frequency and coverage than source documents or paraphrases.This weakens lexical-overlap explanations for their performance.
  • Broader Generalization: Auxiliary views retain their advantage when matched upsampling is halved or removed, including in a pre-training-faithful setting.The same ordering is also observed with human-written views and on Qwen-2.5-7B.
  • Mechanistic Analysis: Auxiliary views are associated with less parameter movement than paraphrasing and stronger layer-wise biases in middle, final, and upper-middle regions.These signatures emerge clearly at larger scales, whereas the 1B model shows little auxiliary-view benefit or corresponding layer-wise pattern.

5 Auxiliary Views Do Not Require a Strong Teacher

The paper tests whether auxiliary-view benefits depend on the capability of the generating teacher. Across eleven generator configurations, downstream effectiveness is not tied to teacher strength, while the amount of generated view text correlates with accuracy.

  • Teacher Strength: Auxiliary-view effectiveness is not contingent on teacher-model strength across eleven generator configurations.The configurations span multiple model families, reasoning-effort levels, and sizes while holding the training schedule and token budget fixed.
  • Teacher Strength: Generated view-text volume correlates with downstream accuracy, with r = +0.62 and p ≈0.04.Reasoning effort changes the generator’s own accuracy but leaves downstream performance essentially unchanged.
  • Teacher Strength: The findings suggest auxiliary views function as general data augmentation rather than distillation from a strong teacher.The generator need only reformulate the provided text, including in complex domains.

6 When Does Paraphrasing Help?

Paraphrasing helps knowledge acquisition conditionally rather than uniformly. Its benefits are strongest at smaller batch sizes and larger model scales, while larger batches can suppress the same overfitting that paraphrasing prevents.

  • Inference Acquisition: Paraphrasing prevents the collapse that follows Source saturation after roughly 20 exposures, sustaining improvement through roughly 40 exposures.Source sharply degrades after saturation, whereas Para. 9 continues improving.
  • Inference Acquisition: Paraphrasing benefits emerge at 7B and above but are slightly harmful at 1B.This scale dependence appears in the inference experiments.
  • Batch-Size Dependence: As batch size grows, Source nearly matches Para. 9 by batch size 256 while Para. 9 remains roughly flat.The authors interpret the interventions as redundant because both prevent degeneration associated with overfitting.
  • Factual Acquisition: For factual acquisition, paraphrasing helps at small batch sizes but reverses at batch size 256 for 7B and 13B models.The proposed explanation is that larger batches dilute the gradient signal from paraphrased text through increased general-data mixing.
  • Factual Acquisition: At batch size 1,024, paraphrasing provides no consistent improvement over Source, while auxiliary views retain a substantial advantage.This result supports a distinction between paraphrase benefits and auxiliary-view benefits.

7 Prior Knowledge Matters

The study examines how contextual and prerequisite knowledge affect acquisition when models lack foundational knowledge. Contextual knowledge produces larger factual gains, while prerequisite knowledge produces greater inference improvements, though lexical overlap partly explains the factual advantage.

  • Prior-Knowledge Gap: The prior-knowledge gap is measured with prerequisite-topic MCQA before and after continued pre-training on synthetic textbooks.The comparison is labeled Base versus CPT in Table 6.
  • Complementary Benefits: Providing contextual or prerequisite knowledge substantially improves acquisition relative to standard paraphrase training.Under token-matched conditions, surrounding knowledge does not consistently outperform paraphrases but remains comparably beneficial.
  • Complementary Benefits: Contextual knowledge drives larger factual gains across both domains, whereas prerequisite knowledge yields greater improvements in inference.These knowledge types therefore exhibit distinct strengths rather than identical effects.
  • Complementary Benefits: Lexical overlap may partly explain contextual knowledge’s factual advantage, but it cannot explain the inference advantage of prerequisite knowledge.Cited works contain factual targets about twice as often as prerequisite textbooks, while prerequisite materials contain inference targets no more often than contextual materials.

8 Ablations

The ablations show that the fixed learning-rate choice and the ordering of prerequisite data affect how results should be interpreted. The study leaves curriculum effects unresolved and may underestimate auxiliary-view benefits under its learning-rate setting.

  • Learning Rate: Using a fixed peak learning rate of 4e-5 may underestimate model-size effects and the overall effects of auxiliary views and paraphrasing.The 13B and 32B base models used higher pre-training rates, 9e-5 and 6e-5, than the 7B model’s 3e-5.
  • Learning Rate: Auxiliary-view and paraphrasing advantages over source-only training widen as the learning rate increases.This pattern is reported in the analysis corresponding to Figure 12.
  • Order of Prior Knowledge: No placement of prerequisite data at the beginning, middle, or end of training is consistently best across metrics.The observed differences are small, and fuller curriculum investigation is left to future work.

9 Discussion & Conclusion

The discussion frames auxiliary views as a way to operationalize data diversity around individual knowledge rather than only at the corpus level. Conceptual reformulations improve learning and factual recall while revealing scale-dependent encoding effects, but their value remains uncertain for specialized long-tailed knowledge.

  • Auxiliary Views: Auxiliary views are presented as explanations, analogies, and reformulations that provide conceptually diverse representations of the same knowledge.The study treats this as a more operational account of what diverse data can mean in pre-training.
  • Discussion: Conceptual reformulations substantially improve learning and induce distinct layer-wise bias and compression, unlike mere paraphrasing.The reformulations include forms such as textbooks and blogs.
  • Discussion: Auxiliary views improve factual recall even when the original document appears less frequently, linking broader conceptual understanding with memorization of specific facts.The paper presents this as an underlying effect of conceptual reformulation.
  • Discussion: Larger models integrate diverse views more effectively, encode them with greater parameter efficiency, and shift learning toward middle and final layers.The ability to encode auxiliary views emerges only in the larger models studied and grows with model size.
  • Practical Takeaways: For domain adaptation, the paper recommends prerequisite pre-training, paraphrasing when data is scarce, and synthetic augmentation with auxiliary views.The paraphrasing recommendation includes the qualification that its benefit diminishes as batch size grows.
  • Limitations: Whether auxiliary views remain effective for specialized, long-tailed knowledge is unresolved when models may lack expertise to generate high-quality views.The paper identifies this as an open question, particularly for scientific domains.
  • Limitations: The study’s scope may limit the generalizability of its insights.The authors frame this as a scope boundary for applying the findings beyond the study.

Limitations

The study’s limitations concern domain coverage, experimental setting, generator applicability, and model scale, restricting how broadly its findings should be generalized.

  • Scope: The analysis covers only three domains, and secondary findings on contextual and prerequisite knowledge are not consistent across them.The authors explicitly state that this limits generalization.
  • Scope: The contextual-knowledge experiment excludes medical case reports because they lack an analogous citation structure.Its results show different factual and inference biases, but no consistent advantage over token-matched paraphrasing.
  • Experimental setting: The pre-training-faithful experiment uses 100 steps at a later checkpoint rather than pre-training from scratch.The setup preserves the original optimizer state, learning-rate schedule, and batch size, but may not generalize to pre-training proper.
  • Generator applicability: Teacher-strength conclusions may not generalize to low-resource, highly specialized domains where generators struggle to comprehend source material.The authors leave teacher-model irrelevance unresolved in those settings.
  • Model scale: The studied models reach only 32B parameters, so the findings may not extend to substantially larger models.The authors identify model scale as a primary determinant of LLM capabilities.

A.1 Dataset Construction

The dataset combines manually collected arXiv papers, medical case reports, and legal opinions, then filters, verifies, and cleans them into suitable source documents and contextual materials.

  • Document collection: The corpus includes manually collected arXiv papers, PubMed Central case reports retrieved through NCBI E-utilities, and U.S. federal appellate opinions retrieved through CourtListener.Candidate documents were filtered to twelve target documents of suitable length across domains.
  • Corpus verification: All 36 source documents had zero title or sampled-sentence matches in the checked OLMo-2 pre-training mixture index.The check used each title and ten randomly sampled body sentences through the Infini-gram API.
  • Cleaning: ArXiv LaTeX, medical JATS XML, and legal PDF/OCR materials received domain-specific cleaning before modeling.Cleaning removes presentation artifacts and repairs formatting while retaining relevant document structure.
  • Contextual sources: Contextual materials consist of cited arXiv papers and cited judicial opinions collected through Semantic Scholar, OpenAlex, and CourtListener links.Retrieved sources undergo the corresponding arXiv or legal-opinion cleaning procedures.

A.2 Synthetic Data Generation

The study generates paraphrases, auxiliary views, probes, and knowledge-context materials, then evaluates them across domains, model sizes, training settings, and metrics.

  • Synthetic data: The pipeline generates 49 paraphrases per source document and counts auxiliary views as posts, questions, or textbook chapters.The auxiliary-view count N is based on these split generated units.
  • Paraphrases: Paraphrases preserve domain-specific content, formatting, terminology, and factual details through tailored prompts.Prompts differ for academic, legal, and medical documents and avoid paraphrasing LaTeX-only paragraphs.
  • Auxiliary views: Auxiliary views restate, explain, or pedagogically reorganize document content as textbooks, Stack Exchange-style questions and answers, and technical blogs.These views target different audiences while remaining conditioned on the source document.
  • Prerequisite views: Prerequisite views teach foundations needed to understand a source while excluding its novel ideas and case-specific reasoning.The generation process uses curriculum-style prompts and, for legal material, may include landmark or doctrinally foundational opinions.
  • Evaluation: Under a fixed token budget, auxiliary views improve factual recall and inference, with larger effects as model size increases.The pattern generally holds across domains, except for factual probes on medical documents.
  • Evaluation: The experiments evaluate view families, contextual and prerequisite knowledge, placement, generators, model families, and learning-rate effects using log probability and MCQA metrics.Reported comparisons include mixed view families, token-matched contextual data, Qwen-2.5-7B, generator correlations, and peak learning rates of 2e-5, 4e-5, and 8e-5.
Loading 2609.04180v1…