PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic continued pretraining

Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, Tatsunori Hashimoto

arXiv:2409.07431v2cs.LGcs.AIcs.CLstat.ML

TL;DR

Small specialized corpora pose a challenge for data-inefficient language-model learning. The paper uses EntiGraph to synthesize diverse training text from a small corpus and then continue pretraining on it. The approach scales on closed-book QA and its acquired knowledge composes with instruction tuning and retrieval.

  • Problem

    Learning knowledge from small specialized corpora is difficult when facts appear rarely, while language models may require many diverse representations per fact.

    Source: §1 · pp. 1–2

  • Method

    Synthetic continued pretraining uses EntiGraph to extract salient entities, generate text about their relations, and continue pretraining on the resulting corpus.

    Source: §1 · p. 2, §2.2 · p. 5, §7.3 · p. 14

  • Results

    EntiGraph CPT shows consistent closed-book QA scaling through a 455M-token synthetic corpus, outperforming raw-corpus and paraphrase baselines, with knowledge that composes with instruction tuning and retrieval.

    Source: §1 · p. 3, §7.3 · p. 14

  • Takeaways & Limitations

    The results show that synthetic continued pretraining can extract more knowledge from small corpora and make the acquired knowledge usable alongside retrieved knowledge.

    Source: §7.2 · p. 13, §7.3 · p. 14

    EntiGraph relies on the assumption that its prompted language model generates faithful synthetic data; fact-checking covered only a subset of QuALITY data.

    Source: §7.1 · p. 13

Abstract

from arXiv · show

Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.

1 INTRODUCTION

The paper targets data-inefficient learning from small, specialized corpora and proposes synthetic continued pretraining to turn them into diverse training data. EntiGraph generates text by connecting extracted entities; experiments report scaling and broader instruction-following benefits.

1 INTRODUCTION

  • Small, specialized corpora pose a data-efficiency challenge because facts may appear rarely, while language models can require hundreds to thousands of diverse representations per fact.The paper frames niche and private data as increasingly important as publicly available data is exhausted.Source: §1 · p. 1
  • Synthetic continued pretraining converts a small domain corpus into a larger corpus with diverse knowledge representations before continuing pretraining.The paper instantiates this approach with EntiGraph, which extracts entities and prompts a language model to describe their relations.Source: §1 · p. 2, Figure 1 · p. 2
  • Repeated paraphrasing alone does not provide enough diversity: its incremental value quickly decreases and downstream performance scales poorly.The paper contrasts paraphrases with deeper analysis and applications, such as those found in lecture notes and discussions.Source: §1 · p. 2
  • EntiGraph CPT accuracy scales log-linearly up to 455M synthetic tokens, and 455M tokens provide 80% of the accuracy improvement from source access at inference.Instruction tuning also enables open-ended instructions related to the QuALITY books.Source: §1 · pp. 2–3

1.1 RELATED WORK

  • Prior continued-pretraining work adapts models using massive broad-domain corpora, whereas this work targets small specialized domains; related synthetic-data approaches differ in their focus on generic knowledge of a small corpus.The paper distinguishes its goal from task-specific QA generation and from knowledge editing on atomic facts.Source: §1.1 · pp. 3–4

2 OUR METHOD

Synthetic continued pretraining first generates a larger corpus from a small source corpus, then continues pretraining on the generated data. EntiGraph uses extracted entities and their relations to build diverse text grounded in the source documents.

2.1 PROBLEM SETUP

  • Synthetic continued pretraining applies a generation algorithm to a small source corpus and continues pretraining on the resulting synthetic corpus instead of the source corpus.The method conditions generation on source documents to improve faithfulness.Source: §2.1 · p. 4
  • The evaluation tests whether a model trained on synthetic data can answer unambiguous queries about the source corpus without accessing it at test time.The paired evaluation inputs are a source corpus and related test queries.Source: §2.1 · pp. 4–5

2.2 ENTIGRAPH

  • EntiGraph prompts a language model to extract salient entities, then analyze relations among subsets of those entities in the source-document context.This hierarchical strategy uses a graph relating entities to structure the generation of diverse knowledge representations.Source: §2.2 · p. 5
  • EntiGraph generates relation descriptions for entity pairs and triplets rather than exhaustively enumerating all subsets of entities.The resulting synthetic texts are collected as the EntiGraph corpus.Source: §2.2 · pp. 5–6

3 EXPERIMENT SETUP

The experiments evaluate parametric knowledge acquisition using QuALITY documents and related questions, with closed-book testing to prevent access to the source texts. They also assess instruction-tuned summarization and establish that the corpus is niche relative to strong API-based models.

3 EXPERIMENT SETUP

  • The main evaluation tests whether models answer source-related queries without access to source documents, while an open-book setup tests composition with retrieval.The open-book setting provides access to the source documents while answering queries.Source: §3 · p. 6
  • The QuALITY source corpus contains 265 articles and short books across genres, averaging about 5,000 tokens each.The dataset includes science fiction and journalism.Source: §3 · p. 6
  • The 4,609 QuALITY test queries are contextualized with article title and author so they remain unambiguous without the source text.The queries are drawn from the 10–20 multiple-choice questions accompanying each article.Source: §3 · p. 6
  • The instruction-tuned models are evaluated on closed-book summarization by prompting them with only an article title and author.The paper also describes using automated measures of hallucination and salient-claim coverage for summarization.Source: §3 · p. 6
  • 44.81% (GPT-3.5 closed-book) and 51.30% (GPT-4 closed-book) compare with 72.60% and 86.09%, respectively, when the source is available.The authors use the roughly 30% improvement with source access to conclude that QuALITY is sufficiently niche for their testbed.Source: §3 · p. 6

4 MAIN EXPERIMENTS

EntiGraph continued pretraining improves closed-book QuALITY accuracy and scales with synthetic-token count through 455M tokens, while Raw CPT and paraphrase baselines fare worse. The acquired knowledge also supports instruction-following tasks without test-time access to the source articles.

4.1 CONTINUED PRETRAINING PROCEDURE

  • The main experiment trains Llama 3 8B on a 455M-token EntiGraph corpus generated from 1.3M QuALITY tokens, with Raw CPT and Rephrase CPT as baselines.EntiGraph CPT uses two epochs with RedPajama replay; the rephrase corpus generation stopped at 38M tokens.Source: §4 · p. 6, §4.1 · p. 7

4.2 QUESTION-ANSWERING EVALUATIONS

  • 56.22% (EntiGraph CPT) versus 39.49% (Llama 3 8B Base) closed-book QA accuracy accompanies log-linear scaling through 455M synthetic tokens.Figure 2 plots QuALITY accuracy against synthetic-token count.Source: Figure 2 · p. 7, §4.2 · p. 8
  • Raw CPT performs worse than the base model, while Rephrase CPT scales poorly relative to EntiGraph CPT.The authors suggest that limited diversity in raw and paraphrased data may constrain acquisition; EntiGraph uses entity-graph relations to structure diversity.Source: §4.2 · p. 8

4.3 INSTRUCTION FOLLOWING EVALUATIONS

  • EntiGraph CPT is compatible with instruction tuning, and EntiGraph Instruct can use acquired knowledge for instruction following without test-time access to the books.Examples include summarizing a cited article, responding without an explicit article reference, and comparing knowledge across two articles.Source: §4.3 · p. 8
  • The paper evaluates closed-book summaries against human summaries using GPT-4 to count false claims and true claims salient to the article.Those counts are normalized by the corresponding counts from human summaries; the authors note that common similarity metrics may not accurately reflect abstractive quality.Source: §4.3 · pp. 8–9

5 OPEN-BOOK EXPERIMENTS

The open-book experiments test whether EntiGraph continued pretraining adds parametric knowledge to retrieval-augmented generation. EntiGraph CPT complements RAG, and CPT alone achieves more than 80% of RAG’s absolute accuracy improvement in this small-corpus setting.

5 OPEN-BOOK EXPERIMENTS

  • The experiments ask whether EntiGraph’s parametric knowledge complements the non-parametric knowledge accessed through RAG.The RAG pipeline indexes document chunks offline, then retrieves and reranks chunks for few-shot prompting at inference.Source: §5 · p. 9
  • EntiGraph CPT outperforms the base model in a RAG pipeline, supporting complementarity between internally learned and retrieved knowledge.The experiments motivate a small-corpus QA recipe combining synthetic data augmentation, continued pretraining, and RAG.Source: §5 · p. 9
  • The evaluation compares QuALITY question-answering accuracy and Recall@8 in open-book RAG, with oracle RAG providing a perfect-retriever upper bound.Recall@8 measures whether the salient article appears among the top eight reranked document chunks.Source: Table 3 · p. 10
  • EntiGraph CPT provides >80% of RAG’s absolute performance improvement, even where RAG recall is nearly perfect.Accuracy rises from 39.49% to 56.22% with EntiGraph CPT, versus 39.49% to 60.35% with RAG.Source: §5 · p. 10

6 THEORETICAL ANALYSIS OF ENTIGRAPH SCALING

The analysis models EntiGraph as rearranging source relations into a form more amenable to learning, then studies the resulting link-density scaling. Its branching-process approximation yields a mixture-of-exponentials trend and three growth regimes.

6 THEORETICAL ANALYSIS OF ENTIGRAPH SCALING

  • EntiGraph is modeled as rearranging existing knowledge, enabling relations such as A–C to become learnable from source relations A–B and B–C.The model predicts mixture-of-exponential scaling, which fits the observed EntiGraph CPT trend.Source: §6 · p. 10

6.1 TOY MODEL SETUP

  • The toy model represents known relations as a binary matrix and models training as memorizing observed source-document relations without meaningful generalization.Synthetic generation samples entity pairs and uses breadth-first search over the source graph to find connecting paths and add relations.Source: §6.1 · pp. 10–11

6.2 RIGOROUS UPPER AND LOWER BOUND

  • The analysis derives rigorous upper and lower bounds for link-density scaling, while noting that these bounds do not establish that accuracy itself grows exponentially.A Poisson branching-process approximation is used to obtain a more precise growth formula.Source: §6.2 · p. 11, §6.3 · p. 11

6.3 AN ANALYTICAL FORMULA

  • The mixture-of-exponential formula closely fits EntiGraph CPT accuracy scaling through 455M synthetic tokens.The model’s limiting link density is governed by the proportion of reachable vertex pairs in the initial relation matrix.Source: §6.3 · p. 12, Figure 4 · p. 12
  • The mixture reflects different relation-specific growth rates, associated with vertex distance from the search root and the number of vertices at each distance.Pairs closer to the root are more likely to be added during each iteration than pairs farther away.Source: §6.3 · p. 12
  • The analytical model predicts fast growth, slower growth, and a plateau, with the phases scaling as Θ(p + t), Θ(log t), and Θ(1).The choice of log t for the middle phase is not canonical; the bound also permits well-behaved monotone increasing concave functions.Source: §6.3 · pp. 12–13

7 DISCUSSION

The discussion identifies limits in EntiGraph’s reliance on a prompted model and points to broader uses of synthetic continued pretraining. It concludes that EntiGraph scales closed-book QA on small corpora and that its acquired knowledge can compose with instruction tuning and retrieval.

7.1 LIMITATIONS

  • EntiGraph’s synthetic-data generation depends on a prompted language model and may fabricate nonexistent entity relations, with greater hallucination risk possible on complex research papers.Manual checks of a few QuALITY books and a subset of generated text found no factual errors; the authors attribute this tentatively to using gpt-4-turbo.Source: §7.1 · p. 13
  • The approach does not yet bootstrap synthetic data generation with a model generating data for its own small target domain.The authors note that gains cannot be explained by distilling the prompted model’s knowledge alone, since their closed-book results exceed GPT-4’s.Source: §7.1 · p. 13

7.2 FUTURE DIRECTIONS

  • Synthetic continued pretraining with EntiGraph extracts more knowledge from small corpora, potentially supporting learning from proprietary data or tail knowledge appearing only once or twice online.Whether such methods improve data efficiency on standard pretraining data without a stronger prompted model remains open.Source: §7.2 · p. 13
  • For repeated long prefixes shared across queries, continuing pretraining on the prefix could internalize its knowledge and leave shorter queries for standard attention.This trades a fixed training cost for shorter contexts; the authors suggest synthetic continued pretraining could make this practical with as little as 1.3M tokens.Source: §7.2 · p. 13

7.3 CONCLUSION

  • The paper adapts continued pretraining from broad domains using 10B–100B+ tokens to specialized corpora of about 1M tokens by synthesizing diverse representations and training on them.Source: §7.3 · p. 14
  • EntiGraph’s closed-book QA performance scales through a 455M-token synthetic corpus, while raw-corpus continued pretraining and synthetic paraphrases show no improvement or asymptote early.The authors also report that acquired knowledge composes with instruction tuning and retrieved knowledge in an open-book setting.Source: §7.3 · p. 14
  • The authors present a simplified mathematical model whose derived scaling form closely matches the empirical trend, and hypothesize that knowledge-graph externalization may help synthesize diverse data.Source: §7.3 · p. 14

CODEBASE, DATASET, AND MODEL WEIGHTS

The paper provides its codebase, 455M-token EntiGraph corpus, and EntiGraph continued-pretraining model weights. Its additional related-work discussion covers synthetic-data generation and continual learning, including continued pretraining.

CODEBASE, DATASET, AND MODEL WEIGHTS

A ADDITIONAL RELATED WORK

B DETAILS ON THE QUALITY DATASET

For each QuALITY book, the appendix describes entity extraction and relation analysis, and reports corpus statistics in Figure 6. The figure summarizes article tokens, extracted entities, and synthetic-data tokens across 265 articles and books.

B DETAILS ON THE QUALITY DATASET

  • For each book, the authors extract entities, analyze all pairwise entity relations and a subset of triplet relations, and report raw and EntiGraph corpus statistics.Source: B DETAILS ON THE QUALITY DATASET · p. 28
  • Figure 6 presents histograms for 265 QuALITY articles and books: raw-article token counts, extracted-entity counts, and EntiGraph synthetic-data token counts per book.Source: Figure 6 · p. 28

C TRAINING DETAILS FOR THE MAIN EXPERIMENTS

The appendix specifies continued-pretraining, replay, raw-corpus training, instruction-tuning, and compute settings. It reports a single-node throughput and an estimated runtime for the main EntiGraph run.

C TRAINING DETAILS FOR THE MAIN EXPERIMENTS

  • Continued pretraining uses Llama 3 8B Base with context length 2048, batch size 16, full-parameter FSDP training, and peak learning rate 5e-6.The learning rate warms up linearly for 5% of steps, then decays with a cosine schedule.Source: C TRAINING DETAILS FOR THE MAIN… · p. 28
  • EntiGraph continued pretraining uses 1B RedPajama tokens for replay at a 0.1 rate, replacing synthetic data in 10% of batches.Source: C TRAINING DETAILS FOR THE MAIN… · p. 29
  • Raw CPT uses 4 epochs and a 0.1 RedPajama replay rate, selected by validation accuracy tuning of epochs and replay rate.Source: C TRAINING DETAILS FOR THE MAIN… · p. 29
  • Instruction tuning uses a 250M-token UltraChat dataset for one epoch, and the tuned model’s AlpacaEval win rate against GPT-4 rises from 0% to 6.25%.The passage gives a 7.7% baseline win rate for Llama 2 Chat 13B.Source: C TRAINING DETAILS FOR THE MAIN… · p. 29
  • Continued-pretraining experiments run on one 8×H100 node at 6090 tokens per second; the 455M-token EntiGraph run uses two epochs and is estimated at about 41 hours.Source: C TRAINING DETAILS FOR THE MAIN… · p. 29

D TASK-SPECIFIC FINETUNING FOR QUALITY QUESTION SET

The section compares task-specific QA fine-tuning with EntiGraph’s task-agnostic continued pretraining. EntiGraph reaches similar QA performance with lower dataset-generation cost, while supporting broader multitask goals.

D TASK-SPECIFIC FINETUNING FOR QUALITY QUESTION SET

  • The authors limit comparisons to task-specific fine-tuning because EntiGraph targets more general multitask goals, and report preliminary results favoring EntiGraph’s scaling and generation costs.They note that task-specific fine-tuning can perform extremely well in-domain but may degrade outside its fine-tuning distribution.Source: D TASK-SPECIFIC FINETUNING FOR QUALITY… · p. 29
  • QA SFT directly generates question-answer pairs and continues pretraining Llama 3 8B on 28M synthetic tokens.Its generation required over $5k because the input-to-output token ratio was large, making further scaling prohibitively expensive.Source: D TASK-SPECIFIC FINETUNING FOR QUALITY… · pp. 29–30
  • EntiGraph attains similar QA performance to QA SFT while remaining task-agnostic, and its dataset-generation cost is much lower.Figure 7 plots accuracy against synthetic token count; the comparison is based on tokens rather than dollars.Source: D TASK-SPECIFIC FINETUNING FOR QUALITY… · p. 30, Figure 7 · p. 30

E ADDITIONAL DETAILS ON OPEN-BOOK EXPERIMENTS

The open-book setup indexes the QuALITY documents and retrieves and reranks chunks for inference-time prompts. The authors tune retrieval settings on a validation split.

E ADDITIONAL DETAILS ON OPEN-BOOK EXPERIMENTS

E.1 STAGE 1: OFFLINE INDEXING

  • Documents are split into non-overlapping chunks that preserve paragraph, sentence, and word boundaries where possible, with title, author, and year prepended.The chunks are embedded with text-embedding-3-large and indexed as embedding–chunk pairs in FAISS.Source: §E.1 · pp. 30–31

E.2 STAGE 2: INFERENCE-TIME RETRIEVAL AND RERANKING

  • At inference, the query includes the article title, author, and four answer choices; the system retrieves K chunks and reranks them to select k < K.The retriever uses approximate nearest-neighbor search, and the reranker chooses the most semantically relevant chunks for the prompt.Source: §E.2 · p. 31

E.3 HYPERPARAMETER TUNING

  • The study fixes K = 128 and tunes the number of reranked chunks and other RAG hyperparameters separately for each model and pipeline using validation accuracy.The compared models are EntiGraph CPT and Llama 3 8B Base.Source: §E.3 · p. 31

F PROOF OF THEOREM 1 AND OTHER ANALYTICAL FORMULAS

This appendix section develops analytical and evaluation details for EntiGraph, including a proof and behavioral approximations alongside QA filtering, summary scoring, and illustrative examples.

F PROOF OF THEOREM 1 AND OTHER ANALYTICAL FORMULAS

  • With high probability, the reachable-pair count is bounded within ε of (1 − ρ(λ))^2V(V − 1).The upper and lower bounds are (1 − ρ(λ))^2(1 + ε/3)V(V − 1) and (1 − ρ(λ))^2(1 − ε)V(V − 1), respectively.Source: F PROOF OF THEOREM 1 AND OTHER… · p. 33
  • The appendix’s analysis combines reachable-pair bounds with branching-process approximations, while its empirical curve fit uses EntiGraph token count to model QuALITY QA accuracy.Source: F PROOF OF THEOREM 1 AND OTHER… · pp. 33–34, §F.1 · p. 35
  • For Erdős–Rényi graphs with p = λ/V, a Poisson(λ) branching process approximates cluster growth and supports an estimate of Acc(Mt).The analysis uses a Galton–Watson tree to approximate breadth-first exploration from a vertex.Source: F PROOF OF THEOREM 1 AND OTHER… · p. 34
  • For a general source graph, the formula can use the reachable-pair proportion C and the distribution μ of edge-inclusion probabilities, but μ is difficult to compute explicitly.Source: F PROOF OF THEOREM 1 AND OTHER… · p. 34
  • The mixture-of-exponentials function grows as Θ(p + t) for 0 ≤ t ≤ t1, Θ(log t) for t1 ≤ t ≤ t2, and Θ(1) for t ≥ t2.This qualitative result assumes fixed 0 < C < 1 and a probability measure μ on Z+ with finite mean.Source: F PROOF OF THEOREM 1 AND OTHER… · p. 34

F.1 CURVE FITTING WITH MIXTURE OF EXPONENTIAL FORMULA

  • The curve-fitting procedure approximates the infinite mixture sum with three terms and fits QuALITY QA accuracy against EntiGraph token count using nonlinear least squares.The fitted formula’s variables are x, the EntiGraph token count in millions, and y(x), QuALITY QA accuracy.Source: §F.1 · p. 35

H.1 QUALITY QA QUESTION SET

  • The QA examples ask for Big Brother as the leader in “1984” and Captain Ahab as the whale-hunting captain in “Moby-Dick.”Source: §H.1 · p. 40
  • QA evaluation samples 64 responses, keeps those matching the required answer format, and randomly selects among valid answers rather than using majority voting.The passage identifies this as different from self-consistency prompting (Wang et al., 2023a).Source: §H.1 · p. 40

H.2 CLOSED-BOOK SUMMARIZATION

  • Automated summary scoring breaks summaries into claims, judges their truth and salience against the source article, then normalizes false and salient claim counts by human-summary counts.The paper reports averages of these normalized metrics across QuALITY articles in Figure 3.Source: §H.2 · p. 40
  • For “Cosmic Yo-Yo,” EntiGraph Instruct describes Bob and Queazy’s asteroid-delivery race, while Raw Instruct invents a time-travel plot about Susan.EntiGraph’s summary also mentions the rivals and a yo-yo-like device; Raw Instruct instead presents the work as a novel about travel between planets.Source: §H.2 · pp. 41–42
  • For “Beach Scene,” EntiGraph Instruct summarizes Purnie’s time-stopping story, while Raw Instruct describes a painting.The paper explicitly notes that EntiGraph correctly summarizes the story while Raw Instruct mistakes the book for a painting.Source: §H.2 · pp. 42–43
  • For “The Madman From Earth,” EntiGraph Instruct’s example summary describes Retief investigating a conspiracy surrounding a missing cruiser and crew.It also characterizes the story as a satire of diplomacy and bureaucracy.Source: §H.2 · p. 43

H.3 ADDITIONAL QUANTITATIVE SUMMARY EVALUATION

EntiGraph summaries had fewer false claims than Rephrase while retaining comparable salient-claim counts, and scaling EntiGraph from 29M tokens to the full corpus further reduced false claims. The evaluation also notes that differing token counts made a fair comparison with Rephrase difficult.

H.3 ADDITIONAL QUANTITATIVE SUMMARY EVALUATION

  • Different total token counts made a fair comparison between Rephrase and the strongest CPT and GPT summarization models difficult.Source: §H.3 · p. 45
  • Under synthetic-dataset subsampling, EntiGraph matched or improved on Rephrase CPT, though its gains were generally smaller than in QA.Source: §H.3 · p. 45
  • The comparison adds instruction-tuned Raw CPT and Rephrase CPT models and a token-matched EntiGraph-29M model sampled from the 455M-token corpus.Source: §H.3 · p. 45
  • EntiGraph summaries had significantly fewer false claims than Rephrase for short prompts, with comparable salient-claim counts; the false-claim gap persisted for long prompts.For long prompts, the error bars separated for false claims but overlapped for salient claims.Source: Figure 8 · p. 45, §H.3 · p. 45
  • Scaling EntiGraph from 29M tokens to the full model significantly reduced false claims for both short and long prompts.The authors suggest that added scale could improve knowledge-intensive downstream tasks.Source: §H.3 · p. 45
Loading 2409.07431v2…