PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Instruction Tuning with Pre-training

Yiming Liang, Tianyu Zheng, Xinrun Du, Ge Zhang, Jiaheng Liu, Xingwei Qu, Wenqiang Zu, Xingrun Xing, Chujie Zheng, Lei Ma, Guoyin Wang, Zhaoxiang Zhang, Wenhao Huang, Xiang Yue, Jiajun Zhang

arXiv:2501.09368v4cs.AI

TL;DR

Instruction-tuning datasets are often narrow compared with pre-training distributions, leaving a gap in dataset coverage and alignment. AITP identifies underrepresented pre-training data, rewrites it as instruction-response pairs, and integrates it with existing SFT data. Evaluations report consistent improvements across models and benchmarks, with the conclusion describing a solution using less than 10% difference data.

  • Problem

    Instruction-tuning datasets are often narrowly focused and differ from broader pre-training distributions, limiting dataset coverage and alignment.

    Source: §1 · p. 1

  • Method

    AITP identifies gaps between SFT and pre-training data, rewrites underrepresented data into instruction-response pairs, and integrates them into the SFT dataset.

    Source: §1 · p. 1, §2.1.3 · p. 3, §2.3 · p. 3

  • Results

    AITP consistently improves performance across three open-source LLMs and eight benchmarks; the conclusion reports bridging the gap with less than 10% difference data.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors present AITP as a cost-effective, practical way to supplement existing SFT datasets with a small amount of difference data.

    Source: §6 · p. 8

    Resource constraints prevent using the full pre-training corpus, so the method samples a subset with reservoir sampling.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Instruction tuning enhances large language models (LLMs) to follow human instructions across diverse tasks, relying on high-quality datasets to guide behavior. However, these datasets, whether manually curated or synthetically generated, are often narrowly focused and misaligned with the broad distributions captured during pre-training, limiting LLM generalization and effective use of pre-trained knowledge. We propose Aligning Instruction Tuning with Pre-training (AITP), a method that bridges this gap by identifying coverage shortfalls in instruction-tuning datasets and rewriting underrepresented pre-training data into high-quality instruction-response pairs. This approach enriches dataset diversity while preserving task-specific objectives. Evaluations on three fully open LLMs across eight benchmarks demonstrate consistent performance improvements with AITP. Ablations highlight the benefits of adaptive data selection, controlled rewriting, and balanced integration, emphasizing the importance of aligning instruction tuning with pre-training distributions to unlock the full potential of LLMs.

1. Introduction

Instruction-tuning datasets can be narrow and poorly aligned with broad pre-training distributions. AITP finds underrepresented pre-training data, rewrites it as instruction-response pairs, and integrates it with the original dataset; experiments report consistent gains across models and benchmarks.

1. Introduction

  • Instruction-tuning datasets are often narrowly focused, while pre-training corpora reflect broader real-world distributions and LLM internal knowledge.Manual datasets are costly and difficult to scale, and synthetic datasets can depend on expensive APIs and tightly coupled generation pipelines.Source: §1 · p. 1
  • AITP identifies instruction-tuning coverage gaps against pre-training data, rewrites underrepresented raw text into instruction-response pairs, and combines them with the original dataset.Its three stages are difference-set generation, rewriting, and integration for fine-tuning.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • AITP consistently improves performance across three open-source LLMs and eight benchmarks.Ablations also highlight adaptive data selection and integration.Source: §1 · p. 2

2. Methods

AITP identifies underrepresented regions in the SFT dataset relative to pre-training data, rewrites selected text into instruction-response pairs, and combines them with the original SFT data for training.

2.1.1. DATA REPRESENTATION

  • Each data point is represented by a final-layer embedding, and dimensionality reduction projects pre-training and SFT embeddings into shared two-dimensional coordinates.The shared projection facilitates visualization and density comparison across datasets.Source: §2.1.1 · p. 2

2.1.2. DENSITY ESTIMATION

  • Kernel Density Estimation visualizes the density of pre-training and SFT points, with bandwidth parameters controlling smoothness in each coordinate direction and the kernel.These visualizations highlight regions where the two distributions diverge.Source: §2.1.2 · p. 2

2.1. Difference Set Generation

  • The difference set targets pre-training data points that are absent or sparsely represented in the SFT distribution, identified using SFT density estimates and a threshold.The process is organized into data representation, density estimation, and difference-set identification.Source: §2.1 · p. 2, §2.1.3 · p. 3

2.1.3. FINDING DIFFERENCE SET

  • A pre-training point with a low density estimate under the SFT distribution is classified as absent or sparsely populated in SFT and included in the difference set.The SFT density estimate determines which points qualify.Source: §2.1.3 · p. 3

2.2. Data Transformation of Difference Set

  • Data transformation generates queries from difference-set text, scores and filters low-quality queries, then generates responses for the remaining queries.The response-generation prompt is applied after query filtering.Source: §2.2 · p. 3

2.3. Training

  • AITP combines rewritten data from the difference set with the original SFT dataset for training, using the same model trained on the pre-training corpus.The paper says this supports consistency with the model’s internal knowledge and helps correct semantic inaccuracies from pre-training formatting errors.Source: §2.3 · p. 3

3. Experiment Settings

The experiments evaluate AITP on three open-source model families using benchmark tasks, paired baselines, distribution-based data selection, controlled rewriting, and full-parameter fine-tuning. The settings also compare dataset-size and rewriting ablations.

3.1. Evaluation

  • AITP is evaluated on IFEval and seven standard benchmarks spanning knowledge, scientific reasoning, coding, commonsense, and math tasks.IFEval reports prompt- and instruction-level strict and loose accuracy; evaluations use OpenCompass.Source: §3.1 · p. 3

3.2. Main Setting

  • Experiments use OLMo, MAP-Neo, and Pythia with their corresponding pre-training corpora and SFT baselines; Pythia uses Tulu-v2 because its SFT dataset is unreleased.Resource constraints lead to reservoir sampling rather than using the full pre-training corpus.Source: §3.2 · pp. 3–4

3.3. Difference Set Generation Setting

  • Difference-set selection compares pre-training and SFT data densities, using density ratios to select difference points; τ is set to 0.7 for Equation 7 and 1.0 for Equation 14.Text embeddings are reduced to two dimensions with PCA, and KDE is used to visualize distributions.Source: §3.3 · p. 4

3.4. Data Transformation Setting

  • Qwen2.5-72B-Instruct generates contextually relevant, self-contained instructions, scores them for quality, difficulty, and information needs, then produces comprehensive answers.Instruction quality is rated on a 1–10 scale.Source: §3.4 · p. 4

3.5. Ablation Setting

  • Ablations test whether gains reflect a larger SFT dataset or distillation by size-matching the combined data and replacing original SFT data with similarly distributed rewritten pre-training data.The latter setup follows LongForm’s fully rewritten-data approach while testing the contribution of retaining existing high-quality datasets.Source: §3.5 · p. 4

3.6. Training Setting

  • AITP trains OLMo, MAP-Neo, and Pythia on combined datasets containing rewritten differences between each pre-training corpus and its corresponding SFT dataset.TuluV2 substitutes for Pythia’s unreleased SFT dataset; training uses full-parameter fine-tuning.Source: §3.6 · pp. 4–5
  • Figure 3 depicts Dolma in red and dataset projections in blue, with darker regions indicating higher data-point concentration.The blue projections represent Tulu V2, the difference set, the rewritten set, and the combined set.Source: Figure 3 · p. 5

4. Results

AITP expands SFT coverage toward pre-training distributions, and the reported experiments show gains across models, generation settings, data-size controls, and rewritten-data ratios.

4.1. Distribution Change Analysis

  • The combined dataset expands original SFT coverage, with dense regions aligning closely to Dolma after rewriting narrows the distribution.The difference set includes pre-training data lacking in SFT, such as code and scientific literature.Source: §4.2 · p. 5, §4.1 · p. 5

4.2. Main Results

  • Average performance improves by 3.8 for OLMo, 1.1 for MAP-Neo, and 0.9 for Pythia across eight benchmarks with AITP.The authors suggest gains may come from supplementing SFT data with missing examples and optimizing its distribution.Source: §4.2 · p. 5, Table 1 · p. 6

4.3. Difference Set Generation Setting Results

  • Across reported generation variants, average absolute improvements over OLMo-SFT are 3.8, 4.3, 3.0, and 3.1.The passage reports these for bge with density estimation, bge with density comparison, MiniLM with density comparison, and a further bge with density comparison setting, respectively.Source: §4.3 · p. 6, Table 2 · p. 6

4.4. Ablation Results

  • At the same dataset size, AITP improves average absolute performance by 2.1; the same-size setting improves less than final AITP.Replacing original SFT with rewritten same-distribution data does not outperform OLMo-SFT, which the authors attribute tentatively to lower rewritten-data quality.Source: §4.4 · pp. 6–7, Table 3 · p. 6

4.5. Ratio Results

  • AITP performs well with rewritten data below 10% of the original SFT dataset, while performance declines as the rewritten set grows.The authors hypothesize that larger ratios may lower overall data quality because rewritten data quality may be lower than original SFT data.Source: §4.5 · p. 7, Figure 4 · p. 7

4.6. Visualization

  • Rewritten data clusters in areas underrepresented by original SFT and avoids regions where original SFT is dense.In the visualization, red marks original SFT data and blue marks rewritten data.Source: §4.6 · p. 7, Figure 5 · p. 7

5. Related Work

Related work traces instruction tuning and open models toward greater accessibility, diversity, and synthetic-data use, while noting persistent quality and distribution challenges.

5.1. Open-Source Large Language Model

  • Fully transparent models expose pre-training corpora, SFT datasets, cleaning processes, checkpoints, and reproducible code beyond model weights.The paper primarily evaluates such models because their pre-training and SFT data are accessible.Source: §5.1 · pp. 7–8

5.2. Instruction Tuning

  • Instruction tuning progressed from manually curated instruction-response datasets to multi-task and larger synthetic datasets intended to improve generalization and robustness.Examples include FLAN, T0, UnifiedQA, FLAN-T5, and Super-Natural Instructions.Source: §5.2 · p. 8

5.3. Improving LLM Using Synthetic Data

  • Synthetic-data methods use varied external signals and generation strategies, but may introduce hallucinations, low quality, limited diversity, and distribution misalignment.Prior work also targets quality, diversity, hallucinations, and synthetic-data distribution; this paper focuses on diversity after manual dataset combination.Source: §5.3 · p. 8

6. Conclusion

AITP addresses coverage and distribution gaps between SFT datasets and pre-training corpora by adding a small amount of difference data.

6. Conclusion

  • AITP identifies the difference set between pre-training corpora and manually assembled SFT datasets to adaptively fill coverage gaps.The approach builds on existing high-quality SFT data and guides synthesis of missing data.Source: §6 · p. 8
  • Adding less than 10% difference data can bridge the gap between SFT and pre-training datasets.The paper describes this as a cost-effective, practical solution for real-world applications.Source: §6 · p. 8
  • The paper reports significant coverage and distribution differences between existing SFT datasets and the pre-training corpus.Source: §6 · p. 8

A. Visualization of SFT Dataset Projections onto the Pre-training Corpus

The appendix visualizes how SFT data sits relative to pre-training data and describes reservoir sampling for selecting a subset of streaming data.

A. Visualization of SFT Dataset Projections onto the Pre-training Corpus

  • Figure 6 uses red bottom regions for the pre-training corpus and light-blue regions above for SFT datasets, with darker areas indicating denser data.Lighter areas indicate sparser distributions.Source: Figure 6 · p. 16

B. Reservoir sampling algorithm

  • Reservoir Sampling uniformly samples k items from a stream without requiring its total size to be known.It is suited to memory-constrained or uncertain stream sizes and allows equal-probability sampling in one pass.Source: B. Reservoir sampling algorithm · p. 17
  • Algorithm 1 takes a data stream and sample size k, outputs a random sample of size k, and initializes its reservoir with the first k items.The shown procedure then iterates over later stream items.Source: B. Reservoir sampling algorithm · p. 17

C. Prompts for data transformation phase

The data-transformation prompts generate questions from raw text, score them, and generate answers. Their instructions emphasize relevance, self-containment, and quality checks.

C. Prompts for data transformation phase

D. Training parameters

The appendix identifies the training hyperparameters used for AITP and states that they match those of the original model’s SFT version.

D. Training parameters

E. Performances across different ratios

Table 5 compares results across ratios using four accuracy measures, distinguishing prompt-level from instance-level and strict from loose accuracy.

E. Performances across different ratios

  • Table 5 reports results across various ratios.The caption does not provide the table's numerical values.Source: Table 5 · p. 19
  • The four metrics are prompt-level strict accuracy, instance-level strict accuracy, prompt-level loose accuracy, and instance-level loose accuracy.Source: Table 5 · p. 19
  • The metric labels distinguish evaluation granularity (prompt or instance) and strict versus loose accuracy.Source: Table 5 · p. 19

F. Examples

The examples contrast dense regions in pre-training and SFT data with rewritten examples, illustrating varied instruction-response content across tasks.

F. Examples

  • Examples 1–3 represent dense pre-training regions for code, scientific literature, and general text; Example 4 represents a dense SFT region.Examples 5–7 represent dense regions in the rewritten set, while Examples 8–10 illustrate differing relative data densities.Source: F. Examples · pp. 19, 22, 23
  • The rewritten L1 regularization example explains its penalty on model coefficients and its role in setting less important feature coefficients to zero.Source: F. Examples · p. 22
  • Rewritten examples also answer factual questions about carbon dioxide composition and the signless Laplacian spectrum of a graph.Source: F. Examples · p. 23
  • Original SFT examples use app reviews to elicit recommendations on a five-choice scale.Source: F. Examples · p. 23
  • A further example solves a spring-constant problem from the spring’s length change under a hanging weight.Source: F. Examples · p. 24
Loading 2501.09368v4…