Instruction Pre-Training: Language Models are Supervised Multitask Learners

1 Introduction

• Unsupervised multitask pre-training became standard because raw-corpus causal language modeling facilitates scaling, while instruction tuning has been reported to enhance task generalization.The passage cites Radford et al. (2019), Brown et al. (2020), and Chowdhery et al. (2023) on pre-training, and Wei et al. (2021), Sanh et al. (2021), and Chung et al. (2024) on instruction tuning.Source: §1 · p. 1
• Instruction Pre-Training augments raw texts with content-based instruction-response pairs, then pre-trains LMs on the augmented corpora.The synthesizer uses open-source models and scales generation to 200M pairs across more than 40 task categories.Source: §1 · pp. 1–2
• Instruction Pre-Training improves Llama3-8B on finance and biomedicine, reaching parity with or surpassing Llama3-70B in continual pre-training.From scratch, a 500M model trained on 100B tokens reaches the performance of a 1B model trained on 300B tokens, and gains more from instruction tuning.Source: §1 · p. 2
• The authors report contributions in supervised multitask pre-training, scalable instruction synthesis, and analysis of factors behind the method's success.The paper says its experiments verify the method's effectiveness and that its analysis examines the synthesizer and synthetic data.Source: §1 · p. 2

2 Instruction Pre-Training

• Instruction Pre-Training appends instruction-response pairs to raw texts and pre-trains language models on the augmented corpora.The instruction is the task input and the response is the task output.Source: §2 · p. 2

2.1 Instruction Synthesizer

• The instruction synthesizer is multitask fine-tuned to generate pairs from raw text, with diverse tuning data intended to support generalization to unseen corpora.It can generate both one-shot and few-shot examples for subsequent pre-training.Source: §2.1 · p. 2, Figure 2 · p. 3
• Synthesizer tuning data reformats diverse context-based task datasets, using their contexts as raw texts and downstream tasks as instruction-response pairs.The contexts span domains such as encyclopedias, social media, and academic tests; tasks include commonsense reasoning and sentiment analysis.Source: §2.1 · p. 2
• Tuning sequences concatenate multiple examples from one dataset, and the loss is calculated only on instruction-response pairs.Same-dataset examples maintain consistency in task format or category, and pair-only loss focuses the model on those pairs.Source: §2.1 · pp. 2–3
• During multi-round inference, each new raw text is combined with previous texts and synthesized pairs to generate additional pairs.The intended pattern of new pairs follows those from prior rounds.Source: §2.1 · p. 3, Figure 3 · p. 3

2.2 LM Pre-Training

• Synthesized pairs are formatted and concatenated with raw texts from multiple rounds, while LM training retains the next-token prediction objective and computes loss on all tokens.Templates diversify instruction formats and create M-shot examples for pre-training; other pre-training settings remain the same as Vanilla Pre-Training.Source: §2.2 · p. 3
• From-scratch training converts only part of the raw corpora and mixes in synthesizer fine-tuning data, whereas continual domain-adaptive training converts all raw corpora and mixes in general instructions.For general training, the remaining raw corpora are left unchanged; for continual training, synthesizer fine-tuning data are excluded from the general-instruction collection.Source: §2.2 · pp. 3–4

3.1 Instruction Synthesizer

• The synthesizer is fine-tuned from open-source Mistral-7Bv0.1, and inference creates about five instruction-response pairs per raw text, each containing about 52 tokens.The paper characterizes this 7B-parameter model as more cost-effective than large-scale or closed-source models used for synthetic data generation.Source: §3.1 · p. 4

3.2 General Pre-training From Scratch

• General pre-training uses 200M RefinedWeb texts containing about 100B tokens, converts 40M texts, and produces 200M pairs totaling about 10B tokens.Two synthesis rounds convert 20M texts each; synthesizer fine-tuning data are repeated four times during pre-training.Source: §3.2 · p. 4
• From-scratch models use Mistral architecture and tokenizer at 500M and 1.3B parameters; evaluation uses lm-evaluation-harness, with 500M models also instruction-tuned and evaluated on MMLU.The pre-training settings largely follow Brown et al. (2020).Source: §3.2 · p. 4

3.3 Domain-Adaptive Continual Pre-Training

• Continual pre-training converts all PubMed Abstracts and financial-news corpora in three rounds, mixes in general instructions, and continues training Llama3-8B separately for each domain.Evaluation uses domain-specific tasks with prompting settings following Cheng et al. (2023).Source: §3.3 · p. 4

4.1 General Pre-Training From Scratch

• With 100B tokens, the 500M model matches Pythia-1B trained on 300B tokens, while the 1.3B model matches BLOOM-3B trained on 341B tokens.The authors describe this as consistent data efficiency across model scales.Source: Table 2 · p. 5, §4.1 · p. 5
• Instruct PT outperforms Vanilla PT on several benchmarks, including datasets absent from the synthesizer’s fine-tuning data.Mix PT, which adds the synthesizer’s fine-tuning data to raw corpora, also improves on Vanilla PT; all methods use the same number of training tokens.Source: §4.1 · pp. 4–5, Table 1 · p. 5
• During MMLU instruction tuning, Instruct PT quickly outperforms Vanilla PT and shows a stable increasing trend; the authors infer task alignment supports faster downstream learning.The authors suggest this may reduce the number of further fine-tuning steps.Source: Figure 4 · p. 5, §4.1 · p. 5

4.2 Domain-Adaptive Continual Pre-Training

• Instruction Pre-Training consistently beats Vanilla Pre-Training on almost all domain-specific tasks, bringing Llama3-8B to parity with or above Llama3-70B.Table 3 compares Llama3-8B before and after continued pre-training with both methods, alongside Llama3-70B as a reference.Source: §4.2 · pp. 5–6, Table 3 · p. 6
• Removing domain-specific corpora reduces domain-specific performance, rule-based synthesis constrains performance through limited diversity, and single-turn synthesis decreases prompting performance.Table 4 reports average task scores within each domain for these training-data ablations.Source: Table 4 · p. 6, §4.2 · p. 6
• Finance NER is an exception where Instruct PT underperforms Vanilla PT; high variance, including Llama3-70B scoring below Llama3-8B, raises reliability concerns.The authors suggest this benchmark may not be reliable.Source: §4.2 · p. 6

5.1 Instruction Synthesizer

• The fine-tuned synthesizer also significantly outperforms the base model on response accuracy for both seen and unseen datasets.Response accuracy is evaluated using F1 similarity between generated and gold responses.Source: §5.1 · pp. 6–7
• The fine-tuned synthesizer outperforms the base model across zero-shot, few-shot, seen, and unseen settings for instruction-response pair quality.On unseen datasets, few-shot substantially outperforms zero-shot.Source: Table 5 · p. 7, §5.1 · p. 7
• The evaluation tests whether pairs conditioned on testing raw text help a base Mistral-7B LM, comparing the authors’ synthesizer with base-model and random-pair alternatives.The random pairs come from a different context; both the base model and the authors’ synthesizer generate pairs based on the testing text.Source: §5.1 · p. 7
• On seen and unseen datasets, synthesized pairs consistently improve the LM’s testing-task performance over all baselines.The comparison tests performance with and without synthesized pairs in the prompt.Source: Figure 5 · p. 7, §5.1 · p. 7

5.2 Instruction-Augmented Corpora

• The instruction-augmented corpora cover 49 task categories, with over 85% context relevance and 70% response accuracy.The synthesized tasks cover all general task scenarios represented in the corpora.Source: §5.2 · pp. 7–8, Table 6 · p. 8, Figure 6 · p. 8

Synthetic Instruction Generation

• Prior synthetic instruction-generation work mainly targets post-training, whereas this work targets pre-training; the authors describe the approaches as complementary.Instruction-pre-trained models also gain more from instruction post-training.Source: §6 · p. 8
• Unlike approaches that distill knowledge from strong models, this method learns from raw corpora and can be task-agnostic rather than dependent on a few gold examples.The authors contrast it with task-specific approaches that rely on such examples.Source: §6 · p. 8
• The authors say their method increases instruction diversity over rule-based methods and could potentially complement iterative synthetic-data techniques.They identify exploring those iterative techniques as future work.Source: §6 · p. 8

Data Curation for LM Pre-Training

• Common pre-training data curation collects, cleans, and organizes corpora; this work explores the orthogonal direction of augmenting raw corpora with large-scale supervised signals.The passage notes that web-scraped data can contain low-quality and duplicate content.Source: §6 · p. 8

Conclusion

• Instruction Pre-Training augments raw corpora with synthesizer-generated instruction-response pairs and uses them to pre-train language models.The synthesizer is fine-tuned on a diverse collection and is described as generating pairs from various corpora.Source: §7 · p. 8
• From-scratch Instruction Pre-Training outperforms Vanilla Pre-Training on base models and yields greater benefit from further instruction tuning.In continual pre-training, it substantially enhances Llama3-8B in two domains.Source: §7 · p. 8

Limitations

• Synthetic instruction-augmented corpora have approximately 70% accuracy, so hallucinations could potentially mislead the pre-trained model.The authors suggest post-verification or improving synthesizer reliability as possible future directions.Source: Limitations · p. 9
• The paper’s pre-training is limited to billions of tokens, while cited works have used trillions; synthetic-data scaling laws and the quantity-quality balance remain open.The cited works are Touvron et al. (2023) and Jiang et al. (2023).Source: Limitations · p. 9

Ethics Statement

• The work uses publicly available datasets and language models, except for pre-training corpora in the finance domain.Source: Ethics Statement · p. 9

A Data Collection for Fine-Tuning Instruction Synthesizer

• Contexts from diverse datasets serve as raw texts, paired with their downstream tasks as instruction-response examples for synthesizer fine-tuning.The collection samples up to 10K examples per dataset, prioritizing examples with more instruction-response pairs to promote task diversity and avoid dataset predominance.Source: A Data Collection for Fine-Tuning… · p. 15

B Tuning and Inference Settings for Instruction Synthesizer

• Examples wrap raw text in <CON> tags and use format-specific templates to separate instruction-response pairs.N-shot sequences concatenate examples; the appendix also describes filling the maximum sequence length with formatted examples from the same dataset for tuning.Source: B Tuning and Inference Settings for… · p. 15, Table 7 · p. 15
• Inference feeds prior-round examples alongside the current raw text, with a maximum sequence length matching the target pre-training LM.The appendix reports about one day to synthesize pairs for 1B raw-corpus tokens on one A100-80GB GPU.Source: B Tuning and Inference Settings for… · p. 15

• General evaluation uses zero-shot WinoGrande, PIQA, and HellaSwag, and five-shot ARC, BoolQ, SIQA, OBQA, and MMLU.Source: C LM Evaluation · pp. 15–16
• General benchmark results use acc-norm scores from lm-evaluation-harness, following Brown et al. (2020).Source: C LM Evaluation · p. 16
• Domain-specific evaluation covers biomedical PubMedQA and USMLE zero-shot, plus ChemProt, MQP, and RCT few-shot; finance evaluation covers ConvFinQA zero-shot and FPB, FiQA SA, Headline, and NER few-shot.Source: C LM Evaluation · p. 16

D Data Contamination Analysis

• Contamination is measured by whether an evaluated example has a substring in training data, with synthesized-pair contamination calculated as the increase from raw to instruction-augmented corpora.Table 9 reports total evaluation examples and contamination counts for raw corpora, augmented corpora, and synthesized pairs.Source: D Data Contamination Analysis · p. 16, Table 9 · p. 17
• Synthesized pairs introduce minimal contamination to the training data.Source: D Data Contamination Analysis · p. 16

E Human Evaluation on Instruction-Augmented Corpora

• Human evaluation scores response accuracy and context relevance as binary measures and counts distinct task categories using a predefined list.Source: E Human Evaluation on… · p. 17
• The synthesized instruction-response pairs span 51 task categories, with over 85% context relevance and 70% response accuracy.Source: E Human Evaluation on… · p. 17, Table 10 · p. 17

• The analysis compares domain coverage and overlap between raw text and instruction domains, and examines distributions in fine-tuning data, raw corpora, and synthesized pairs.Source: F Domain Distribution Analysis · p. 17
• Synthesized pairs cover most domains in raw texts and have high domain overlap with them.For multi-domain texts, the synthesizer generates an average of 5 instruction-response pairs per raw text, and these pairs cover most text domains.Source: F Domain Distribution Analysis · p. 17
• Synthesized pairs closely follow the raw corpora’s domain distribution, although the fine-tuning data and raw corpora distributions differ substantially.Source: F Domain Distribution Analysis · p. 17, Table 12 · p. 18