PaperlayerView exact revision on arXiv

Source-linked AI summary

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Ajay Patel, Colin Raffel, Chris Callison-Burch

arXiv:2601.22146v3cs.CLcs.LG

TL;DR

FineInstructions addresses the limited scale and realism of supervised instruction-tuning data by transforming pre-training documents into synthetic instruction-answer pairs. It creates ~18M templates from real user queries, matches them to documents, and instantiates grounded answers at scale. Pre-training on the resulting data outperforms standard pre-training and other synthetic pipelines on response-quality benchmarks, while the authors note limitations in tuning choices, benchmark coverage, and perplexity-based evaluation.

  • Problem

    Supervised instruction-tuning data is limited, while standard pre-training uses vast unstructured corpora and instruction-tuning datasets are often small or narrow.

  • Method

    FineInstructions converts ~18M real user-written queries into reusable templates, matches them to compatible pre-training documents, and generates grounded instruction-answer pairs.

  • Results

    FineInstructions outperforms standard pre-training and other synthetic baselines on benchmarks measuring free-form response quality across academic and realistic user tasks.

  • Takeaways & Limitations

    The resulting 1B+ synthetic instruction-answer dataset can support supervised, instruction-aligned pre-training from scratch.

  • Takeaways & Limitations

    The method’s procedural choices reflect compute constraints and manual calibration, and the benchmark suite lacks many long-tail realistic knowledge tasks.

Abstract

from arXiv · show

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

1 Introduction

FineInstructions addresses the limited scale and realism of supervised instruction-tuning data by converting pre-training documents into diverse, realistic instruction-answer pairs. Its ~18M templates come from real user queries, enabling billion-scale synthetic data and instruction-only pre-training.

  • Motivation: Existing instruction-tuning datasets are often small, narrow, or based on unrealistic academic task templates.Some contain only a few thousand examples, while others use relatively few task templates.
  • Motivation: Pre-training corpora provide knowledge and indirect task supervision, but next-token prediction may not be the most efficient way to absorb capabilities.Prior synthetic rephrasing and transformation pipelines suggest that alternative data formats can improve knowledge absorption and model performance.
  • Approach: FineInstructions transforms pre-training documents into diverse, realistic instruction-response pairs by matching user-like queries to document knowledge and extracting grounded answers.The pipeline instantiates templates on a per-document basis.
  • Approach: ~18M instruction templates are mined from real user-written queries and tasks, supporting synthetic data that is diverse and representative of real user tasks.The templates are converted into generic forms for reuse across documents.
  • Contributions: Pre-training solely on FineInstructions outperforms standard pre-training and other synthetic pipelines on three benchmarks spanning academic and realistic user tasks.The paper releases a dataset containing 1B+ synthetic instruction-answer pairs.

2 Related Work

Prior work has explored synthetic data generation, document filtering, and mixture reweighting as alternatives or complements to conventional language-model pre-training.

  • Related Work: Previous approaches generate synthetic training data with LLMs, filter documents using LLM judgments, or reweight and resample document mixtures.These methods target changes to the composition or format of pre-training data.

3 FineInstructions

FineInstructions converts user-written queries into reusable templates, matches them to compatible document regions, and generates and filters grounded instruction-answer pairs at scale. Specialized retrieval and distilled models support coverage, instantiation, and quality control.

  • Pipeline Overview: The pipeline transforms user queries into reusable templates, matches them with documents containing sufficient information, and uses a judge model to assess generated pairs.A document must support both realistic query instantiation and a grounded answer.
  • Pipeline Overview: The procedure converts large unlabeled corpora into supervised instruction-answer data using programmatic labeling and distilled models for pipeline stages.The resulting pairs support supervised pre-training by conditioning answer generation on each instruction.
  • Generating Instruction Templates: ~18M templates are produced by genericizing ~18M queries with tagged entity or scenario spans, using ~50K silver examples to train a Query Genericizer Model.The model is fine-tuned from Llama-3.2 1B Instruct and also produces compatible document descriptions.
  • Matching Documents to Instruction Templates: BGE-M3 embeddings and FAISS retrieval identify potentially compatible templates, while hard compatibility examples fine-tune document-template matching.Gaussian-pooled representations capture multiple document regions for retrieval.
  • Matching Documents to Instruction Templates: Gaussian pooling distributes K = 5 kernels across documents, producing six retrieval representations that improve coverage of distinct semantic regions.The representations include one global embedding and five local embeddings.
  • Matching Documents to Instruction Templates: Pearson correlation of 0.99 links retrieval chunk indices to answer-excerpt locations after local-semantic fine-tuning.Most excerpts come from the 19%–71% document range, scaling linearly with chunk index.
  • Instantiating Answers: Retrieved templates are instantiated with document excerpts, allowing slight rephrasing or introductory wording while preserving supporting context.The process generates silver examples, trains a distilled Instantiator Model, and filters outputs with judges.
  • Quality Control: A judging and filtering stage uses the Flow Judge model and a 1–5 rubric to select higher-quality synthetic instructions and answers.Standalone instruction-answer formatting also makes the data compatible with off-the-shelf reward and judge models.

4 Experimental Setup

The experiments compare FineInstructions with standard and synthetic pre-training under matched source corpora and token budgets. Evaluation covers knowledge absorption and realistic user-query response quality across academic and conversational benchmarks.

  • Experimental Design: The study tests whether training from scratch on FineInstructions improves knowledge absorption and model performance.The experimental setup is designed to validate synthetic instruction-answer pre-training.
  • Experimental Design: All methods use the same unstructured source corpora and number of training tokens, with original-document pre-training serving as the standard baseline.Additional baselines use synthetic transformations from prior work.
  • Baselines: The comparisons include IPT data based on RefinedWeb and synthetic versions of that corpus.IPT provides a prior instruction-and-response synthesis baseline.
  • Baselines: Nemotron-CC comparisons include vanilla data, mixed synthetic transformations, standalone diverse Q&A, and WRAP rephrasing data.These datasets are drawn at approximately 300B tokens for the relevant comparisons.
  • Benchmarks: The evaluations format questions according to each method’s training template and assess both knowledge absorption and realistic user queries.The suite includes MixEval, MT-Bench-101, and AlpacaEval.

5 Results

FineInstructions outperforms standard pre-training and other synthetic baselines across knowledge-focused and open-ended benchmarks, although MT-Bench-101 differentiates models less clearly.

  • FineInstructions outperforms standard pre-training and other synthetic baselines on both datasets.
  • ~69% relative improvement on MixEval compared to standard pre-training on the IPT dataset.
  • ~39% improvement on Nemotron-CC compared to standard pre-training.
  • FineInstructions outputs are consistently preferred on AlpacaEval and achieve a higher MT-Bench-101 score than any other method.
  • Improvements hold across knowledge-focused MixEval and open-ended MT-Bench-101 and AlpacaEval benchmarks.

6 Discussion

The discussion examines instruction diversity, judging effects, evaluation choices, and scaling limits. It reports broad diversity and improved performance from filtering while identifying tuning, benchmark, and scaling constraints.

  • 6.1 Diversity of Instructions: 4.3M unique instruction templates instantiated ~1.08B total instructions, with no template comprising more than 0.09% of generated instructions.
  • 6.1 Diversity of Instructions: The template distribution is highly diverse, with most templates used fewer than 1,000 times and long-tail templates represented alongside simpler ones.
  • 6.1 Diversity of Instructions: Instruction types are analyzed across domains, knowledge reasoning, and task-oriented versus simple recall categories using zero-shot LLM classification.
  • 6.1 Diversity of Instructions: Zero-shot classification may yield errors, so the task-composition analysis provides high-level insights rather than definitive annotations.
  • 6.2 Judging and Filtering: Judging and filtering overall improves performance, especially on AlpacaEval.
  • 6.3 Limitations and Future Directions: The authors do not report perplexity because FineInstructions uses instruction-answer pairs, prioritizing downstream user utility instead.
  • 6.3 Limitations and Future Directions: Several pipeline choices reflect compute constraints and manual calibration, while complex templates remain challenging to match and instantiate.
  • 6.3 Limitations and Future Directions: Log probability-based classification benchmarks are ill-suited to models that consistently produce long-form answers and assign low probability to short responses.

7 Conclusion

FineInstructions transforms real user queries and pre-training documents into scalable, instruction-aligned synthetic data. The paper reports that this format supports training that better reflects downstream usage and improves knowledge absorption efficiency.

  • FineInstructions transforms real user queries into templates that generate in-distribution synthetic data at scale.
  • The resulting data trains LLMs in a supervised, instruction-aligned format rather than through self-supervised next-token prediction on documents.
  • The transformed objective and data structure better reflect downstream usage patterns and improve knowledge absorption efficiency.

Ethics Statement

The ethics discussion notes that synthetic-data training can amplify generator biases and errors, although near-exact source excerpts mitigate hallucinated content.

  • Synthetic training data may amplify biases and errors from the generating model, leaving some marginal risk of systematic data biasing.
  • FineInstructions mitigates hallucinated content by using near-exact source-document excerpts and mainly transforming naturally occurring text.

A Dataset Sources

FineInstructions draws on user-written queries and prompts, converts them into reusable templates, and instantiates compatible templates with documents to create grounded instruction-answer pairs.

  • Document matching: Compatible documents must contain enough information to fill template variables and answer the resulting query.The template and document jointly produce grounded, realistic, diverse instruction-tuning questions and answers.
  • Dataset uses: With more than 1B instructions, a 0.58% share of math instructions yields more than 6M math instructions for topic-mixture and domain-specialist training experiments.This supports mining task-specific or domain-specific instruction sets from FineInstructions.

F Ablation of Judging and Filtering Stage

The judging and filtering stage improves FineInstructions pre-training performance, with the strongest additional effect reported on AlpacaEval.

  • Ablation result: Adding judging and filtering further improves performance, especially on AlpacaEval.The ablation compares pre-training with and without the stage on the IPT and Nemotron-CC datasets.

G Performance of Training on FineInstructions at Different Model Scales

Across 300M, 1.8B, and 7B models, FineInstructions remains competitive with or outperforms larger-scale baselines under equivalent token and compute budgets.

  • Scale comparison: FineInstructions consistently achieves higher performance across MixEval, MT-Bench-101, and AlpacaEval under equivalent token and compute budgets.At fixed model sizes, FineInstructions models are competitive with or outperform baselines trained at the next larger scale.

H Detailed Breakdown of Sub-Benchmarks in MixEval

The appendix reports detailed MixEval and MixEval-Hard sub-benchmark results, including accuracy, sample counts, and comparisons among standard pre-training, FineInstructions, and Nemotron-CC.

  • Breakdown scope: Tables 8 and 9 break down MixEval and MixEval-Hard performance by sub-benchmark for every evaluated pre-training method.Sample counts n appear in the sub-benchmark column headers.
  • MixEval Standard: Table 8 reports MixEval Standard sub-benchmark performance as percentage accuracy with per-sub-benchmark sample counts.Methods are abbreviated as Std. PT, FI, and Nemo-CC.
  • Method comparison: The breakdown compares Standard Pre-Training, FineInstructions, and Nemotron-CC, highlighting the strongest method within each corpus group.The stated comparison applies to the detailed MixEval results table.
Loading 2601.22146v3…