The Best Instruction-Tuning Data are Those That Fit

1 Introduction

• Effective SFT requires tailoring data to target models, not only scaling data or choosing which questions models should learn.The introduction contrasts response tailoring with prior work focused on questions, while noting response tailoring in later post-training phases.Source: §1 · p. 2
• GRAPE selects each instruction’s response by highest length-normalized probability under the target model, then uses standard SFT.The selected responses are intended to be closer to the model’s pretrained distribution.Source: §1 · p. 2
• GRAPE curates model-dependent SFT datasets, addressing distribution-shift risks while requiring only one forward pass over candidate responses.The paper connects those risks to spurious correlations and catastrophic forgetting.Source: §1 · p. 2
• Up to 17.3% absolute gain on average performance came from GRAPE-selected responses versus training on a 3× larger dataset in controlled experiments.GRAPE-selected responses also outperformed responses from the strongest teacher under consideration, LLAMA3.1-405B-INSTRUCT.Source: §1 · p. 2
• 4.6% average improvement over state-of-the-art data selection approaches and up to 6.1% over a baseline using 4.5 times more data were reported in realistic settings.With one-third of the data and half the epochs, GRAPE also enabled LLaMA3.1-8B to exceed Tulu-8B-SFT performance.Source: §1 · p. 2

2.1 Data Engineering for Instruction Tuning

• Instruction-data selection has emphasized quality, diversity, synthesis, or instruction customization, leaving response selection for pretrained-distribution alignment comparatively overlooked.The paper identifies choosing responses that align with the base model’s pretrained distribution as the overlooked aspect.Source: §2.1 · p. 3

2.2 Toward Distribution-Aligned Supervised Fine-Tuning

• On-policy alignment findings motivate matching SFT data to the pretrained model’s distribution, though SFT occurs earlier than reinforcement learning.The paper notes that sampling only from the base model at this earlier stage can lead to model collapse, motivating responses gathered from varied sources.Source: §2.2 · pp. 3–4
• The paper hypothesizes that preserving proximity between pretrained and fine-tuning distributions can reduce distortion risks, including catastrophic forgetting and spurious correlations.It argues that SFT’s comparatively limited examples make maintaining this proximity important.Source: §2.2 · p. 3

3 Methodology

• GRAPE finds responses in a candidate pool that align with the base model’s pretrained distribution, then follows with standard SFT.The method is organized as response collection and model-specific customization before SFT.Source: §3 · p. 4

3.1 Collecting Responses from Existing Resources

• Existing instruction datasets often reuse prompts with varied responses, and GRAPE can use these candidates or generate new ones when existing resources are insufficient.Multiple responses from various datasets form the candidate set associated with each instruction.Source: §3.1 · pp. 4–5

3.2 Customize Dataset For Models

• For each instruction, GRAPE ranks candidate responses by the target model’s length-normalized conditional log-probability and selects the highest-probability responses for supervision.This is equivalent to ranking by lowest perplexity; the same candidate pool can yield different model-specific selections.Source: §3.2 · p. 5
• GRAPE needs only a forward pass to compute candidate log-probabilities, with no gradients or optimization, making it simple to integrate into SFT pipelines.The training-loss curves show the ordering Best < Random < Worst for highest-, random-, and lowest-probability responses.Source: Figure 2 · p. 4, §3.2 · p. 5

4.1 Experimental Setup

• The controlled experiment isolates distribution matching on verifiable coding, logic, and math tasks using UltraInteract-SFT.The dataset contains about 80,800 instructions and over 280,000 step-wise training examples; GRAPE selects from original and additional responses while matching the original response count.Source: §4 · p. 5, §4.1 · p. 5
• Evaluation spans four base models and coding and math reasoning benchmarks, with training configurations generally matched to the original dataset’s instruction and response counts.The tested models are Llama 3.1-8B, Llama 3.2-3B, Mistral-7B, and Qwen2.5-7B.Source: §4.1 · p. 6
• Baselines compare GRAPE with the original dataset, responses from Llama 3.1-405B-Instruct, and three times as many validated responses.The 3×-data comparison holds training hyperparameters fixed.Source: §4.1 · p. 6

4.2 Results and Analysis

• GRAPE improves average benchmark performance by up to 13.8% over strongest-model responses and by 3.6%–17.3% over 3×-data training.Table 1 reports the synthetic experiment’s results and GRAPE’s absolute improvement over each row.Source: Table 1 · p. 6, §4.2 · pp. 6–7

5.1 Data Mixture Details

• The candidate pool contains 350.4K unique instructions and about 1.03 million instruction-response pairs, retaining preference-data instructions with at least two distinct responses.Tulu-3 and Olmo-2 share their supervised fine-tuning data mixture; the study collects overlapping instructions and their responses.Source: §5.1 · p. 7

5.2 Evaluation

• Evaluation covers coding, math, knowledge, and instruction-following benchmarks; Figure 4 reports that selected responses vary significantly across base models for 1K Tulu instructions.The benchmarks are LeetCode, MATH, BBH, MMLU, and AlpacaEval-V2.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Baselines

• Baselines isolate response selection, scale training up to 1.58M pairs (about 4.5× GRAPE’s data), or apply LESS, Emb-NV, and S2L selection.Controlled baselines include SFT-only, random selection, and Reverse-GRAPE; scaling baselines include Tulu3-SFT and All Responses.Source: §5.3 · pp. 7–8

5.4 Results

• GRAPE-selected models outperform strong baselines, including all-data training; LLaMA3.1-8B also exceeds Tulu3-8B-SFT at roughly one-sixth the training computation.The comparison uses the Tulu-Olmo collection; Table 2 includes Tulu3-SFT results for LLaMA3.1-8B.Source: Table 2 · p. 8, §5.4 · pp. 8–9

5.5 Ablations

• Using responses sampled from one generator, GRAPE further boosts Mistral-v0.3-7B performance over Magpie-Zoo replicas, with no additional engineering overhead.The experiment selects among 10 Qwen2.5-72B-Instruct responses per instruction.Source: Table 3 · p. 8, §5.5 · p. 9
• Self-generation consistently degrades performance; the authors attribute this to narrow, repetitive responses reinforcing biases and reducing reasoning diversity.They contrast this with GRAPE’s selection of external responses that are both diverse and distribution-aligned.Source: Table 4 · p. 9, §5.5 · p. 9

• GRAPE improves supervised fine-tuning by selecting responses aligned with the base model’s pretrained distribution.Source: §6 · p. 10
• Selection requires only a forward pass over candidate responses, making GRAPE efficient and easy to integrate.Source: §6 · p. 10
• GRAPE consistently outperforms stronger baselines trained on significantly larger datasets and surpasses more complex, costly data-selection methods.Source: §6 · p. 10

• GRAPE selects responses for a fixed instruction set, whereas instance-level methods choose instructions based on coverage, skill balance, or difficulty.Source: §7 · p. 10
• Response selection is complementary to instruction-level selection, and the two approaches could be combined to improve data quality and training efficiency.Source: §7 · p. 10
• GRAPE assumes a quality-controlled candidate pool, and its selection effectiveness may be influenced by the base model’s inherent capabilities.Source: §7 · p. 10

NeurIPS Paper Checklist

• The checklist says the paper discusses limitations, including performance degradation when pursuing in-distribution answers in the wrong way, and points to appendices for reproducibility details.It also reports publicly available datasets and models, and that the method requires computing normalized training-data probabilities.Source: NeurIPS Paper Checklist · pp. 23–25

A Parameter Distance

• Well-matched training data shifts model parameters less than ill-matched data, as measured by the L2 distance between fine-tuned and pretrained checkpoints.The authors use parameter difference as a signal of distribution drift during SFT.Source: A Parameter Distance · p. 30

B.1 Comparing with Reward Based Selection

• GRAPE outperforms reward-based selection across both models and all benchmarks, favoring base-model distribution alignment over external reward selection.Reward-based selection ranks each instruction’s responses using the Skywork-Reward-Llama3.1-8B-v0.2 reward model before standard SFT; the comparison reports no numeric margins here.Source: §B.1 · p. 30
• Table 7 compares GRAPE with reward-based selection across benchmarks for LLaMA3.1-8B and Mistral-v0.3-7B, using benchmark-specific scores where higher is better.Source: Table 7 · p. 30

B.2 Experiment on OpenHermes

• The OpenHermes-2.5 corpus contains approximately 1 million instructions; the resulting selection contains 575K unique instructions and 1.34M instruction-response pairs.Responses are aggregated across sources, and preference-based datasets retain only winning responses.Source: §B.2 · p. 30
• GRAPE outperforms naive combination strategies on OpenHermes-2.5, with Table 8 reporting results for the dataset.The table caption notes that its Subset row trains exclusively on the subset’s SFT responses.Source: Table 8 · p. 30, §B.2 · p. 31

B.3 Data Selection For Long Chain-of-Thoughts

• The long-chain-of-thought experiment retains verified-correct R1-Distill-Qwen-1.5B trajectories for a subset of LUFFY and compares lowest- versus highest-perplexity instances.Table 9 reports performance metrics on MATH; the supplied passages do not state its outcome.Source: §B.3 · p. 31, Table 9 · p. 31

B.4 Token-level GRAPE

• Token-level likelihood-weighted training yields consistent improvements across several benchmarks on OpenThoughts-114k.Evaluation includes competition-level math datasets and general reasoning tests; Table 10 reports Qwen2.5-3B performance.Source: §B.4 · p. 31, Table 10 · p. 31

C Further Details On Baselines

• The further-baselines section describes experimental setups for S2L, LESS, and NV-Embed.These are presented as data-selection baselines.Source: C Further Details On Baselines · p. 31

C.1 S2L

• S2L trains a reference model to capture training dynamics, clusters example trajectories, and samples equally across clusters for a balanced subset.The setup uses model-family-matched references and K-means clustering into 100 clusters.Source: §C.1 · pp. 31–32

C.2 LESS

• LESS scores training examples by gradient-based influence on validation examples, then selects examples greedily by their scores.Influence scores aggregate across validation examples within each task and take the maximum across tasks.Source: §C.2 · p. 32

C.3 NV-Embed

• NV-Embed ranks examples by embedding similarity to validation examples and uses round-robin selection to maintain task coverage.The method is model-agnostic and uses a fixed pretrained embedding model.Source: §C.3 · p. 32

• Models are trained on a 4-GPU Nvidia-GH200 node, with batch size 256 and micro batch size 2.Source: D Further Training Details · p. 32

• Further ablation results are referenced in Tables 11 and 4.Source: E Further Ablations on UltraInteract · p. 32

F.2 TLDR: The Final Table

• GRAPE’s cost analysis assumes candidate responses are already available; it requires no additional training and one conditional-probability computation per response.The comparison describes GRAPE as achieving superior performance with minimal computational cost among model-based selection approaches.Source: §F.2 · p. 34, Table 13 · p. 34

F.3.1 Gradient-based Influence

• Gradient-based influence selects examples using training–validation gradient similarity; in-run variants avoid separate gradient computation but require full-dataset training and reflect only a particular iteration’s preference.The LESS pipeline uses warmup checkpoints and per-example gradients, with additional training and repeated gradient passes.Source: §F.3.1 · pp. 34–35

F.3.2 Gradient Matching; F.3.3 Gradient Norm

• Gradient matching clusters examples by gradient features for greedy selection, while GraNd scores examples using gradient norms averaged across model initializations.TAGCOS follows the warmup and gradient-computation pipeline of LESS, with the same stated computational cost.Source: §F.3.2 · p. 35, §F.3.3 · p. 35

F.4 Embedding-based Methods

• Embedding-based selection uses target-model embeddings, matching training examples to validation data or selecting a diverse subset by embedding-space geometry.These approaches require no additional model training and compute per-sample embeddings.Source: §F.4 · p. 35

4.1 Simple Uncertainty-based Indicators; F.5.1 Perplexity (PPL)

• Uncertainty indicators use target-model logits without additional training, whereas perplexity selection can require training a reference model before scoring examples.PPL may use the final model or a smaller reference model.Source: §4.1 · p. 36, §F.5.1 · p. 36

F.5.2 Learnability

• Learnability measures per-example loss reduction from before to after full training, excluding uncertain but unlearnable points; its computation uses two forward passes per example.The method also requires training the final model on the full dataset to obtain a reference model.Source: §F.5.2 · p. 36

F.5.3 Loss Trajectory

• S2L clusters examples by loss trajectories from a reference model, but trajectory computation can cost more than gradient-based methods because its epoch count is typically much larger.The choice of reference model matters because models from the final model’s family tend to have similar training loss trajectories.Source: §F.5.3 · p. 36

G Multi-Round GRAPE

• 37.3% versus 35.9% average benchmark scores after a second GRAPE round show further gains across multiple benchmarks for LLAMA3.1-8B.The benchmarks include AlpacaEval, WizardEval, MATH, MMLU, and BBH.Source: Table 14 · p. 37, G Multi-Round GRAPE · p. 37

H Details Of Correlation Analysis

• The study randomly sampled 1,000 instances from the Tulu-v3 training set and generated multiple responses for each instruction.A range of generator models produced the responses.Source: H Details Of Correlation Analysis · p. 37
• Responses were ranked by perplexity under each base model, with lower perplexity treated as a better match to that model’s distribution.Source: H Details Of Correlation Analysis · p. 37
• The analysis identified each base model’s top-1 response per instance and visualized the resulting rankings in a heatmap.The heatmap is Figure 4.Source: H Details Of Correlation Analysis · p. 37